1 <html> 2 <head> 3 <title>TestNG</title> 4 5 <link rel="stylesheet" href="testng.css" type="text/css" /> 6 <link type="text/css" rel="stylesheet" href="http://beust.com/beust.css" /> 7 <script type="text/javascript" src="banner.js"></script> 8 9 <script type="text/javascript" src="http://beust.com/scripts/shCore.js"></script> 10 <script type="text/javascript" src="http://beust.com/scripts/shBrushJava.js"></script> 11 <script type="text/javascript" src="http://beust.com/scripts/shBrushXml.js"></script> 12 <script type="text/javascript" src="http://beust.com/scripts/shBrushBash.js"></script> 13 <script type="text/javascript" src="http://beust.com/scripts/shBrushPlain.js"></script> 14 <link type="text/css" rel="stylesheet" href="http://beust.com/styles/shCore.css"/> 15 <link type="text/css" rel="stylesheet" href="http://beust.com/styles/shThemeCedric.css"/> 16 <script type="text/javascript"> 17 SyntaxHighlighter.config.clipboardSwf = 'scripts/clipboard.swf'; 18 SyntaxHighlighter.defaults['gutter'] = false; 19 SyntaxHighlighter.all(); 20 </script> 21 <script type="text/javascript" src="http://beust.com/toc.js"></script> 22 23 <style type="text/css"> 24 /* Set the command-line table option column width. */ 25 #command-line colgroup.option { 26 width: 7em; 27 } 28 </style> 29 </head> 30 <body onLoad="generateToc();"> 31 32 <script type="text/javascript"> 33 displayMenu("documentation-main.html") 34 </script> 35 36 <h2 align="center">TestNG</h2> 37 38 <!-- -------------------------- 39 40 <table class="float-right"> 41 <tr> 42 <td> 43 <script type="text/javascript"><!-- 44 google_ad_client = "pub-1467757024002850"; 45 google_ad_width = 120; 46 google_ad_height = 600; 47 google_ad_format = "120x600_as"; 48 google_ad_channel ="5560744946"; 49 //--> 50 <!-- 51 52 </script> 53 54 <script type="text/javascript" 55 src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> 56 </script> 57 </td> 58 </tr> 59 </table> 60 61 ------------------------------- --> 62 63 <!------------------------------------- 64 TOC 65 ------------------------------------> 66 67 <h3>Table of Contents</h3> 68 69 <div id="table-of-contents"> 70 </div> 71 72 73 <!------------------------------------- 74 INTRODUCTION 75 ------------------------------------> 76 <h3><a class="section" name="introduction">Introduction</a></h3> 77 78 TestNG is a testing framework designed to simplify a broad range of testing needs, from unit testing (testing a class in isolation of the others) to integration testing (testing entire systems made of several classes, several packages and even several external frameworks, such as application servers). 79 80 <p> 81 82 Writing a test is typically a three-step process: 83 84 <ul> 85 <li>Write the business logic of your test and insert <a href="#annotations">TestNG annotations</a> in your code. 86 </li> 87 <li>Add the information about your test (e.g. the class name, the groups you wish to run, etc...) in a <tt><a href="#testng-xml">testng.xml</a></tt> file or in build.xml. 88 </li> 89 <li><a href="ant.html">Run TestNG</a>. 90 </li> 91 </ul> 92 You can find a quick example on the <a href="index.html">Welcome page</a>. 93 <p> 94 The concepts used in this documentation are as follows: 95 96 <ul> 97 <li> 98 A suite is represented by one XML file. It can contain one or more tests and is defined by the <tt><suite></tt> tag. 99 </li> 100 <li> 101 A test is represented by <tt><test></tt> and can contain one or more TestNG classes. 102 </li> 103 <li> 104 A TestNG class is a Java class that contains at least one TestNG annotation. It is represented by the <tt><class></tt> tag and can contain one or more test methods. 105 </li> 106 <li> 107 A test method is a Java method annotated by <tt>@Test</tt> in your source. 108 </li></ul>A TestNG test can be configured by <tt>@BeforeXXX and @AfterXXX </tt>annotations which allows to perform some Java logic before and after a certain point, these points being either of the items listed above.<p> 109 The rest of this manual will explain the following: 110 <p> 111 <ul> 112 <li>A list of all the annotations with a brief explanation. This will give you an idea of the various functionalities offered by TestNG but you will probably want to consult the section dedicated to each of these annotations to learn the details. 113 </li> 114 <li>A description of the testng.xml file, its syntax and what you can specify in it. 115 </li> 116 <li>A detailed list of the various features and how to use them with a combination of annotations and testng.xml. 117 </li> 118 </ul> 119 120 121 <!------------------------------------- 122 ANNOTATIONS 123 ------------------------------------> 124 125 <h3><a class="section" name="annotations">Annotations</a></h3> 126 127 Here is a quick overview of the annotations available in TestNG along with their attributes. 128 129 <p> 130 131 <table> 132 133 <tr> 134 <td colspan="2"><b><tt>@BeforeSuite<br>@AfterSuite<br>@BeforeTest<br>@AfterTest<br>@BeforeGroups<br>@AfterGroups<br>@BeforeClass<br>@AfterClass<br>@BeforeMethod<br>@AfterMethod</tt></b></td><td><b>Configuration information for a TestNG class:</b> 135 136 <br> 137 138 <br><b>@BeforeSuite: </b>The annotated method will be run before all tests in this suite have run. 139 140 <br><b>@AfterSuite: </b> The annotated method will be run after all tests in this suite have run. 141 142 <br><b>@BeforeTest</b>: The annotated method will be run before any test method belonging to the classes inside the <test> tag is run. 143 144 <br><b>@AfterTest</b>: The annotated method will be run after all the test methods belonging to the classes inside the <test> tag have run. 145 146 <br><b>@BeforeGroups</b>: The list of groups that this configuration method will run before. This method is guaranteed to run shortly before the first test method that belongs to any of these groups is invoked. 147 148 <br><b>@AfterGroups</b>: The list of groups that this configuration method will run after. This method is guaranteed to run shortly after the last test method that belongs to any of these groups is invoked. 149 150 <br><b>@BeforeClass</b>: The annotated method will be run before the first test method in the current class is invoked. 151 152 <br><b>@AfterClass</b>: The annotated method will be run after all the test methods in the current class have been run. 153 154 <br><b>@BeforeMethod</b>: The annotated method will be run before each test method. 155 156 <br><b>@AfterMethod</b>: The annotated method will be run after each test method. 157 158 </td> 159 </tr> 160 161 <tr> 162 <td> 163 </td> 164 <td> 165 <tt>alwaysRun</tt> 166 </td> 167 <td> 168 For before methods (beforeSuite, beforeTest, beforeTestClass and 169 beforeTestMethod, but not beforeGroups): 170 If set to true, this configuration method will be run 171 regardless of what groups it belongs to. 172 <br> 173 For after methods (afterSuite, afterClass, ...): 174 If set to true, this configuration method will be run 175 even if one or more methods invoked previously failed or 176 was skipped. 177 </td> 178 </tr> 179 180 <tr> 181 <td> 182 </td> 183 <td> 184 <tt>dependsOnGroups</tt> 185 </td> 186 <td> 187 The list of groups this method depends on. 188 </td> 189 </tr> 190 191 <tr> 192 <td> 193 </td> 194 <td> 195 <tt>dependsOnMethods</tt> 196 </td> 197 <td> 198 The list of methods this method depends on. 199 </td> 200 </tr> 201 202 <tr> 203 <td> 204 </td> 205 <td> 206 <tt>enabled</tt> 207 </td> 208 <td> 209 Whether methods on this class/method are enabled. 210 </td> 211 </tr> 212 213 <tr> 214 <td> 215 </td> 216 <td> 217 <tt>groups</tt> 218 </td> 219 <td> 220 The list of groups this class/method belongs to. 221 </td> 222 </tr> 223 224 <tr> 225 <td> 226 </td> 227 <td> 228 <tt>inheritGroups</tt> 229 </td> 230 <td> 231 If true, this method will belong to groups specified in the @Test annotation at the class level. 232 </td> 233 </tr> 234 235 <tr class="separator"> 236 <td colspan="3"> </td> 237 </tr> 238 239 <tr> 240 <td colspan="2"><tt><b>@DataProvider</b></tt></td><td><b>Marks a method as supplying data for a test method. The annotated method must return an Object[][] where each Object[] can be assigned the parameter list of the test method. The @Test method that wants to receive data from this DataProvider needs to use a dataProvider name equals to the name of this annotation.</b></td></tr><tr> 241 <td> 242 </td> 243 <td> 244 <tt>name</tt> 245 </td> 246 <td> 247 The name of this data provider. If it's not supplied, the name of this data provider will automatically be set to the name of the method. 248 </td> 249 </tr> 250 <tr> 251 <td> 252 </td> 253 <td> 254 <tt>parallel</tt> 255 </td> 256 <td> 257 If set to true, tests generated using this data provider are run in parallel. Default value is false. 258 </td> 259 </tr> 260 <tr> 261 <td colspan="3"> </td> 262 </tr> 263 264 <tr> 265 <td colspan="2"><b><tt>@Factory</tt></b></td><td><b> Marks a method as a factory that returns objects that will be used by TestNG as Test classes. The method must return Object[].</b></td></tr><tr> 266 <td colspan="3"> </td> 267 </tr> 268 269 <tr> 270 <td colspan="2"><b><tt>@Listeners</tt></b></td><td><b>Defines listeners on a test class.</b></td></tr> 271 <tr> 272 <td></td> 273 <td> 274 <tt>value</tt> 275 </td> 276 <td> 277 An array of classes that extend <tt>org.testng.ITestNGListener</tt>. 278 </td> 279 </tr> 280 281 <tr> 282 <td colspan="3"> </td> 283 </tr> 284 285 <td colspan="2"><b><tt>@Parameters</tt></b></td><td><b>Describes how to pass parameters to a @Test method.</b></td></tr><tr> 286 <td> 287 </td> 288 <td> 289 <tt>value</tt> 290 </td> 291 <td> 292 The list of variables used to fill the parameters of this method. 293 </td> 294 </tr> 295 296 <tr> 297 <td colspan="3"> </td> 298 </tr> 299 300 <tr> 301 <td colspan="2"><b>@Test</b></td><td><b>Marks a class or a method as part of the test.</b></td></tr><tr> 302 <td> 303 </td> 304 <td> 305 <tt>alwaysRun</tt> 306 </td> 307 <td> 308 If set to true, this test method will always be run even if it depends on a method that failed. 309 </td> 310 </tr> 311 312 <tr> 313 <td> 314 </td> 315 <td> 316 <tt>dataProvider</tt> 317 </td> 318 <td> 319 The name of the data provider for this test method. 320 </td> 321 </tr> 322 323 324 <tr> 325 <td> 326 </td> 327 <td> 328 <tt>dataProviderClass</tt> 329 </td> 330 <td> 331 The class where to look for the data provider. If not specified, the data provider will be looked on the class of the current test method or one of its base classes. If this attribute is specified, the data provider method needs to be static on the specified class. 332 </td> 333 </tr> 334 335 336 337 338 339 340 <tr> 341 <td> 342 </td> 343 <td> 344 <tt>dependsOnGroups</tt> 345 </td> 346 <td> 347 The list of groups this method depends on. 348 </td> 349 </tr> 350 351 <tr> 352 <td> 353 </td> 354 <td> 355 <tt>dependsOnMethods</tt> 356 </td> 357 <td> 358 The list of methods this method depends on. 359 </td> 360 </tr> 361 362 <tr> 363 <td> 364 </td> 365 <td> 366 <tt>description</tt> 367 </td> 368 <td> 369 The description for this method. 370 </td> 371 </tr> 372 373 <tr> 374 <td> 375 </td> 376 <td> 377 <tt>enabled</tt> 378 </td> 379 <td> 380 Whether methods on this class/method are enabled. 381 </td> 382 </tr> 383 384 <tr> 385 <td> 386 </td> 387 <td> 388 <tt>expectedExceptions</tt> 389 </td> 390 <td> 391 The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure. 392 </td> 393 </tr> 394 395 <tr> 396 <td> 397 </td> 398 <td> 399 <tt>groups</tt> 400 </td> 401 <td> 402 The list of groups this class/method belongs to. 403 </td> 404 </tr> 405 406 <tr> 407 <td> 408 </td> 409 <td> 410 <tt>invocationCount</tt> 411 </td> 412 <td> 413 The number of times this method should be invoked. 414 </td> 415 </tr> 416 417 <tr> 418 <td> 419 </td> 420 <td> 421 <tt>invocationTimeOut</tt> 422 </td> 423 <td> 424 The maximum number of milliseconds this test should take for the cumulated time of all the invocationcounts. This attribute will be ignored if invocationCount is not specified. 425 </td> 426 </tr> 427 428 <tr> 429 <td> 430 </td> 431 <td> 432 <tt>priority</tt> 433 </td> 434 <td> 435 The priority for this test method. Lower priorities will be scheduled first. 436 </td> 437 </tr> 438 439 <tr> 440 <td> 441 </td> 442 <td> 443 444 <tt>successPercentage</tt> 445 </td> 446 <td> 447 The percentage of success expected from this method 448 </td> 449 </tr> 450 451 <tr> 452 <td> 453 </td> 454 <td> 455 <tt>singleThreaded</tt> 456 </td> 457 <td> 458 If set to true, all the methods on this test class are guaranteed to run in the same thread, even if the tests are currently being run with parallel="methods". This attribute can only be used at the class level and it will be ignored if used at the method level. Note: this attribute used to be called <tt>sequential</tt> (now deprecated). 459 460 </td> 461 </tr> 462 463 <tr> 464 <td> 465 </td> 466 <td> 467 <tt>timeOut</tt> 468 </td> 469 <td> 470 The maximum number of milliseconds this test should take. 471 </td> 472 </tr> 473 474 <tr> 475 <td> 476 </td> 477 <td> 478 <tt>threadPoolSize</tt> 479 </td> 480 <td> 481 The size of the thread pool for this method. 482 The method will be invoked from multiple threads as specified by 483 invocationCount. <br>Note: this attribute is ignored if invocationCount is not specified 484 485 </td> 486 </tr> 487 488 </table> 489 490 491 492 493 </pre> 494 <!------------------------------------- 495 TESTNG.XML 496 ------------------------------------> 497 <h3><a class="section" name="testng-xml">testng.xml</a></h3> 498 499 <p>You can invoke TestNG in several different ways:</p><ul> 500 <li>With a <tt>testng.xml</tt> file</li><li><a href="http://testng.org/doc/ant.html">With ant</a></li><li>From the command line</li></ul><p>This section describes the format of <tt>testng.xml</tt> (you will find documentation 501 on ant and the command line below).</p><p>The current DTD for <tt>testng.xml</tt> can be found on the main Web site: 502 <a href="http://testng.org/testng-1.0.dtd">http://testng.org/testng-1.0.dtd</a> 503 (for your convenience, you might prefer to browse the 504 <a href="http://testng.org/dtd">HTML version</a>).</p> 505 506 Here is an example <tt>testng.xml</tt> file: 507 508 <p> 509 <h3 class="sourcetitle">testng.xml</h3> 510 <pre class="brush: xml"> 511 <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" > 512 513 <suite name="Suite1" verbose="1" > 514 <test name="Nopackage" > 515 <classes> 516 <class name="NoPackageTest" /> 517 </classes> 518 </test> 519 520 <test name="Regression1"> 521 <classes> 522 <class name="test.sample.ParameterSample"/> 523 <class name="test.sample.ParameterTest"/> 524 </classes> 525 </test> 526 </suite> 527 </pre> 528 529 You can specify package names instead of class names: 530 531 <h3 class="sourcetitle">testng.xml</h3> 532 <pre class="brush: xml; highlight: [5,6,7]"> 533 <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" > 534 535 <suite name="Suite1" verbose="1" > 536 <test name="Regression1" > 537 <packages> 538 <package name="test.sample" /> 539 </packages> 540 </test> 541 </suite> 542 </pre> 543 544 545 <p>In this example, TestNG will look at all the classes in the package 546 <tt>test.sample</tt> and will retain only classes that have TestNG annotations.</p> 547 548 You can also specify groups and methods to be included and excluded: 549 550 <h3 class="sourcetitle">testng.xml</h3> 551 <pre class="brush: xml"> 552 <test name="Regression1"> 553 <groups> 554 <run> 555 <exclude name="brokenTests" /> 556 <include name="checkinTests" /> 557 </run> 558 </groups> 559 560 <classes> 561 <class name="test.IndividualMethodsTest"> 562 <methods> 563 <include name="testMethod" /> 564 </methods> 565 </class> 566 </classes> 567 </test> 568 </pre> 569 570 <p>You can also define new groups inside <tt>testng.xml</tt> and specify additional details in attributes, such as whether to run the tests in parallel, how many threads to use, whether you are running JUnit tests, etc... 571 <p> 572 573 By default, TestNG will run your tests in the order they are found in the XML 574 file. If you want the classes and methods listed in this file to be 575 run in an unpredictible order, set the <tt>preserve-order</tt> 576 attribute to <tt>false</tt> 577 578 <h3 class="sourcetitle">testng.xml</h3> 579 <pre class="brush: xml"> 580 <test name="Regression1" preserve-order="false"> 581 <classes> 582 583 <class name="test.Test1"> 584 <methods> 585 <include name="m1" /> 586 <include name="m2" /> 587 </methods> 588 </class> 589 590 <class name="test.Test2" /> 591 592 </classes> 593 </test> 594 </pre> 595 596 <p> 597 598 Please see the DTD for a complete list of the features, or read on.</p> 599 600 <!------------------------------------- 601 RUNNING TESTNG 602 ------------------------------------> 603 604 <h3><a class="section" name="running-testng">Running TestNG</a></h3> 605 606 TestNG can be invoked in different ways: 607 608 <ul> 609 <li>Command line 610 </li> 611 <li><a href="ant.html">ant</a> 612 </li> 613 <li><a href="eclipse.html">Eclipse</a> 614 </li> 615 <li><a href="idea.html">IntelliJ's IDEA</a> 616 </li> 617 </ul> 618 619 This section only explains how to invoke TestNG from the command line. Please click on one of the links above if you are interested in one of the other ways. 620 <p> 621 Assuming that you have TestNG in your class path, the simplest way to invoke TestNG is as follows: 622 623 <pre class="brush: text"> 624 java org.testng.TestNG testng1.xml [testng2.xml testng3.xml ...] 625 </pre> 626 627 You need to specify at least one XML file describing the TestNG suite you are trying to run. Additionally, the following command-line switches are available: 628 629 </p> 630 631 <table id="command-line"> 632 <caption>Command Line Parameters</caption><colgroup class="option"/> 633 <colgroup class="argument"/> 634 <colgroup class="documentation"/> 635 <thead> 636 637 <tr> 638 <th>Option</th> 639 <th>Argument</th> 640 <th>Documentation</th> 641 </tr> 642 </thead> 643 644 <tbody> 645 <tr> 646 <td>-configfailurepolicy</td> 647 <td><tt>skip</tt>|<tt>continue</tt></td> 648 <td>Whether TestNG should <tt>continue</tt> to execute the remaining tests in the suite or <tt>skip</tt> them if 649 an @Before* method fails. Default behavior is <tt>skip</tt>.</td> 650 </tr> 651 652 <tr> 653 <td>-d</td> 654 <td>A directory</td> 655 <td>The directory where the reports will be generated (defaults to <tt>test-output</tt>).</td> 656 </tr> 657 658 <tr> 659 <td>-dataproviderthreadcount</td> 660 <td>The default number of threads to use for data 661 providers when running tests in parallel.</td> 662 <td>This sets the default maximum number of threads to use 663 for data providers when running tests in parallel. It will only take effect if the parallel mode has been selected (for example, with the -parallel option). This can be overridden in the suite definition.</td> 664 </tr> 665 666 <tr> 667 <td>-excludegroups</td> 668 <td>A comma-separated list of groups.</td><td>The list of groups you want to be excluded from this run.</td> 669 </tr> 670 671 <tr> 672 <td>-groups</td> 673 <td>A comma-separated list of groups.</td> 674 <td>The list of groups you want to run (e.g. <tt>"windows,linux,regression"</tt>).</td> 675 </tr> 676 677 <tr> 678 <td>-listener</td> 679 <td>A comma-separated list of Java classes that can be found on your classpath.</td> 680 <td>Lets you specify your own test listeners. The classes need to implement <a href="../javadocs/org/testng/ITestListener.html"> <tt>org.testng.ITestListener</tt></a></td> 681 </tr> 682 683 <tr> 684 <td>-methods</td> 685 <td>A comma separated list of fully qualified class name and method. For example <tt>com.example.Foo.f1,com.example.Bar.f2</tt>.</td> 686 <td>Lets you specify individual methods to run.</tt></a></td> 687 </tr> 688 689 <tr> 690 <td>-methodselectors</td> 691 <td>A comma-separated list of Java classes and method 692 priorities that define method selectors.</td> 693 <td>Lets you specify method selectors on the command 694 line. For example: <tt>com.example.Selector1:3,com.example.Selector2:2</tt></td> 695 </tr> 696 697 <tr> 698 <td>-parallel</td> 699 <td>methods|tests|classes</td> 700 <td>If specified, sets the default mechanism used to determine how to use parallel threads when running tests. If not set, default mechanism is not to use parallel threads at all. This can be overridden in the suite definition.</td> 701 </tr> 702 703 <tr> 704 <td>-reporter</td> 705 <td>The extended configuration for a custom report listener.</td> 706 <td>Similar to the <tt>-listener</tt> option, except that it allows the configuration of JavaBeans-style properties on the reporter instance. 707 <br> 708 Example: <tt>-reporter com.test.MyReporter:methodFilter=*insert*,enableFiltering=true</tt> 709 <br> 710 You can have as many occurences of this option, one for each reporter that needs to be added.</td> 711 </tr> 712 713 <tr> 714 <td>-sourcedir</td> 715 <td>A semi-colon separated list of directories.</td> 716 <td>The directories where your javadoc annotated test sources are. This option is only necessary if you are using javadoc type annotations. (e.g. <tt>"src/test"</tt> or <tt>"src/test/org/testng/eclipse-plugin;src/test/org/testng/testng"</tt>).</td> 717 </tr> 718 719 <tr> 720 <td>-suitename</td> 721 <td>The default name to use for a test suite.</td> 722 <td>This specifies the suite name for a test suite defined on the command line. This option is ignored if the suite.xml file or the source code specifies a different suite name. It is possible to create a suite name with spaces in it if you surround it with double-quotes "like this".</td> 723 </tr> 724 725 <tr> 726 <td>-testclass</td> 727 <td>A comma-separated list of classes that can be found in your classpath.</td><td>A list of class files separated by commas (e.g. <tt>"org.foo.Test1,org.foo.test2"</tt>).</td> 728 </tr> 729 730 <tr> 731 <td>-testjar</td> 732 <td>A jar file.</td> 733 <td>Specifies a jar file that contains test classes. If a <tt>testng.xml</tt> file is found at the root of that jar file, it will be used, otherwise, all the test classes found in this jar file will be considered test classes.</td> 734 </tr> 735 736 <tr> 737 <td>-testname</td> 738 <td>The default name to use for a test.</td> 739 <td>This specifies the name for a test defined on the command line. This option is ignored if the suite.xml file or the source code specifies a different test name. It is possible to create a test name with spaces in it if you surround it with double-quotes "like this".</td> 740 </tr> 741 742 <tr> 743 <td>-testnames</td> 744 <td>A comma separated list of test names.</td> 745 <td>Only tests defined in a <test> tag matching one of these names will be run.</td> 746 </tr> 747 748 <tr> 749 <td>-testrunfactory</td> 750 <td>A Java classes that can be found on your classpath.</td> 751 <td>Lets you specify your own test runners. The class needs to implement <a href="../javadocs/org/testng/ITestRunnerFactory.html"> <tt>org.testng.ITestRunnerFactory</tt></a>.</td> 752 </tr> 753 754 <tr> 755 <td>-threadcount</td> 756 <td>The default number of threads to use when running tests in parallel.</td> 757 <td>This sets the default maximum number of threads to use for running tests in parallel. It will only take effect if the parallel mode has been selected (for example, with the -parallel option). This can be overridden in the suite definition.</td> 758 </tr> 759 760 <tr> 761 <td>-xmlpathinjar</td> 762 <td>The path of the XML file inside the jar file.</td> 763 <td>This attribute should contain the path to a valid XML file inside the test jar (e.g. <tt>"resources/testng.xml"</tt>). The default is <tt>"testng.xml"</tt>, which means a file called "<tt>testng.xml</tt>" at the root of the jar file. This option will be ignored unless <tt>-testjar</tt> is specified.</td> 764 </tr> 765 766 </tbody> 767 </table> 768 769 <p> 770 771 This documentation can be obtained by invoking TestNG without any arguments. 772 773 <p> 774 775 You can also put the command line switches in a text file, say <tt>c:\command.txt</tt>, and tell TestNG to use that file to retrieve its parameters: 776 777 <pre class="brush: text"> 778 C:> more c:\command.txt 779 -d test-output testng.xml 780 C:> java org.testng.TestNG @c:\command.txt 781 </pre> 782 783 <p> 784 785 Additionally, TestNG can be passed properties on the command line of the Java Virtual Machine, for example 786 787 <pre class="brush: text"> 788 java -Dtestng.test.classpath="c:/build;c:/java/classes;" org.testng.TestNG testng.xml 789 </pre> 790 791 Here are the properties that TestNG understands: 792 793 <table id="system=properties"> 794 <caption>System properties</caption> 795 <colgroup class="option"/> 796 <colgroup class="type"/> 797 <colgroup class="documentation"/> 798 <thead> 799 <tr><th>Property</th> 800 <th>Type</th> 801 <th>Documentation</th></tr> 802 </thead> 803 804 <tr> 805 <td>testng.test.classpath</td> 806 <td>A semi-colon separated series of directories that contain your test classes.</td> 807 <td>If this property is set, TestNG will use it to look for your test classes instead of the class path. This is convenient if you are using the <tt>package</tt> tag in your XML file and you have a lot of classes in your classpath, most of them not being test classes. 808 </tr> 809 </table> 810 811 <br> 812 813 <b>Example:</b> 814 815 <pre class="brush: text"> 816 java org.testng.TestNG -groups windows,linux -testclass org.test.MyTest 817 </pre> 818 819 The <a href="ant.html">ant task</a> and <a href="#testng-xml">testng.xml</a> allow you to launch TestNG with more parameters (methods to include, specifying parameters, etc...), so you should consider using the command line only when you are trying to learn about TestNG and you want to get up and running quickly. 820 821 <p> 822 823 <em>Important</em>: The command line flags that specify what tests should be run will be ignored if you also specify a <tt>testng.xml</tt> file, with the exception of <tt>-includedgroups</tt> and <tt>-excludedgroups</tt>, which will override all the group inclusions/exclusions found in <tt>testng.xml</tt>. 824 825 <!------------------------------------- 826 METHODS 827 ------------------------------------> 828 829 <h3><a class="section" name="methods">Test methods, Test classes and Test groups</a></h3> 830 831 <h4><a class="section" indent=".." name="test-methods">Test methods</a></h4> 832 833 Test methods are annotated with <tt>@Test</tt>. Methods annotated with <tt>@Test</tt> that happen to return a value will be ignored, unless you set <tt>allow-return-values</tt> to <tt>true</tt> in your <tt>testng.xml</tt>: 834 835 <pre class="brush: xml"> 836 <suite allow-return-values="true"> 837 838 or 839 840 <test allow-return-values="true"> 841 </pre> 842 843 844 <h4><a class="section" indent=".." name="test-groups">Test groups</a></h4> 845 846 <p> 847 848 TestNG allows you to perform sophisticated groupings of test methods. Not 849 only can you declare that methods belong to groups, but you can also specify 850 groups that contain other groups. Then TestNG can be invoked and asked to 851 include a certain set of groups (or regular expressions) while excluding another 852 set. This gives you maximum flexibility in how you partition your tests 853 and doesn't require you to recompile anything if you want to run two different 854 sets of tests back to back.</p> 855 856 <p> 857 Groups are specified in your <tt>testng.xml</tt> file and can be found either under the <tt><test></tt> or <tt><suite></tt> tag. Groups specified in the <tt><suite></tt> tag apply to all the <tt><test></tt> tags underneath. Note that groups are accumulative in these tags: if you specify group "a" in <tt><suite></tt> and "b" in <tt><test></tt>, then both "a" and "b" will be included. 858 859 <p> 860 861 <p>For example, it is quite common to have at least two categories of tests</p><ul> 862 <li>Check-in tests. These tests should be run before you submit new 863 code. They should typically be fast and just make sure no basic 864 functionality was broken.<br> 865 </li> 866 <li>Functional tests. These tests should cover all the functionalities 867 of your software and be run at least once a day, although ideally you would 868 want to run them continuously.</li></ul> 869 870 871 Typically, check-in tests are a subset of functional tests. TestNG 872 allows you to specify this in a very intuitive way with test groups. For 873 example, you could structure your test by saying that your entire test class 874 belongs to the "functest" group, and additionally that a couple of methods 875 belong to the group "checkintest": 876 877 <h3 class="sourcetitle">Test1.java</h3> 878 <pre class="brush: java"> 879 public class Test1 { 880 @Test(groups = { "functest", "checkintest" }) 881 public void testMethod1() { 882 } 883 884 @Test(groups = {"functest", "checkintest"} ) 885 public void testMethod2() { 886 } 887 888 @Test(groups = { "functest" }) 889 public void testMethod3() { 890 } 891 } 892 </pre> 893 894 Invoking TestNG with 895 <br> 896 897 <h3 class="sourcetitle">testng.xml</h3> 898 <pre class="brush: xml"> 899 <test name="Test1"> 900 <groups> 901 <run> 902 <include name="functest"/> 903 </run> 904 </groups> 905 <classes> 906 <class name="example1.Test1"/> 907 </classes> 908 </test> 909 </pre> 910 911 <p>will run all the test methods in that classes, while invoking it with <tt>checkintest</tt> will only run 912 <tt>testMethod1()</tt> and <tt>testMethod2()</tt>.</p> 913 914 915 Here is another example, using regular expressions this time. Assume 916 that some of your test methods should not be run on Linux, your test would look 917 like: 918 919 <h3 class="sourcetitle">Test1.java</h3> 920 <pre class="brush: java"> 921 @Test 922 public class Test1 { 923 @Test(groups = { "windows.checkintest" }) 924 public void testWindowsOnly() { 925 } 926 927 @Test(groups = {"linux.checkintest"} ) 928 public void testLinuxOnly() { 929 } 930 931 @Test(groups = { "windows.functest" ) 932 public void testWindowsToo() { 933 } 934 } 935 </pre> 936 937 938 You could use the following testng.xml to launch only the Windows methods: 939 940 <h3 class="sourcetitle">testng.xml</h3> 941 <pre class="brush: xml; highlight: [4,9]"> 942 <test name="Test1"> 943 <groups> 944 <run> 945 <include name="windows.*"/> 946 </run> 947 </groups> 948 949 <classes> 950 <class name="example1.Test1"/> 951 </classes> 952 </test> 953 </pre> 954 955 <blockquote> 956 <em>Note: TestNG uses <a href="http://en.wikipedia.org/wiki/Regular_expression">regular expressions</a>, and not <a href="http://en.wikipedia.org/wiki/Wildmat">wildmats</a>. Be aware of the difference (for example, "anything" is matched by ".*" -- dot star -- and not "*").</em> 957 </blockquote> 958 959 <h4><a name="method-groups">Method groups</a></h4> 960 961 You can also exclude or include individual methods: 962 963 <h3 class="sourcetitle">testng.xml</h3> 964 <pre class="brush: xml"> 965 <test name="Test1"> 966 <classes> 967 <class name="example1.Test1"> 968 <methods> 969 <include name=".*enabledTestMethod.*"/> 970 <exclude name=".*brokenTestMethod.*"/> 971 </methods> 972 </class> 973 </classes> 974 </test> 975 </pre> 976 977 This can come in handy to deactivate a single method without having to recompile 978 anything, but I don't recommend using this technique too much since it makes 979 your testing framework likely to break if you start refactoring your Java code 980 (the regular expressions used in the tags might not match your methods any 981 more). 982 983 984 <h4><a class="section" indent=".." name="groups-of-groups">Groups of groups</a></h4> 985 986 Groups can also include other groups. These groups are called "MetaGroups". 987 For example, you might want to define a group "all" that includes "checkintest" 988 and "functest". "functest" itself will contain the groups "windows" and 989 "linux" while "checkintest will only contain "windows". Here is how you 990 would define this in your property file: 991 992 993 <h3 class="sourcetitle">testng.xml</h3> 994 <pre class="brush: xml"> 995 <test name="Regression1"> 996 <groups> 997 <define name="functest"> 998 <include name="windows"/> 999 <include name="linux"/> 1000 </define> 1001 1002 <define name="all"> 1003 <include name="functest"/> 1004 <include name="checkintest"/> 1005 </define> 1006 1007 <run> 1008 <include name="all"/> 1009 </run> 1010 </groups> 1011 1012 <classes> 1013 <class name="test.sample.Test1"/> 1014 </classes> 1015 </test> 1016 </pre> 1017 1018 </p><!------------------------------------- 1019 EXCLUSION 1020 ------------------------------------> 1021 1022 <h4><a class="section" indent=".." name="exclusions">Exclusion groups</a></h4> 1023 1024 <p>TestNG allows you to include groups as well as exclude them.</p> 1025 1026 1027 For example, it is quite usual to have tests that temporarily break because 1028 of a recent change, and you don't have time to fix the breakage yet. 4 1029 However, you do want to have clean runs of your functional tests, so you need to 1030 deactivate these tests but keep in mind they will need to be reactivated.</p><p>A simple way to solve this problem is to create a group called "broken" and 1031 make these test methods belong to it. For example, in the above example, I 1032 know that testMethod2() is now broken so I want to disable it: 1033 1034 1035 <h3 class="sourcetitle">Java</h3> 1036 <pre class="brush: java"> 1037 @Test(groups = {"checkintest", "broken"} ) 1038 public void testMethod2() { 1039 } 1040 </pre> 1041 1042 All I need to do now is to exclude this group from the run: 1043 1044 <h3 class="sourcetitle">testng.xml</h3> 1045 <pre class="brush: xml; highlight: 5"> 1046 <test name="Simple example"> 1047 <groups> 1048 <run> 1049 <include name="checkintest"/> 1050 <exclude name="broken"/> 1051 </run> 1052 </groups> 1053 1054 <classes> 1055 <class name="example1.Test1"/> 1056 </classes> 1057 </test> 1058 </pre> 1059 1060 <p>This way, I will get a clean test run while keeping track of what tests are 1061 broken and need to be fixed later.</p> 1062 1063 <blockquote> 1064 <p><i>Note: you can also disable tests on an individual basis by using the 1065 "enabled" property available on both @Test and @Before/After 1066 annotations.</i></p> 1067 </blockquote> 1068 1069 1070 <!------------------------------------- 1071 PARTIAL GROUPS 1072 ------------------------------------> 1073 1074 <h4><a class="section" indent=".." name="partial-groups">Partial groups</a></h4> 1075 1076 You can define groups at the class level and then add groups at the method level: 1077 1078 <h3 class="sourcetitle">All.java</h3> 1079 <pre class="brush: java"> 1080 @Test(groups = { "checkin-test" }) 1081 public class All { 1082 1083 @Test(groups = { "func-test" ) 1084 public void method1() { ... } 1085 1086 public void method2() { ... } 1087 } 1088 </pre> 1089 1090 In this class, method2() is part of the group "checkin-test", which is defined 1091 at the class level, while method1() belongs to both "checkin-test" and 1092 "func-test". 1093 1094 <!------------------------------------- 1095 PARAMETERS 1096 ------------------------------------> 1097 1098 <h4><a class="section" indent=".." name="parameters">Parameters</a></h4> 1099 1100 <p> 1101 1102 1103 Test methods don't have to be parameterless. You can use an arbitrary 1104 number of parameters on each of your test method, and you instruct TestNG to 1105 pass you the correct parameters with the <tt>@Parameters</tt> annotation.</p><p> 1106 1107 1108 There are two ways to set these parameters: with <tt>testng.xml</tt> or 1109 programmatically.</p> 1110 1111 1112 <h5><a class="section" indent="..." name="parameters-testng-xml">Parameters from <tt>testng.xml</tt></a></h5> 1113 1114 1115 If you are using simple values for your parameters, you can specify them in your 1116 <tt>testng.xml</tt>: 1117 1118 1119 <h3 class="sourcetitle">Java</h3> 1120 <pre class="brush: java"> 1121 @Parameters({ "first-name" }) 1122 @Test 1123 public void testSingleString(String firstName) { 1124 System.out.println("Invoked testString " + firstName); 1125 assert "Cedric".equals(firstName); 1126 } 1127 </pre> 1128 1129 In this code, we specify that the parameter <tt>firstName</tt> of your Java method 1130 should receive the value of the XML parameter called <tt>first-name</tt><i>.</i> 1131 This XML parameter is defined in <tt>testng.xml</tt>:<p> 1132 1133 <h3 class="sourcetitle">testng.xml</h3> 1134 <pre class="brush: xml"> 1135 <suite name="My suite"> 1136 <parameter name="first-name" value="Cedric"/> 1137 <test name="Simple example"> 1138 <-- ... --> 1139 </pre> 1140 1141 <h4><span style="font-weight: 400">The same technique can be used for <tt>@Before/After </tt>and <tt>@Factory</tt> annotations:</span></h4> 1142 1143 <pre class="brush: java"> 1144 @Parameters({ "datasource", "jdbcDriver" }) 1145 @BeforeMethod 1146 public void beforeTest(String ds, String driver) { 1147 m_dataSource = ...; // look up the value of datasource 1148 m_jdbcDriver = driver; 1149 } 1150 </pre> 1151 1152 This time, the two Java parameter <i>ds</i> 1153 and <i>driver</i> will receive the value given to the properties <tt>datasource</tt> 1154 and <tt>jdbc-driver </tt>respectively. 1155 1156 <p> 1157 1158 Parameters can be declared optional with the <a href="../javadocs/org/testng/annotations/Optional.html"><tt>Optional</tt></a> annotation: 1159 1160 <pre class="brush: java"> 1161 @Parameters("db") 1162 @Test 1163 public void testNonExistentParameter(@Optional("mysql") String db) { ... } 1164 </pre> 1165 1166 If no parameter named "db" is found in your <tt>testng.xml</tt> file, your test method will receive the default value specified inside the <tt>@Optional</tt> annotation: "mysql". 1167 1168 <p>The <tt>@Parameters</tt> annotation can be placed at the following locations:</p><ul> 1169 <li>On any method that already has a <tt>@Test</tt>, <tt>@Before/After</tt> 1170 or <tt>@Factory</tt> annotation.</li><li>On at most one constructor of your test class. In this case, 1171 TestNG will invoke this particular constructor with the parameters 1172 initialized to the values specified in <tt>testng.xml</tt> whenever it needs 1173 to instantiate your test class. This feature can be used to initialize fields 1174 inside your classes to values that will then be used by your 1175 test methods.</li></ul> 1176 <blockquote> 1177 <p><i>Notes: 1178 1179 </i> 1180 <ul> 1181 <li><i>The XML parameters are mapped to the Java parameters in the same order as 1182 they are found in the annotation, and TestNG will issue an error if the numbers 1183 don't match. </i> 1184 <li><i>Parameters are scoped. In <tt>testng.xml</tt>, you can declare them either under a 1185 <tt><suite></tt> tag or under <tt><test></tt>. If two parameters have the same name, it's the one 1186 defined in <tt><test></tt> that has precedence. This is convenient if you need to specify 1187 a parameter applicable to all your tests and override its value only for certain 1188 tests. </i> 1189 </ul> 1190 <p></p> 1191 </blockquote> 1192 1193 1194 <h5><a class="section" indent="..." name="parameters-dataproviders">Parameters with DataProviders</a></h5> 1195 1196 1197 <p>Specifying parameters in <tt>testng.xml</tt> might not be sufficient if you need to pass complex parameters, or parameters that need to be created from Java (complex objects, objects read from a property file or a database, etc...). In this case, you can use a Data Provider to supply the values you need to test. A Data Provider is a method on your class that returns an array of array of objects. This method is annotated with <tt>@DataProvider</tt>: 1198 1199 <h3 class="sourcetitle">Java</h3> 1200 <pre class="brush: java"> 1201 //This method will provide data to any test method that declares that its Data Provider 1202 //is named "test1" 1203 @DataProvider(name = "test1") 1204 public Object[][] createData1() { 1205 return new Object[][] { 1206 { "Cedric", new Integer(36) }, 1207 { "Anne", new Integer(37)}, 1208 }; 1209 } 1210 1211 //This test method declares that its data should be supplied by the Data Provider 1212 //named "test1" 1213 @Test(dataProvider = "test1") 1214 public void verifyData1(String n1, Integer n2) { 1215 System.out.println(n1 + " " + n2); 1216 } 1217 </pre> 1218 will print 1219 1220 <pre class="brush: text"> 1221 Cedric 36 1222 Anne 37 1223 </pre> 1224 1225 A <tt>@Test</tt> method specifies its Data Provider with the <tt>dataProvider</tt> attribute. 1226 This name must correspond to a method on the same class annotated with <tt>@DataProvider(name="...")</tt> 1227 with a matching name. 1228 1229 <p> 1230 By default, the data provider will be looked for in the current test class or one of its base classes. If you want to put your data provider in a different class, it needs to be a static method or a class with a non-arg constructor, and you specify the class where it can be found in the <tt>dataProviderClass</tt> attribute: 1231 1232 <h3 class="sourcetitle">StaticProvider.java</h3> 1233 <pre class="brush: java"> 1234 public class StaticProvider { 1235 @DataProvider(name = "create") 1236 public static Object[][] createData() { 1237 return new Object[][] { 1238 new Object[] { new Integer(42) } 1239 }; 1240 } 1241 } 1242 1243 public class MyTest { 1244 @Test(dataProvider = "create", dataProviderClass = StaticProvider.class) 1245 public void test(Integer n) { 1246 // ... 1247 } 1248 } 1249 </pre> 1250 1251 The data provider supports injection too. TestNG will use the test context for the injection. 1252 1253 The Data Provider method can return one of the following two types: 1254 1255 <ul> 1256 <li>An array of array of objects (<tt>Object[][]</tt>) where the first dimension's size is the number of times the test method will be invoked and the second dimension size contains an array of objects that must be compatible with the parameter types of the test method. This is the cast illustrated by the example above.</li><li>An <tt>Iterator<Object[]></tt>. The only difference with <tt>Object[][]</tt> is that an <tt>Iterator</tt> lets you create your test data lazily. TestNG will invoke the iterator and then the test method with the parameters returned by this iterator one by one. This is particularly useful if you have a lot of parameter sets to pass to the method and you don't want to create all of them upfront. 1257 </ul> 1258 Here is an example of this feature: 1259 1260 <pre class="brush: java"> 1261 @DataProvider(name = "test1") 1262 public Iterator<Object[]> createData() { 1263 return new MyIterator(DATA); 1264 } 1265 </pre> 1266 1267 If you declare your <tt>@DataProvider</tt> as taking a <tt>java.lang.reflect.Method</tt> 1268 as first parameter, TestNG will pass the current test method for this 1269 first parameter. This is particularly useful when several test methods 1270 use the same <tt>@DataProvider</tt> and you want it to return different 1271 values depending on which test method it is supplying data for. 1272 <p> 1273 For example, the following code prints the name of the test method inside its <tt>@DataProvider</tt>: 1274 1275 <pre class="brush: java"> 1276 @DataProvider(name = "dp") 1277 public Object[][] createData(Method m) { 1278 System.out.println(m.getName()); // print test method name 1279 return new Object[][] { new Object[] { "Cedric" }}; 1280 } 1281 1282 @Test(dataProvider = "dp") 1283 public void test1(String s) { 1284 } 1285 1286 @Test(dataProvider = "dp") 1287 public void test2(String s) { 1288 } 1289 </pre> 1290 1291 and will therefore display: 1292 1293 <pre class="brush: text"> 1294 test1 1295 test2 1296 </pre> 1297 1298 Data providers can run in parallel with the attribute <tt>parallel</tt>: 1299 1300 <pre class="brush: java"> 1301 @DataProvider(parallel = true) 1302 // ... 1303 </pre> 1304 1305 Parallel data providers running from an XML file share the same pool of threads, which has a size of 10 by default. You can modify this value in the <tt><suite></tt> tag of your XML file: 1306 1307 1308 <pre class="brush: xml"> 1309 <suite name="Suite1" data-provider-thread-count="20" > 1310 ... 1311 </pre> 1312 1313 If you want to run a few specific data providers in a different thread pool, you need to run them from a different XML file. 1314 1315 <p> 1316 1317 <h5><a class="section" indent="..." name="parameters-reports">Parameters in reports</a></h5> 1318 1319 <p> 1320 1321 Parameters used to invoke your test methods are shown in the HTML reports generated by TestNG. Here is an example: 1322 1323 <p align="center"> 1324 <img src="pics/parameters.png" /> 1325 </p> 1326 1327 1328 </p> 1329 1330 1331 <!------------------------------------- 1332 DEPENDENCIES 1333 ------------------------------------> 1334 1335 1336 <h4><a class="section" indent=".." name="dependent-methods">Dependencies</a></h4> 1337 1338 <p>Sometimes, you need 1339 your test methods to be invoked in a certain order. Here are a 1340 few examples: 1341 1342 <ul> 1343 <li>To make sure a certain number of test methods have completed and succeeded 1344 before running more test methods. 1345 <li>To initialize your tests while wanting this initialization methods to be 1346 test methods as well (methods tagged with <tt>@Before/After</tt> will not be part of the 1347 final report). 1348 </ul> 1349 1350 TestNG allows you to specify dependencies either with annotations or 1351 in XML. 1352 1353 <h5><a class="section" indent="..." name="dependencies-with-annotations">Dependencies with annotations</a></h5> 1354 1355 <p>You can use the attributes <tt>dependsOnMethods</tt> or <tt>dependsOnGroups</tt>, found on the <tt>@Test</tt> annotation.</p>There are two kinds of dependencies: 1356 1357 <ul> 1358 <li><b>Hard dependencies</b>. All the methods you depend on must have run and succeeded for you to run. If at least one failure occurred in your dependencies, you will not be invoked and marked as a SKIP in the report. 1359 </li> 1360 <li><b>Soft dependencies</b>. You will always be run after the methods you depend on, even if some of them have failed. This is useful when you just want to make sure that your test methods are run in a certain order but their success doesn't really depend on the success of others. A soft dependency is obtained by adding <tt>"alwaysRun=true"</tt> in your <tt>@Test</tt> annotation. 1361 </ul> 1362 1363 Here is an example of a hard dependency: 1364 1365 <pre class="brush: java"> 1366 @Test 1367 public void serverStartedOk() {} 1368 1369 @Test(dependsOnMethods = { "serverStartedOk" }) 1370 public void method1() {} 1371 </pre> 1372 1373 <p>In this example, <tt>method1()</tt> is declared as depending on method 1374 serverStartedOk(), which guarantees that serverStartedOk() 1375 will always be invoked first.</p><p>You can also have methods that depend on entire groups:</p> 1376 1377 <pre class="brush: java"> 1378 @Test(groups = { "init" }) 1379 public void serverStartedOk() {} 1380 1381 @Test(groups = { "init" }) 1382 public void initEnvironment() {} 1383 1384 @Test(dependsOnGroups = { "init.*" }) 1385 public void method1() {} 1386 </pre> 1387 1388 <p>In this example, method1() is declared as depending on any group matching the 1389 regular expression "init.*", which guarantees that the methods <tt>serverStartedOk()</tt> 1390 and <tt>initEnvironment()</tt> will always be invoked before <tt>method1()</tt>. </p> 1391 <blockquote> 1392 <p><i>Note: as stated before, the order of invocation for methods that 1393 belong in the same group is not guaranteed to be the same across test runs.</i></p></blockquote><p>If a method depended upon fails and you have a hard dependency on it (<tt>alwaysRun=false</tt>, which is the default), the methods that depend on it are <b>not</b> 1394 marked as <tt>FAIL</tt> but as <tt>SKIP</tt>. Skipped methods will be reported as such in 1395 the final report (in a color that is neither red nor green in HTML), 1396 which is important since skipped methods are not necessarily failures.</p><p>Both <tt>dependsOnGroups</tt> and <tt>dependsOnMethods</tt> accept regular 1397 expressions as parameters. For <tt>dependsOnMethods</tt>, if you are 1398 depending on a method which happens to have several overloaded versions, all the 1399 overloaded methods will be invoked. If you only want to invoke one of the 1400 overloaded methods, you should use <tt>dependsOnGroups</tt>.</p><p>For a more advanced example of dependent methods, please refer to 1401 <a href="http://beust.com/weblog/archives/000171.html">this article</a>, which 1402 uses inheritance to provide an elegant solution to the problem of multiple 1403 dependencies.</p> 1404 1405 By default, dependent methods are grouped by class. For example, if method <tt>b()</tt> depends on method <tt>a()</tt> and you have several instances of the class that contains these methods (because of a factory of a data provider), then the invocation order will be as follows: 1406 1407 <pre class="brush: plain"> 1408 a(1) 1409 a(2) 1410 b(2) 1411 b(2) 1412 </pre> 1413 1414 TestNG will not run <tt>b()</tt> until all the instances have invoked their <tt>a()</tt> method. 1415 1416 <p> 1417 1418 This behavior might not be desirable in certain scenarios, such as for example testing a sign in and sign out of a web browser for various countries. In such a case, you would like the following ordering: 1419 1420 <pre class="brush: plain"> 1421 signIn("us") 1422 signOut("us") 1423 signIn("uk") 1424 signOut("uk") 1425 </pre> 1426 1427 For this ordering, you can use the XML attribute <tt>group-by-instances</tt>. This attribute is valid either on <suite> or <test>: 1428 1429 <pre class="brush: xml"> 1430 <suite name="Factory" group-by-instances="true"> 1431 or 1432 <test name="Factory" group-by-instances="true"> 1433 </pre> 1434 1435 1436 <h5><a class="section" indent="..." name="dependencies-in-xml">Dependencies in XML</a></h5> 1437 1438 Alternatively, you can specify your group dependencies in the <tt>testng.xml</tt> file. You use the <tt><dependencies></tt> tag to achieve this: 1439 1440 <pre class="brush: xml"> 1441 <test name="My suite"> 1442 <groups> 1443 <dependencies> 1444 <group name="c" depends-on="a b" /> 1445 <group name="z" depends-on="c" /> 1446 </dependencies> 1447 </groups> 1448 </test> 1449 </pre> 1450 1451 The <tt><depends-on></tt> attribute contains a space-separated list of groups. 1452 1453 1454 <!------------------------------------- 1455 FACTORIES 1456 ------------------------------------> 1457 1458 <h4><a class="section" indent=".." name="factories">Factories</a></h4> 1459 1460 Factories allow you to create tests dynamically. For example, imagine you 1461 want to create a test method that will access a page on a Web site several 1462 times, and you want to invoke it with different values: 1463 1464 <h3 class="sourcetitle">TestWebServer.java</h3> 1465 <pre class="brush: java"> 1466 public class TestWebServer { 1467 @Test(parameters = { "number-of-times" }) 1468 public void accessPage(int numberOfTimes) { 1469 while (numberOfTimes-- > 0) { 1470 // access the web page 1471 } 1472 } 1473 } 1474 </pre> 1475 1476 <h3 class="sourcetitle">testng.xml</h3> 1477 <pre class="brush: java"> 1478 <test name="T1"> 1479 <parameter name="number-of-times" value="10"/> 1480 <class name= "TestWebServer" /> 1481 </test> 1482 1483 <test name="T2"> 1484 <parameter name="number-of-times" value="20"/> 1485 <class name= "TestWebServer"/> 1486 </test> 1487 1488 <test name="T3"> 1489 <parameter name="number-of-times" value="30"/> 1490 <class name= "TestWebServer"/> 1491 </test> 1492 </pre> 1493 1494 This can become quickly impossible to manage, so instead, you should use a factory: 1495 1496 <h3 class="sourcetitle">WebTestFactory.java</h3> 1497 <pre class="brush: java"> 1498 public class WebTestFactory { 1499 @Factory 1500 public Object[] createInstances() { 1501 Object[] result = new Object[10]; 1502 for (int i = 0; i < 10; i++) { 1503 result[i] = new WebTest(i * 10); 1504 } 1505 return result; 1506 } 1507 } 1508 </pre> 1509 1510 and the new test class is now: 1511 1512 <h3 class="sourcetitle">WebTest.java</h3> 1513 <pre class="brush: java"> 1514 public class WebTest { 1515 private int m_numberOfTimes; 1516 public WebTest(int numberOfTimes) { 1517 m_numberOfTimes = numberOfTimes; 1518 } 1519 1520 @Test 1521 public void testServer() { 1522 for (int i = 0; i < m_numberOfTimes; i++) { 1523 // access the web page 1524 } 1525 } 1526 } 1527 </pre> 1528 1529 <p>Your <tt>testng.xml</tt> only needs to reference the class that 1530 contains the factory method, since the test instances themselves will be created 1531 at runtime:</p> 1532 1533 <pre class="brush: java"> 1534 <class name="WebTestFactory" /> 1535 </pre> 1536 1537 <p>The factory method can receive parameters just like <tt>@Test</tt> and <tt>@Before/After</tt> and it must return <tt>Object[]</tt>. 1538 The objects returned can be of any class (not necessarily the same class as the 1539 factory class) and they don't even need to contain TestNG annotations (in which 1540 case they will be ignored by TestNG).</p> 1541 1542 <p> 1543 1544 Factories can also be used with data providers, and you can leverage this functionality by putting the <tt>@Factory</tt> annotation either on a regular method or on a constructor. Here is an example of a constructor factory: 1545 1546 <pre class="brush:java"> 1547 @Factory(dataProvider = "dp") 1548 public FactoryDataProviderSampleTest(int n) { 1549 super(n); 1550 } 1551 1552 @DataProvider 1553 static public Object[][] dp() { 1554 return new Object[][] { 1555 new Object[] { 41 }, 1556 new Object[] { 42 }, 1557 }; 1558 } 1559 </pre> 1560 1561 The example will make TestNG create two test classes, on with the constructor invoked with the value 41 and the other with 42. 1562 1563 1564 1565 <!------------------------------------- 1566 CLASS LEVEL ANNOTATIONS 1567 ------------------------------------> 1568 1569 <h4><a class="section" indent=".." name="class-level">Class level annotations</a></h4> 1570 1571 The <tt>@Test</tt> annotation can be put on a class instead of a test method: 1572 1573 <h3 class="sourcetitle">Test1.java</h3> 1574 <pre class="brush: java"> 1575 @Test 1576 public class Test1 { 1577 public void test1() { 1578 } 1579 1580 public void test2() { 1581 } 1582 } 1583 </pre> 1584 The effect of a class level <tt>@Test</tt> annotation is to make all the public methods of this class to become test methods even if they are not annotated. You can still repeat the <tt>@Test</tt> annotation on a method if you want to add certain attributes. 1585 <p> 1586 1587 For example: 1588 1589 <h3 class="sourcetitle">Test1.java</h3> 1590 <pre class="brush: java"> 1591 @Test 1592 public class Test1 { 1593 public void test1() { 1594 } 1595 1596 @Test(groups = "g1") 1597 public void test2() { 1598 } 1599 } 1600 </pre> 1601 will make both <tt>test1()</tt> and <tt>test2()</tt> test methods but on top of that, <tt>test2()</tt> now belongs to the group "g1". 1602 <p> 1603 1604 1605 1606 <!------------------------------------- 1607 PARALLEL RUNNING 1608 ------------------------------------> 1609 1610 <h4><a class="section" indent=".." name="parallel-running">Parallelism and time-outs</a></h4> 1611 1612 You can instruct TestNG to run your tests in separate threads in various ways. 1613 1614 <h5><a class="section" indent="..." name="parallel-suites">Parallel suites</a></h5> 1615 1616 This is useful if you are running several suite files (e.g. "<tt>java org.testng.TestNG testng1.xml testng2.xml"</tt>) and you want each of these suites to be run in a separate thread. You can use the following command line flag to specify the size of a thread pool: 1617 1618 <pre class="brush: plain"> 1619 java org.testng.TestNG -suitethreadpoolsize 3 testng1.xml testng2.xml testng3.xml 1620 </pre> 1621 1622 The corresponding ant task name is <tt>suitethreadpoolsize</tt>. 1623 1624 <h5><a class="section" indent="..." name="parallel-tests">Parallel tests, classes and methods</a></h5> 1625 The <i>parallel</i> attribute on the <suite> tag can take one of following values: 1626 1627 <pre class="brush: xml"> 1628 <suite name="My suite" parallel="methods" thread-count="5"> 1629 </pre> 1630 1631 <pre class="brush: xml"> 1632 <suite name="My suite" parallel="tests" thread-count="5"> 1633 </pre> 1634 1635 <pre class="brush: xml"> 1636 <suite name="My suite" parallel="classes" thread-count="5"> 1637 </pre> 1638 1639 <pre class="brush: xml"> 1640 <suite name="My suite" parallel="instances" thread-count="5"> 1641 </pre> 1642 1643 <ul> 1644 <li> 1645 <b><tt>parallel="methods"</tt></b>: TestNG will run all your test methods in separate threads. Dependent methods will also run in separate threads but they will respect the order that you specified. 1646 </li> 1647 1648 <br> 1649 1650 <li> 1651 <b><tt>parallel="tests"</tt></b>: TestNG will run all the methods in the same <test> tag in the same thread, but each <test> tag will be in a separate thread. This allows you to group all your classes that are not thread safe in the same <test> and guarantee they will all run in the same thread while taking advantage of TestNG using as many threads as possible to run your tests. 1652 </li> 1653 1654 <br> 1655 1656 <li> 1657 <b><tt>parallel="classes"</tt></b>: TestNG will run all the methods in the same class in the same thread, but each class will be run in a separate thread. 1658 </li> 1659 1660 <br/> 1661 1662 <li> 1663 <b><tt>parallel="instances"</tt></b>: TestNG will run all the methods in the same instance in the same thread, but two methods on two different instances will be running in different threads. 1664 </li> 1665 1666 </ul> 1667 1668 <p> 1669 1670 1671 Additionally, the attribute <i> 1672 thread-count</i> allows you to specify how many threads should be allocated for 1673 this execution.<blockquote> 1674 <p><i>Note: the <tt>@Test</tt> attribute <tt>timeOut</tt> works in both 1675 parallel and non-parallel mode.</i></p></blockquote>You can also specify that a <tt>@Test</tt> method should be invoked from different threads. You can use the attribute <tt>threadPoolSize</tt> to achieve this result: 1676 1677 <pre class="brush: java"> 1678 @Test(threadPoolSize = 3, invocationCount = 10, timeOut = 10000) 1679 public void testServer() { 1680 </pre> 1681 In this example, the function <tt>testServer</tt> will be invoked ten times from three different threads. Additionally, a time-out of ten seconds guarantees that none of the threads will block on this thread forever. 1682 1683 <!------------------------------------- 1684 RERUNNING 1685 ------------------------------------> 1686 1687 1688 <h4><a class="section" indent=".." name="rerunning">Rerunning failed tests</a></h4> 1689 1690 Every time tests fail in a suite, TestNG creates a file called <tt>testng-failed.xml</tt> in the output directory. 1691 This XML file contains the necessary information to rerun only these methods 1692 that failed, allowing you to quickly reproduce the failures without having to 1693 run the entirety of your tests. Therefore, a typical session would look 1694 like this: 1695 1696 <pre class="brush: text"> 1697 java -classpath testng.jar;%CLASSPATH% org.testng.TestNG -d test-outputs testng.xml 1698 java -classpath testng.jar;%CLASSPATH% org.testng.TestNG -d test-outputs test-outputs\testng-failed.xml 1699 </pre> 1700 1701 <p>Note that <tt>testng-failed.xml</tt> will contain all the necessary dependent 1702 methods so that you are guaranteed to run the methods that failed without any 1703 SKIP failures.</p> 1704 1705 <h4><a class="section" indent=".." name="junit">JUnit tests</a></h4> 1706 1707 TestNG can run JUnit 3 and JUnit 4 tests. All you need to do is 1708 put the JUnit jar file on the classpath, specify your JUnit test classes in the <tt>testng.classNames</tt> 1709 property and set the <tt>testng.junit</tt> property to true: 1710 1711 <p></p> 1712 <h3 class="sourcetitle">testng.xml</h3> 1713 <pre class="brush: xml"> 1714 <test name="Test1" junit="true"> 1715 <classes> 1716 <!-- ... --> 1717 </pre> 1718 1719 <p>The behavior of TestNG in this case is similar to JUnit depending on the JUnit version found on the class path:<br> 1720 </p> 1721 <ul> 1722 <li>JUnit 3: 1723 <ul> 1724 <li>All methods starting with test* in your classes will be run</li><li>If there is a method setUp() on your test class, it will be invoked before 1725 every test method</li><li>If there is a method tearDown() on your test class, it will be invoked 1726 before after every test method</li><li>If your test class contains a method suite(), all the tests returned by 1727 this method will be invoked</li></ul> 1728 </li> 1729 <li>JUnit 4: 1730 <ul> 1731 <li>TestNG will use the <tt>org.junit.runner.JUnitCore</tt> runner to run your tests</li> 1732 </ul> 1733 </li> 1734 </ul> 1735 1736 <!------------------------------------- 1737 JUNIT 1738 ---------------------------------------> 1739 1740 1741 <!------------------------------------- 1742 RUNNING TESTNG 1743 ------------------------------------> 1744 <h4><a class="section" indent=".." name="running-testng-programmatically">Running TestNG programmatically</a></h4> 1745 1746 You can invoke TestNG from your own programs very easily: 1747 1748 <h3 class="sourcetitle">Java</h3> 1749 <pre class="brush: java"> 1750 TestListenerAdapter tla = new TestListenerAdapter(); 1751 TestNG testng = new TestNG(); 1752 testng.setTestClasses(new Class[] { Run2.class }); 1753 testng.addListener(tla); 1754 testng.run(); 1755 </pre> 1756 1757 This example creates a <tt><a href="http://testng.org/javadocs/org/testng/TestNG.html">TestNG</a></tt> object and runs the test class <tt>Run2</tt>. It also adds a <tt>TestListener</tt>. You can either use the adapter class <tt><a href="http://testng.org/javadocs/org/testng/TestListenerAdapter.html">org.testng.TestListenerAdapter</a></tt> or implement <tt><a href="http://testng.org/javadocs/org/testng/ITestListener.html">org.testng.ITestListener</a></tt> yourself. This interface contains various callback methods that let you keep track of when a test starts, succeeds, fails, etc... 1758 <p> 1759 Similary, you can invoke TestNG on a <tt>testng.xml</tt> file or you can create a virtual <tt>testng.xml</tt> file yourself. In order to do this, you can use the classes found the package <tt><a href="http://testng.org/javadocs/org/testng/xml/package-frame.html">org.testng.xml</a></tt>: <tt><a href="http://testng.org/javadocs/org/testng/xml/XmlClass.html">XmlClass</a></tt>, <tt><a href="http://testng.org/javadocs/org/testng/xml/XmlTest.html">XmlTest</a></tt>, etc... Each of these classes correspond to their XML tag counterpart. 1760 <p> 1761 For example, suppose you want to create the following virtual file: 1762 1763 <pre class="brush: java"> 1764 <suite name="TmpSuite" > 1765 <test name="TmpTest" > 1766 <classes> 1767 <class name="test.failures.Child" /> 1768 <classes> 1769 </test> 1770 </suite> 1771 </pre> 1772 1773 You would use the following code: 1774 1775 <pre class="brush: java"> 1776 XmlSuite suite = new XmlSuite(); 1777 suite.setName("TmpSuite"); 1778 1779 XmlTest test = new XmlTest(suite); 1780 test.setName("TmpTest"); 1781 List<XmlClass> classes = new ArrayList<XmlClass>(); 1782 classes.add(new XmlClass("test.failures.Child")); 1783 test.setXmlClasses(classes) ; 1784 </pre> 1785 And then you can pass this <tt>XmlSuite</tt> to TestNG: 1786 1787 <pre class="brush: java"> 1788 List<XmlSuite> suites = new ArrayList<XmlSuite>(); 1789 suites.add(suite); 1790 TestNG tng = new TestNG(); 1791 tng.setXmlSuites(suites); 1792 tng.run(); 1793 </pre> 1794 1795 <p>Please see the <a href="../doc/javadocs/org/testng/package-summary.html" target="mainFrame">JavaDocs</a> for the entire API.</p><p> 1796 1797 1798 <!------------------------------------- 1799 BEANSHELL 1800 ------------------------------------> 1801 1802 <h4><a class="section" indent=".." name="beanshell">BeanShell and advanced group selection</a></h4> 1803 1804 1805 <p>If the <tt><include></tt> and <tt><exclude></tt> tags in <tt>testng.xml</tt> are not enough for your needs, you can use a <a href="http://beanshell.org">BeanShell</a> expression to decide whether a certain test method should be included in a test run or not. You specify this expression just under the <tt><test></tt> tag:</p> 1806 1807 <h3 class="sourcetitle">testng.xml</h3> 1808 <pre class="brush: xml"> 1809 <test name="BeanShell test"> 1810 <method-selectors> 1811 <method-selector> 1812 <script language="beanshell"><![CDATA[ 1813 groups.containsKey("test1") 1814 ]]></script> 1815 </method-selector> 1816 </method-selectors> 1817 <!-- ... --> 1818 </pre> 1819 1820 When a <tt><script></tt> tag is found in <tt>testng.xml</tt>, TestNG will ignore subsequent <tt><include></tt> and <tt><exclude></tt> of groups and methods in the current <tt><test></tt> tag: your BeanShell expression will be the only way to decide whether a test method is included or not.</p><p>Here are additional information on the BeanShell script:</p><ul> 1821 <li> 1822 It must return a boolean value. Except for this constraint, any valid BeanShell code is allowed (for example, you might want to return <tt>true </tt>during week days and false during weekends, which would allow you to run tests differently depending on the date).<br> 1823 </li> 1824 <li> 1825 TestNG defines the following variables for your convenience:<br> <b><tt>java.lang.reflect.Method method</tt></b>: the current test method.<br> <b>org.testng.ITestNGMethod testngMethod</b>: the description of the current test method.<br> <b><tt>java.util.Map<String, String> groups</tt></b>: a map of the groups the current test method belongs to.<br> 1826 </li> 1827 <li> 1828 You might want to surround your expression with a <tt>CDATA</tt> declaration (as shown above) to avoid tedious quoting of reserved XML characters).<br> 1829 </li> 1830 </ul> 1831 1832 <!------------------------------------- 1833 ANNOTATION TRANSFORMERS 1834 ------------------------------------> 1835 1836 <h4><a class="section" indent=".." name="annotationtransformers">Annotation Transformers</a></h4> 1837 1838 TestNG allows you to modify the content of all the annotations at runtime. This is especially useful if the annotations in the source code are right most of the time, but there are a few situations where you'd like to override their value. 1839 <p> 1840 1841 In order to achieve this, you need to use an Annotation Transformer. 1842 1843 <p> 1844 1845 An Annotation Transformer is a class that implements the following interface: 1846 1847 <pre class="brush: java"> 1848 public interface IAnnotationTransformer { 1849 1850 /** 1851 * This method will be invoked by TestNG to give you a chance 1852 * to modify a TestNG annotation read from your test classes. 1853 * You can change the values you need by calling any of the 1854 * setters on the ITest interface. 1855 * 1856 * Note that only one of the three parameters testClass, 1857 * testConstructor and testMethod will be non-null. 1858 * 1859 * @param annotation The annotation that was read from your 1860 * test class. 1861 * @param testClass If the annotation was found on a class, this 1862 * parameter represents this class (null otherwise). 1863 * @param testConstructor If the annotation was found on a constructor, 1864 * this parameter represents this constructor (null otherwise). 1865 * @param testMethod If the annotation was found on a method, 1866 * this parameter represents this method (null otherwise). 1867 */ 1868 public void transform(ITest annotation, Class testClass, 1869 Constructor testConstructor, Method testMethod); 1870 } 1871 </pre> 1872 1873 Like all the other TestNG listeners, you can specify this class either on the command line or with ant: 1874 1875 <p> 1876 1877 <pre class="brush: java"> 1878 java org.testng.TestNG -listener MyTransformer testng.xml 1879 </pre> 1880 1881 or programmatically: 1882 1883 <p> 1884 1885 <pre class="brush: java"> 1886 TestNG tng = new TestNG(); 1887 tng.setAnnotationTransformer(new MyTransformer()); 1888 // ... 1889 </pre> 1890 1891 When the method <tt>transform()</tt> is invoked, you can call any of the setters on the <tt>ITest test</tt> parameter to alter its value before TestNG proceeds further. 1892 <p> 1893 For example, here is how you would override the attribute <tt>invocationCount</tt> but only on the test method <tt>invoke()</tt> of one of your test classes: 1894 1895 <pre class="brush: java"> 1896 public class MyTransformer implements IAnnotationTransformer { 1897 public void transform(ITest annotation, Class testClass, 1898 Constructor testConstructor, Method testMethod) 1899 { 1900 if ("invoke".equals(testMethod.getName())) { 1901 annotation.setInvocationCount(5); 1902 } 1903 } 1904 } 1905 </pre> 1906 1907 <tt>IAnnotationTransformer</tt> only lets you modify a <tt>@Test</tt> annotation. If you need to modify another TestNG annotation (a configuration annotation, <tt>@Factory</tt> or <tt>@DataProvider</tt>), use an <tt>IAnnotationTransformer2</tt>. 1908 1909 <!------------------------------------- 1910 METHOD INTERCEPTORS 1911 ------------------------------------> 1912 1913 <h4><a class="section" indent=".." name="methodinterceptors">Method Interceptors</a></h4> 1914 1915 Once TestNG has calculated in what order the test methods will be invoked, these methods are split in two groups: 1916 1917 <ul> 1918 <li><em>Methods run sequentially</em>. These are all the test methods that have dependencies or dependents. These methods will be run in a specific order. 1919 <li><em>Methods run in no particular order</em>. These are all the methods that don't belong in the first category. The order in which these test methods are run is random and can vary from one run to the next (although by default, TestNG will try to group test methods by class). 1920 </ul> 1921 1922 In order to give you more control on the methods that belong to the second category, TestNG defines the following interface: 1923 1924 <pre class="brush: java"> 1925 public interface IMethodInterceptor { 1926 1927 List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context); 1928 1929 } 1930 1931 </pre> 1932 1933 The list of methods passed in parameters are all the methods that can be run in any order. Your <tt>intercept</tt> method is expected to return a similar list of <tt>IMethodInstance</tt>, which can be either of the following: 1934 1935 <ul> 1936 <li>The same list you received in parameter but in a different order. 1937 <li>A smaller list of <tt>IMethodInstance</tt> objects. 1938 <li>A bigger list of <tt>IMethodInstance</tt> objects. 1939 </ul> 1940 1941 Once you have defined your interceptor, you pass it to TestNG as a listener. For example: 1942 1943 <p> 1944 1945 <h3 class="sourcetitle">Shell</h3> 1946 <pre class="brush: text"> 1947 java -classpath "testng-jdk15.jar:test/build" org.testng.TestNG -listener test.methodinterceptors.NullMethodInterceptor 1948 -testclass test.methodinterceptors.FooTest 1949 </pre> 1950 1951 For the equivalent <tt>ant</tt> syntax, see the <tt>listeners</tt> attribute in the <a href="ant.html">ant documentation</a>. 1952 <p> 1953 For example, here is a Method Interceptor that will reorder the methods so that test methods that belong to the group "fast" are always run first: 1954 1955 <pre class="brush: java"> 1956 public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) { 1957 List<IMethodInstance> result = new ArrayList<IMethodInstance>(); 1958 for (IMethodInstance m : methods) { 1959 Test test = m.getMethod().getConstructorOrMethod().getAnnotation(Test.class); 1960 Set<String> groups = new HashSet<String>(); 1961 for (String group : test.groups()) { 1962 groups.add(group); 1963 } 1964 if (groups.contains("fast")) { 1965 result.add(0, m); 1966 } 1967 else { 1968 result.add(m); 1969 } 1970 } 1971 return result; 1972 } 1973 </pre> 1974 1975 1976 <!------------------------------------- 1977 TESTNG LISTENERS 1978 ------------------------------------> 1979 1980 <h4><a class="section" indent=".." name="testng-listeners">TestNG Listeners</a></h4> 1981 1982 There are several interfaces that allow you to modify TestNG's behavior. These interfaces are broadly called "TestNG Listeners". Here are a few listeners: 1983 1984 <ul> 1985 <li><tt>IAnnotationTransformer</tt> (<a href="#annotationtransformers">doc</a>, <a href="../javadocs/org/testng/IAnnotationTransformer.html">javadoc</a>) 1986 <li><tt>IAnnotationTransformer2</tt> (<a href="#annotationtransformers">doc</a>, <a href="../javadocs/org/testng/IAnnotationTransformer2.html">javadoc</a>) 1987 <li><tt>IHookable</tt> (<a href="#ihookable">doc</a>, <a href="../javadocs/org/testng/IHookable.html">javadoc</a>) 1988 <li><tt>IInvokedMethodListener</tt> (doc, <a href="../javadocs/org/testng/IInvokedMethodListener.html">javadoc</a>) 1989 <li><tt>IMethodInterceptor</tt> (<a href="#methodinterceptors">doc</a>, <a href="../javadocs/org/testng/IMethodInterceptor.html">javadoc</a>) 1990 <li><tt>IReporter</tt> (<a href="#logging-reporters">doc</a>, <a href="../javadocs/org/testng/IReporter.html">javadoc</a>) 1991 <li><tt>ISuiteListener</tt> (doc, <a href="../javadocs/org/testng/ISuiteListener.html">javadoc</a>) 1992 <li><tt>ITestListener</tt> (<a href="#logging-listeners">doc</a>, <a href="../javadocs/org/testng/ITestListener.html">javadoc</a>) 1993 </ul> 1994 1995 When you implement one of these interfaces, you can let TestNG know about it with either of the following ways: 1996 1997 <ul> 1998 <li><a href="#running-testng">Using -listener on the command line.</a> 1999 <li><a href="ant.html">Using <listeners> with ant.</a> 2000 <li>Using <listeners> in your <tt>testng.xml</tt> file. 2001 <li>Using the <tt>@Listeners</tt> annotation on any of your test classes. 2002 <li>Using <tt>ServiceLoader</tt>. 2003 </ul> 2004 2005 <h5><a class="section" indent="..." name="listeners-testng-xml">Specifying listeners with <tt>testng.xml</tt> or in Java</a></h5> 2006 2007 Here is how you can define listeners in your <tt>testng.xml</tt> file: 2008 2009 <pre class="brush: xml"> 2010 <suite> 2011 2012 <listeners> 2013 <listener class-name="com.example.MyListener" /> 2014 <listener class-name="com.example.MyMethodInterceptor" /> 2015 </listeners> 2016 2017 ... 2018 2019 </pre> 2020 2021 Or if you prefer to define these listeners in Java: 2022 2023 <pre class="brush: java"> 2024 @Listeners({ com.example.MyListener.class, com.example.MyMethodInterceptor.class }) 2025 public class MyTest { 2026 // ... 2027 } 2028 </pre> 2029 2030 The <tt>@Listeners</tt> annotation can contain any class that extends <tt>org.testng.ITestNGListener</tt> <b>except</b> <tt>IAnnotationTransformer</tt> and <tt>IAnnotationTransformer2</tt>. The reason is that these listeners need to be known very early in the process so that TestNG can use them to rewrite your annotations, therefore you need to specify these listeners in your <tt>testng.xml</tt> file. 2031 2032 <p> 2033 2034 Note that the <tt>@Listeners</tt> annotation will apply to your entire suite file, just as if you had specified it in a <tt>testng.xml</tt> file. If you want to restrict its scope (for example, only running on the current class), the code in your listener could first check the test method that's about to run and decide what to do then. 2035 2036 <h5><a class="section" indent="..." name="listeners-service-loader">Specifying listeners with <tt>ServiceLoader</tt></a></h5> 2037 2038 Finally, the JDK offers a very elegant mechanism to specify implementations of interfaces on the class path via the <tt><a href="http://download.oracle.com/javase/6/docs/api/java/util/ServiceLoader.html">ServiceLoader</a></tt> class. 2039 2040 <p> 2041 With ServiceLoader, all you need to do is create a jar file that contains your listener(s) and a few configuration files, put that jar file on the classpath when you run TestNG and TestNG will automatically find them. 2042 <p> 2043 2044 Here is a concrete example of how it works. 2045 2046 <p> 2047 2048 Let's start by creating a listener (any TestNG listener should work): 2049 2050 <pre class="brush: java"> 2051 package test.tmp; 2052 2053 public class TmpSuiteListener implements ISuiteListener { 2054 @Override 2055 public void onFinish(ISuite suite) { 2056 System.out.println("Finishing"); 2057 } 2058 2059 @Override 2060 public void onStart(ISuite suite) { 2061 System.out.println("Starting"); 2062 } 2063 } 2064 </pre> 2065 2066 Compile this file, then create a file at the location <tt>META-INF/services/org.testng.ITestNGListener</tt>, which will name the implementation(s) you want for this interface. 2067 2068 <p> 2069 2070 You should end up with the following directory structure, with only two files: 2071 2072 <pre class="brush: plain; highlight: [4, 7]"> 2073 $ tree 2074 |____META-INF 2075 | |____services 2076 | | |____org.testng.ITestNGListener 2077 |____test 2078 | |____tmp 2079 | | |____TmpSuiteListener.class 2080 2081 $ cat META-INF/services/org.testng.ITestNGListener 2082 test.tmp.TmpSuiteListener 2083 </pre> 2084 2085 Create a jar of this directory: 2086 2087 <pre class="brush: plain"> 2088 $ jar cvf ../sl.jar . 2089 added manifest 2090 ignoring entry META-INF/ 2091 adding: META-INF/services/(in = 0) (out= 0)(stored 0%) 2092 adding: META-INF/services/org.testng.ITestNGListener(in = 26) (out= 28)(deflated -7%) 2093 adding: test/(in = 0) (out= 0)(stored 0%) 2094 adding: test/tmp/(in = 0) (out= 0)(stored 0%) 2095 adding: test/tmp/TmpSuiteListener.class(in = 849) (out= 470)(deflated 44%) 2096 </pre> 2097 2098 Next, put this jar file on your classpath when you invoke TestNG: 2099 2100 <pre class="brush: plain"> 2101 $ java -classpath sl.jar:testng.jar org.testng.TestNG testng-single.yaml 2102 Starting 2103 f2 11 2 2104 PASSED: f2("2") 2105 Finishing 2106 </pre> 2107 2108 This mechanism allows you to apply the same set of listeners to an entire organization just by adding a jar file to the classpath, instead of asking every single developer to remember to specify these listeners in their testng.xml file. 2109 2110 <!------------------------------------- 2111 DEPENDENCY INJECTION 2112 ------------------------------------> 2113 2114 <h4><a class="section" indent=".." name="dependency-injection">Dependency injection</a></h4> 2115 2116 TestNG supports two different kinds of dependency injection: native (performed by TestNG itself) and external (performed by a dependency injection framework such as Guice). 2117 2118 <h5><a class="section" indent="..." name="native-dependency-injection">Native dependency injection</a></h5> 2119 2120 TestNG lets you declare additional parameters in your methods. When this happens, TestNG will automatically fill these parameters with the right value. Dependency injection can be used in the following places: 2121 2122 <ul> 2123 2124 <li> 2125 Any @Before method or @Test method can declare a parameter of type <tt>ITestContext</tt>. 2126 <li> 2127 Any @AfterMethod method can declare a parameter of type <tt>ITestResult</tt>, which will reflect the result of the test method that was just run. 2128 <li> 2129 Any @Before and @After methods can declare a parameter of type <tt>XmlTest</tt>, which contain the current <tt><test></tt> tag. 2130 <li> 2131 Any @BeforeMethod (and @AfterMethod) can declare a parameter of type 2132 <tt>java.lang.reflect.Method</tt>. This parameter will receive the 2133 test method that will be called once this @BeforeMethod finishes (or 2134 after the method as run for @AfterMethod). 2135 <li> 2136 Any @BeforeMethod can declare a parameter of type <tt>Object[]</tt>. This parameter will receive the list of parameters that are about to be fed to the upcoming test method, which could be either injected by TestNG, such as <tt>java.lang.reflect.Method</tt> or come from a <tt>@DataProvider</tt>. 2137 <li> 2138 Any @DataProvider can declare a parameter of type 2139 <tt>ITestContext</tt> or <tt>java.lang.reflect.Method</tt>. The 2140 latter parameter will receive the test method that is about to be invoked. 2141 </ul> 2142 2143 You can turn off injection with the <tt>@NoInjection</tt> annotation: 2144 2145 <pre class="brush: java; highlight: [9]"> 2146 public class NoInjectionTest { 2147 2148 @DataProvider(name = "provider") 2149 public Object[][] provide() throws Exception { 2150 return new Object[][] { { CC.class.getMethod("f") } }; 2151 } 2152 2153 @Test(dataProvider = "provider") 2154 public void withoutInjection(@NoInjection Method m) { 2155 Assert.assertEquals(m.getName(), "f"); 2156 } 2157 2158 @Test(dataProvider = "provider") 2159 public void withInjection(Method m) { 2160 Assert.assertEquals(m.getName(), "withInjection"); 2161 } 2162 } 2163 </pre> 2164 2165 <h5><a class="section" indent="..." name="guice-dependency-injection">Guice dependency injection</a></h5> 2166 2167 If you use Guice, TestNG gives you an easy way to inject your test objects with a Guice module: 2168 2169 <pre class="brush: java"> 2170 @Guice(modules = GuiceExampleModule.class) 2171 public class GuiceTest extends SimpleBaseTest { 2172 2173 @Inject 2174 ISingleton m_singleton; 2175 2176 @Test 2177 public void singletonShouldWork() { 2178 m_singleton.doSomething(); 2179 } 2180 2181 } 2182 </pre> 2183 2184 In this example, <tt>GuiceExampleModule</tt> is expected to bind the interface <tt>ISingleton</tt> to some concrete class: 2185 2186 <pre class="brush: java"> 2187 public class GuiceExampleModule implements Module { 2188 2189 @Override 2190 public void configure(Binder binder) { 2191 binder.bind(ISingleton.class).to(ExampleSingleton.class).in(Singleton.class); 2192 } 2193 2194 } 2195 </pre> 2196 2197 If you need more flexibility in specifying which modules should be used to instantiate your test classes, you can specify a module factory: 2198 2199 <pre class="brush: java"> 2200 @Guice(moduleFactory = ModuleFactory.class) 2201 public class GuiceModuleFactoryTest { 2202 2203 @Inject 2204 ISingleton m_singleton; 2205 2206 @Test 2207 public void singletonShouldWork() { 2208 m_singleton.doSomething(); 2209 } 2210 } 2211 </pre> 2212 2213 The module factory needs to implement the interface <a href="../javadocs/org/testng/IModuleFactory.html">IModuleFactory</a>: 2214 2215 <pre class="brush: java"> 2216 public interface IModuleFactory { 2217 /** 2218 * @param context The current test context 2219 * @param testClass The test class 2220 * 2221 * @return The Guice module that should be used to get an instance of this 2222 * test class. 2223 */ 2224 Module createModule(ITestContext context, Class<?> testClass); 2225 } 2226 </pre> 2227 2228 Your factory will be passed an instance of the test context and the test class that TestNG needs to instantiate. Your <tt>createModule</tt> method should return a Guice Module that will know how to instantiate this test class. You can use the test context to find out more information about your environment, such as parameters specified in <tt>testng.xml</tt>, etc... 2229 2230 You will get even more flexibility and Guice power with <tt>parent-module</tt> and <tt>guice-stage</tt> suite parameters. 2231 <tt>guice-stage</tt> allow you to chose the <a href="https://github.com/google/guice/wiki/Bootstrap"><tt>Stage</tt></a> used to create the parent injector. 2232 The default one is <tt>DEVELOPMENT</tt>. Other allowed values are <tt>PRODUCTION</tt> and <tt>TOOL</tt>. 2233 Here is how you can define parent-module in your test.xml file: 2234 2235 <pre class="brush: xml"> 2236 <suite parent-module="com.example.SuiteParenModule" guice-stage="PRODUCTION"> 2237 </suite> 2238 </pre> 2239 2240 TestNG will create this module only once for given suite. Will also use this module for obtaining instances of test specific Guice modules and module factories, then will create child injector for each test class. 2241 2242 With such approach you can declare all common bindings in parent-module also you can inject binding declared in parent-module in module and module factory. Here is an example of this functionality: 2243 2244 <pre class="brush: java"> 2245 package com.example; 2246 2247 public class ParentModule extends AbstractModule { 2248 @Override 2249 protected void conigure() { 2250 bind(MyService.class).toProvider(MyServiceProvider.class); 2251 bind(MyContext.class).to(MyContextImpl.class).in(Singleton.class); 2252 } 2253 } 2254 </pre> 2255 2256 <pre class="brush: java"> 2257 package com.example; 2258 2259 public class TestModule extends AbstractModule { 2260 private final MyContext myContext; 2261 2262 @Inject 2263 TestModule(MyContext myContext) { 2264 this.myContext = myContext 2265 } 2266 2267 @Override 2268 protected void configure() { 2269 bind(MySession.class).toInstance(myContext.getSession()); 2270 } 2271 } 2272 </pre> 2273 2274 <pre class="brush: xml"> 2275 <suite parent-module="com.example.ParentModule"> 2276 </suite> 2277 </pre> 2278 2279 <pre class="brush: java"> 2280 package com.example; 2281 2282 @Test 2283 @Guice(modules = TestModule.class) 2284 public class TestClass { 2285 @Inject 2286 MyService myService; 2287 @Inject 2288 MySession mySession; 2289 2290 public void testServiceWithSession() { 2291 myService.serve(mySession); 2292 } 2293 } 2294 </pre> 2295 2296 As you see ParentModule declares binding for MyService and MyContext classes. Then MyContext is injected using constructor injection into TestModule class, which also declare binding for MySession. Then parent-module in test XML file is set to ParentModule class, this enables injection in TestModule. Later in TestClass you see two injections: 2297 * MyService - binding taken from ParentModule 2298 * MySession - binding taken from TestModule 2299 This configuration ensures you that all tests in this suite will be run with same session instance, the MyContextImpl object is only created once per suite, this give you possibility to configure common environment state for all tests in suite. 2300 2301 <!------------------------------------- 2302 INVOKED METHOD LISTENERS 2303 ------------------------------------> 2304 2305 <h4><a class="section" indent=".." name="invokedmethodlistener">Listening to method invocations</a></h4> 2306 2307 The listener <tt><a href="../javadocs/org/testng/IInvokedMethodListener.html">IInvokedMethodListener</a></tt> allows you to be notified whenever TestNG is about to invoke a test (annotated with <tt>@Test</tt>) or configuration (annotated with any of the <tt>@Before</tt> or <tt>@After</tt> annotation) method. You need to implement the following interface: 2308 2309 <pre class="brush: java"> 2310 public interface IInvokedMethodListener extends ITestNGListener { 2311 void beforeInvocation(IInvokedMethod method, ITestResult testResult); 2312 void afterInvocation(IInvokedMethod method, ITestResult testResult); 2313 } 2314 </pre> 2315 2316 and declare it as a listener, as explained in <a href="#testng-listeners">the section about TestNG listeners</a>. 2317 2318 <p> 2319 2320 2321 <!------------------------------------- 2322 IHOOKABLE AND ICONFIGURABLE 2323 ------------------------------------> 2324 2325 <h4><a class="section" indent=".." name="ihookable">Overriding test methods</a></h4> 2326 2327 TestNG allows you to override and possibly skip the invocation of test methods. One example of where this is useful is if you need to your test methods with a specific security manager. You achieve this by providing a listener that implements <a href="../javadocs/org/testng/IHookable.html"><tt>IHookable</tt></a>. 2328 <p> 2329 Here is an example with JAAS: 2330 2331 <pre class="brush: java"> 2332 public class MyHook implements IHookable { 2333 public void run(final IHookCallBack icb, ITestResult testResult) { 2334 // Preferably initialized in a @Configuration method 2335 mySubject = authenticateWithJAAs(); 2336 2337 Subject.doAs(mySubject, new PrivilegedExceptionAction() { 2338 public Object run() { 2339 icb.callback(testResult); 2340 } 2341 }; 2342 } 2343 } 2344 </pre> 2345 2346 <p> 2347 <!------------------------------------- 2348 IALTERSUITELISTENER 2349 ------------------------------------> 2350 2351 <h4><a class="section" indent=".." name="ialtersuite">Altering suites (or) tests</a></h4> 2352 Sometimes you may need to just want to alter a suite (or) a test tag in a suite xml in runtime without having to 2353 change the contents of a suite file. 2354 2355 <p> 2356 A classic example for this would be to try and leverage your existing suite file and try using it for simulating a load test 2357 on your "Application under test". 2358 At the minimum you would end up duplicating the contents of your <test> tag multiple 2359 times and create a new suite xml file and work with. But this doesn't seem to scale a lot. 2360 2361 <p> 2362 TestNG allows you to alter a suite (or) a test tag in your suite xml file at runtime via listeners. 2363 You achieve this by providing a listener that implements <a href="../javadocs/org/testng/IAlterSuiteListener.html"><tt>IAlterSuiteListener</tt></a>. 2364 Please refer to <a href="#testng-listeners">Listeners section</a> to learn about listeners. 2365 <p> 2366 Here is an example that shows how the suite name is getting altered in runtime: 2367 2368 <pre class="brush: java"> 2369 public class AlterSuiteNameListener implements IAlterSuiteListener { 2370 2371 @Override 2372 public void alter(List<XmlSuite> suites) { 2373 XmlSuite suite = suites.get(0); 2374 suite.setName(getClass().getSimpleName()); 2375 } 2376 } 2377 </pre> 2378 2379 This listener can only be added with either of the following ways: 2380 <ul> 2381 <li>Through the <tt><listeners></tt> tag in the suite xml file.</li> 2382 <li>Through a <a href="#listeners-service-loader">Service Loader</a></li> 2383 </ul> 2384 2385 This listener cannot be added to execution using the <tt>@Listeners</tt> annotation. 2386 2387 <!------------------------------------ 2388 TEST SUCCESS 2389 ------------------------------------> 2390 2391 <h3><a class="section" indent="." name="test-results">Test results</a></h3> 2392 2393 2394 <h4><a class="section" indent=".." name="success-failure">Success, failure and assert</a></h4> 2395 2396 2397 <p>A test is considered successful if it completed without throwing any 2398 exception or if it threw an exception that was expected (see the 2399 documentation for the <tt>expectedExceptions</tt> attribute found on the <tt>@Test</tt> annotation). 2400 </p> 2401 2402 <p>Your test methods will typically be made of calls that can throw an 2403 exception, or of various assertions (using the Java "assert" keyword). An 2404 "assert" failing will trigger an AssertionErrorException, which in turn will 2405 mark the method as failed (remember to use -ea on the JVM if you are not seeing 2406 the assertion errors).</p><p>Here is an example test method:</p> 2407 2408 <pre class="brush: java"> 2409 @Test 2410 public void verifyLastName() { 2411 assert "Beust".equals(m_lastName) : "Expected name Beust, for" + m_lastName; 2412 } 2413 </pre> 2414 2415 TestNG also include JUnit's Assert class, which lets you perform 2416 assertions on complex objects: 2417 2418 <pre class="brush: java"> 2419 import static org.testng.AssertJUnit.*; 2420 //... 2421 @Test 2422 public void verify() { 2423 assertEquals("Beust", m_lastName); 2424 } 2425 </pre> 2426 <p>Note that the above code use a static import in order to be able to use the 2427 <tt>assertEquals</tt> method without having to prefix it by its class. 2428 2429 <!------------------------------------- 2430 LOGGING 2431 ------------------------------------> 2432 </p> 2433 2434 <h4><a class="section" indent=".." name="logging">Logging and results</a></h4> 2435 2436 The results of the test run are created in a file called <tt>index.html</tt> in the 2437 directory specified when launching SuiteRunner. This file points to 2438 various other HTML and text files that contain the result of the entire test 2439 run. You can see a typical example 2440 <a href="http://testng.org/test-output/index.html">here</a>. 2441 2442 <p> 2443 It's very easy to generate your own reports with TestNG with Listeners and Reporters: 2444 2445 <ul> 2446 <li><b>Listeners</b> implement the interface <a href="../javadocs/org/testng/ITestListener.html"><tt>org.testng.ITestListener</tt></a> and are notified in real time of when a test starts, passes, fails, etc...</li><li><b>Reporters</b> implement the interface <a href="../javadocs/org/testng/IReporter.html"><tt>org.testng.IReporter</tt></a> and are notified when all the suites have been run by TestNG. The IReporter instance receives a list of objects that describe the entire test run.</li></ul>For example, if you want to generate a PDF report of your test run, you don't need to be notified in real time of the test run so you should probably use an <tt>IReporter</tt>. If you'd like to write a real-time reporting of your tests, such as a GUI with a progress bar or a text reporter displaying dots (".") as each test is invoked (as is explained below), <tt>ITestListener</tt> is your best choice. 2447 2448 <h5><a class="section" indent="..." name="logging-listeners">Logging Listeners</a></h5> 2449 2450 Here is a listener that displays a "." for each passed test, a "F" for each failure and a "S" for each skip: 2451 2452 <pre class="brush: java"> 2453 public class DotTestListener extends TestListenerAdapter { 2454 private int m_count = 0; 2455 2456 @Override 2457 public void onTestFailure(ITestResult tr) { 2458 log("F"); 2459 } 2460 2461 @Override 2462 public void onTestSkipped(ITestResult tr) { 2463 log("S"); 2464 } 2465 2466 @Override 2467 public void onTestSuccess(ITestResult tr) { 2468 log("."); 2469 } 2470 2471 private void log(String string) { 2472 System.out.print(string); 2473 if (++m_count % 40 == 0) { 2474 System.out.println(""); 2475 } 2476 } 2477 } 2478 </pre> 2479 2480 In this example, I chose to extend <a href="../javadocs/org/testng/TestListenerAdapter.html"><tt>TestListenerAdapter</tt></a>, which implements <a href="../javadocs/org/testng/ITestListener.html"><tt>ITestListener</tt></a> with empty methods, so I don't have to override other methods from the interface that I have no interest in. You can implement the interface directly if you prefer. 2481 2482 <p> 2483 Here is how I invoke TestNG to use this new listener: 2484 2485 <h3 class="sourcetitle">Shell</h3> 2486 <pre class="brush: text"> 2487 java -classpath testng.jar;%CLASSPATH% org.testng.TestNG -listener org.testng.reporters.DotTestListener test\testng.xml 2488 </pre> 2489 2490 and the output: 2491 2492 <p> 2493 2494 <h3 class="sourcetitle">Shell</h3> 2495 <pre class="brush: text"> 2496 ........................................ 2497 ........................................ 2498 ........................................ 2499 ........................................ 2500 ........................................ 2501 ......................... 2502 =============================================== 2503 TestNG JDK 1.5 2504 Total tests run: 226, Failures: 0, Skips: 0 2505 =============================================== 2506 </pre> 2507 2508 Note that when you use <tt>-listener</tt>, TestNG will automatically determine the type of listener you want to use. 2509 2510 <h5><a class="section" indent="..." name="logging-reporters">Logging Reporters</a></h5> 2511 2512 The <a href="../javadocs/org/testng/IReporter.html"><tt>org.testng.IReporter</tt></a> interface only has one method: 2513 2514 <pre class="brush: java"> 2515 public void generateReport(List<ISuite</a>> suites, String outputDirectory) 2516 </pre> 2517 2518 This method will be invoked by TestNG when all the suites have been run and you can inspect its parameters to access all the information on the run that was just completed. 2519 2520 <p> 2521 2522 <h5><a class="section" indent="..." name="logging-junitreports">JUnitReports</a></h5> 2523 2524 <p> 2525 2526 2527 TestNG contains a listener that takes the TestNG results 2528 and outputs an XML file that can then be fed to JUnitReport. <a href="http://testng.org/test-report/junit-noframes.html"> 2529 Here</a> is an example, and the ant task to create this report: 2530 2531 <h3 class="sourcetitle">build.xml</h3> 2532 <pre class="brush: xml"> 2533 <target name="reports"> 2534 <junitreport todir="test-report"> 2535 <fileset dir="test-output"> 2536 <include name="*/*.xml"/> 2537 </fileset> 2538 2539 <report format="noframes" todir="test-report"/> 2540 </junitreport> 2541 </target> 2542 </pre> 2543 <blockquote> 2544 <em>Note: a current incompatibility between the JDK 1.5 and JUnitReports 2545 prevents the frame version from working, so you need to specify "noframes" to 2546 get this to work for now.</em> 2547 </blockquote> 2548 2549 <h5><a class="section" indent="..." name="logging-reporter-api">Reporter API</a></h5> 2550 2551 <p> 2552 If you need to log messages that should appear in the generated HTML reports, you can use the class <tt><a href="../javadocs/org/testng/Reporter.html">org.testng.Reporter</a></tt>: 2553 2554 <blockquote class="brush: text"> 2555 <font color="#ffffff"> </font><font color="#000000">Reporter.log</font><font color="#000000">(</font><font color="#2a00ff">"M3 WAS CALLED"</font><font color="#000000">)</font><font color="#000000">;</font> 2556 2557 </blockquote> 2558 2559 <p align="center"> 2560 2561 <img src="pics/show-output1.png" /> 2562 <img src="pics/show-output2.png" /> 2563 2564 </p> 2565 2566 <h5><a class="section" indent="..." name="logging-xml-reports">XML Reports</a></h5> 2567 2568 <p> 2569 TestNG offers an XML reporter capturing TestNG specific information that is not available in JUnit reports. This is particulary useful when the user's test environment needs to consume XML results with TestNG-specific data that the JUnit format can't provide. Below is a sample of the output of such a reporter: 2570 </p> 2571 2572 <pre class="brush: xml"> 2573 <testng-results> 2574 <suite name="Suite1"> 2575 <groups> 2576 <group name="group1"> 2577 <method signature="com.test.TestOne.test2()" name="test2" class="com.test.TestOne"/> 2578 <method signature="com.test.TestOne.test1()" name="test1" class="com.test.TestOne"/> 2579 </group> 2580 <group name="group2"> 2581 <method signature="com.test.TestOne.test2()" name="test2" class="com.test.TestOne"/> 2582 </group> 2583 </groups> 2584 <test name="test1"> 2585 <class name="com.test.TestOne"> 2586 <test-method status="FAIL" signature="test1()" name="test1" duration-ms="0" 2587 started-at="2007-05-28T12:14:37Z" description="someDescription2" 2588 finished-at="2007-05-28T12:14:37Z"> 2589 <exception class="java.lang.AssertionError"> 2590 <short-stacktrace> 2591 <![CDATA[ 2592 java.lang.AssertionError 2593 ... Removed 22 stack frames 2594 ]]> 2595 </short-stacktrace> 2596 </exception> 2597 </test-method> 2598 <test-method status="PASS" signature="test2()" name="test2" duration-ms="0" 2599 started-at="2007-05-28T12:14:37Z" description="someDescription1" 2600 finished-at="2007-05-28T12:14:37Z"> 2601 </test-method> 2602 <test-method status="PASS" signature="setUp()" name="setUp" is-config="true" duration-ms="15" 2603 started-at="2007-05-28T12:14:37Z" finished-at="2007-05-28T12:14:37Z"> 2604 </test-method> 2605 </class> 2606 </test> 2607 </suite> 2608 </testng-results> 2609 </pre> 2610 <p>This reporter is injected along with the other default listeners so you can get this type of output by default. The listener provides some properties that can tweak the reporter to fit your needs. The following table contains a list of these properties with a short explanation: 2611 </p> 2612 <table border="1" width="100%" id="table6"> 2613 <tr> 2614 <th>Property</th> 2615 <th>Comment</th> 2616 <th>Default value</th> 2617 </tr> 2618 <tr> 2619 <td>outputDirectory</td> 2620 <td>A <tt>String</tt> indicating the directory where should the XML files be outputed.</td> 2621 <td>The TestNG output directory</td> 2622 </tr> 2623 <tr> 2624 <td>timestampFormat</td> 2625 <td>Specifies the format of date fields that are generated by this reporter</td> 2626 <td>yyyy-MM-dd'T'HH:mm:ss'Z'</td> 2627 </tr> 2628 <tr> 2629 <td>fileFragmentationLevel</td> 2630 <td>An integer having the values 1, 2 or 3, indicating the way that the XML files are generated: 2631 <br> 2632 <pre> 2633 1 - will generate all the results in one file. 2634 2 - each suite is generated in a separate XML file that is linked to the main file. 2635 3 - same as 2 plus separate files for test-cases that are referenced from the suite files. 2636 </pre> 2637 </td> 2638 <td>1</td> 2639 </tr> 2640 <tr> 2641 <td>splitClassAndPackageNames</td> 2642 <td>This boolean specifies the way that class names are generated for the <tt><class></tt> element. 2643 For example, you will get <tt><class class="com.test.MyTest"></tt> for false and <tt><class class="MyTest" package="com.test"></tt> for true. 2644 </td> 2645 <td>false</td> 2646 </tr> 2647 <tr> 2648 <td>generateGroupsAttribute</td> 2649 <td>A boolean indicating if a <tt>groups</tt> attribute should be generated for the <tt><test-method></tt> element. This feature aims at providing a 2650 straight-forward method of retrieving the groups that include a test method without having to surf through the <tt><group></tt> elements. 2651 </td> 2652 <td>false</td> 2653 </tr> 2654 <tr> 2655 <td>generateTestResultAttributes</td> 2656 <td> 2657 A boolean indicating if an <tt><attributes></tt> tag should be generated for each <tt><test-method></tt> element, containing the test result 2658 attributes (See <tt>ITestResult.setAttribute()</tt> about setting test result attributes). Each attribute <tt>toString()</tt> representation will be 2659 written in a <tt><attribute name="[attribute name]"></tt> tag. 2660 </td> 2661 <td>false</td> 2662 </tr> 2663 <tr> 2664 <td>stackTraceOutputMethod</td> 2665 <td>Specifies the type of stack trace that is to be generated for exceptions and has the following values: 2666 <br> 2667 <pre> 2668 0 - no stacktrace (just Exception class and message). 2669 1 - a short version of the stack trace keeping just a few lines from the top 2670 2 - the complete stacktrace with all the inner exceptions 2671 3 - both short and long stacktrace 2672 </pre> 2673 </td> 2674 <td>2</td> 2675 </tr> 2676 <tr> 2677 <td>generateDependsOnMethods</td> 2678 <td>Use this attribute to enable/disable the generation of a <tt>depends-on-methods</tt> attribute for the <tt><test-method></tt> element. 2679 </td> 2680 <td>true</td> 2681 </tr> 2682 <tr> 2683 <td>generateDependsOnGroups</td> 2684 <td>Enable/disable the generation of a <tt>depends-on-groups</tt> attribute for the <tt><test-method></tt> element. 2685 </td> 2686 <td>true</td> 2687 </tr> 2688 </table> 2689 <p> 2690 In order to configure this reporter you can use the <tt>-reporter</tt> option in the command line or the <a href="http://testng.org/doc/ant.html">Ant</a> 2691 task with the nested <tt><reporter></tt> element. For each of these you must specify the class <tt>org.testng.reporters.XMLReporter</tt>. 2692 Please note that you cannot configure the built-in reporter because this one will only use default settings. If you need just the XML report with custom settings 2693 you will have to add it manually with one of the two methods and disable the default listeners. 2694 </p> 2695 2696 <!------------------------------------ 2697 YAML 2698 ------------------------------------> 2699 2700 <h3><a class="section" name="yaml">YAML</a></h3> 2701 2702 TestNG supports <a href="http://www.yaml.org/">YAML</a> as an alternate way of specifying your suite file. For example, the following XML file: 2703 2704 <pre class="brush: xml"> 2705 <!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" > 2706 2707 <suite name="SingleSuite" verbose="2" thread-count="4" > 2708 2709 <parameter name="n" value="42" /> 2710 2711 <test name="Regression2"> 2712 <groups> 2713 <run> 2714 <exclude name="broken" /> 2715 </run> 2716 </groups> 2717 2718 <classes> 2719 <class name="test.listeners.ResultEndMillisTest" /> 2720 </classes> 2721 </test> 2722 </suite> 2723 </pre> 2724 <p>and here is its YAML version:</p> 2725 <pre class="brush: plain"> 2726 name: SingleSuite 2727 threadCount: 4 2728 parameters: { n: 42 } 2729 2730 tests: 2731 - name: Regression2 2732 parameters: { count: 10 } 2733 excludedGroups: [ broken ] 2734 classes: 2735 - test.listeners.ResultEndMillisTest 2736 </pre> 2737 2738 Here is <a href="https://github.com/cbeust/testng/blob/master/src/test/resources/testng.xml">TestNG's own suite file</a>, and its <a href="https://github.com/cbeust/testng/blob/master/src/test/resources/testng.yaml">YAML counterpart</a>. 2739 2740 <p> 2741 2742 You might find the YAML file format easier to read and to maintain. YAML files are also recognized by the TestNG Eclipse plug-in. You can find more information about YAML and TestNG in this <a href="http://beust.com/weblog/2010/08/15/yaml-the-forgotten-victim-of-the-format-wars/">blog post</a>. 2743 2744 2745 <!----------------------------------------------------------------------> 2746 2747 <a name="testng-dtd"> 2748 <hr width="100%"> 2749 <p>Back to my <a href="http://beust.com/weblog">home page</a>.</p><p>Or check out some of my other projects:</p><ul> 2750 <li><a href="http://beust.com/ejbgen">EJBGen</a>: an EJB tag 2751 generator.</li><li><a href="http://testng.org">TestNG</a>: A testing framework using annotations, test groups and method parameters. </li><li><a href="http://beust.com/doclipse">Doclipse</a>: a JavaDoc tag 2752 Eclipse plug-in.</li><li><a href="http://beust.com/j15">J15</a>: an Eclipse plug-in to help 2753 you migrate your code to the new JDK 1.5 constructs.</li><li><a href="http://beust.com/sgen">SGen</a>: a replacement for 2754 XDoclet with an easy plug-in architecture.</li><li><a href="http://beust.com/canvas">Canvas</a>: a template generator 2755 based on the Groovy language.</li></ul><p> 2756 </p> 2757 2758 <script src="http://www.google-analytics.com/urchin.js" type="text/javascript"> 2759 </script> 2760 <script type="text/javascript"> 2761 _uacct = "UA-238215-2"; 2762 urchinTracker(); 2763 </script> 2764 2765
