1 TODO for TestNG 2 3 * Pass the XmlTest in @Before/@After methods 4 * Allow a testng.xml file to be passed when -testjar is used 5 * Add onStart to IConfigurationListener (create a new interface, actually) 6 * Add timeout to @Before/@After 7 * Pass parameters from ant 8 * Make it possible to specify groups on command line and classes in testng.xml 9 (and any combinations thereof: command line, ant, testng.xml) 10 * DataProvider index in testng.xml 11 * Create a servlet for remote driving 12 * Add time-outs at the testng.xml. Also: test and suite time-outs? http://tinyurl.com/kbwxq 13 * Add working dir to the ant task 14 * Add a servlet so TestNG can be invoked from a web browser 15 * Make it possible to add listeners from the Eclipse plug-in 16 17 Doc: 18 19 * Document the fact that @Test methods with return values are ignored. 20 21 =========================================================================== 22 Older TODO's: 23 24 * Show enabled=false methods in the reports, as well as methods 25 in groups that were not run 26 * Multi-threading for invocationCount and maybe for <test> too 27 * Annotation to specify that a method should be called concurrently by n threads 28 (on second thought, we should do that for an entire group) 29 * more thread ideas: http://www.theserverside.com/news/thread.tss?thread_id=38922 30 * package support for command line and ant 31 * Parameters for classes (to be passed as parameters to constructors) 32 * testng-dist.zip should contain a top-level directory 33 * For dependent methods, if the user is trying to run an incomplete graph 34 (A depends on B but B is being excluded from the run), what to do? Ignore 35 the exclusion on B and run it anyway, or abort with an exception explaining 36 what's happening?) 37 * Make timeOut() work with milliseconds (but keep seconds for backward 38 compatibility) 39 * Improve the plug-in API so people can add listeners without having to 40 modify TestRunner 41 * Use factories for the programmatic API. 42 * Add dynamic generation of tests 43 * Make Javadoc comments over methods appear in the final report 44 45 Documentation: 46 * IHookable 47 * List<IReporter> 48 49 50 DONE 51 52 * Retry patch 53 * If a method with invocationCount fails, don't run the others 54 * Allow multiple listeners in ant task 55 * Add working dir to the ant task 56 * Introduce "test" and "suite" parameters to @Test at the class level to 57 avoid having to use testng.xml 58 * Remove TestNG stack traces from the report 59 * When 0 tests were run, exit with an error http://tinyurl.com/ftng6 60 * Show all the extra output for all methods in a single 61 dedicated page 62 * report API 63 * Show parameters used to invoke a specific test 64 * show skipped groups/methods in HTML report 65 * beforeTestGroups 66 * <testng classfileset> doesn't add to classpath 67 * threadPoolSize 68 * Parameter logging 69 * JavaDoc for org.testng.TestNG 70 * org.testng.Reporter in the HTML report (screenshot ready) 71 * Document @Parameters only works for @Test (should mention @Factory and @Configuration) 72 * Document: <testng classfileset> doesn't add to classpath 73 * Document org.testng.Reporter in the HTML report (screenshot ready) 74 * Document parameter logging 75 * JavaDoc for org.testng.TestNG 76 * Document threadPoolSize 77 * Make sure it can run several times in the same JVM 78 * Implement invocationCount and successPercentage 79 * Support multiple testng.xml (TestNG allows it but not the reporters 80 yet) 81 * The HTML reporter collapses all the suites into one report, we should 82 create one HTML report per suite 83 * testng-failed.xml should contain the parameters of testng.xml (if any) 84 * Create a testng-failed.xml that includes dependent methods 85 * Generic reported with compare(ITestResult, ITestResult) for 86 easier reporter for "slowest method first" or generate testng-failed.xml 87 * Iterator factories 88 * configuration methods don't respect inheritance 89 - build.xml should issue a clear error message if trying to build with JDK1.4 90 - Implement <tasdkdef resource="testnganttasks"> so we can define 91 ant tasks for TestNG and JUnitConverter automatically 92 - Write documentation to declare ant task in section 3.2.8 93 - Documentation for alwaysRun 94 - Allow to specify packages or prefix in the <classes> tag: 95 <classes prefix="com.beust.testng"><class name="A"><class name="B"> /> 96 - New assert classes 97 - New ways to launch 98 - JUnitConverter documentation 99 - new beforeSuite/afterSuite 100 101 * in testng-failures.xml include the beforeSuite/afterSuite methods (very tricky) 102 * Provide log.properties configuration (not using log any more) 103 * Make @ExpectedExceptions fail if no exception is thrown 104 * Make timeOut() work in non-parallel mode (the default mode needs to become 105 parallel=true thread-count=1 106 * The exception thrown when a test passes with a @ExpectedExceptions is not 107 available via the TestNG API: ITestResult.getThrowable(). 108 * Add assert API for arrays and collections (undecided yet: partial asserting) 109 * dependsOnMethods 110 Allow to specify <groups> at the <suite> level 111 Make TestNG run on class files and not just on testng.xml 112 Make TestNG run on a jar file that has a testng.xml file in its root or just on all 113 the classes inside that jar file. 114 115 Implement parameter passing of tests: define a property in the properties 116 file and pass it to the test method: 117 118 @Test(params = { "${fn}", "${ln}" } 119 public void testNames(String firstName, String lastName) { 120 } 121 122 Run groups of groups 123 List all tests that will be run, or show methods per group 124 HTML generation 125 Make test and class methods discoverable 126 JUnit adapter 127 Multiple ExpectedException 128 Inheritance 129 Test listeners 130 Group regexps for launching 131 132 ==== 133 134 A new comment has been posted on your blog Otaku, Cedric's weblog, on entry 135 #149 (The poor shape of Unit Testing). 136 http://beust.com/weblog/archives/000149.html 137 138 IP Address: 68.72.49.189 139 Name: Curt Cox 140 Email Address: curtcox (a] gmail.com 141 URL: 142 143 Comments: 144 145 For whatever its worth, here are my problems with JUnit. I've largely developed work-arounds. 146 147 1. Interface-based testing is awkward. 148 2. Only fast-fail tests are supported. 149 3. The design is brittle, poorly documented, and thus hard to extend. 150 151 How should interface-based testing be handled in TestNG? A nice feature would be bundled tests for interfaces in the java* namespace that are automatically applied. In other words: 152 - all classes will fail a (supressible) test if they violate the contract for equals() and hashCode() 153 - classes that implement Comparable will fail a test if they fail to implement Comparable 154 - and likewise for java.io.Serializable, java.util.Map, java.util.List, etc... 155 156 When I say all tests are fast-fail in JUnit, what I mean is that the first failed assertion in a method causes the test to exit. While this is usually desirable, a more conversational style of tests can sometimes be much easier to read and write. Such a conversational test doesn't generate a simple failure, but rather a score. The score could be either x of y passed or x percent passed. The important part is that the first failure doesn't terminate the test. 157 158 As I said, I've largely developed work-arounds for doing these in JUnit, but developing tools for conversational tests that play nice with the various JUnit runners was a real challenge. The exact contract that Eclipse expects of JUnit tests turns out to be different than what either of the bundled runners or the Ant task expect. Anyone who considers either the JUnit code or interfaces well-documented has a much different concept of well-documented than I do. 159 160 161
