1 ================ 2 lit TODO Items 3 ================ 4 5 Infrastructure 6 ============== 7 8 1. Change to always load suites, then resolve command line arguments? 9 10 Currently we expect each input argument to be a path on disk; we do a 11 recursive search to find the test suite for each item, but then we only do a 12 local search based at the input path to find tests. Additionally, for any path 13 that matches a file on disk we explicitly construct a test instance (bypassing 14 the formats on discovery implementation). 15 16 This has a couple problems: 17 18 * The test format doesn't have control over the test instances that result 19 from file paths. 20 21 * It isn't possible to specify virtual tests as inputs. For example, it is not 22 possible to specify an individual subtest to run with the googletest format. 23 24 * The test format doesn't have full control over the discovery of tests in 25 subdirectories. 26 27 Instead, we should move to a model whereby first all of the input specifiers 28 are resolved to test suites, and then the resolution of the input specifier is 29 delegated to each test suite. This could take a couple forms: 30 31 * We could resolve to test suites, then fully load each test suite, then have 32 a fixed process to map input specifiers to tests in the test suite 33 (presumably based on path-in-suite derivations). This has the benefit of 34 being consistent across all test formats, but the downside of requiring 35 loading the entire test suite. 36 37 * We could delegate all of the resolution of specifiers to the test 38 suite. This would allow formats that anticipate large test suites to manage 39 their own resolution for better performance. We could provide a default 40 resolution strategy that was similar to what we do now (start at subpaths 41 for directories, but allow the test format control over what happens for 42 individual tests). 43 44 2. Consider move to identifying all tests by path-to-test-suite and then path to 45 subtest, and don't use test suite names. 46 47 Currently the test suite name is presented as part of test names, but it has 48 no other useful function, and it is something that has to be skipped over to 49 cut-and-paste a name to subsequently use to rerun a test. If we just 50 represented each test suite by the path to its suite, then it would allow more 51 easy cut-and-paste of the test output lines. This has the downside that the 52 lines might get rather long. 53 54 3. Allow 'lit' driver to cooperate with test formats and suites to add options 55 (or at least sanitize accepted params). 56 57 We have started to use the --params method more and more extensively, and it is 58 cumbersome and error prone. Additionally, there are currently various options 59 ``lit`` honors that should more correctly be specified as belonging to the 60 ShTest test format. 61 62 It would be really nice if we could allow test formats and test suites to add 63 their own options to be parsed. The difficulty here, of course, is that we 64 don't know what test formats or test suites are in use until we have parsed the 65 input specifiers. For test formats we could ostensibly require all the possible 66 formats to be registered in order to have options, but for test suites we would 67 certainly have to load the suite before we can query it for what options it 68 understands. 69 70 That leaves us with the following options: 71 72 * Currently we could almost get away with parsing the input specifiers without 73 having done option parsing first (the exception is ``--config-prefix``) but 74 that isn't a very extensible design. 75 76 * We could make a distinction in the command line syntax for test format and 77 test suite options. For example, we could require something like:: 78 79 lit -j 1 -sv input-specifier -- --some-format-option 80 81 which would be relatively easy to implement with optparser (I think). 82 83 * We could allow fully interspersed arguments by first extracting the options 84 lit knows about and parsing them, then dispatching the remainder to the 85 formats. This seems the most convenient for users, who are unlikely to care 86 about (or even be aware of) the distinction between the generic lit 87 infrastructure and format or suite specific options. 88 89 4. Eliminate duplicate execution models for ShTest tests. 90 91 Currently, the ShTest format uses tests written with shell-script like syntax, 92 and executes them in one of two ways. The first way is by converting them into 93 a bash script and literally executing externally them using bash. The second 94 way is through the use of an internal shell parser and shell execution code 95 (built on the subprocess module). The external execution mode is used on most 96 Unix systems that have bash, the internal execution mode is used on Windows. 97 98 Having two ways to do the same thing is error prone and leads to unnecessary 99 complexity in the testing environment. Additionally, because the mode that 100 converts scripts to bash doesn't try and validate the syntax, it is possible 101 to write tests that use bash shell features unsupported by the internal 102 shell. Such tests won't work on Windows but this may not be obvious to the 103 developer writing the test. 104 105 Another limitation is that when executing the scripts externally, the ShTest 106 format has no idea which commands fail, or what output comes from which 107 commands, so this limits how convenient the output of ShTest failures can be 108 and limits other features (for example, knowing what temporary files were 109 written). 110 111 We should eliminate having two ways of executing the same tests to reduce 112 platform differences and make it easier to develop new features in the ShTest 113 module. This is currently blocked on: 114 115 * The external execution mode is faster in some situations, because it avoids 116 being bottlenecked on the GIL. This can hopefully be obviated simply by 117 using --use-processes. 118 119 * Some tests in LLVM/Clang are explicitly disabled with the internal shell 120 (because they use features specific to bash). We would need to rewrite these 121 tests, or add additional features to the internal shell handling to allow 122 them to pass. 123 124 5. Consider changing core to support setup vs. execute distinction. 125 126 Many of the existing test formats are cleanly divided into two phases, once 127 parses the test format and extracts XFAIL and REQUIRES information, etc., and 128 the other code actually executes the test. 129 130 We could make this distinction part of the core infrastructure and that would 131 enable a couple things: 132 133 * The REQUIREs handling could be lifted to the core, which is nice. 134 135 * This would provide a clear place to insert subtest support, because the 136 setup phase could be responsible for providing subtests back to the 137 core. That would provide part of the infrastructure to parallelize them, for 138 example, and would probably interact well with other possible features like 139 parameterized tests. 140 141 * This affords a clean implementation of --no-execute. 142 143 * One possible downside could be for test formats that cannot determine their 144 subtests without having executed the test. Supporting such formats would 145 either force the test to actually be executed in the setup stage (which 146 might be ok, as long as the API was explicitly phrased to support that), or 147 would mean we are forced into supporting subtests as return values from the 148 execute phase. 149 150 Any format can just keep all of its code in execute, presumably, so the only 151 cost of implementing this is its impact on the API and futures changes. 152 153 154 Miscellaneous 155 ============= 156 157 * Move temp directory name into local test config. 158 159 * Support valgrind in all configs, and LLVM style valgrind. 160 161 * Support a timeout / ulimit. 162 163 * Create an explicit test suite object (instead of using the top-level 164 TestingConfig object). 165
