1 =pod 2 3 =head1 NAME 4 5 lit - LLVM Integrated Tester 6 7 =head1 SYNOPSIS 8 9 B<lit> [I<options>] [I<tests>] 10 11 =head1 DESCRIPTION 12 13 B<lit> is a portable tool for executing LLVM and Clang style test suites, 14 summarizing their results, and providing indication of failures. B<lit> is 15 designed to be a lightweight testing tool with as simple a user interface as 16 possible. 17 18 B<lit> should be run with one or more I<tests> to run specified on the command 19 line. Tests can be either individual test files or directories to search for 20 tests (see L<"TEST DISCOVERY">). 21 22 Each specified test will be executed (potentially in parallel) and once all 23 tests have been run B<lit> will print summary information on the number of tests 24 which passed or failed (see L<"TEST STATUS RESULTS">). The B<lit> program will 25 execute with a non-zero exit code if any tests fail. 26 27 By default B<lit> will use a succinct progress display and will only print 28 summary information for test failures. See L<"OUTPUT OPTIONS"> for options 29 controlling the B<lit> progress display and output. 30 31 B<lit> also includes a number of options for controlling how tests are exected 32 (specific features may depend on the particular test format). See L<"EXECUTION 33 OPTIONS"> for more information. 34 35 Finally, B<lit> also supports additional options for only running a subset of 36 the options specified on the command line, see L<"SELECTION OPTIONS"> for 37 more information. 38 39 Users interested in the B<lit> architecture or designing a B<lit> testing 40 implementation should see L<"LIT ARCHITECTURE"> 41 42 =head1 GENERAL OPTIONS 43 44 =over 45 46 =item B<-h>, B<--help> 47 48 Show the B<lit> help message. 49 50 =item B<-j> I<N>, B<--threads>=I<N> 51 52 Run I<N> tests in parallel. By default, this is automatically chosen to match 53 the number of detected available CPUs. 54 55 =item B<--config-prefix>=I<NAME> 56 57 Search for I<NAME.cfg> and I<NAME.site.cfg> when searching for test suites, 58 instead of I<lit.cfg> and I<lit.site.cfg>. 59 60 =item B<--param> I<NAME>, B<--param> I<NAME>=I<VALUE> 61 62 Add a user defined parameter I<NAME> with the given I<VALUE> (or the empty 63 string if not given). The meaning and use of these parameters is test suite 64 dependent. 65 66 =back 67 68 =head1 OUTPUT OPTIONS 69 70 =over 71 72 =item B<-q>, B<--quiet> 73 74 Suppress any output except for test failures. 75 76 =item B<-s>, B<--succinct> 77 78 Show less output, for example don't show information on tests that pass. 79 80 =item B<-v>, B<--verbose> 81 82 Show more information on test failures, for example the entire test output 83 instead of just the test result. 84 85 =item B<--no-progress-bar> 86 87 Do not use curses based progress bar. 88 89 =back 90 91 =head1 EXECUTION OPTIONS 92 93 =over 94 95 =item B<--path>=I<PATH> 96 97 Specify an addition I<PATH> to use when searching for executables in tests. 98 99 =item B<--vg> 100 101 Run individual tests under valgrind (using the memcheck tool). The 102 I<--error-exitcode> argument for valgrind is used so that valgrind failures will 103 cause the program to exit with a non-zero status. 104 105 =item B<--vg-arg>=I<ARG> 106 107 When I<--vg> is used, specify an additional argument to pass to valgrind itself. 108 109 =item B<--time-tests> 110 111 Track the wall time individual tests take to execute and includes the results in 112 the summary output. This is useful for determining which tests in a test suite 113 take the most time to execute. Note that this option is most useful with I<-j 114 1>. 115 116 =back 117 118 =head1 SELECTION OPTIONS 119 120 =over 121 122 =item B<--max-tests>=I<N> 123 124 Run at most I<N> tests and then terminate. 125 126 =item B<--max-time>=I<N> 127 128 Spend at most I<N> seconds (approximately) running tests and then terminate. 129 130 =item B<--shuffle> 131 132 Run the tests in a random order. 133 134 =back 135 136 =head1 ADDITIONAL OPTIONS 137 138 =over 139 140 =item B<--debug> 141 142 Run B<lit> in debug mode, for debugging configuration issues and B<lit> itself. 143 144 =item B<--show-suites> 145 146 List the discovered test suites as part of the standard output. 147 148 =item B<--no-tcl-as-sh> 149 150 Run Tcl scripts internally (instead of converting to shell scripts). 151 152 =item B<--repeat>=I<N> 153 154 Run each test I<N> times. Currently this is primarily useful for timing tests, 155 other results are not collated in any reasonable fashion. 156 157 =back 158 159 =head1 EXIT STATUS 160 161 B<lit> will exit with an exit code of 1 if there are any FAIL or XPASS 162 results. Otherwise, it will exit with the status 0. Other exit codes used for 163 non-test related failures (for example a user error or an internal program 164 error). 165 166 =head1 TEST DISCOVERY 167 168 The inputs passed to B<lit> can be either individual tests, or entire 169 directories or hierarchies of tests to run. When B<lit> starts up, the first 170 thing it does is convert the inputs into a complete list of tests to run as part 171 of I<test discovery>. 172 173 In the B<lit> model, every test must exist inside some I<test suite>. B<lit> 174 resolves the inputs specified on the command line to test suites by searching 175 upwards from the input path until it finds a I<lit.cfg> or I<lit.site.cfg> 176 file. These files serve as both a marker of test suites and as configuration 177 files which B<lit> loads in order to understand how to find and run the tests 178 inside the test suite. 179 180 Once B<lit> has mapped the inputs into test suites it traverses the list of 181 inputs adding tests for individual files and recursively searching for tests in 182 directories. 183 184 This behavior makes it easy to specify a subset of tests to run, while still 185 allowing the test suite configuration to control exactly how tests are 186 interpreted. In addition, B<lit> always identifies tests by the test suite they 187 are in, and their relative path inside the test suite. For appropriately 188 configured projects, this allows B<lit> to provide convenient and flexible 189 support for out-of-tree builds. 190 191 =head1 TEST STATUS RESULTS 192 193 Each test ultimately produces one of the following six results: 194 195 =over 196 197 =item B<PASS> 198 199 The test succeeded. 200 201 =item B<XFAIL> 202 203 The test failed, but that is expected. This is used for test formats which allow 204 specifying that a test does not currently work, but wish to leave it in the test 205 suite. 206 207 =item B<XPASS> 208 209 The test succeeded, but it was expected to fail. This is used for tests which 210 were specified as expected to fail, but are now succeeding (generally because 211 the feautre they test was broken and has been fixed). 212 213 =item B<FAIL> 214 215 The test failed. 216 217 =item B<UNRESOLVED> 218 219 The test result could not be determined. For example, this occurs when the test 220 could not be run, the test itself is invalid, or the test was interrupted. 221 222 =item B<UNSUPPORTED> 223 224 The test is not supported in this environment. This is used by test formats 225 which can report unsupported tests. 226 227 =back 228 229 Depending on the test format tests may produce additional information about 230 their status (generally only for failures). See the L<Output|"LIT OUTPUT"> 231 section for more information. 232 233 =head1 LIT INFRASTRUCTURE 234 235 This section describes the B<lit> testing architecture for users interested in 236 creating a new B<lit> testing implementation, or extending an existing one. 237 238 B<lit> proper is primarily an infrastructure for discovering and running 239 arbitrary tests, and to expose a single convenient interface to these 240 tests. B<lit> itself doesn't know how to run tests, rather this logic is 241 defined by I<test suites>. 242 243 =head2 TEST SUITES 244 245 As described in L<"TEST DISCOVERY">, tests are always located inside a I<test 246 suite>. Test suites serve to define the format of the tests they contain, the 247 logic for finding those tests, and any additional information to run the tests. 248 249 B<lit> identifies test suites as directories containing I<lit.cfg> or 250 I<lit.site.cfg> files (see also B<--config-prefix>. Test suites are initially 251 discovered by recursively searching up the directory hierarchy for all the input 252 files passed on the command line. You can use B<--show-suites> to display the 253 discovered test suites at startup. 254 255 Once a test suite is discovered, its config file is loaded. Config files 256 themselves are Python modules which will be executed. When the config file is 257 executed, two important global variables are predefined: 258 259 =over 260 261 =item B<lit> 262 263 The global B<lit> configuration object (a I<LitConfig> instance), which defines 264 the builtin test formats, global configuration parameters, and other helper 265 routines for implementing test configurations. 266 267 =item B<config> 268 269 This is the config object (a I<TestingConfig> instance) for the test suite, 270 which the config file is expected to populate. The following variables are also 271 available on the I<config> object, some of which must be set by the config and 272 others are optional or predefined: 273 274 B<name> I<[required]> The name of the test suite, for use in reports and 275 diagnostics. 276 277 B<test_format> I<[required]> The test format object which will be used to 278 discover and run tests in the test suite. Generally this will be a builtin test 279 format available from the I<lit.formats> module. 280 281 B<test_src_root> The filesystem path to the test suite root. For out-of-dir 282 builds this is the directory that will be scanned for tests. 283 284 B<test_exec_root> For out-of-dir builds, the path to the test suite root inside 285 the object directory. This is where tests will be run and temporary output files 286 places. 287 288 B<environment> A dictionary representing the environment to use when executing 289 tests in the suite. 290 291 B<suffixes> For B<lit> test formats which scan directories for tests, this 292 variable as a list of suffixes to identify test files. Used by: I<ShTest>, 293 I<TclTest>. 294 295 B<substitutions> For B<lit> test formats which substitute variables into a test 296 script, the list of substitutions to perform. Used by: I<ShTest>, I<TclTest>. 297 298 B<unsupported> Mark an unsupported directory, all tests within it will be 299 reported as unsupported. Used by: I<ShTest>, I<TclTest>. 300 301 B<parent> The parent configuration, this is the config object for the directory 302 containing the test suite, or None. 303 304 B<on_clone> The config is actually cloned for every subdirectory inside a test 305 suite, to allow local configuration on a per-directory basis. The I<on_clone> 306 variable can be set to a Python function which will be called whenever a 307 configuration is cloned (for a subdirectory). The function should takes three 308 arguments: (1) the parent configuration, (2) the new configuration (which the 309 I<on_clone> function will generally modify), and (3) the test path to the new 310 directory being scanned. 311 312 =back 313 314 =head2 TEST DISCOVERY 315 316 Once test suites are located, B<lit> recursively traverses the source directory 317 (following I<test_src_root>) looking for tests. When B<lit> enters a 318 sub-directory, it first checks to see if a nest test suite is defined in that 319 directory. If so, it loads that test suite recursively, otherwise it 320 instantiates a local test config for the directory (see L<"LOCAL CONFIGURATION 321 FILES">). 322 323 Tests are identified by the test suite they are contained within, and the 324 relative path inside that suite. Note that the relative path may not refer to an 325 actual file on disk; some test formats (such as I<GoogleTest>) define "virtual 326 tests" which have a path that contains both the path to the actual test file and 327 a subpath to identify the virtual test. 328 329 =head2 LOCAL CONFIGURATION FILES 330 331 When B<lit> loads a subdirectory in a test suite, it instantiates a local test 332 configuration by cloning the configuration for the parent direction -- the root 333 of this configuration chain will always be a test suite. Once the test 334 configuration is cloned B<lit> checks for a I<lit.local.cfg> file in the 335 subdirectory. If present, this file will be loaded and can be used to specialize 336 the configuration for each individual directory. This facility can be used to 337 define subdirectories of optional tests, or to change other configuration 338 parameters -- for example, to change the test format, or the suffixes which 339 identify test files. 340 341 =head2 LIT EXAMPLE TESTS 342 343 The B<lit> distribution contains several example implementations of test suites 344 in the I<ExampleTests> directory. 345 346 =head1 SEE ALSO 347 348 L<valgrind(1)> 349 350 =head1 AUTHOR 351 352 Written by Daniel Dunbar and maintained by the LLVM Team (L<http://llvm.org/>). 353 354 =cut 355
