1 lit - LLVM Integrated Tester 2 ============================ 3 4 SYNOPSIS 5 -------- 6 7 :program:`lit` [*options*] [*tests*] 8 9 DESCRIPTION 10 ----------- 11 12 :program:`lit` is a portable tool for executing LLVM and Clang style test 13 suites, summarizing their results, and providing indication of failures. 14 :program:`lit` is designed to be a lightweight testing tool with as simple a 15 user interface as possible. 16 17 :program:`lit` should be run with one or more *tests* to run specified on the 18 command line. Tests can be either individual test files or directories to 19 search for tests (see :ref:`test-discovery`). 20 21 Each specified test will be executed (potentially in parallel) and once all 22 tests have been run :program:`lit` will print summary information on the number 23 of tests which passed or failed (see :ref:`test-status-results`). The 24 :program:`lit` program will execute with a non-zero exit code if any tests 25 fail. 26 27 By default :program:`lit` will use a succinct progress display and will only 28 print summary information for test failures. See :ref:`output-options` for 29 options controlling the :program:`lit` progress display and output. 30 31 :program:`lit` also includes a number of options for controlling how tests are 32 executed (specific features may depend on the particular test format). See 33 :ref:`execution-options` for more information. 34 35 Finally, :program:`lit` also supports additional options for only running a 36 subset of the options specified on the command line, see 37 :ref:`selection-options` for more information. 38 39 Users interested in the :program:`lit` architecture or designing a 40 :program:`lit` testing implementation should see :ref:`lit-infrastructure`. 41 42 GENERAL OPTIONS 43 --------------- 44 45 .. option:: -h, --help 46 47 Show the :program:`lit` help message. 48 49 .. option:: -j N, --threads=N 50 51 Run ``N`` tests in parallel. By default, this is automatically chosen to 52 match the number of detected available CPUs. 53 54 .. option:: --config-prefix=NAME 55 56 Search for :file:`{NAME}.cfg` and :file:`{NAME}.site.cfg` when searching for 57 test suites, instead of :file:`lit.cfg` and :file:`lit.site.cfg`. 58 59 .. option:: --param NAME, --param NAME=VALUE 60 61 Add a user defined parameter ``NAME`` with the given ``VALUE`` (or the empty 62 string if not given). The meaning and use of these parameters is test suite 63 dependent. 64 65 .. _output-options: 66 67 OUTPUT OPTIONS 68 -------------- 69 70 .. option:: -q, --quiet 71 72 Suppress any output except for test failures. 73 74 .. option:: -s, --succinct 75 76 Show less output, for example don't show information on tests that pass. 77 78 .. option:: -v, --verbose 79 80 Show more information on test failures, for example the entire test output 81 instead of just the test result. 82 83 .. option:: --no-progress-bar 84 85 Do not use curses based progress bar. 86 87 .. _execution-options: 88 89 EXECUTION OPTIONS 90 ----------------- 91 92 .. option:: --path=PATH 93 94 Specify an additional ``PATH`` to use when searching for executables in tests. 95 96 .. option:: --vg 97 98 Run individual tests under valgrind (using the memcheck tool). The 99 ``--error-exitcode`` argument for valgrind is used so that valgrind failures 100 will cause the program to exit with a non-zero status. 101 102 When this option is enabled, :program:`lit` will also automatically provide a 103 "``valgrind``" feature that can be used to conditionally disable (or expect 104 failure in) certain tests. 105 106 .. option:: --vg-arg=ARG 107 108 When :option:`--vg` is used, specify an additional argument to pass to 109 :program:`valgrind` itself. 110 111 .. option:: --vg-leak 112 113 When :option:`--vg` is used, enable memory leak checks. When this option is 114 enabled, :program:`lit` will also automatically provide a "``vg_leak``" 115 feature that can be used to conditionally disable (or expect failure in) 116 certain tests. 117 118 .. option:: --time-tests 119 120 Track the wall time individual tests take to execute and includes the results 121 in the summary output. This is useful for determining which tests in a test 122 suite take the most time to execute. Note that this option is most useful 123 with ``-j 1``. 124 125 .. _selection-options: 126 127 SELECTION OPTIONS 128 ----------------- 129 130 .. option:: --max-tests=N 131 132 Run at most ``N`` tests and then terminate. 133 134 .. option:: --max-time=N 135 136 Spend at most ``N`` seconds (approximately) running tests and then terminate. 137 138 .. option:: --shuffle 139 140 Run the tests in a random order. 141 142 ADDITIONAL OPTIONS 143 ------------------ 144 145 .. option:: --debug 146 147 Run :program:`lit` in debug mode, for debugging configuration issues and 148 :program:`lit` itself. 149 150 .. option:: --show-suites 151 152 List the discovered test suites as part of the standard output. 153 154 .. option:: --repeat=N 155 156 Run each test ``N`` times. Currently this is primarily useful for timing 157 tests, other results are not collated in any reasonable fashion. 158 159 EXIT STATUS 160 ----------- 161 162 :program:`lit` will exit with an exit code of 1 if there are any FAIL or XPASS 163 results. Otherwise, it will exit with the status 0. Other exit codes are used 164 for non-test related failures (for example a user error or an internal program 165 error). 166 167 .. _test-discovery: 168 169 TEST DISCOVERY 170 -------------- 171 172 The inputs passed to :program:`lit` can be either individual tests, or entire 173 directories or hierarchies of tests to run. When :program:`lit` starts up, the 174 first thing it does is convert the inputs into a complete list of tests to run 175 as part of *test discovery*. 176 177 In the :program:`lit` model, every test must exist inside some *test suite*. 178 :program:`lit` resolves the inputs specified on the command line to test suites 179 by searching upwards from the input path until it finds a :file:`lit.cfg` or 180 :file:`lit.site.cfg` file. These files serve as both a marker of test suites 181 and as configuration files which :program:`lit` loads in order to understand 182 how to find and run the tests inside the test suite. 183 184 Once :program:`lit` has mapped the inputs into test suites it traverses the 185 list of inputs adding tests for individual files and recursively searching for 186 tests in directories. 187 188 This behavior makes it easy to specify a subset of tests to run, while still 189 allowing the test suite configuration to control exactly how tests are 190 interpreted. In addition, :program:`lit` always identifies tests by the test 191 suite they are in, and their relative path inside the test suite. For 192 appropriately configured projects, this allows :program:`lit` to provide 193 convenient and flexible support for out-of-tree builds. 194 195 .. _test-status-results: 196 197 TEST STATUS RESULTS 198 ------------------- 199 200 Each test ultimately produces one of the following six results: 201 202 **PASS** 203 204 The test succeeded. 205 206 **XFAIL** 207 208 The test failed, but that is expected. This is used for test formats which allow 209 specifying that a test does not currently work, but wish to leave it in the test 210 suite. 211 212 **XPASS** 213 214 The test succeeded, but it was expected to fail. This is used for tests which 215 were specified as expected to fail, but are now succeeding (generally because 216 the feature they test was broken and has been fixed). 217 218 **FAIL** 219 220 The test failed. 221 222 **UNRESOLVED** 223 224 The test result could not be determined. For example, this occurs when the test 225 could not be run, the test itself is invalid, or the test was interrupted. 226 227 **UNSUPPORTED** 228 229 The test is not supported in this environment. This is used by test formats 230 which can report unsupported tests. 231 232 Depending on the test format tests may produce additional information about 233 their status (generally only for failures). See the :ref:`output-options` 234 section for more information. 235 236 .. _lit-infrastructure: 237 238 LIT INFRASTRUCTURE 239 ------------------ 240 241 This section describes the :program:`lit` testing architecture for users interested in 242 creating a new :program:`lit` testing implementation, or extending an existing one. 243 244 :program:`lit` proper is primarily an infrastructure for discovering and running 245 arbitrary tests, and to expose a single convenient interface to these 246 tests. :program:`lit` itself doesn't know how to run tests, rather this logic is 247 defined by *test suites*. 248 249 TEST SUITES 250 ~~~~~~~~~~~ 251 252 As described in :ref:`test-discovery`, tests are always located inside a *test 253 suite*. Test suites serve to define the format of the tests they contain, the 254 logic for finding those tests, and any additional information to run the tests. 255 256 :program:`lit` identifies test suites as directories containing ``lit.cfg`` or 257 ``lit.site.cfg`` files (see also :option:`--config-prefix`). Test suites are 258 initially discovered by recursively searching up the directory hierarchy for 259 all the input files passed on the command line. You can use 260 :option:`--show-suites` to display the discovered test suites at startup. 261 262 Once a test suite is discovered, its config file is loaded. Config files 263 themselves are Python modules which will be executed. When the config file is 264 executed, two important global variables are predefined: 265 266 **lit** 267 268 The global **lit** configuration object (a *LitConfig* instance), which defines 269 the builtin test formats, global configuration parameters, and other helper 270 routines for implementing test configurations. 271 272 **config** 273 274 This is the config object (a *TestingConfig* instance) for the test suite, 275 which the config file is expected to populate. The following variables are also 276 available on the *config* object, some of which must be set by the config and 277 others are optional or predefined: 278 279 **name** *[required]* The name of the test suite, for use in reports and 280 diagnostics. 281 282 **test_format** *[required]* The test format object which will be used to 283 discover and run tests in the test suite. Generally this will be a builtin test 284 format available from the *lit.formats* module. 285 286 **test_source_root** The filesystem path to the test suite root. For out-of-dir 287 builds this is the directory that will be scanned for tests. 288 289 **test_exec_root** For out-of-dir builds, the path to the test suite root inside 290 the object directory. This is where tests will be run and temporary output files 291 placed. 292 293 **environment** A dictionary representing the environment to use when executing 294 tests in the suite. 295 296 **suffixes** For **lit** test formats which scan directories for tests, this 297 variable is a list of suffixes to identify test files. Used by: *ShTest*. 298 299 **substitutions** For **lit** test formats which substitute variables into a test 300 script, the list of substitutions to perform. Used by: *ShTest*. 301 302 **unsupported** Mark an unsupported directory, all tests within it will be 303 reported as unsupported. Used by: *ShTest*. 304 305 **parent** The parent configuration, this is the config object for the directory 306 containing the test suite, or None. 307 308 **root** The root configuration. This is the top-most :program:`lit` configuration in 309 the project. 310 311 **on_clone** The config is actually cloned for every subdirectory inside a test 312 suite, to allow local configuration on a per-directory basis. The *on_clone* 313 variable can be set to a Python function which will be called whenever a 314 configuration is cloned (for a subdirectory). The function should takes three 315 arguments: (1) the parent configuration, (2) the new configuration (which the 316 *on_clone* function will generally modify), and (3) the test path to the new 317 directory being scanned. 318 319 **pipefail** Normally a test using a shell pipe fails if any of the commands 320 on the pipe fail. If this is not desired, setting this variable to false 321 makes the test fail only if the last command in the pipe fails. 322 323 TEST DISCOVERY 324 ~~~~~~~~~~~~~~ 325 326 Once test suites are located, :program:`lit` recursively traverses the source 327 directory (following *test_source_root*) looking for tests. When :program:`lit` 328 enters a sub-directory, it first checks to see if a nested test suite is 329 defined in that directory. If so, it loads that test suite recursively, 330 otherwise it instantiates a local test config for the directory (see 331 :ref:`local-configuration-files`). 332 333 Tests are identified by the test suite they are contained within, and the 334 relative path inside that suite. Note that the relative path may not refer to 335 an actual file on disk; some test formats (such as *GoogleTest*) define 336 "virtual tests" which have a path that contains both the path to the actual 337 test file and a subpath to identify the virtual test. 338 339 .. _local-configuration-files: 340 341 LOCAL CONFIGURATION FILES 342 ~~~~~~~~~~~~~~~~~~~~~~~~~ 343 344 When :program:`lit` loads a subdirectory in a test suite, it instantiates a 345 local test configuration by cloning the configuration for the parent direction 346 --- the root of this configuration chain will always be a test suite. Once the 347 test configuration is cloned :program:`lit` checks for a *lit.local.cfg* file 348 in the subdirectory. If present, this file will be loaded and can be used to 349 specialize the configuration for each individual directory. This facility can 350 be used to define subdirectories of optional tests, or to change other 351 configuration parameters --- for example, to change the test format, or the 352 suffixes which identify test files. 353 354 TEST RUN OUTPUT FORMAT 355 ~~~~~~~~~~~~~~~~~~~~~~ 356 357 The :program:`lit` output for a test run conforms to the following schema, in 358 both short and verbose modes (although in short mode no PASS lines will be 359 shown). This schema has been chosen to be relatively easy to reliably parse by 360 a machine (for example in buildbot log scraping), and for other tools to 361 generate. 362 363 Each test result is expected to appear on a line that matches: 364 365 .. code-block:: none 366 367 <result code>: <test name> (<progress info>) 368 369 where ``<result-code>`` is a standard test result such as PASS, FAIL, XFAIL, 370 XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes of IMPROVED and 371 REGRESSED are also allowed. 372 373 The ``<test name>`` field can consist of an arbitrary string containing no 374 newline. 375 376 The ``<progress info>`` field can be used to report progress information such 377 as (1/300) or can be empty, but even when empty the parentheses are required. 378 379 Each test result may include additional (multiline) log information in the 380 following format: 381 382 .. code-block:: none 383 384 <log delineator> TEST '(<test name>)' <trailing delineator> 385 ... log message ... 386 <log delineator> 387 388 where ``<test name>`` should be the name of a preceding reported test, ``<log 389 delineator>`` is a string of "*" characters *at least* four characters long 390 (the recommended length is 20), and ``<trailing delineator>`` is an arbitrary 391 (unparsed) string. 392 393 The following is an example of a test run output which consists of four tests A, 394 B, C, and D, and a log message for the failing test C: 395 396 .. code-block:: none 397 398 PASS: A (1 of 4) 399 PASS: B (2 of 4) 400 FAIL: C (3 of 4) 401 ******************** TEST 'C' FAILED ******************** 402 Test 'C' failed as a result of exit code 1. 403 ******************** 404 PASS: D (4 of 4) 405 406 LIT EXAMPLE TESTS 407 ~~~~~~~~~~~~~~~~~ 408 409 The :program:`lit` distribution contains several example implementations of 410 test suites in the *ExampleTests* directory. 411 412 SEE ALSO 413 -------- 414 415 valgrind(1) 416
