1 ================================= 2 LLVM Testing Infrastructure Guide 3 ================================= 4 5 .. contents:: 6 :local: 7 8 .. toctree:: 9 :hidden: 10 11 TestSuiteMakefileGuide 12 13 Overview 14 ======== 15 16 This document is the reference manual for the LLVM testing 17 infrastructure. It documents the structure of the LLVM testing 18 infrastructure, the tools needed to use it, and how to add and run 19 tests. 20 21 Requirements 22 ============ 23 24 In order to use the LLVM testing infrastructure, you will need all of the 25 software required to build LLVM, as well as `Python <http://python.org>`_ 2.5 or 26 later. 27 28 LLVM testing infrastructure organization 29 ======================================== 30 31 The LLVM testing infrastructure contains two major categories of tests: 32 regression tests and whole programs. The regression tests are contained 33 inside the LLVM repository itself under ``llvm/test`` and are expected 34 to always pass -- they should be run before every commit. 35 36 The whole programs tests are referred to as the "LLVM test suite" (or 37 "test-suite") and are in the ``test-suite`` module in subversion. For 38 historical reasons, these tests are also referred to as the "nightly 39 tests" in places, which is less ambiguous than "test-suite" and remains 40 in use although we run them much more often than nightly. 41 42 Regression tests 43 ---------------- 44 45 The regression tests are small pieces of code that test a specific 46 feature of LLVM or trigger a specific bug in LLVM. The language they are 47 written in depends on the part of LLVM being tested. These tests are driven by 48 the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and 49 are located in the ``llvm/test`` directory. 50 51 Typically when a bug is found in LLVM, a regression test containing just 52 enough code to reproduce the problem should be written and placed 53 somewhere underneath this directory. For example, it can be a small 54 piece of LLVM IR distilled from an actual application or benchmark. 55 56 ``test-suite`` 57 -------------- 58 59 The test suite contains whole programs, which are pieces of code which 60 can be compiled and linked into a stand-alone program that can be 61 executed. These programs are generally written in high level languages 62 such as C or C++. 63 64 These programs are compiled using a user specified compiler and set of 65 flags, and then executed to capture the program output and timing 66 information. The output of these programs is compared to a reference 67 output to ensure that the program is being compiled correctly. 68 69 In addition to compiling and executing programs, whole program tests 70 serve as a way of benchmarking LLVM performance, both in terms of the 71 efficiency of the programs generated as well as the speed with which 72 LLVM compiles, optimizes, and generates code. 73 74 The test-suite is located in the ``test-suite`` Subversion module. 75 76 Debugging Information tests 77 --------------------------- 78 79 The test suite contains tests to check quality of debugging information. 80 The test are written in C based languages or in LLVM assembly language. 81 82 These tests are compiled and run under a debugger. The debugger output 83 is checked to validate of debugging information. See README.txt in the 84 test suite for more information . This test suite is located in the 85 ``debuginfo-tests`` Subversion module. 86 87 Quick start 88 =========== 89 90 The tests are located in two separate Subversion modules. The 91 regressions tests are in the main "llvm" module under the directory 92 ``llvm/test`` (so you get these tests for free with the main LLVM tree). 93 Use ``make check-all`` to run the regression tests after building LLVM. 94 95 The more comprehensive test suite that includes whole programs in C and C++ 96 is in the ``test-suite`` module. See :ref:`test-suite Quickstart 97 <test-suite-quickstart>` for more information on running these tests. 98 99 Regression tests 100 ---------------- 101 102 To run all of the LLVM regression tests, use the master Makefile in the 103 ``llvm/test`` directory. LLVM Makefiles require GNU Make (read the :doc:`LLVM 104 Makefile Guide <MakefileGuide>` for more details): 105 106 .. code-block:: bash 107 108 % make -C llvm/test 109 110 or: 111 112 .. code-block:: bash 113 114 % make check 115 116 If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you 117 can run the LLVM and Clang tests simultaneously using: 118 119 .. code-block:: bash 120 121 % make check-all 122 123 To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make 124 variable to pass the required options to lit. For example, you can use: 125 126 .. code-block:: bash 127 128 % make check LIT_ARGS="-v --vg --vg-leak" 129 130 to enable testing with valgrind and with leak checking enabled. 131 132 To run individual tests or subsets of tests, you can use the ``llvm-lit`` 133 script which is built as part of LLVM. For example, to run the 134 ``Integer/BitPacked.ll`` test by itself you can run: 135 136 .. code-block:: bash 137 138 % llvm-lit ~/llvm/test/Integer/BitPacked.ll 139 140 or to run all of the ARM CodeGen tests: 141 142 .. code-block:: bash 143 144 % llvm-lit ~/llvm/test/CodeGen/ARM 145 146 For more information on using the :program:`lit` tool, see ``llvm-lit --help`` 147 or the :doc:`lit man page <CommandGuide/lit>`. 148 149 Debugging Information tests 150 --------------------------- 151 152 To run debugging information tests simply checkout the tests inside 153 clang/test directory. 154 155 .. code-block:: bash 156 157 % cd clang/test 158 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests 159 160 These tests are already set up to run as part of clang regression tests. 161 162 Regression test structure 163 ========================= 164 165 The LLVM regression tests are driven by :program:`lit` and are located in the 166 ``llvm/test`` directory. 167 168 This directory contains a large array of small tests that exercise 169 various features of LLVM and to ensure that regressions do not occur. 170 The directory is broken into several sub-directories, each focused on a 171 particular area of LLVM. 172 173 Writing new regression tests 174 ---------------------------- 175 176 The regression test structure is very simple, but does require some 177 information to be set. This information is gathered via ``configure`` 178 and is written to a file, ``test/lit.site.cfg`` in the build directory. 179 The ``llvm/test`` Makefile does this work for you. 180 181 In order for the regression tests to work, each directory of tests must 182 have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine 183 how to run the tests. This file is just Python code and thus is very 184 flexible, but we've standardized it for the LLVM regression tests. If 185 you're adding a directory of tests, just copy ``lit.local.cfg`` from 186 another directory to get running. The standard ``lit.local.cfg`` simply 187 specifies which files to look in for tests. Any directory that contains 188 only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit 189 documentation <CommandGuide/lit>` for more information. 190 191 Each test file must contain lines starting with "RUN:" that tell :program:`lit` 192 how to run it. If there are no RUN lines, :program:`lit` will issue an error 193 while running a test. 194 195 RUN lines are specified in the comments of the test program using the 196 keyword ``RUN`` followed by a colon, and lastly the command (pipeline) 197 to execute. Together, these lines form the "script" that :program:`lit` 198 executes to run the test case. The syntax of the RUN lines is similar to a 199 shell's syntax for pipelines including I/O redirection and variable 200 substitution. However, even though these lines may *look* like a shell 201 script, they are not. RUN lines are interpreted by :program:`lit`. 202 Consequently, the syntax differs from shell in a few ways. You can specify 203 as many RUN lines as needed. 204 205 :program:`lit` performs substitution on each RUN line to replace LLVM tool names 206 with the full paths to the executable built for each tool (in 207 ``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does 208 not invoke any stray LLVM tools in the user's path during testing. 209 210 Each RUN line is executed on its own, distinct from other lines unless 211 its last character is ``\``. This continuation character causes the RUN 212 line to be concatenated with the next one. In this way you can build up 213 long pipelines of commands without making huge line lengths. The lines 214 ending in ``\`` are concatenated until a RUN line that doesn't end in 215 ``\`` is found. This concatenated set of RUN lines then constitutes one 216 execution. :program:`lit` will substitute variables and arrange for the pipeline 217 to be executed. If any process in the pipeline fails, the entire line (and 218 test case) fails too. 219 220 Below is an example of legal RUN lines in a ``.ll`` file: 221 222 .. code-block:: llvm 223 224 ; RUN: llvm-as < %s | llvm-dis > %t1 225 ; RUN: llvm-dis < %s.bc-13 > %t2 226 ; RUN: diff %t1 %t2 227 228 As with a Unix shell, the RUN lines permit pipelines and I/O 229 redirection to be used. 230 231 There are some quoting rules that you must pay attention to when writing 232 your RUN lines. In general nothing needs to be quoted. :program:`lit` won't 233 strip off any quote characters so they will get passed to the invoked program. 234 To avoid this use curly braces to tell :program:`lit` that it should treat 235 everything enclosed as one value. 236 237 In general, you should strive to keep your RUN lines as simple as possible, 238 using them only to run tools that generate textual output you can then examine. 239 The recommended way to examine output to figure out if the test passes is using 240 the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN 241 lines is deprecated - please do not send or commit patches that use it.]* 242 243 Fragile tests 244 ------------- 245 246 It is easy to write a fragile test that would fail spuriously if the tool being 247 tested outputs a full path to the input file. For example, :program:`opt` by 248 default outputs a ``ModuleID``: 249 250 .. code-block:: console 251 252 $ cat example.ll 253 define i32 @main() nounwind { 254 ret i32 0 255 } 256 257 $ opt -S /path/to/example.ll 258 ; ModuleID = '/path/to/example.ll' 259 260 define i32 @main() nounwind { 261 ret i32 0 262 } 263 264 ``ModuleID`` can unexpetedly match against ``CHECK`` lines. For example: 265 266 .. code-block:: llvm 267 268 ; RUN: opt -S %s | FileCheck 269 270 define i32 @main() nounwind { 271 ; CHECK-NOT: load 272 ret i32 0 273 } 274 275 This test will fail if placed into a ``download`` directory. 276 277 To make your tests robust, always use ``opt ... < %s`` in the RUN line. 278 :program:`opt` does not output a ``ModuleID`` when input comes from stdin. 279 280 Platform-Specific Tests 281 ----------------------- 282 283 Whenever adding tests that require the knowledge of a specific platform, 284 either related to code generated, specific output or back-end features, 285 you must make sure to isolate the features, so that buildbots that 286 run on different architectures (and don't even compile all back-ends), 287 don't fail. 288 289 The first problem is to check for target-specific output, for example sizes 290 of structures, paths and architecture names, for example: 291 292 * Tests containing Windows paths will fail on Linux and vice-versa. 293 * Tests that check for ``x86_64`` somewhere in the text will fail anywhere else. 294 * Tests where the debug information calculates the size of types and structures. 295 296 Also, if the test rely on any behaviour that is coded in any back-end, it must 297 go in its own directory. So, for instance, code generator tests for ARM go 298 into ``test/CodeGen/ARM`` and so on. Those directories contain a special 299 ``lit`` configuration file that ensure all tests in that directory will 300 only run if a specific back-end is compiled and available. 301 302 For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is: 303 304 .. code-block:: python 305 306 config.suffixes = ['.ll', '.c', '.cpp', '.test'] 307 if not 'ARM' in config.root.targets: 308 config.unsupported = True 309 310 Other platform-specific tests are those that depend on a specific feature 311 of a specific sub-architecture, for example only to Intel chips that support ``AVX2``. 312 313 For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture 314 variants: 315 316 .. code-block:: llvm 317 318 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2 319 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1 320 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2 321 322 And the checks are different: 323 324 .. code-block:: llvm 325 326 ; SSE2: @test1 327 ; SSE2: psubusw LCPI0_0(%rip), %xmm0 328 ; AVX1: @test1 329 ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 330 ; AVX2: @test1 331 ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 332 333 So, if you're testing for a behaviour that you know is platform-specific or 334 depends on special features of sub-architectures, you must add the specific 335 triple, test with the specific FileCheck and put it into the specific 336 directory that will filter out all other architectures. 337 338 339 Substitutions 340 ------------- 341 342 Besides replacing LLVM tool names the following substitutions are performed in 343 RUN lines: 344 345 ``%%`` 346 Replaced by a single ``%``. This allows escaping other substitutions. 347 348 ``%s`` 349 File path to the test case's source. This is suitable for passing on the 350 command line as the input to an LLVM tool. 351 352 Example: ``/home/user/llvm/test/MC/ELF/foo_test.s`` 353 354 ``%S`` 355 Directory path to the test case's source. 356 357 Example: ``/home/user/llvm/test/MC/ELF`` 358 359 ``%t`` 360 File path to a temporary file name that could be used for this test case. 361 The file name won't conflict with other test cases. You can append to it 362 if you need multiple temporaries. This is useful as the destination of 363 some redirected output. 364 365 Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp`` 366 367 ``%T`` 368 Directory of ``%t``. 369 370 Example: ``/home/user/llvm.build/test/MC/ELF/Output`` 371 372 ``%{pathsep}`` 373 374 Expands to the path separator, i.e. ``:`` (or ``;`` on Windows). 375 376 377 **LLVM-specific substitutions:** 378 379 ``%shlibext`` 380 The suffix for the host platforms shared library files. This includes the 381 period as the first character. 382 383 Example: ``.so`` (Linux), ``.dylib`` (OS X), ``.dll`` (Windows) 384 385 ``%exeext`` 386 The suffix for the host platforms executable files. This includes the 387 period as the first character. 388 389 Example: ``.exe`` (Windows), empty on Linux. 390 391 ``%(line)``, ``%(line+<number>)``, ``%(line-<number>)`` 392 The number of the line where this substitution is used, with an optional 393 integer offset. This can be used in tests with multiple RUN lines, which 394 reference test file's line numbers. 395 396 397 **Clang-specific substitutions:** 398 399 ``%clang`` 400 Invokes the Clang driver. 401 402 ``%clang_cpp`` 403 Invokes the Clang driver for C++. 404 405 ``%clang_cl`` 406 Invokes the CL-compatible Clang driver. 407 408 ``%clangxx`` 409 Invokes the G++-compatible Clang driver. 410 411 ``%clang_cc1`` 412 Invokes the Clang frontend. 413 414 ``%itanium_abi_triple``, ``%ms_abi_triple`` 415 These substitutions can be used to get the current target triple adjusted to 416 the desired ABI. For example, if the test suite is running with the 417 ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to 418 ``i686-pc-mingw32``. This allows a test to run with a specific ABI without 419 constraining it to a specific triple. 420 421 To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``. 422 423 424 Other Features 425 -------------- 426 427 To make RUN line writing easier, there are several helper programs. These 428 helpers are in the PATH when running tests, so you can just call them using 429 their name. For example: 430 431 ``not`` 432 This program runs its arguments and then inverts the result code from it. 433 Zero result codes become 1. Non-zero result codes become 0. 434 435 Sometimes it is necessary to mark a test case as "expected fail" or 436 XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:`` 437 on a line near the top of the file. This signals that the test case 438 should succeed if the test fails. Such test cases are counted separately 439 by the testing tool. To specify an expected fail, use the XFAIL keyword 440 in the comments of the test program followed by a colon and one or more 441 failure patterns. Each failure pattern can be either ``*`` (to specify 442 fail everywhere), or a part of a target triple (indicating the test 443 should fail on that platform), or the name of a configurable feature 444 (for example, ``loadable_module``). If there is a match, the test is 445 expected to fail. If not, the test is expected to succeed. To XFAIL 446 everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL`` 447 line: 448 449 .. code-block:: llvm 450 451 ; XFAIL: darwin,sun 452 453 To make the output more useful, :program:`lit` will scan 454 the lines of the test case for ones that contain a pattern that matches 455 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number 456 that is related to the test case. The number after "PR" specifies the 457 LLVM bugzilla number. When a PR number is specified, it will be used in 458 the pass/fail reporting. This is useful to quickly get some context when 459 a test fails. 460 461 Finally, any line that contains "END." will cause the special 462 interpretation of lines to terminate. This is generally done right after 463 the last RUN: line. This has two side effects: 464 465 (a) it prevents special interpretation of lines that are part of the test 466 program, not the instructions to the test case, and 467 468 (b) it speeds things up for really big test cases by avoiding 469 interpretation of the remainder of the file. 470 471 ``test-suite`` Overview 472 ======================= 473 474 The ``test-suite`` module contains a number of programs that can be 475 compiled and executed. The ``test-suite`` includes reference outputs for 476 all of the programs, so that the output of the executed program can be 477 checked for correctness. 478 479 ``test-suite`` tests are divided into three types of tests: MultiSource, 480 SingleSource, and External. 481 482 - ``test-suite/SingleSource`` 483 484 The SingleSource directory contains test programs that are only a 485 single source file in size. These are usually small benchmark 486 programs or small programs that calculate a particular value. Several 487 such programs are grouped together in each directory. 488 489 - ``test-suite/MultiSource`` 490 491 The MultiSource directory contains subdirectories which contain 492 entire programs with multiple source files. Large benchmarks and 493 whole applications go here. 494 495 - ``test-suite/External`` 496 497 The External directory contains Makefiles for building code that is 498 external to (i.e., not distributed with) LLVM. The most prominent 499 members of this directory are the SPEC 95 and SPEC 2000 benchmark 500 suites. The ``External`` directory does not contain these actual 501 tests, but only the Makefiles that know how to properly compile these 502 programs from somewhere else. When using ``LNT``, use the 503 ``--test-externals`` option to include these tests in the results. 504 505 .. _test-suite-quickstart: 506 507 ``test-suite`` Quickstart 508 ------------------------- 509 510 The modern way of running the ``test-suite`` is focused on testing and 511 benchmarking complete compilers using the 512 `LNT <http://llvm.org/docs/lnt>`_ testing infrastructure. 513 514 For more information on using LNT to execute the ``test-suite``, please 515 see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_ 516 documentation. 517 518 ``test-suite`` Makefiles 519 ------------------------ 520 521 Historically, the ``test-suite`` was executed using a complicated setup 522 of Makefiles. The LNT based approach above is recommended for most 523 users, but there are some testing scenarios which are not supported by 524 the LNT approach. In addition, LNT currently uses the Makefile setup 525 under the covers and so developers who are interested in how LNT works 526 under the hood may want to understand the Makefile based setup. 527 528 For more information on the ``test-suite`` Makefile setup, please see 529 the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`. 530
