1 ============== 2 Testing libc++ 3 ============== 4 5 .. contents:: 6 :local: 7 8 Getting Started 9 =============== 10 11 libc++ uses LIT to configure and run its tests. The primary way to run the 12 libc++ tests is by using make check-libcxx. However since libc++ can be used 13 in any number of possible configurations it is important to customize the way 14 LIT builds and runs the tests. This guide provides information on how to use 15 LIT directly to test libc++. 16 17 Please see the `Lit Command Guide`_ for more information about LIT. 18 19 .. _LIT Command Guide: http://llvm.org/docs/CommandGuide/lit.html 20 21 Setting up the Environment 22 -------------------------- 23 24 After building libc++ you must setup your environment to test libc++ using 25 LIT. 26 27 #. Create a shortcut to the actual lit executable so that you can invoke it 28 easily from the command line. 29 30 .. code-block:: bash 31 32 $ alias lit='python path/to/llvm/utils/lit/lit.py' 33 34 #. Tell LIT where to find your build configuration. 35 36 .. code-block:: bash 37 38 $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg 39 40 Example Usage 41 ------------- 42 43 Once you have your environment set up and you have built libc++ you can run 44 parts of the libc++ test suite by simply running `lit` on a specified test or 45 directory. For example: 46 47 .. code-block:: bash 48 49 $ cd path/to/src/libcxx 50 $ lit -sv test/std/re # Run all of the std::regex tests 51 $ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test 52 $ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic 53 54 Sometimes you'll want to change the way LIT is running the tests. Custom options 55 can be specified using the `--param=<name>=<val>` flag. The most common option 56 you'll want to change is the standard dialect (ie -std=c++XX). By default the 57 test suite will select the newest C++ dialect supported by the compiler and use 58 that. However if you want to manually specify the option like so: 59 60 .. code-block:: bash 61 62 $ lit -sv test/std/containers # Run the tests with the newest -std 63 $ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03 64 65 Occasionally you'll want to add extra compile or link flags when testing. 66 You can do this as follows: 67 68 .. code-block:: bash 69 70 $ lit -sv --param=compile_flags='-Wcustom-warning' 71 $ lit -sv --param=link_flags='-L/custom/library/path' 72 73 Some other common examples include: 74 75 .. code-block:: bash 76 77 # Specify a custom compiler. 78 $ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std 79 80 # Enable warnings in the test suite 81 $ lit -sv --param=enable_warnings=true test/std 82 83 # Use UBSAN when running the tests. 84 $ lit -sv --param=use_sanitizer=Undefined 85 86 87 LIT Options 88 =========== 89 90 :program:`lit` [*options*...] [*filenames*...] 91 92 Command Line Options 93 -------------------- 94 95 To use these options you pass them on the LIT command line as --param NAME or 96 --param NAME=VALUE. Some options have default values specified during CMake's 97 configuration. Passing the option on the command line will override the default. 98 99 .. program:: lit 100 101 .. option:: cxx_under_test=<path/to/compiler> 102 103 Specify the compiler used to build the tests. 104 105 .. option:: cxx_stdlib_under_test=<stdlib name> 106 107 **Values**: libc++, libstdc++ 108 109 Specify the C++ standard library being tested. Unless otherwise specified 110 libc++ is used. This option is intended to allow running the libc++ test 111 suite against other standard library implementations. 112 113 .. option:: std=<standard version> 114 115 **Values**: c++98, c++03, c++11, c++14, c++17, c++2a 116 117 Change the standard version used when building the tests. 118 119 .. option:: libcxx_site_config=<path/to/lit.site.cfg> 120 121 Specify the site configuration to use when running the tests. This option 122 overrides the environment variable LIBCXX_SITE_CONFIG. 123 124 .. option:: cxx_headers=<path/to/headers> 125 126 Specify the c++ standard library headers that are tested. By default the 127 headers in the source tree are used. 128 129 .. option:: cxx_library_root=<path/to/lib/> 130 131 Specify the directory of the libc++ library to be tested. By default the 132 library folder of the build directory is used. This option cannot be used 133 when use_system_cxx_lib is provided. 134 135 136 .. option:: cxx_runtime_root=<path/to/lib/> 137 138 Specify the directory of the libc++ library to use at runtime. This directory 139 is not added to the linkers search path. This can be used to compile tests 140 against one version of libc++ and run them using another. The default value 141 for this option is `cxx_library_root`. 142 143 .. option:: use_system_cxx_lib=<bool> 144 145 **Default**: False 146 147 Enable or disable testing against the installed version of libc++ library. 148 Note: This does not use the installed headers. 149 150 .. option:: use_lit_shell=<bool> 151 152 Enable or disable the use of LIT's internal shell in ShTests. If the 153 environment variable LIT_USE_INTERNAL_SHELL is present then that is used as 154 the default value. Otherwise the default value is True on Windows and False 155 on every other platform. 156 157 .. option:: compile_flags="<list-of-args>" 158 159 Specify additional compile flags as a space delimited string. 160 Note: This options should not be used to change the standard version used. 161 162 .. option:: link_flags="<list-of-args>" 163 164 Specify additional link flags as a space delimited string. 165 166 .. option:: debug_level=<level> 167 168 **Values**: 0, 1 169 170 Enable the use of debug mode. Level 0 enables assertions and level 1 enables 171 assertions and debugging of iterator misuse. 172 173 .. option:: use_sanitizer=<sanitizer name> 174 175 **Values**: Memory, MemoryWithOrigins, Address, Undefined 176 177 Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when 178 building libc++ then that sanitizer will be used by default. 179 180 .. option:: color_diagnostics 181 182 Enable the use of colorized compile diagnostics. If the color_diagnostics 183 option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is 184 present then color diagnostics will be enabled. 185 186 187 Environment Variables 188 --------------------- 189 190 .. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg> 191 192 Specify the site configuration to use when running the tests. 193 Also see `libcxx_site_config`. 194 195 .. envvar:: LIBCXX_COLOR_DIAGNOSTICS 196 197 If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt 198 to use color diagnostic outputs from the compiler. 199 Also see `color_diagnostics`. 200 201 Benchmarks 202 ========== 203 204 Libc++ contains benchmark tests separately from the test of the test suite. 205 The benchmarks are written using the `Google Benchmark`_ library, a copy of which 206 is stored in the libc++ repository. 207 208 For more information about using the Google Benchmark library see the 209 `official documentation <https://github.com/google/benchmark>`_. 210 211 .. _`Google Benchmark`: https://github.com/google/benchmark 212 213 Building Benchmarks 214 ------------------- 215 216 The benchmark tests are not built by default. The benchmarks can be built using 217 the ``cxx-benchmarks`` target. 218 219 An example build would look like: 220 221 .. code-block:: bash 222 223 $ cd build 224 $ cmake [options] <path to libcxx sources> 225 $ make cxx-benchmarks 226 227 This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be 228 built against the just-built libc++. The compiled tests are output into 229 ``build/benchmarks``. 230 231 The benchmarks can also be built against the platforms native standard library 232 using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This 233 is useful for comparing the performance of libc++ to other standard libraries. 234 The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and 235 ``<test>.native.out`` otherwise. 236 237 Also See: 238 239 * :ref:`Building Libc++ <build instructions>` 240 * :ref:`CMake Options` 241 242 Running Benchmarks 243 ------------------ 244 245 The benchmarks must be run manually by the user. Currently there is no way 246 to run them as part of the build. 247 248 For example: 249 250 .. code-block:: bash 251 252 $ cd build/benchmarks 253 $ make cxx-benchmarks 254 $ ./algorithms.libcxx.out # Runs all the benchmarks 255 $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks 256 257 For more information about running benchmarks see `Google Benchmark`_. 258
