1 <a id="top"></a> 2 # Known limitations 3 4 Over time, some limitations of Catch2 emerged. Some of these are due 5 to implementation details that cannot be easily changed, some of these 6 are due to lack of development resources on our part, and some of these 7 are due to plain old 3rd party bugs. 8 9 10 ## Implementation limits 11 ### Sections nested in loops 12 13 If you are using `SECTION`s inside loops, you have to create them with 14 different name per loop's iteration. The recommended way to do so is to 15 incorporate the loop's counter into section's name, like so: 16 17 ```cpp 18 TEST_CASE( "Looped section" ) { 19 for (char i = '0'; i < '5'; ++i) { 20 SECTION(std::string("Looped section ") + i) { 21 SUCCEED( "Everything is OK" ); 22 } 23 } 24 } 25 ``` 26 27 or with a `DYNAMIC_SECTION` macro (that was made for exactly this purpose): 28 29 ```cpp 30 TEST_CASE( "Looped section" ) { 31 for (char i = '0'; i < '5'; ++i) { 32 DYNAMIC_SECTION( "Looped section " << i) { 33 SUCCEED( "Everything is OK" ); 34 } 35 } 36 } 37 ``` 38 39 ### Tests might be run again if last section fails 40 41 If the last section in a test fails, it might be run again. This is because 42 Catch2 discovers `SECTION`s dynamically, as they are about to run, and 43 if the last section in test case is aborted during execution (e.g. via 44 the `REQUIRE` family of macros), Catch2 does not know that there are no 45 more sections in that test case and must run the test case again. 46 47 48 ## Features 49 This section outlines some missing features, what is their status and their possible workarounds. 50 51 ### Thread safe assertions 52 Catch2's assertion macros are not thread safe. This does not mean that 53 you cannot use threads inside Catch's test, but that only single thread 54 can interact with Catch's assertions and other macros. 55 56 This means that this is ok 57 ```cpp 58 std::vector<std::thread> threads; 59 std::atomic<int> cnt{ 0 }; 60 for (int i = 0; i < 4; ++i) { 61 threads.emplace_back([&]() { 62 ++cnt; ++cnt; ++cnt; ++cnt; 63 }); 64 } 65 for (auto& t : threads) { t.join(); } 66 REQUIRE(cnt == 16); 67 ``` 68 because only one thread passes the `REQUIRE` macro and this is not 69 ```cpp 70 std::vector<std::thread> threads; 71 std::atomic<int> cnt{ 0 }; 72 for (int i = 0; i < 4; ++i) { 73 threads.emplace_back([&]() { 74 ++cnt; ++cnt; ++cnt; ++cnt; 75 CHECK(cnt == 16); 76 }); 77 } 78 for (auto& t : threads) { t.join(); } 79 REQUIRE(cnt == 16); 80 ``` 81 82 Because C++11 provides the necessary tools to do this, we are planning 83 to remove this limitation in the future. 84 85 ### Process isolation in a test 86 Catch does not support running tests in isolated (forked) processes. While this might in the future, the fact that Windows does not support forking and only allows full-on process creation and the desire to keep code as similar as possible across platforms, mean that this is likely to take significant development time, that is not currently available. 87 88 ### Running multiple tests in parallel 89 Catch's test execution is strictly serial. If you find yourself with a test suite that takes too long to run and you want to make it parallel, there are 2 feasible solutions 90 * You can split your tests into multiple binaries and then run these binaries in parallel. 91 * You can have Catch list contained test cases and then run the same test binary multiple times in parallel, passing each instance list of test cases it should run. 92 93 Both of these solutions have their problems, but should let you wring parallelism out of your test suite. 94 95 ## 3rd party bugs 96 This section outlines known bugs in 3rd party components (this means compilers, standard libraries, standard runtimes). 97 98 ### Visual Studio 2017 -- raw string literal in assert fails to compile 99 There is a known bug in Visual Studio 2017 (VC 15), that causes compilation error when preprocessor attempts to stringize a raw string literal (`#` preprocessor is applied to it). This snippet is sufficient to trigger the compilation error: 100 ```cpp 101 #define CATCH_CONFIG_MAIN 102 #include "catch.hpp" 103 104 TEST_CASE("test") { 105 CHECK(std::string(R"("\)") == "\"\\"); 106 } 107 ``` 108 109 Catch provides a workaround, it is possible to disable stringification of original expressions by defining `CATCH_CONFIG_DISABLE_STRINGIFICATION`: 110 ```cpp 111 #define CATCH_CONFIG_FAST_COMPILE 112 #define CATCH_CONFIG_DISABLE_STRINGIFICATION 113 #include "catch.hpp" 114 115 TEST_CASE("test") { 116 CHECK(std::string(R"("\)") == "\"\\"); 117 } 118 ``` 119 120 _Do note that this changes the output somewhat_ 121 ``` 122 catchwork\test1.cpp(6): 123 PASSED: 124 CHECK( Disabled by CATCH_CONFIG_DISABLE_STRINGIFICATION ) 125 with expansion: 126 ""\" == ""\" 127 ``` 128 129 ### Visual Studio 2015 -- Alignment compilation error (C2718) 130 131 VS 2015 has a known bug, where `declval<T>` can cause compilation error 132 if `T` has alignment requirements that it cannot meet. 133 134 135 A workaround is to explicitly specialize `Catch::is_range` for given 136 type (this avoids code path that uses `declval<T>` in a SFINAE context). 137 138 139 ### Visual Studio 2015 -- Wrong line number reported in debug mode 140 VS 2015 has a known bug where `__LINE__` macro can be improperly expanded under certain circumstances, while compiling multi-file project in Debug mode. 141 142 A workaround is to compile the binary in Release mode. 143 144 ### Clang/G++ -- skipping leaf sections after an exception 145 Some versions of `libc++` and `libstdc++` (or their runtimes) have a bug with `std::uncaught_exception()` getting stuck returning `true` after rethrow, even if there are no active exceptions. One such case is this snippet, which skipped the sections "a" and "b", when compiled against `libcxxrt` from master 146 ```cpp 147 #define CATCH_CONFIG_MAIN 148 #include <catch.hpp> 149 150 TEST_CASE("a") { 151 CHECK_THROWS(throw 3); 152 } 153 154 TEST_CASE("b") { 155 int i = 0; 156 SECTION("a") { i = 1; } 157 SECTION("b") { i = 2; } 158 CHECK(i > 0); 159 } 160 ``` 161 162 If you are seeing a problem like this, i.e. a weird test paths that trigger only under Clang with `libc++`, or only under very specific version of `libstdc++`, it is very likely you are seeing this. The only known workaround is to use a fixed version of your standard library. 163 164 ### Clang/G++ -- `Matches` string matcher always returns false 165 This is a bug in `libstdc++-4.8`, where all matching methods from `<regex>` return false. Since `Matches` uses `<regex>` internally, if the underlying implementation does not work, it doesn't work either. 166 167 Workaround: Use newer version of `libstdc++`. 168 169 170 ### libstdc++, `_GLIBCXX_DEBUG` macro and random ordering of tests 171 172 Running a Catch2 binary compiled against libstdc++ with `_GLIBCXX_DEBUG` 173 macro defined with `--order rand` will cause a debug check to trigger and 174 abort the run due to self-assignment. 175 [This is a known bug inside libstdc++](https://stackoverflow.com/questions/22915325/avoiding-self-assignment-in-stdshuffle/23691322) 176 177 Workaround: Don't use `--order rand` when compiling against debug-enabled 178 libstdc++. 179
