1 ============================= 2 Advanced Build Configurations 3 ============================= 4 5 .. contents:: 6 :local: 7 8 Introduction 9 ============ 10 11 `CMake <http://www.cmake.org/>`_ is a cross-platform build-generator tool. CMake 12 does not build the project, it generates the files needed by your build tool 13 (GNU make, Visual Studio, etc.) for building LLVM. 14 15 If **you are a new contributor**, please start with the :doc:`GettingStarted` or 16 :doc:`CMake` pages. This page is intended for users doing more complex builds. 17 18 Many of the examples below are written assuming specific CMake Generators. 19 Unless otherwise explicitly called out these commands should work with any CMake 20 generator. 21 22 Bootstrap Builds 23 ================ 24 25 The Clang CMake build system supports bootstrap (aka multi-stage) builds. At a 26 high level a multi-stage build is a chain of builds that pass data from one 27 stage into the next. The most common and simple version of this is a traditional 28 bootstrap build. 29 30 In a simple two-stage bootstrap build, we build clang using the system compiler, 31 then use that just-built clang to build clang again. In CMake this simplest form 32 of a bootstrap build can be configured with a single option, 33 CLANG_ENABLE_BOOTSTRAP. 34 35 .. code-block:: console 36 37 $ cmake -G Ninja -DCLANG_ENABLE_BOOTSTRAP=On <path to source> 38 $ ninja stage2 39 40 This command itself isn't terribly useful because it assumes default 41 configurations for each stage. The next series of examples utilize CMake cache 42 scripts to provide more complex options. 43 44 The clang build system refers to builds as stages. A stage1 build is a standard 45 build using the compiler installed on the host, and a stage2 build is built 46 using the stage1 compiler. This nomenclature holds up to more stages too. In 47 general a stage*n* build is built using the output from stage*n-1*. 48 49 Apple Clang Builds (A More Complex Bootstrap) 50 ============================================= 51 52 Apple's Clang builds are a slightly more complicated example of the simple 53 bootstrapping scenario. Apple Clang is built using a 2-stage build. 54 55 The stage1 compiler is a host-only compiler with some options set. The stage1 56 compiler is a balance of optimization vs build time because it is a throwaway. 57 The stage2 compiler is the fully optimized compiler intended to ship to users. 58 59 Setting up these compilers requires a lot of options. To simplify the 60 configuration the Apple Clang build settings are contained in CMake Cache files. 61 You can build an Apple Clang compiler using the following commands: 62 63 .. code-block:: console 64 65 $ cmake -G Ninja -C <path to clang>/cmake/caches/Apple-stage1.cmake <path to source> 66 $ ninja stage2-distribution 67 68 This CMake invocation configures the stage1 host compiler, and sets 69 CLANG_BOOTSTRAP_CMAKE_ARGS to pass the Apple-stage2.cmake cache script to the 70 stage2 configuration step. 71 72 When you build the stage2-distribution target it builds the minimal stage1 73 compiler and required tools, then configures and builds the stage2 compiler 74 based on the settings in Apple-stage2.cmake. 75 76 This pattern of using cache scripts to set complex settings, and specifically to 77 make later stage builds include cache scripts is common in our more advanced 78 build configurations. 79 80 Multi-stage PGO 81 =============== 82 83 Profile-Guided Optimizations (PGO) is a really great way to optimize the code 84 clang generates. Our multi-stage PGO builds are a workflow for generating PGO 85 profiles that can be used to optimize clang. 86 87 At a high level, the way PGO works is that you build an instrumented compiler, 88 then you run the instrumented compiler against sample source files. While the 89 instrumented compiler runs it will output a bunch of files containing 90 performance counters (.profraw files). After generating all the profraw files 91 you use llvm-profdata to merge the files into a single profdata file that you 92 can feed into the LLVM_PROFDATA_FILE option. 93 94 Our PGO.cmake cache script automates that whole process. You can use it by 95 running: 96 97 .. code-block:: console 98 99 $ cmake -G Ninja -C <path_to_clang>/cmake/caches/PGO.cmake <source dir> 100 $ ninja stage2-instrumented-generate-profdata 101 102 If you let that run for a few hours or so, it will place a profdata file in your 103 build directory. This takes a really long time because it builds clang twice, 104 and you *must* have compiler-rt in your build tree. 105 106 This process uses any source files under the perf-training directory as training 107 data as long as the source files are marked up with LIT-style RUN lines. 108 109 After it finishes you can use find . -name clang.profdata to find it, but it 110 should be at a path something like: 111 112 .. code-block:: console 113 114 <build dir>/tools/clang/stage2-instrumented-bins/utils/perf-training/clang.profdata 115 116 You can feed that file into the LLVM_PROFDATA_FILE option when you build your 117 optimized compiler. 118 119 The PGO came cache has a slightly different stage naming scheme than other 120 multi-stage builds. It generates three stages; stage1, stage2-instrumented, and 121 stage2. Both of the stage2 builds are built using the stage1 compiler. 122 123 The PGO came cache generates the following additional targets: 124 125 **stage2-instrumented** 126 Builds a stage1 x86 compiler, runtime, and required tools (llvm-config, 127 llvm-profdata) then uses that compiler to build an instrumented stage2 compiler. 128 129 **stage2-instrumented-generate-profdata** 130 Depends on "stage2-instrumented" and will use the instrumented compiler to 131 generate profdata based on the training files in <clang>/utils/perf-training 132 133 **stage2** 134 Depends of "stage2-instrumented-generate-profdata" and will use the stage1 135 compiler with the stage2 profdata to build a PGO-optimized compiler. 136 137 **stage2-check-llvm** 138 Depends on stage2 and runs check-llvm using the stage2 compiler. 139 140 **stage2-check-clang** 141 Depends on stage2 and runs check-clang using the stage2 compiler. 142 143 **stage2-check-all** 144 Depends on stage2 and runs check-all using the stage2 compiler. 145 146 **stage2-test-suite** 147 Depends on stage2 and runs the test-suite using the stage3 compiler (requires 148 in-tree test-suite). 149 150 3-Stage Non-Determinism 151 ======================= 152 153 In the ancient lore of compilers non-determinism is like the multi-headed hydra. 154 Whenever it's head pops up, terror and chaos ensue. 155 156 Historically one of the tests to verify that a compiler was deterministic would 157 be a three stage build. The idea of a three stage build is you take your sources 158 and build a compiler (stage1), then use that compiler to rebuild the sources 159 (stage2), then you use that compiler to rebuild the sources a third time 160 (stage3) with an identical configuration to the stage2 build. At the end of 161 this, you have a stage2 and stage3 compiler that should be bit-for-bit 162 identical. 163 164 You can perform one of these 3-stage builds with LLVM & clang using the 165 following commands: 166 167 .. code-block:: console 168 169 $ cmake -G Ninja -C <path_to_clang>/cmake/caches/3-stage.cmake <source dir> 170 $ ninja stage3 171 172 After the build you can compare the stage2 & stage3 compilers. We have a bot 173 setup `here <http://lab.llvm.org:8011/builders/clang-3stage-ubuntu>`_ that runs 174 this build and compare configuration. 175
