1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" 2 "http://www.w3.org/TR/html4/strict.dtd"> 3 <html> 4 <head> 5 <title>Clang Driver Manual</title> 6 <link type="text/css" rel="stylesheet" href="../menu.css"> 7 <link type="text/css" rel="stylesheet" href="../content.css"> 8 <style type="text/css"> 9 td { 10 vertical-align: top; 11 } 12 </style> 13 </head> 14 <body> 15 16 <!--#include virtual="../menu.html.incl"--> 17 18 <div id="content"> 19 20 <h1>Driver Design & Internals</h1> 21 22 <ul> 23 <li><a href="#intro">Introduction</a></li> 24 <li><a href="#features">Features and Goals</a> 25 <ul> 26 <li><a href="#gcccompat">GCC Compatibility</a></li> 27 <li><a href="#components">Flexible</a></li> 28 <li><a href="#performance">Low Overhead</a></li> 29 <li><a href="#simple">Simple</a></li> 30 </ul> 31 </li> 32 <li><a href="#design">Design</a> 33 <ul> 34 <li><a href="#int_intro">Internals Introduction</a></li> 35 <li><a href="#int_overview">Design Overview</a></li> 36 <li><a href="#int_notes">Additional Notes</a> 37 <ul> 38 <li><a href="#int_compilation">The Compilation Object</a></li> 39 <li><a href="#int_unified_parsing">Unified Parsing & Pipelining</a></li> 40 <li><a href="#int_toolchain_translation">ToolChain Argument Translation</a></li> 41 <li><a href="#int_unused_warnings">Unused Argument Warnings</a></li> 42 </ul> 43 </li> 44 <li><a href="#int_gcc_concepts">Relation to GCC Driver Concepts</a></li> 45 </ul> 46 </li> 47 </ul> 48 49 50 <!-- ======================================================================= --> 51 <h2 id="intro">Introduction</h2> 52 <!-- ======================================================================= --> 53 54 <p>This document describes the Clang driver. The purpose of this 55 document is to describe both the motivation and design goals 56 for the driver, as well as details of the internal 57 implementation.</p> 58 59 <!-- ======================================================================= --> 60 <h2 id="features">Features and Goals</h2> 61 <!-- ======================================================================= --> 62 63 <p>The Clang driver is intended to be a production quality 64 compiler driver providing access to the Clang compiler and 65 tools, with a command line interface which is compatible with 66 the gcc driver.</p> 67 68 <p>Although the driver is part of and driven by the Clang 69 project, it is logically a separate tool which shares many of 70 the same goals as Clang:</p> 71 72 <p><b>Features</b>:</p> 73 <ul> 74 <li><a href="#gcccompat">GCC Compatibility</a></li> 75 <li><a href="#components">Flexible</a></li> 76 <li><a href="#performance">Low Overhead</a></li> 77 <li><a href="#simple">Simple</a></li> 78 </ul> 79 80 <!--=======================================================================--> 81 <h3 id="gcccompat">GCC Compatibility</h3> 82 <!--=======================================================================--> 83 84 <p>The number one goal of the driver is to ease the adoption of 85 Clang by allowing users to drop Clang into a build system 86 which was designed to call GCC. Although this makes the driver 87 much more complicated than might otherwise be necessary, we 88 decided that being very compatible with the gcc command line 89 interface was worth it in order to allow users to quickly test 90 clang on their projects.</p> 91 92 <!--=======================================================================--> 93 <h3 id="components">Flexible</h3> 94 <!--=======================================================================--> 95 96 <p>The driver was designed to be flexible and easily accommodate 97 new uses as we grow the clang and LLVM infrastructure. As one 98 example, the driver can easily support the introduction of 99 tools which have an integrated assembler; something we hope to 100 add to LLVM in the future.</p> 101 102 <p>Similarly, most of the driver functionality is kept in a 103 library which can be used to build other tools which want to 104 implement or accept a gcc like interface. </p> 105 106 <!--=======================================================================--> 107 <h3 id="performance">Low Overhead</h3> 108 <!--=======================================================================--> 109 110 <p>The driver should have as little overhead as possible. In 111 practice, we found that the gcc driver by itself incurred a 112 small but meaningful overhead when compiling many small 113 files. The driver doesn't do much work compared to a 114 compilation, but we have tried to keep it as efficient as 115 possible by following a few simple principles:</p> 116 <ul> 117 <li>Avoid memory allocation and string copying when 118 possible.</li> 119 120 <li>Don't parse arguments more than once.</li> 121 122 <li>Provide a few simple interfaces for efficiently searching 123 arguments.</li> 124 </ul> 125 126 <!--=======================================================================--> 127 <h3 id="simple">Simple</h3> 128 <!--=======================================================================--> 129 130 <p>Finally, the driver was designed to be "as simple as 131 possible", given the other goals. Notably, trying to be 132 completely compatible with the gcc driver adds a significant 133 amount of complexity. However, the design of the driver 134 attempts to mitigate this complexity by dividing the process 135 into a number of independent stages instead of a single 136 monolithic task.</p> 137 138 <!-- ======================================================================= --> 139 <h2 id="design">Internal Design and Implementation</h2> 140 <!-- ======================================================================= --> 141 142 <ul> 143 <li><a href="#int_intro">Internals Introduction</a></li> 144 <li><a href="#int_overview">Design Overview</a></li> 145 <li><a href="#int_notes">Additional Notes</a></li> 146 <li><a href="#int_gcc_concepts">Relation to GCC Driver Concepts</a></li> 147 </ul> 148 149 <!--=======================================================================--> 150 <h3><a name="int_intro">Internals Introduction</a></h3> 151 <!--=======================================================================--> 152 153 <p>In order to satisfy the stated goals, the driver was designed 154 to completely subsume the functionality of the gcc executable; 155 that is, the driver should not need to delegate to gcc to 156 perform subtasks. On Darwin, this implies that the Clang 157 driver also subsumes the gcc driver-driver, which is used to 158 implement support for building universal images (binaries and 159 object files). This also implies that the driver should be 160 able to call the language specific compilers (e.g. cc1) 161 directly, which means that it must have enough information to 162 forward command line arguments to child processes 163 correctly.</p> 164 165 <!--=======================================================================--> 166 <h3><a name="int_overview">Design Overview</a></h3> 167 <!--=======================================================================--> 168 169 <p>The diagram below shows the significant components of the 170 driver architecture and how they relate to one another. The 171 orange components represent concrete data structures built by 172 the driver, the green components indicate conceptually 173 distinct stages which manipulate these data structures, and 174 the blue components are important helper classes. </p> 175 176 <div style="text-align:center"> 177 <a href="DriverArchitecture.png"> 178 <img width=400 src="DriverArchitecture.png" 179 alt="Driver Architecture Diagram"> 180 </a> 181 </div> 182 183 <!--=======================================================================--> 184 <h3><a name="int_stages">Driver Stages</a></h3> 185 <!--=======================================================================--> 186 187 <p>The driver functionality is conceptually divided into five stages:</p> 188 189 <ol> 190 <li> 191 <b>Parse: Option Parsing</b> 192 193 <p>The command line argument strings are decomposed into 194 arguments (<tt>Arg</tt> instances). The driver expects to 195 understand all available options, although there is some 196 facility for just passing certain classes of options 197 through (like <tt>-Wl,</tt>).</p> 198 199 <p>Each argument corresponds to exactly one 200 abstract <tt>Option</tt> definition, which describes how 201 the option is parsed along with some additional 202 metadata. The Arg instances themselves are lightweight and 203 merely contain enough information for clients to determine 204 which option they correspond to and their values (if they 205 have additional parameters).</p> 206 207 <p>For example, a command line like "-Ifoo -I foo" would 208 parse to two Arg instances (a JoinedArg and a SeparateArg 209 instance), but each would refer to the same Option.</p> 210 211 <p>Options are lazily created in order to avoid populating 212 all Option classes when the driver is loaded. Most of the 213 driver code only needs to deal with options by their 214 unique ID (e.g., <tt>options::OPT_I</tt>),</p> 215 216 <p>Arg instances themselves do not generally store the 217 values of parameters. In many cases, this would 218 simply result in creating unnecessary string 219 copies. Instead, Arg instances are always embedded inside 220 an ArgList structure, which contains the original vector 221 of argument strings. Each Arg itself only needs to contain 222 an index into this vector instead of storing its values 223 directly.</p> 224 225 <p>The clang driver can dump the results of this 226 stage using the <tt>-ccc-print-options</tt> flag (which 227 must precede any actual command line arguments). For 228 example:</p> 229 <pre> 230 $ <b>clang -ccc-print-options -Xarch_i386 -fomit-frame-pointer -Wa,-fast -Ifoo -I foo t.c</b> 231 Option 0 - Name: "-Xarch_", Values: {"i386", "-fomit-frame-pointer"} 232 Option 1 - Name: "-Wa,", Values: {"-fast"} 233 Option 2 - Name: "-I", Values: {"foo"} 234 Option 3 - Name: "-I", Values: {"foo"} 235 Option 4 - Name: "<input>", Values: {"t.c"} 236 </pre> 237 238 <p>After this stage is complete the command line should be 239 broken down into well defined option objects with their 240 appropriate parameters. Subsequent stages should rarely, 241 if ever, need to do any string processing.</p> 242 </li> 243 244 <li> 245 <b>Pipeline: Compilation Job Construction</b> 246 247 <p>Once the arguments are parsed, the tree of subprocess 248 jobs needed for the desired compilation sequence are 249 constructed. This involves determining the input files and 250 their types, what work is to be done on them (preprocess, 251 compile, assemble, link, etc.), and constructing a list of 252 Action instances for each task. The result is a list of 253 one or more top-level actions, each of which generally 254 corresponds to a single output (for example, an object or 255 linked executable).</p> 256 257 <p>The majority of Actions correspond to actual tasks, 258 however there are two special Actions. The first is 259 InputAction, which simply serves to adapt an input 260 argument for use as an input to other Actions. The second 261 is BindArchAction, which conceptually alters the 262 architecture to be used for all of its input Actions.</p> 263 264 <p>The clang driver can dump the results of this 265 stage using the <tt>-ccc-print-phases</tt> flag. For 266 example:</p> 267 <pre> 268 $ <b>clang -ccc-print-phases -x c t.c -x assembler t.s</b> 269 0: input, "t.c", c 270 1: preprocessor, {0}, cpp-output 271 2: compiler, {1}, assembler 272 3: assembler, {2}, object 273 4: input, "t.s", assembler 274 5: assembler, {4}, object 275 6: linker, {3, 5}, image 276 </pre> 277 <p>Here the driver is constructing seven distinct actions, 278 four to compile the "t.c" input into an object file, two to 279 assemble the "t.s" input, and one to link them together.</p> 280 281 <p>A rather different compilation pipeline is shown here; in 282 this example there are two top level actions to compile 283 the input files into two separate object files, where each 284 object file is built using <tt>lipo</tt> to merge results 285 built for two separate architectures.</p> 286 <pre> 287 $ <b>clang -ccc-print-phases -c -arch i386 -arch x86_64 t0.c t1.c</b> 288 0: input, "t0.c", c 289 1: preprocessor, {0}, cpp-output 290 2: compiler, {1}, assembler 291 3: assembler, {2}, object 292 4: bind-arch, "i386", {3}, object 293 5: bind-arch, "x86_64", {3}, object 294 6: lipo, {4, 5}, object 295 7: input, "t1.c", c 296 8: preprocessor, {7}, cpp-output 297 9: compiler, {8}, assembler 298 10: assembler, {9}, object 299 11: bind-arch, "i386", {10}, object 300 12: bind-arch, "x86_64", {10}, object 301 13: lipo, {11, 12}, object 302 </pre> 303 304 <p>After this stage is complete the compilation process is 305 divided into a simple set of actions which need to be 306 performed to produce intermediate or final outputs (in 307 some cases, like <tt>-fsyntax-only</tt>, there is no 308 "real" final output). Phases are well known compilation 309 steps, such as "preprocess", "compile", "assemble", 310 "link", etc.</p> 311 </li> 312 313 <li> 314 <b>Bind: Tool & Filename Selection</b> 315 316 <p>This stage (in conjunction with the Translate stage) 317 turns the tree of Actions into a list of actual subprocess 318 to run. Conceptually, the driver performs a top down 319 matching to assign Action(s) to Tools. The ToolChain is 320 responsible for selecting the tool to perform a particular 321 action; once selected the driver interacts with the tool 322 to see if it can match additional actions (for example, by 323 having an integrated preprocessor). 324 325 <p>Once Tools have been selected for all actions, the driver 326 determines how the tools should be connected (for example, 327 using an inprocess module, pipes, temporary files, or user 328 provided filenames). If an output file is required, the 329 driver also computes the appropriate file name (the suffix 330 and file location depend on the input types and options 331 such as <tt>-save-temps</tt>). 332 333 <p>The driver interacts with a ToolChain to perform the Tool 334 bindings. Each ToolChain contains information about all 335 the tools needed for compilation for a particular 336 architecture, platform, and operating system. A single 337 driver invocation may query multiple ToolChains during one 338 compilation in order to interact with tools for separate 339 architectures.</p> 340 341 <p>The results of this stage are not computed directly, but 342 the driver can print the results via 343 the <tt>-ccc-print-bindings</tt> option. For example:</p> 344 <pre> 345 $ <b>clang -ccc-print-bindings -arch i386 -arch ppc t0.c</b> 346 # "i386-apple-darwin9" - "clang", inputs: ["t0.c"], output: "/tmp/cc-Sn4RKF.s" 347 # "i386-apple-darwin9" - "darwin::Assemble", inputs: ["/tmp/cc-Sn4RKF.s"], output: "/tmp/cc-gvSnbS.o" 348 # "i386-apple-darwin9" - "darwin::Link", inputs: ["/tmp/cc-gvSnbS.o"], output: "/tmp/cc-jgHQxi.out" 349 # "ppc-apple-darwin9" - "gcc::Compile", inputs: ["t0.c"], output: "/tmp/cc-Q0bTox.s" 350 # "ppc-apple-darwin9" - "gcc::Assemble", inputs: ["/tmp/cc-Q0bTox.s"], output: "/tmp/cc-WCdicw.o" 351 # "ppc-apple-darwin9" - "gcc::Link", inputs: ["/tmp/cc-WCdicw.o"], output: "/tmp/cc-HHBEBh.out" 352 # "i386-apple-darwin9" - "darwin::Lipo", inputs: ["/tmp/cc-jgHQxi.out", "/tmp/cc-HHBEBh.out"], output: "a.out" 353 </pre> 354 355 <p>This shows the tool chain, tool, inputs and outputs which 356 have been bound for this compilation sequence. Here clang 357 is being used to compile t0.c on the i386 architecture and 358 darwin specific versions of the tools are being used to 359 assemble and link the result, but generic gcc versions of 360 the tools are being used on PowerPC.</p> 361 </li> 362 363 <li> 364 <b>Translate: Tool Specific Argument Translation</b> 365 366 <p>Once a Tool has been selected to perform a particular 367 Action, the Tool must construct concrete Jobs which will be 368 executed during compilation. The main work is in translating 369 from the gcc style command line options to whatever options 370 the subprocess expects.</p> 371 372 <p>Some tools, such as the assembler, only interact with a 373 handful of arguments and just determine the path of the 374 executable to call and pass on their input and output 375 arguments. Others, like the compiler or the linker, may 376 translate a large number of arguments in addition.</p> 377 378 <p>The ArgList class provides a number of simple helper 379 methods to assist with translating arguments; for example, 380 to pass on only the last of arguments corresponding to some 381 option, or all arguments for an option.</p> 382 383 <p>The result of this stage is a list of Jobs (executable 384 paths and argument strings) to execute.</p> 385 </li> 386 387 <li> 388 <b>Execute</b> 389 <p>Finally, the compilation pipeline is executed. This is 390 mostly straightforward, although there is some interaction 391 with options 392 like <tt>-pipe</tt>, <tt>-pass-exit-codes</tt> 393 and <tt>-time</tt>.</p> 394 </li> 395 396 </ol> 397 398 <!--=======================================================================--> 399 <h3><a name="int_notes">Additional Notes</a></h3> 400 <!--=======================================================================--> 401 402 <h4 id="int_compilation">The Compilation Object</h4> 403 404 <p>The driver constructs a Compilation object for each set of 405 command line arguments. The Driver itself is intended to be 406 invariant during construction of a Compilation; an IDE should be 407 able to construct a single long lived driver instance to use 408 for an entire build, for example.</p> 409 410 <p>The Compilation object holds information that is particular 411 to each compilation sequence. For example, the list of used 412 temporary files (which must be removed once compilation is 413 finished) and result files (which should be removed if 414 compilation fails).</p> 415 416 <h4 id="int_unified_parsing">Unified Parsing & Pipelining</h4> 417 418 <p>Parsing and pipelining both occur without reference to a 419 Compilation instance. This is by design; the driver expects that 420 both of these phases are platform neutral, with a few very well 421 defined exceptions such as whether the platform uses a driver 422 driver.</p> 423 424 <h4 id="int_toolchain_translation">ToolChain Argument Translation</h4> 425 426 <p>In order to match gcc very closely, the clang driver 427 currently allows tool chains to perform their own translation of 428 the argument list (into a new ArgList data structure). Although 429 this allows the clang driver to match gcc easily, it also makes 430 the driver operation much harder to understand (since the Tools 431 stop seeing some arguments the user provided, and see new ones 432 instead).</p> 433 434 <p>For example, on Darwin <tt>-gfull</tt> gets translated into two 435 separate arguments, <tt>-g</tt> 436 and <tt>-fno-eliminate-unused-debug-symbols</tt>. Trying to write Tool 437 logic to do something with <tt>-gfull</tt> will not work, because Tool 438 argument translation is done after the arguments have been 439 translated.</p> 440 441 <p>A long term goal is to remove this tool chain specific 442 translation, and instead force each tool to change its own logic 443 to do the right thing on the untranslated original arguments.</p> 444 445 <h4 id="int_unused_warnings">Unused Argument Warnings</h4> 446 <p>The driver operates by parsing all arguments but giving Tools 447 the opportunity to choose which arguments to pass on. One 448 downside of this infrastructure is that if the user misspells 449 some option, or is confused about which options to use, some 450 command line arguments the user really cared about may go 451 unused. This problem is particularly important when using 452 clang as a compiler, since the clang compiler does not support 453 anywhere near all the options that gcc does, and we want to make 454 sure users know which ones are being used.</p> 455 456 <p>To support this, the driver maintains a bit associated with 457 each argument of whether it has been used (at all) during the 458 compilation. This bit usually doesn't need to be set by hand, 459 as the key ArgList accessors will set it automatically.</p> 460 461 <p>When a compilation is successful (there are no errors), the 462 driver checks the bit and emits an "unused argument" warning for 463 any arguments which were never accessed. This is conservative 464 (the argument may not have been used to do what the user wanted) 465 but still catches the most obvious cases.</p> 466 467 <!--=======================================================================--> 468 <h3><a name="int_gcc_concepts">Relation to GCC Driver Concepts</a></h3> 469 <!--=======================================================================--> 470 471 <p>For those familiar with the gcc driver, this section provides 472 a brief overview of how things from the gcc driver map to the 473 clang driver.</p> 474 475 <ul> 476 <li> 477 <b>Driver Driver</b> 478 <p>The driver driver is fully integrated into the clang 479 driver. The driver simply constructs additional Actions to 480 bind the architecture during the <i>Pipeline</i> 481 phase. The tool chain specific argument translation is 482 responsible for handling <tt>-Xarch_</tt>.</p> 483 484 <p>The one caveat is that this approach 485 requires <tt>-Xarch_</tt> not be used to alter the 486 compilation itself (for example, one cannot 487 provide <tt>-S</tt> as an <tt>-Xarch_</tt> argument). The 488 driver attempts to reject such invocations, and overall 489 there isn't a good reason to abuse <tt>-Xarch_</tt> to 490 that end in practice.</p> 491 492 <p>The upside is that the clang driver is more efficient and 493 does little extra work to support universal builds. It also 494 provides better error reporting and UI consistency.</p> 495 </li> 496 497 <li> 498 <b>Specs</b> 499 <p>The clang driver has no direct correspondent for 500 "specs". The majority of the functionality that is 501 embedded in specs is in the Tool specific argument 502 translation routines. The parts of specs which control the 503 compilation pipeline are generally part of 504 the <i>Pipeline</i> stage.</p> 505 </li> 506 507 <li> 508 <b>Toolchains</b> 509 <p>The gcc driver has no direct understanding of tool 510 chains. Each gcc binary roughly corresponds to the 511 information which is embedded inside a single 512 ToolChain.</p> 513 514 <p>The clang driver is intended to be portable and support 515 complex compilation environments. All platform and tool 516 chain specific code should be protected behind either 517 abstract or well defined interfaces (such as whether the 518 platform supports use as a driver driver).</p> 519 </li> 520 </ul> 521 </div> 522 </body> 523 </html> 524
