Sean Silva | 0a7d2fe | 2012-12-21 00:16:53 +0000 | [diff] [blame] | 1 | ========================= |
| 2 | Driver Design & Internals |
| 3 | ========================= |
| 4 | |
| 5 | .. contents:: |
| 6 | :local: |
| 7 | |
| 8 | Introduction |
| 9 | ============ |
| 10 | |
| 11 | This document describes the Clang driver. The purpose of this document |
| 12 | is to describe both the motivation and design goals for the driver, as |
| 13 | well as details of the internal implementation. |
| 14 | |
| 15 | Features and Goals |
| 16 | ================== |
| 17 | |
| 18 | The Clang driver is intended to be a production quality compiler driver |
| 19 | providing access to the Clang compiler and tools, with a command line |
| 20 | interface which is compatible with the gcc driver. |
| 21 | |
| 22 | Although the driver is part of and driven by the Clang project, it is |
| 23 | logically a separate tool which shares many of the same goals as Clang: |
| 24 | |
| 25 | .. contents:: Features |
| 26 | :local: |
| 27 | |
| 28 | GCC Compatibility |
| 29 | ----------------- |
| 30 | |
| 31 | The number one goal of the driver is to ease the adoption of Clang by |
| 32 | allowing users to drop Clang into a build system which was designed to |
| 33 | call GCC. Although this makes the driver much more complicated than |
| 34 | might otherwise be necessary, we decided that being very compatible with |
| 35 | the gcc command line interface was worth it in order to allow users to |
| 36 | quickly test clang on their projects. |
| 37 | |
| 38 | Flexible |
| 39 | -------- |
| 40 | |
| 41 | The driver was designed to be flexible and easily accommodate new uses |
| 42 | as we grow the clang and LLVM infrastructure. As one example, the driver |
| 43 | can easily support the introduction of tools which have an integrated |
| 44 | assembler; something we hope to add to LLVM in the future. |
| 45 | |
| 46 | Similarly, most of the driver functionality is kept in a library which |
| 47 | can be used to build other tools which want to implement or accept a gcc |
| 48 | like interface. |
| 49 | |
| 50 | Low Overhead |
| 51 | ------------ |
| 52 | |
| 53 | The driver should have as little overhead as possible. In practice, we |
| 54 | found that the gcc driver by itself incurred a small but meaningful |
| 55 | overhead when compiling many small files. The driver doesn't do much |
| 56 | work compared to a compilation, but we have tried to keep it as |
| 57 | efficient as possible by following a few simple principles: |
| 58 | |
| 59 | - Avoid memory allocation and string copying when possible. |
| 60 | - Don't parse arguments more than once. |
| 61 | - Provide a few simple interfaces for efficiently searching arguments. |
| 62 | |
| 63 | Simple |
| 64 | ------ |
| 65 | |
| 66 | Finally, the driver was designed to be "as simple as possible", given |
| 67 | the other goals. Notably, trying to be completely compatible with the |
| 68 | gcc driver adds a significant amount of complexity. However, the design |
| 69 | of the driver attempts to mitigate this complexity by dividing the |
| 70 | process into a number of independent stages instead of a single |
| 71 | monolithic task. |
| 72 | |
| 73 | Internal Design and Implementation |
| 74 | ================================== |
| 75 | |
| 76 | .. contents:: |
| 77 | :local: |
| 78 | :depth: 1 |
| 79 | |
| 80 | Internals Introduction |
| 81 | ---------------------- |
| 82 | |
| 83 | In order to satisfy the stated goals, the driver was designed to |
| 84 | completely subsume the functionality of the gcc executable; that is, the |
| 85 | driver should not need to delegate to gcc to perform subtasks. On |
| 86 | Darwin, this implies that the Clang driver also subsumes the gcc |
| 87 | driver-driver, which is used to implement support for building universal |
| 88 | images (binaries and object files). This also implies that the driver |
| 89 | should be able to call the language specific compilers (e.g. cc1) |
| 90 | directly, which means that it must have enough information to forward |
| 91 | command line arguments to child processes correctly. |
| 92 | |
| 93 | Design Overview |
| 94 | --------------- |
| 95 | |
| 96 | The diagram below shows the significant components of the driver |
| 97 | architecture and how they relate to one another. The orange components |
| 98 | represent concrete data structures built by the driver, the green |
| 99 | components indicate conceptually distinct stages which manipulate these |
| 100 | data structures, and the blue components are important helper classes. |
| 101 | |
| 102 | .. image:: DriverArchitecture.png |
| 103 | :align: center |
| 104 | :alt: Driver Architecture Diagram |
| 105 | |
| 106 | Driver Stages |
| 107 | ------------- |
| 108 | |
| 109 | The driver functionality is conceptually divided into five stages: |
| 110 | |
| 111 | #. **Parse: Option Parsing** |
| 112 | |
| 113 | The command line argument strings are decomposed into arguments |
| 114 | (``Arg`` instances). The driver expects to understand all available |
| 115 | options, although there is some facility for just passing certain |
| 116 | classes of options through (like ``-Wl,``). |
| 117 | |
| 118 | Each argument corresponds to exactly one abstract ``Option`` |
| 119 | definition, which describes how the option is parsed along with some |
| 120 | additional metadata. The Arg instances themselves are lightweight and |
| 121 | merely contain enough information for clients to determine which |
| 122 | option they correspond to and their values (if they have additional |
| 123 | parameters). |
| 124 | |
| 125 | For example, a command line like "-Ifoo -I foo" would parse to two |
| 126 | Arg instances (a JoinedArg and a SeparateArg instance), but each |
| 127 | would refer to the same Option. |
| 128 | |
| 129 | Options are lazily created in order to avoid populating all Option |
| 130 | classes when the driver is loaded. Most of the driver code only needs |
| 131 | to deal with options by their unique ID (e.g., ``options::OPT_I``), |
| 132 | |
| 133 | Arg instances themselves do not generally store the values of |
| 134 | parameters. In many cases, this would simply result in creating |
| 135 | unnecessary string copies. Instead, Arg instances are always embedded |
| 136 | inside an ArgList structure, which contains the original vector of |
| 137 | argument strings. Each Arg itself only needs to contain an index into |
| 138 | this vector instead of storing its values directly. |
| 139 | |
| 140 | The clang driver can dump the results of this stage using the |
| 141 | ``-ccc-print-options`` flag (which must precede any actual command |
| 142 | line arguments). For example: |
| 143 | |
| 144 | .. code-block:: console |
| 145 | |
| 146 | $ clang -ccc-print-options -Xarch_i386 -fomit-frame-pointer -Wa,-fast -Ifoo -I foo t.c |
| 147 | Option 0 - Name: "-Xarch_", Values: {"i386", "-fomit-frame-pointer"} |
| 148 | Option 1 - Name: "-Wa,", Values: {"-fast"} |
| 149 | Option 2 - Name: "-I", Values: {"foo"} |
| 150 | Option 3 - Name: "-I", Values: {"foo"} |
| 151 | Option 4 - Name: "<input>", Values: {"t.c"} |
| 152 | |
| 153 | After this stage is complete the command line should be broken down |
| 154 | into well defined option objects with their appropriate parameters. |
| 155 | Subsequent stages should rarely, if ever, need to do any string |
| 156 | processing. |
| 157 | |
| 158 | #. **Pipeline: Compilation Job Construction** |
| 159 | |
| 160 | Once the arguments are parsed, the tree of subprocess jobs needed for |
| 161 | the desired compilation sequence are constructed. This involves |
| 162 | determining the input files and their types, what work is to be done |
| 163 | on them (preprocess, compile, assemble, link, etc.), and constructing |
| 164 | a list of Action instances for each task. The result is a list of one |
| 165 | or more top-level actions, each of which generally corresponds to a |
| 166 | single output (for example, an object or linked executable). |
| 167 | |
| 168 | The majority of Actions correspond to actual tasks, however there are |
| 169 | two special Actions. The first is InputAction, which simply serves to |
| 170 | adapt an input argument for use as an input to other Actions. The |
| 171 | second is BindArchAction, which conceptually alters the architecture |
| 172 | to be used for all of its input Actions. |
| 173 | |
| 174 | The clang driver can dump the results of this stage using the |
| 175 | ``-ccc-print-phases`` flag. For example: |
| 176 | |
| 177 | .. code-block:: console |
| 178 | |
| 179 | $ clang -ccc-print-phases -x c t.c -x assembler t.s |
| 180 | 0: input, "t.c", c |
| 181 | 1: preprocessor, {0}, cpp-output |
| 182 | 2: compiler, {1}, assembler |
| 183 | 3: assembler, {2}, object |
| 184 | 4: input, "t.s", assembler |
| 185 | 5: assembler, {4}, object |
| 186 | 6: linker, {3, 5}, image |
| 187 | |
| 188 | Here the driver is constructing seven distinct actions, four to |
| 189 | compile the "t.c" input into an object file, two to assemble the |
| 190 | "t.s" input, and one to link them together. |
| 191 | |
| 192 | A rather different compilation pipeline is shown here; in this |
| 193 | example there are two top level actions to compile the input files |
| 194 | into two separate object files, where each object file is built using |
| 195 | ``lipo`` to merge results built for two separate architectures. |
| 196 | |
| 197 | .. code-block:: console |
| 198 | |
| 199 | $ clang -ccc-print-phases -c -arch i386 -arch x86_64 t0.c t1.c |
| 200 | 0: input, "t0.c", c |
| 201 | 1: preprocessor, {0}, cpp-output |
| 202 | 2: compiler, {1}, assembler |
| 203 | 3: assembler, {2}, object |
| 204 | 4: bind-arch, "i386", {3}, object |
| 205 | 5: bind-arch, "x86_64", {3}, object |
| 206 | 6: lipo, {4, 5}, object |
| 207 | 7: input, "t1.c", c |
| 208 | 8: preprocessor, {7}, cpp-output |
| 209 | 9: compiler, {8}, assembler |
| 210 | 10: assembler, {9}, object |
| 211 | 11: bind-arch, "i386", {10}, object |
| 212 | 12: bind-arch, "x86_64", {10}, object |
| 213 | 13: lipo, {11, 12}, object |
| 214 | |
| 215 | After this stage is complete the compilation process is divided into |
| 216 | a simple set of actions which need to be performed to produce |
| 217 | intermediate or final outputs (in some cases, like ``-fsyntax-only``, |
| 218 | there is no "real" final output). Phases are well known compilation |
| 219 | steps, such as "preprocess", "compile", "assemble", "link", etc. |
| 220 | |
| 221 | #. **Bind: Tool & Filename Selection** |
| 222 | |
| 223 | This stage (in conjunction with the Translate stage) turns the tree |
| 224 | of Actions into a list of actual subprocess to run. Conceptually, the |
| 225 | driver performs a top down matching to assign Action(s) to Tools. The |
| 226 | ToolChain is responsible for selecting the tool to perform a |
| 227 | particular action; once selected the driver interacts with the tool |
| 228 | to see if it can match additional actions (for example, by having an |
| 229 | integrated preprocessor). |
| 230 | |
| 231 | Once Tools have been selected for all actions, the driver determines |
| 232 | how the tools should be connected (for example, using an inprocess |
| 233 | module, pipes, temporary files, or user provided filenames). If an |
| 234 | output file is required, the driver also computes the appropriate |
| 235 | file name (the suffix and file location depend on the input types and |
| 236 | options such as ``-save-temps``). |
| 237 | |
| 238 | The driver interacts with a ToolChain to perform the Tool bindings. |
| 239 | Each ToolChain contains information about all the tools needed for |
| 240 | compilation for a particular architecture, platform, and operating |
| 241 | system. A single driver invocation may query multiple ToolChains |
| 242 | during one compilation in order to interact with tools for separate |
| 243 | architectures. |
| 244 | |
| 245 | The results of this stage are not computed directly, but the driver |
| 246 | can print the results via the ``-ccc-print-bindings`` option. For |
| 247 | example: |
| 248 | |
| 249 | .. code-block:: console |
| 250 | |
| 251 | $ clang -ccc-print-bindings -arch i386 -arch ppc t0.c |
| 252 | # "i386-apple-darwin9" - "clang", inputs: ["t0.c"], output: "/tmp/cc-Sn4RKF.s" |
| 253 | # "i386-apple-darwin9" - "darwin::Assemble", inputs: ["/tmp/cc-Sn4RKF.s"], output: "/tmp/cc-gvSnbS.o" |
| 254 | # "i386-apple-darwin9" - "darwin::Link", inputs: ["/tmp/cc-gvSnbS.o"], output: "/tmp/cc-jgHQxi.out" |
| 255 | # "ppc-apple-darwin9" - "gcc::Compile", inputs: ["t0.c"], output: "/tmp/cc-Q0bTox.s" |
| 256 | # "ppc-apple-darwin9" - "gcc::Assemble", inputs: ["/tmp/cc-Q0bTox.s"], output: "/tmp/cc-WCdicw.o" |
| 257 | # "ppc-apple-darwin9" - "gcc::Link", inputs: ["/tmp/cc-WCdicw.o"], output: "/tmp/cc-HHBEBh.out" |
| 258 | # "i386-apple-darwin9" - "darwin::Lipo", inputs: ["/tmp/cc-jgHQxi.out", "/tmp/cc-HHBEBh.out"], output: "a.out" |
| 259 | |
| 260 | This shows the tool chain, tool, inputs and outputs which have been |
| 261 | bound for this compilation sequence. Here clang is being used to |
| 262 | compile t0.c on the i386 architecture and darwin specific versions of |
| 263 | the tools are being used to assemble and link the result, but generic |
| 264 | gcc versions of the tools are being used on PowerPC. |
| 265 | |
| 266 | #. **Translate: Tool Specific Argument Translation** |
| 267 | |
| 268 | Once a Tool has been selected to perform a particular Action, the |
| 269 | Tool must construct concrete Jobs which will be executed during |
| 270 | compilation. The main work is in translating from the gcc style |
| 271 | command line options to whatever options the subprocess expects. |
| 272 | |
| 273 | Some tools, such as the assembler, only interact with a handful of |
| 274 | arguments and just determine the path of the executable to call and |
| 275 | pass on their input and output arguments. Others, like the compiler |
| 276 | or the linker, may translate a large number of arguments in addition. |
| 277 | |
| 278 | The ArgList class provides a number of simple helper methods to |
| 279 | assist with translating arguments; for example, to pass on only the |
| 280 | last of arguments corresponding to some option, or all arguments for |
| 281 | an option. |
| 282 | |
| 283 | The result of this stage is a list of Jobs (executable paths and |
| 284 | argument strings) to execute. |
| 285 | |
| 286 | #. **Execute** |
| 287 | |
| 288 | Finally, the compilation pipeline is executed. This is mostly |
| 289 | straightforward, although there is some interaction with options like |
| 290 | ``-pipe``, ``-pass-exit-codes`` and ``-time``. |
| 291 | |
| 292 | Additional Notes |
| 293 | ---------------- |
| 294 | |
| 295 | The Compilation Object |
| 296 | ^^^^^^^^^^^^^^^^^^^^^^ |
| 297 | |
| 298 | The driver constructs a Compilation object for each set of command line |
| 299 | arguments. The Driver itself is intended to be invariant during |
| 300 | construction of a Compilation; an IDE should be able to construct a |
| 301 | single long lived driver instance to use for an entire build, for |
| 302 | example. |
| 303 | |
| 304 | The Compilation object holds information that is particular to each |
| 305 | compilation sequence. For example, the list of used temporary files |
| 306 | (which must be removed once compilation is finished) and result files |
| 307 | (which should be removed if compilation fails). |
| 308 | |
| 309 | Unified Parsing & Pipelining |
| 310 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 311 | |
| 312 | Parsing and pipelining both occur without reference to a Compilation |
| 313 | instance. This is by design; the driver expects that both of these |
| 314 | phases are platform neutral, with a few very well defined exceptions |
| 315 | such as whether the platform uses a driver driver. |
| 316 | |
| 317 | ToolChain Argument Translation |
| 318 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 319 | |
| 320 | In order to match gcc very closely, the clang driver currently allows |
| 321 | tool chains to perform their own translation of the argument list (into |
| 322 | a new ArgList data structure). Although this allows the clang driver to |
| 323 | match gcc easily, it also makes the driver operation much harder to |
| 324 | understand (since the Tools stop seeing some arguments the user |
| 325 | provided, and see new ones instead). |
| 326 | |
| 327 | For example, on Darwin ``-gfull`` gets translated into two separate |
| 328 | arguments, ``-g`` and ``-fno-eliminate-unused-debug-symbols``. Trying to |
| 329 | write Tool logic to do something with ``-gfull`` will not work, because |
| 330 | Tool argument translation is done after the arguments have been |
| 331 | translated. |
| 332 | |
| 333 | A long term goal is to remove this tool chain specific translation, and |
| 334 | instead force each tool to change its own logic to do the right thing on |
| 335 | the untranslated original arguments. |
| 336 | |
| 337 | Unused Argument Warnings |
| 338 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
| 339 | |
| 340 | The driver operates by parsing all arguments but giving Tools the |
| 341 | opportunity to choose which arguments to pass on. One downside of this |
| 342 | infrastructure is that if the user misspells some option, or is confused |
| 343 | about which options to use, some command line arguments the user really |
| 344 | cared about may go unused. This problem is particularly important when |
| 345 | using clang as a compiler, since the clang compiler does not support |
| 346 | anywhere near all the options that gcc does, and we want to make sure |
| 347 | users know which ones are being used. |
| 348 | |
| 349 | To support this, the driver maintains a bit associated with each |
| 350 | argument of whether it has been used (at all) during the compilation. |
| 351 | This bit usually doesn't need to be set by hand, as the key ArgList |
| 352 | accessors will set it automatically. |
| 353 | |
| 354 | When a compilation is successful (there are no errors), the driver |
| 355 | checks the bit and emits an "unused argument" warning for any arguments |
| 356 | which were never accessed. This is conservative (the argument may not |
| 357 | have been used to do what the user wanted) but still catches the most |
| 358 | obvious cases. |
| 359 | |
| 360 | Relation to GCC Driver Concepts |
| 361 | ------------------------------- |
| 362 | |
| 363 | For those familiar with the gcc driver, this section provides a brief |
| 364 | overview of how things from the gcc driver map to the clang driver. |
| 365 | |
| 366 | - **Driver Driver** |
| 367 | |
| 368 | The driver driver is fully integrated into the clang driver. The |
| 369 | driver simply constructs additional Actions to bind the architecture |
| 370 | during the *Pipeline* phase. The tool chain specific argument |
| 371 | translation is responsible for handling ``-Xarch_``. |
| 372 | |
| 373 | The one caveat is that this approach requires ``-Xarch_`` not be used |
| 374 | to alter the compilation itself (for example, one cannot provide |
| 375 | ``-S`` as an ``-Xarch_`` argument). The driver attempts to reject |
| 376 | such invocations, and overall there isn't a good reason to abuse |
| 377 | ``-Xarch_`` to that end in practice. |
| 378 | |
| 379 | The upside is that the clang driver is more efficient and does little |
| 380 | extra work to support universal builds. It also provides better error |
| 381 | reporting and UI consistency. |
| 382 | |
| 383 | - **Specs** |
| 384 | |
| 385 | The clang driver has no direct correspondent for "specs". The |
| 386 | majority of the functionality that is embedded in specs is in the |
| 387 | Tool specific argument translation routines. The parts of specs which |
| 388 | control the compilation pipeline are generally part of the *Pipeline* |
| 389 | stage. |
| 390 | |
| 391 | - **Toolchains** |
| 392 | |
| 393 | The gcc driver has no direct understanding of tool chains. Each gcc |
| 394 | binary roughly corresponds to the information which is embedded |
| 395 | inside a single ToolChain. |
| 396 | |
| 397 | The clang driver is intended to be portable and support complex |
| 398 | compilation environments. All platform and tool chain specific code |
| 399 | should be protected behind either abstract or well defined interfaces |
| 400 | (such as whether the platform supports use as a driver driver). |