Gitiles
Code Review Sign In
gerrit-public.fairphone.software / fp2-dev / platform / external / clang / 0084ead1f867b745d13121b2560441e355cb347b / . / docs / PCHInternals.html
blob: 74ef403808672a7fd8f9638401a640cf05b9dc4c [file] [log] [blame]
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]1<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
2<html> <head>
3<title>Precompiled Headers (PCH)</title>
4</head>
5
6<body>
7
8<!--#include virtual="../menu.html.incl"-->
9
10<div id="content">
11
12<h1>Precompiled Headers</h1>
13
14 <p>This document describes the design and implementation of Clang's
15 precompiled headers (PCH). If you are interested in the end-user
16 view, please see the <a
17 href="UsersManual.html#precompiledheaders">User's Manual</a>.</p>
18
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]19 <p><b>Table of Contents</b></p>
20 <ul>
21 <li><a href="#usage">Using Precompiled Headers with
22 <tt>clang-cc</tt></a></li>
23 <li><a href="#philosophy">Design Philosophy</a></li>
24 <li><a href="#contents">Precompiled Header Contents</a>
25 <ul>
26 <li><a href="#metadata">Metadata Block</a></li>
27 <li><a href="#sourcemgr">Source Manager Block</a></li>
28 <li><a href="#preprocessor">Preprocessor Block</a></li>
29 <li><a href="#types">Types Block</a></li>
30 <li><a href="#decls">Declarations Block</a></li>
31 <li><a href="#stmt">Statements and Expressions</a></li>
32 <li><a href="#idtable">Identifier Table Block</a></li>
33 <li><a href="#method-pool">Method Pool Block</a></li>
34 </ul>
35 </li>
Douglas Gregor0084ead2009-06-03 21:41:31 +0000[diff] [blame^]36 <li><a href="#tendrils">Precompiled Header Integration Points</a></li>
37</ul>
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]38
39<h2 id="usage">Using Precompiled Headers with <tt>clang-cc</tt></h2>
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]40
41<p>The low-level Clang compiler, <tt>clang-cc</tt>, supports two command
42line options for generating and using PCH files.<p>
43
44<p>To generate PCH files using <tt>clang-cc</tt>, use the option
45<b><tt>-emit-pch</tt></b>:
46
47<pre> $ clang-cc test.h -emit-pch -o test.h.pch </pre>
48
49<p>This option is transparently used by <tt>clang</tt> when generating
50PCH files. The resulting PCH file contains the serialized form of the
51compiler's internal representation after it has completed parsing and
52semantic analysis. The PCH file can then be used as a prefix header
53with the <b><tt>-include-pch</tt></b> option:</p>
54
55<pre>
56 $ clang-cc -include-pch test.h.pch test.c -o test.s
57</pre>
58
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]59<h2 id="philosophy">Design Philosophy</h2>
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]60
61<p>Precompiled headers are meant to improve overall compile times for
62 projects, so the design of precompiled headers is entirely driven by
63 performance concerns. The use case for precompiled headers is
64 relatively simple: when there is a common set of headers that is
65 included in nearly every source file in the project, we
66 <i>precompile</i> that bundle of headers into a single precompiled
67 header (PCH file). Then, when compiling the source files in the
68 project, we load the PCH file first (as a prefix header), which acts
69 as a stand-in for that bundle of headers.</p>
70
71<p>A precompiled header implementation improves performance when:</p>
72<ul>
73 <li>Loading the PCH file is significantly faster than re-parsing the
74 bundle of headers stored within the PCH file. Thus, a precompiled
75 header design attempts to minimize the cost of reading the PCH
76 file. Ideally, this cost should not vary with the size of the
77 precompiled header file.</li>
78
79 <li>The cost of generating the PCH file initially is not so large
80 that it counters the per-source-file performance improvement due to
81 eliminating the need to parse the bundled headers in the first
82 place. This is particularly important on multi-core systems, because
83 PCH file generation serializes the build when all compilations
84 require the PCH file to be up-to-date.</li>
85</ul>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]86
87<p>Clang's precompiled headers are designed with a compact on-disk
88representation, which minimizes both PCH creation time and the time
89required to initially load the PCH file. The PCH file itself contains
90a serialized representation of Clang's abstract syntax trees and
91supporting data structures, stored using the same compressed bitstream
92as <a href="http://llvm.org/docs/BitCodeFormat.html">LLVM's bitcode
93file format</a>.</p>
94
95<p>Clang's precompiled headers are loaded "lazily" from disk. When a
96PCH file is initially loaded, Clang reads only a small amount of data
97from the PCH file to establish where certain important data structures
98are stored. The amount of data read in this initial load is
99independent of the size of the PCH file, such that a larger PCH file
100does not lead to longer PCH load times. The actual header data in the
101PCH file--macros, functions, variables, types, etc.--is loaded only
102when it is referenced from the user's code, at which point only that
103entity (and those entities it depends on) are deserialized from the
104PCH file. With this approach, the cost of using a precompiled header
105for a translation unit is proportional to the amount of code actually
106used from the header, rather than being proportional to the size of
107the header itself.</p> </body>
108
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]109<h2 id="contents">Precompiled Header Contents</h2>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]110
111<img src="PCHLayout.png" align="right" alt="Precompiled header layout">
112
113<p>Clang's precompiled headers are organized into several different
114blocks, each of which contains the serialized representation of a part
115of Clang's internal representation. Each of the blocks corresponds to
116either a block or a record within <a
117 href="http://llvm.org/docs/BitCodeFormat.html">LLVM's bitstream
118format</a>. The contents of each of these logical blocks are described
119below.</p>
120
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]121<h3 id="metadata">Metadata Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]122
123<p>The metadata block contains several records that provide
124information about how the precompiled header was built. This metadata
125is primarily used to validate the use of a precompiled header. For
Douglas Gregorfe3f2232009-06-03 18:26:16 +0000[diff] [blame]126example, a precompiled header built for a 32-bit x86 target cannot be used
127when compiling for a 64-bit x86 target. The metadata block contains
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]128information about:</p>
129
130<dl>
131 <dt>Language options</dt>
132 <dd>Describes the particular language dialect used to compile the
133PCH file, including major options (e.g., Objective-C support) and more
134minor options (e.g., support for "//" comments). The contents of this
135record correspond to the <code>LangOptions</code> class.</dd>
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]136
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]137 <dt>Target architecture</dt>
138 <dd>The target triple that describes the architecture, platform, and
139ABI for which the PCH file was generated, e.g.,
140<code>i386-apple-darwin9</code>.</dd>
141
142 <dt>PCH version</dt>
143 <dd>The major and minor version numbers of the precompiled header
144format. Changes in the minor version number should not affect backward
145compatibility, while changes in the major version number imply that a
146newer compiler cannot read an older precompiled header (and
147vice-versa).</dd>
148
149 <dt>Original file name</dt>
150 <dd>The full path of the header that was used to generate the
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]151precompiled header.</dd>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]152
153 <dt>Predefines buffer</dt>
154 <dd>Although not explicitly stored as part of the metadata, the
155predefines buffer is used in the validation of the precompiled header.
156The predefines buffer itself contains code generated by the compiler
157to initialize the preprocessor state according to the current target,
158platform, and command-line options. For example, the predefines buffer
159will contain "<code>#define __STDC__ 1</code>" when we are compiling C
160without Microsoft extensions. The predefines buffer itself is stored
161within the <a href="#sourcemgr">source manager block</a>, but its
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]162contents are verified along with the rest of the metadata.</dd>
163
164</dl>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]165
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]166<h3 id="sourcemgr">Source Manager Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]167
168<p>The source manager block contains the serialized representation of
169Clang's <a
170 href="InternalsManual.html#SourceLocation">SourceManager</a> class,
171which handles the mapping from source locations (as represented in
172Clang's abstract syntax tree) into actual column/line positions within
173a source file or macro instantiation. The precompiled header's
174representation of the source manager also includes information about
175all of the headers that were (transitively) included when building the
176precompiled header.</p>
177
178<p>The bulk of the source manager block is dedicated to information
179about the various files, buffers, and macro instantiations into which
180a source location can refer. Each of these is referenced by a numeric
181"file ID", which is a unique number (allocated starting at 1) stored
182in the source location. Clang serializes the information for each kind
183of file ID, along with an index that maps file IDs to the position
184within the PCH file where the information about that file ID is
185stored. The data associated with a file ID is loaded only when
186required by the front end, e.g., to emit a diagnostic that includes a
187macro instantiation history inside the header itself.</p>
188
189<p>The source manager block also contains information about all of the
190headers that were included when building the precompiled header. This
191includes information about the controlling macro for the header (e.g.,
192when the preprocessor identified that the contents of the header
193dependent on a macro like <code>LLVM_CLANG_SOURCEMANAGER_H</code>)
194along with a cached version of the results of the <code>stat()</code>
195system calls performed when building the precompiled header. The
196latter is particularly useful in reducing system time when searching
197for include files.</p>
198
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]199<h3 id="preprocessor">Preprocessor Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]200
201<p>The preprocessor block contains the serialized representation of
202the preprocessor. Specifically, it contains all of the macros that
203have been defined by the end of the header used to build the
204precompiled header, along with the token sequences that comprise each
205macro. The macro definitions are only read from the PCH file when the
206name of the macro first occurs in the program. This lazy loading of
207macro definitions is trigged by lookups into the <a
208 href="#idtable">identifier table</a>.</p>
209
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]210<h3 id="types">Types Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]211
212<p>The types block contains the serialized representation of all of
213the types referenced in the translation unit. Each Clang type node
214(<code>PointerType</code>, <code>FunctionProtoType</code>, etc.) has a
215corresponding record type in the PCH file. When types are deserialized
216from the precompiled header, the data within the record is used to
217reconstruct the appropriate type node using the AST context.</p>
218
219<p>Each type has a unique type ID, which is an integer that uniquely
220identifies that type. Type ID 0 represents the NULL type, type IDs
221less than <code>NUM_PREDEF_TYPE_IDS</code> represent predefined types
222(<code>void</code>, <code>float</code>, etc.), while other
223"user-defined" type IDs are assigned consecutively from
224<code>NUM_PREDEF_TYPE_IDS</code> upward as the types are encountered.
225The PCH file has an associated mapping from the user-defined types
226block to the location within the types block where the serialized
227representation of that type resides, enabling lazy deserialization of
228types. When a type is referenced from within the PCH file, that
229reference is encoded using the type ID shifted left by 3 bits. The
230lower three bits are used to represent the <code>const</code>,
231<code>volatile</code>, and <code>restrict</code> qualifiers, as in
232Clang's <a
233 href="http://clang.llvm.org/docs/InternalsManual.html#Type">QualType</a>
234class.</p>
235
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]236<h3 id="decls">Declarations Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]237
238<p>The declarations block contains the serialized representation of
239all of the declarations referenced in the translation unit. Each Clang
240declaration node (<code>VarDecl</code>, <code>FunctionDecl</code>,
241etc.) has a corresponding record type in the PCH file. When
242declarations are deserialized from the precompiled header, the data
243within the record is used to build and populate a new instance of the
244corresponding <code>Decl</code> node. As with types, each declaration
245node has a numeric ID that is used to refer to that declaration within
246the PCH file. In addition, a lookup table provides a mapping from that
247numeric ID to the offset within the precompiled header where that
248declaration is described.</p>
249
250<p>Declarations in Clang's abstract syntax trees are stored
251hierarchically. At the top of the hierarchy is the translation unit
252(<code>TranslationUnitDecl</code>), which contains all of the
253declarations in the translation unit. These declarations---such as
254functions or struct types---may also contain other declarations inside
255them, and so on. Within Clang, each declaration is stored within a <a
256href="http://clang.llvm.org/docs/InternalsManual.html#DeclContext">declaration
257context</a>, as represented by the <code>DeclContext</code> class.
258Declaration contexts provide the mechanism to perform name lookup
259within a given declaration (e.g., find the member named <code>x</code>
260in a structure) and iterate over the declarations stored within a
261context (e.g., iterate over all of the fields of a structure for
262structure layout).</p>
263
264<p>In Clang's precompiled header format, deserializing a declaration
265that is a <code>DeclContext</code> is a separate operation from
266deserializing all of the declarations stored within that declaration
267context. Therefore, Clang will deserialize the translation unit
268declaration without deserializing the declarations within that
269translation unit. When required, the declarations stored within a
270declaration context will be serialized. There are two representations
271of the declarations within a declaration context, which correspond to
272the name-lookup and iteration behavior described above:</p>
273
274<ul>
275 <li>When the front end performs name lookup to find a name
276 <code>x</code> within a given declaration context (for example,
277 during semantic analysis of the expression <code>p-&gt;x</code>,
278 where <code>p</code>'s type is defined in the precompiled header),
279 Clang deserializes a hash table mapping from the names within that
280 declaration context to the declaration IDs that represent each
281 visible declaration with that name. The entire hash table is
282 deserialized at this point (into the <code>llvm::DenseMap</code>
283 stored within each <code>DeclContext</code> object), but the actual
284 declarations are not yet deserialized. In a second step, those
285 declarations with the name <code>x</code> will be deserialized and
286 will be used as the result of name lookup.</li>
287
288 <li>When the front end performs iteration over all of the
289 declarations within a declaration context, all of those declarations
290 are immediately de-serialized. For large declaration contexts (e.g.,
291 the translation unit), this operation is expensive; however, large
292 declaration contexts are not traversed in normal compilation, since
293 such a traversal is unnecessary. However, it is common for the code
294 generator and semantic analysis to traverse declaration contexts for
295 structs, classes, unions, and enumerations, although those contexts
296 contain relatively few declarations in the common case.</li>
297</ul>
298
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]299<h3 id="stmt">Statements and Expressions</h3>
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]300
301<p>Statements and expressions are stored in the precompiled header in
302both the <a href="#types">types</a> and the <a
303 href="#decls">declarations</a> blocks, because every statement or
304expression will be associated with either a type or declaration. The
305actual statement and expression records are stored immediately
306following the declaration or type that owns the statement or
307expression. For example, the statement representing the body of a
308function will be stored directly following the declaration of the
309function.</p>
310
311<p>As with types and declarations, each statement and expression kind
312in Clang's abstract syntax tree (<code>ForStmt</code>,
313<code>CallExpr</code>, etc.) has a corresponding record type in the
314precompiled header, which contains the serialized representation of
Douglas Gregorfe3f2232009-06-03 18:26:16 +0000[diff] [blame]315that statement or expression. Each substatement or subexpression
316within an expression is stored as a separate record (which keeps most
317records to a fixed size). Within the precompiled header, the
318subexpressions of an expression are stored prior to the expression
319that owns those expression, using a form of <a
320href="http://en.wikipedia.org/wiki/Reverse_Polish_notation">Reverse
321Polish Notation</a>. For example, an expression <code>3 - 4 + 5</code>
322would be represented as follows:</p>
323
324<table border="1">
325 <tr><td><code>IntegerLiteral(3)</code></td></tr>
326 <tr><td><code>IntegerLiteral(4)</code></td></tr>
327 <tr><td><code>BinaryOperator(-)</code></td></tr>
328 <tr><td><code>IntegerLiteral(5)</code></td></tr>
329 <tr><td><code>BinaryOperator(+)</code></td></tr>
330 <tr><td>STOP</td></tr>
331</table>
332
333<p>When reading this representation, Clang evaluates each expression
334record it encounters, builds the appropriate abstract synax tree node,
335and then pushes that expression on to a stack. When a record contains <i>N</i>
336subexpressions--<code>BinaryOperator</code> has two of them--those
337expressions are popped from the top of the stack. The special STOP
338code indicates that we have reached the end of a serialized expression
339or statement; other expression or statement records may follow, but
340they are part of a different expression.</p>
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]341
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]342<h3 id="idtable">Identifier Table Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]343
344<p>The identifier table block contains an on-disk hash table that maps
345each identifier mentioned within the precompiled header to the
346serialized representation of the identifier's information (e.g, the
347<code>IdentifierInfo</code> structure). The serialized representation
348contains:</p>
349
350<ul>
351 <li>The actual identifier string.</li>
352 <li>Flags that describe whether this identifier is the name of a
353 built-in, a poisoned identifier, an extension token, or a
354 macro.</li>
355 <li>If the identifier names a macro, the offset of the macro
356 definition within the <a href="#preprocessor">preprocessor
357 block</a>.</li>
358 <li>If the identifier names one or more declarations visible from
359 translation unit scope, the <a href="#decls">declaration IDs</a> of these
360 declarations.</li>
361</ul>
362
363<p>When a precompiled header is loaded, the precompiled header
364mechanism introduces itself into the identifier table as an external
365lookup source. Thus, when the user program refers to an identifier
366that has not yet been seen, Clang will perform a lookup into the
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]367identifier table. If an identifier is found, its contents---macro definitions, flags, top-level declarations, etc.---will be deserialized, at which point the corresponding <code>IdentifierInfo</code> structure will have the same contents it would have after parsing the headers in the precompiled header.</p>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]368
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]369<p>Within the PCH file, the identifiers used to name declarations are represented with an integral value. A separate table provides a mapping from this integral value (the identifier ID) to the location within the on-disk
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]370hash table where that identifier is stored. This mapping is used when
371deserializing the name of a declaration, the identifier of a token, or
372any other construct in the PCH file that refers to a name.</p>
373
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame]374<h3 id="method-pool">Method Pool Block</h3>
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]375
376<p>The method pool block is represented as an on-disk hash table that
377serves two purposes: it provides a mapping from the names of
378Objective-C selectors to the set of Objective-C instance and class
379methods that have that particular selector (which is required for
380semantic analysis in Objective-C) and also stores all of the selectors
381used by entities within the precompiled header. The design of the
382method pool is similar to that of the <a href="#idtable">identifier
383table</a>: the first time a particular selector is formed during the
384compilation of the program, Clang will search in the on-disk hash
385table of selectors; if found, Clang will read the Objective-C methods
386associated with that selector into the appropriate front-end data
387structure (<code>Sema::InstanceMethodPool</code> and
388<code>Sema::FactoryMethodPool</code> for instance and class methods,
389respectively).</p>
390
391<p>As with identifiers, selectors are represented by numeric values
392within the PCH file. A separate index maps these numeric selector
393values to the offset of the selector within the on-disk hash table,
394and will be used when de-serializing an Objective-C method declaration
395(or other Objective-C construct) that refers to the selector.</p>
396
Douglas Gregor0084ead2009-06-03 21:41:31 +0000[diff] [blame^]397<h2 id="tendrils">Precompiled Header Integration Points</h2>
398
399<p>The "lazy" deserialization behavior of precompiled headers requires
400their integration into several completely different submodules of
401Clang. For example, lazily deserializing the declarations during name
402lookup requires that the name-lookup routines be able to query the
403precompiled header to find entities within the PCH file.</p>
404
405<p>For each Clang data structure that requires direct interaction with
406the precompiled header logic, there is an abstract class that provides
407the interface between the two modules. The <code>PCHReader</code>
408class, which handles the loading of a precompiled header, inherits
409from all of these abstract classes to provide lazy deserialization of
410Clang's data structures. <code>PCHReader</code> implements the
411following abstract classes:</p>
412
413<dl>
414 <dt><code>StatSysCallCache</code></dt>
415 <dd>This abstract interface is associated with the
416 <code>FileManager</code> class, and is used whenever the file
417 manager is going to perform a <code>stat()</code> system call.</dd>
418
419 <dt><code>ExternalSLocEntrySource</code></dt>
420 <dd>This abstract interface is associated with the
421 <code>SourceManager</code> class, and is used whenever the
422 <a href="#sourcemgr">source manager</a> needs to load the details
423 of a file, buffer, or macro instantiation.</dd>
424
425 <dt><code>IdentifierInfoLookup</code></dt>
426 <dd>This abstract interface is associated with the
427 <code>IdentifierTable</code> class, and is used whenever the
428 program source refers to an identifier that has not yet been seen.
429 In this case, the precompiled header implementation searches for
430 this identifier within its <a href="#idtable">identifier table</a>
431 to load any top-level declarations or macros associated with that
432 identifier.</dd>
433
434 <dt><code>ExternalASTSource</code></dt>
435 <dd>This abstract interface is associated with the
436 <code>ASTContext</code> class, and is used whenever the abstract
437 syntax tree nodes need to loaded from the precompiled header. It
438 provides the ability to de-serialize declarations and types
439 identified by their numeric values, read the bodies of functions
440 when required, and read the declarations stored within a
441 declaration context (either for iteration or for name lookup).</dd>
442
443 <dt><code>ExternalSemaSource</code></dt>
444 <dd>This abstract interface is associated with the <code>Sema</code>
445 class, and is used whenever semantic analysis needs to read
446 information from the <a href="#methodpool">global method
447 pool</a>.</dd>
448</dl>
449
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]450</div>
451
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]452</html>