Gitiles
Code Review Sign In
gerrit-public.fairphone.software / fp2-dev / platform / external / clang / 923cb23eabaf800a647dd9466ed68fe5b83c3250 / . / docs / PCHInternals.html
blob: 7c0c1403fabc6f4cf8296e116d99841050db97da [file] [log] [blame]
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]1<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
2<html> <head>
3<title>Precompiled Headers (PCH)</title>
4</head>
5
6<body>
7
8<!--#include virtual="../menu.html.incl"-->
9
10<div id="content">
11
12<h1>Precompiled Headers</h1>
13
14 <p>This document describes the design and implementation of Clang's
15 precompiled headers (PCH). If you are interested in the end-user
16 view, please see the <a
17 href="UsersManual.html#precompiledheaders">User's Manual</a>.</p>
18
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]19 <p><b>Table of Contents</b></p>
20 <ul>
21 <li><a href="#usage">Using Precompiled Headers with
22 <tt>clang-cc</tt></a></li>
23 <li><a href="#philosophy">Design Philosophy</a></li>
24 <li><a href="#contents">Precompiled Header Contents</a>
25 <ul>
26 <li><a href="#metadata">Metadata Block</a></li>
27 <li><a href="#sourcemgr">Source Manager Block</a></li>
28 <li><a href="#preprocessor">Preprocessor Block</a></li>
29 <li><a href="#types">Types Block</a></li>
30 <li><a href="#decls">Declarations Block</a></li>
31 <li><a href="#stmt">Statements and Expressions</a></li>
32 <li><a href="#idtable">Identifier Table Block</a></li>
33 <li><a href="#method-pool">Method Pool Block</a></li>
34 </ul>
35 </li>
36 </ul>
37
38<h2 id="usage">Using Precompiled Headers with <tt>clang-cc</tt></h2>
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]39
40<p>The low-level Clang compiler, <tt>clang-cc</tt>, supports two command
41line options for generating and using PCH files.<p>
42
43<p>To generate PCH files using <tt>clang-cc</tt>, use the option
44<b><tt>-emit-pch</tt></b>:
45
46<pre> $ clang-cc test.h -emit-pch -o test.h.pch </pre>
47
48<p>This option is transparently used by <tt>clang</tt> when generating
49PCH files. The resulting PCH file contains the serialized form of the
50compiler's internal representation after it has completed parsing and
51semantic analysis. The PCH file can then be used as a prefix header
52with the <b><tt>-include-pch</tt></b> option:</p>
53
54<pre>
55 $ clang-cc -include-pch test.h.pch test.c -o test.s
56</pre>
57
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]58<h2 id="philosophy">Design Philosophy</h2>
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]59
60<p>Precompiled headers are meant to improve overall compile times for
61 projects, so the design of precompiled headers is entirely driven by
62 performance concerns. The use case for precompiled headers is
63 relatively simple: when there is a common set of headers that is
64 included in nearly every source file in the project, we
65 <i>precompile</i> that bundle of headers into a single precompiled
66 header (PCH file). Then, when compiling the source files in the
67 project, we load the PCH file first (as a prefix header), which acts
68 as a stand-in for that bundle of headers.</p>
69
70<p>A precompiled header implementation improves performance when:</p>
71<ul>
72 <li>Loading the PCH file is significantly faster than re-parsing the
73 bundle of headers stored within the PCH file. Thus, a precompiled
74 header design attempts to minimize the cost of reading the PCH
75 file. Ideally, this cost should not vary with the size of the
76 precompiled header file.</li>
77
78 <li>The cost of generating the PCH file initially is not so large
79 that it counters the per-source-file performance improvement due to
80 eliminating the need to parse the bundled headers in the first
81 place. This is particularly important on multi-core systems, because
82 PCH file generation serializes the build when all compilations
83 require the PCH file to be up-to-date.</li>
84</ul>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]85
86<p>Clang's precompiled headers are designed with a compact on-disk
87representation, which minimizes both PCH creation time and the time
88required to initially load the PCH file. The PCH file itself contains
89a serialized representation of Clang's abstract syntax trees and
90supporting data structures, stored using the same compressed bitstream
91as <a href="http://llvm.org/docs/BitCodeFormat.html">LLVM's bitcode
92file format</a>.</p>
93
94<p>Clang's precompiled headers are loaded "lazily" from disk. When a
95PCH file is initially loaded, Clang reads only a small amount of data
96from the PCH file to establish where certain important data structures
97are stored. The amount of data read in this initial load is
98independent of the size of the PCH file, such that a larger PCH file
99does not lead to longer PCH load times. The actual header data in the
100PCH file--macros, functions, variables, types, etc.--is loaded only
101when it is referenced from the user's code, at which point only that
102entity (and those entities it depends on) are deserialized from the
103PCH file. With this approach, the cost of using a precompiled header
104for a translation unit is proportional to the amount of code actually
105used from the header, rather than being proportional to the size of
106the header itself.</p> </body>
107
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]108<h2 id="contents">Precompiled Header Contents</h2>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]109
110<img src="PCHLayout.png" align="right" alt="Precompiled header layout">
111
112<p>Clang's precompiled headers are organized into several different
113blocks, each of which contains the serialized representation of a part
114of Clang's internal representation. Each of the blocks corresponds to
115either a block or a record within <a
116 href="http://llvm.org/docs/BitCodeFormat.html">LLVM's bitstream
117format</a>. The contents of each of these logical blocks are described
118below.</p>
119
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]120<h3 id="metadata">Metadata Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]121
122<p>The metadata block contains several records that provide
123information about how the precompiled header was built. This metadata
124is primarily used to validate the use of a precompiled header. For
Douglas Gregorfe3f2232009-06-03 18:26:16 +0000[diff] [blame]125example, a precompiled header built for a 32-bit x86 target cannot be used
126when compiling for a 64-bit x86 target. The metadata block contains
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]127information about:</p>
128
129<dl>
130 <dt>Language options</dt>
131 <dd>Describes the particular language dialect used to compile the
132PCH file, including major options (e.g., Objective-C support) and more
133minor options (e.g., support for "//" comments). The contents of this
134record correspond to the <code>LangOptions</code> class.</dd>
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]135
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]136 <dt>Target architecture</dt>
137 <dd>The target triple that describes the architecture, platform, and
138ABI for which the PCH file was generated, e.g.,
139<code>i386-apple-darwin9</code>.</dd>
140
141 <dt>PCH version</dt>
142 <dd>The major and minor version numbers of the precompiled header
143format. Changes in the minor version number should not affect backward
144compatibility, while changes in the major version number imply that a
145newer compiler cannot read an older precompiled header (and
146vice-versa).</dd>
147
148 <dt>Original file name</dt>
149 <dd>The full path of the header that was used to generate the
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]150precompiled header.</dd>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]151
152 <dt>Predefines buffer</dt>
153 <dd>Although not explicitly stored as part of the metadata, the
154predefines buffer is used in the validation of the precompiled header.
155The predefines buffer itself contains code generated by the compiler
156to initialize the preprocessor state according to the current target,
157platform, and command-line options. For example, the predefines buffer
158will contain "<code>#define __STDC__ 1</code>" when we are compiling C
159without Microsoft extensions. The predefines buffer itself is stored
160within the <a href="#sourcemgr">source manager block</a>, but its
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]161contents are verified along with the rest of the metadata.</dd>
162
163</dl>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]164
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]165<h3 id="sourcemgr">Source Manager Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]166
167<p>The source manager block contains the serialized representation of
168Clang's <a
169 href="InternalsManual.html#SourceLocation">SourceManager</a> class,
170which handles the mapping from source locations (as represented in
171Clang's abstract syntax tree) into actual column/line positions within
172a source file or macro instantiation. The precompiled header's
173representation of the source manager also includes information about
174all of the headers that were (transitively) included when building the
175precompiled header.</p>
176
177<p>The bulk of the source manager block is dedicated to information
178about the various files, buffers, and macro instantiations into which
179a source location can refer. Each of these is referenced by a numeric
180"file ID", which is a unique number (allocated starting at 1) stored
181in the source location. Clang serializes the information for each kind
182of file ID, along with an index that maps file IDs to the position
183within the PCH file where the information about that file ID is
184stored. The data associated with a file ID is loaded only when
185required by the front end, e.g., to emit a diagnostic that includes a
186macro instantiation history inside the header itself.</p>
187
188<p>The source manager block also contains information about all of the
189headers that were included when building the precompiled header. This
190includes information about the controlling macro for the header (e.g.,
191when the preprocessor identified that the contents of the header
192dependent on a macro like <code>LLVM_CLANG_SOURCEMANAGER_H</code>)
193along with a cached version of the results of the <code>stat()</code>
194system calls performed when building the precompiled header. The
195latter is particularly useful in reducing system time when searching
196for include files.</p>
197
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]198<h3 id="preprocessor">Preprocessor Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]199
200<p>The preprocessor block contains the serialized representation of
201the preprocessor. Specifically, it contains all of the macros that
202have been defined by the end of the header used to build the
203precompiled header, along with the token sequences that comprise each
204macro. The macro definitions are only read from the PCH file when the
205name of the macro first occurs in the program. This lazy loading of
206macro definitions is trigged by lookups into the <a
207 href="#idtable">identifier table</a>.</p>
208
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]209<h3 id="types">Types Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]210
211<p>The types block contains the serialized representation of all of
212the types referenced in the translation unit. Each Clang type node
213(<code>PointerType</code>, <code>FunctionProtoType</code>, etc.) has a
214corresponding record type in the PCH file. When types are deserialized
215from the precompiled header, the data within the record is used to
216reconstruct the appropriate type node using the AST context.</p>
217
218<p>Each type has a unique type ID, which is an integer that uniquely
219identifies that type. Type ID 0 represents the NULL type, type IDs
220less than <code>NUM_PREDEF_TYPE_IDS</code> represent predefined types
221(<code>void</code>, <code>float</code>, etc.), while other
222"user-defined" type IDs are assigned consecutively from
223<code>NUM_PREDEF_TYPE_IDS</code> upward as the types are encountered.
224The PCH file has an associated mapping from the user-defined types
225block to the location within the types block where the serialized
226representation of that type resides, enabling lazy deserialization of
227types. When a type is referenced from within the PCH file, that
228reference is encoded using the type ID shifted left by 3 bits. The
229lower three bits are used to represent the <code>const</code>,
230<code>volatile</code>, and <code>restrict</code> qualifiers, as in
231Clang's <a
232 href="http://clang.llvm.org/docs/InternalsManual.html#Type">QualType</a>
233class.</p>
234
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]235<h3 id="decls">Declarations Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]236
237<p>The declarations block contains the serialized representation of
238all of the declarations referenced in the translation unit. Each Clang
239declaration node (<code>VarDecl</code>, <code>FunctionDecl</code>,
240etc.) has a corresponding record type in the PCH file. When
241declarations are deserialized from the precompiled header, the data
242within the record is used to build and populate a new instance of the
243corresponding <code>Decl</code> node. As with types, each declaration
244node has a numeric ID that is used to refer to that declaration within
245the PCH file. In addition, a lookup table provides a mapping from that
246numeric ID to the offset within the precompiled header where that
247declaration is described.</p>
248
249<p>Declarations in Clang's abstract syntax trees are stored
250hierarchically. At the top of the hierarchy is the translation unit
251(<code>TranslationUnitDecl</code>), which contains all of the
252declarations in the translation unit. These declarations---such as
253functions or struct types---may also contain other declarations inside
254them, and so on. Within Clang, each declaration is stored within a <a
255href="http://clang.llvm.org/docs/InternalsManual.html#DeclContext">declaration
256context</a>, as represented by the <code>DeclContext</code> class.
257Declaration contexts provide the mechanism to perform name lookup
258within a given declaration (e.g., find the member named <code>x</code>
259in a structure) and iterate over the declarations stored within a
260context (e.g., iterate over all of the fields of a structure for
261structure layout).</p>
262
263<p>In Clang's precompiled header format, deserializing a declaration
264that is a <code>DeclContext</code> is a separate operation from
265deserializing all of the declarations stored within that declaration
266context. Therefore, Clang will deserialize the translation unit
267declaration without deserializing the declarations within that
268translation unit. When required, the declarations stored within a
269declaration context will be serialized. There are two representations
270of the declarations within a declaration context, which correspond to
271the name-lookup and iteration behavior described above:</p>
272
273<ul>
274 <li>When the front end performs name lookup to find a name
275 <code>x</code> within a given declaration context (for example,
276 during semantic analysis of the expression <code>p-&gt;x</code>,
277 where <code>p</code>'s type is defined in the precompiled header),
278 Clang deserializes a hash table mapping from the names within that
279 declaration context to the declaration IDs that represent each
280 visible declaration with that name. The entire hash table is
281 deserialized at this point (into the <code>llvm::DenseMap</code>
282 stored within each <code>DeclContext</code> object), but the actual
283 declarations are not yet deserialized. In a second step, those
284 declarations with the name <code>x</code> will be deserialized and
285 will be used as the result of name lookup.</li>
286
287 <li>When the front end performs iteration over all of the
288 declarations within a declaration context, all of those declarations
289 are immediately de-serialized. For large declaration contexts (e.g.,
290 the translation unit), this operation is expensive; however, large
291 declaration contexts are not traversed in normal compilation, since
292 such a traversal is unnecessary. However, it is common for the code
293 generator and semantic analysis to traverse declaration contexts for
294 structs, classes, unions, and enumerations, although those contexts
295 contain relatively few declarations in the common case.</li>
296</ul>
297
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]298<h3 id="stmt">Statements and Expressions</h3>
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]299
300<p>Statements and expressions are stored in the precompiled header in
301both the <a href="#types">types</a> and the <a
302 href="#decls">declarations</a> blocks, because every statement or
303expression will be associated with either a type or declaration. The
304actual statement and expression records are stored immediately
305following the declaration or type that owns the statement or
306expression. For example, the statement representing the body of a
307function will be stored directly following the declaration of the
308function.</p>
309
310<p>As with types and declarations, each statement and expression kind
311in Clang's abstract syntax tree (<code>ForStmt</code>,
312<code>CallExpr</code>, etc.) has a corresponding record type in the
313precompiled header, which contains the serialized representation of
Douglas Gregorfe3f2232009-06-03 18:26:16 +0000[diff] [blame]314that statement or expression. Each substatement or subexpression
315within an expression is stored as a separate record (which keeps most
316records to a fixed size). Within the precompiled header, the
317subexpressions of an expression are stored prior to the expression
318that owns those expression, using a form of <a
319href="http://en.wikipedia.org/wiki/Reverse_Polish_notation">Reverse
320Polish Notation</a>. For example, an expression <code>3 - 4 + 5</code>
321would be represented as follows:</p>
322
323<table border="1">
324 <tr><td><code>IntegerLiteral(3)</code></td></tr>
325 <tr><td><code>IntegerLiteral(4)</code></td></tr>
326 <tr><td><code>BinaryOperator(-)</code></td></tr>
327 <tr><td><code>IntegerLiteral(5)</code></td></tr>
328 <tr><td><code>BinaryOperator(+)</code></td></tr>
329 <tr><td>STOP</td></tr>
330</table>
331
332<p>When reading this representation, Clang evaluates each expression
333record it encounters, builds the appropriate abstract synax tree node,
334and then pushes that expression on to a stack. When a record contains <i>N</i>
335subexpressions--<code>BinaryOperator</code> has two of them--those
336expressions are popped from the top of the stack. The special STOP
337code indicates that we have reached the end of a serialized expression
338or statement; other expression or statement records may follow, but
339they are part of a different expression.</p>
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]340
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]341<h3 id="idtable">Identifier Table Block</h3>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]342
343<p>The identifier table block contains an on-disk hash table that maps
344each identifier mentioned within the precompiled header to the
345serialized representation of the identifier's information (e.g, the
346<code>IdentifierInfo</code> structure). The serialized representation
347contains:</p>
348
349<ul>
350 <li>The actual identifier string.</li>
351 <li>Flags that describe whether this identifier is the name of a
352 built-in, a poisoned identifier, an extension token, or a
353 macro.</li>
354 <li>If the identifier names a macro, the offset of the macro
355 definition within the <a href="#preprocessor">preprocessor
356 block</a>.</li>
357 <li>If the identifier names one or more declarations visible from
358 translation unit scope, the <a href="#decls">declaration IDs</a> of these
359 declarations.</li>
360</ul>
361
362<p>When a precompiled header is loaded, the precompiled header
363mechanism introduces itself into the identifier table as an external
364lookup source. Thus, when the user program refers to an identifier
365that has not yet been seen, Clang will perform a lookup into the
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]366identifier table. If an identifier is found, its contents---macro definitions, flags, top-level declarations, etc.---will be deserialized, at which point the corresponding <code>IdentifierInfo</code> structure will have the same contents it would have after parsing the headers in the precompiled header.</p>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]367
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]368<p>Within the PCH file, the identifiers used to name declarations are represented with an integral value. A separate table provides a mapping from this integral value (the identifier ID) to the location within the on-disk
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]369hash table where that identifier is stored. This mapping is used when
370deserializing the name of a declaration, the identifier of a token, or
371any other construct in the PCH file that refers to a name.</p>
372
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]373<h3 id="method-pool">Method Pool Block</h3>
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]374
375<p>The method pool block is represented as an on-disk hash table that
376serves two purposes: it provides a mapping from the names of
377Objective-C selectors to the set of Objective-C instance and class
378methods that have that particular selector (which is required for
379semantic analysis in Objective-C) and also stores all of the selectors
380used by entities within the precompiled header. The design of the
381method pool is similar to that of the <a href="#idtable">identifier
382table</a>: the first time a particular selector is formed during the
383compilation of the program, Clang will search in the on-disk hash
384table of selectors; if found, Clang will read the Objective-C methods
385associated with that selector into the appropriate front-end data
386structure (<code>Sema::InstanceMethodPool</code> and
387<code>Sema::FactoryMethodPool</code> for instance and class methods,
388respectively).</p>
389
390<p>As with identifiers, selectors are represented by numeric values
391within the PCH file. A separate index maps these numeric selector
392values to the offset of the selector within the on-disk hash table,
393and will be used when de-serializing an Objective-C method declaration
394(or other Objective-C construct) that refers to the selector.</p>
395
Douglas Gregor923cb232009-06-03 18:35:59 +0000[diff] [blame^]396<h2 id="tendrils"></h2>
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]397</div>
398
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]399</html>