Gitiles
Code Review Sign In
gerrit-public.fairphone.software / fp2-dev / platform / external / clang / fe3f223de8d670cdfdb1f362242c2b952e1c5f34 / . / docs / PCHInternals.html
blob: f1cfeb3673829fd188f5e91a3e4230f59420866c [file] [log] [blame]
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]1<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
2<html> <head>
3<title>Precompiled Headers (PCH)</title>
4</head>
5
6<body>
7
8<!--#include virtual="../menu.html.incl"-->
9
10<div id="content">
11
12<h1>Precompiled Headers</h1>
13
14 <p>This document describes the design and implementation of Clang's
15 precompiled headers (PCH). If you are interested in the end-user
16 view, please see the <a
17 href="UsersManual.html#precompiledheaders">User's Manual</a>.</p>
18
19<h2>Using precompiled headers with <tt>clang-cc</tt></h2>
20
21<p>The low-level Clang compiler, <tt>clang-cc</tt>, supports two command
22line options for generating and using PCH files.<p>
23
24<p>To generate PCH files using <tt>clang-cc</tt>, use the option
25<b><tt>-emit-pch</tt></b>:
26
27<pre> $ clang-cc test.h -emit-pch -o test.h.pch </pre>
28
29<p>This option is transparently used by <tt>clang</tt> when generating
30PCH files. The resulting PCH file contains the serialized form of the
31compiler's internal representation after it has completed parsing and
32semantic analysis. The PCH file can then be used as a prefix header
33with the <b><tt>-include-pch</tt></b> option:</p>
34
35<pre>
36 $ clang-cc -include-pch test.h.pch test.c -o test.s
37</pre>
38
39<h2>PCH Design Philosophy</h2>
40
41<p>Precompiled headers are meant to improve overall compile times for
42 projects, so the design of precompiled headers is entirely driven by
43 performance concerns. The use case for precompiled headers is
44 relatively simple: when there is a common set of headers that is
45 included in nearly every source file in the project, we
46 <i>precompile</i> that bundle of headers into a single precompiled
47 header (PCH file). Then, when compiling the source files in the
48 project, we load the PCH file first (as a prefix header), which acts
49 as a stand-in for that bundle of headers.</p>
50
51<p>A precompiled header implementation improves performance when:</p>
52<ul>
53 <li>Loading the PCH file is significantly faster than re-parsing the
54 bundle of headers stored within the PCH file. Thus, a precompiled
55 header design attempts to minimize the cost of reading the PCH
56 file. Ideally, this cost should not vary with the size of the
57 precompiled header file.</li>
58
59 <li>The cost of generating the PCH file initially is not so large
60 that it counters the per-source-file performance improvement due to
61 eliminating the need to parse the bundled headers in the first
62 place. This is particularly important on multi-core systems, because
63 PCH file generation serializes the build when all compilations
64 require the PCH file to be up-to-date.</li>
65</ul>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]66
67<p>Clang's precompiled headers are designed with a compact on-disk
68representation, which minimizes both PCH creation time and the time
69required to initially load the PCH file. The PCH file itself contains
70a serialized representation of Clang's abstract syntax trees and
71supporting data structures, stored using the same compressed bitstream
72as <a href="http://llvm.org/docs/BitCodeFormat.html">LLVM's bitcode
73file format</a>.</p>
74
75<p>Clang's precompiled headers are loaded "lazily" from disk. When a
76PCH file is initially loaded, Clang reads only a small amount of data
77from the PCH file to establish where certain important data structures
78are stored. The amount of data read in this initial load is
79independent of the size of the PCH file, such that a larger PCH file
80does not lead to longer PCH load times. The actual header data in the
81PCH file--macros, functions, variables, types, etc.--is loaded only
82when it is referenced from the user's code, at which point only that
83entity (and those entities it depends on) are deserialized from the
84PCH file. With this approach, the cost of using a precompiled header
85for a translation unit is proportional to the amount of code actually
86used from the header, rather than being proportional to the size of
87the header itself.</p> </body>
88
89<h2>Precompiled Header Contents</h2>
90
91<img src="PCHLayout.png" align="right" alt="Precompiled header layout">
92
93<p>Clang's precompiled headers are organized into several different
94blocks, each of which contains the serialized representation of a part
95of Clang's internal representation. Each of the blocks corresponds to
96either a block or a record within <a
97 href="http://llvm.org/docs/BitCodeFormat.html">LLVM's bitstream
98format</a>. The contents of each of these logical blocks are described
99below.</p>
100
101<h3 name="metadata">Metadata Block</h3>
102
103<p>The metadata block contains several records that provide
104information about how the precompiled header was built. This metadata
105is primarily used to validate the use of a precompiled header. For
Douglas Gregorfe3f2232009-06-03 18:26:16 +0000[diff] [blame^]106example, a precompiled header built for a 32-bit x86 target cannot be used
107when compiling for a 64-bit x86 target. The metadata block contains
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]108information about:</p>
109
110<dl>
111 <dt>Language options</dt>
112 <dd>Describes the particular language dialect used to compile the
113PCH file, including major options (e.g., Objective-C support) and more
114minor options (e.g., support for "//" comments). The contents of this
115record correspond to the <code>LangOptions</code> class.</dd>
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]116
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]117 <dt>Target architecture</dt>
118 <dd>The target triple that describes the architecture, platform, and
119ABI for which the PCH file was generated, e.g.,
120<code>i386-apple-darwin9</code>.</dd>
121
122 <dt>PCH version</dt>
123 <dd>The major and minor version numbers of the precompiled header
124format. Changes in the minor version number should not affect backward
125compatibility, while changes in the major version number imply that a
126newer compiler cannot read an older precompiled header (and
127vice-versa).</dd>
128
129 <dt>Original file name</dt>
130 <dd>The full path of the header that was used to generate the
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]131precompiled header.</dd>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]132
133 <dt>Predefines buffer</dt>
134 <dd>Although not explicitly stored as part of the metadata, the
135predefines buffer is used in the validation of the precompiled header.
136The predefines buffer itself contains code generated by the compiler
137to initialize the preprocessor state according to the current target,
138platform, and command-line options. For example, the predefines buffer
139will contain "<code>#define __STDC__ 1</code>" when we are compiling C
140without Microsoft extensions. The predefines buffer itself is stored
141within the <a href="#sourcemgr">source manager block</a>, but its
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]142contents are verified along with the rest of the metadata.</dd>
143
144</dl>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]145
146<h3 name="sourcemgr">Source Manager Block</h3>
147
148<p>The source manager block contains the serialized representation of
149Clang's <a
150 href="InternalsManual.html#SourceLocation">SourceManager</a> class,
151which handles the mapping from source locations (as represented in
152Clang's abstract syntax tree) into actual column/line positions within
153a source file or macro instantiation. The precompiled header's
154representation of the source manager also includes information about
155all of the headers that were (transitively) included when building the
156precompiled header.</p>
157
158<p>The bulk of the source manager block is dedicated to information
159about the various files, buffers, and macro instantiations into which
160a source location can refer. Each of these is referenced by a numeric
161"file ID", which is a unique number (allocated starting at 1) stored
162in the source location. Clang serializes the information for each kind
163of file ID, along with an index that maps file IDs to the position
164within the PCH file where the information about that file ID is
165stored. The data associated with a file ID is loaded only when
166required by the front end, e.g., to emit a diagnostic that includes a
167macro instantiation history inside the header itself.</p>
168
169<p>The source manager block also contains information about all of the
170headers that were included when building the precompiled header. This
171includes information about the controlling macro for the header (e.g.,
172when the preprocessor identified that the contents of the header
173dependent on a macro like <code>LLVM_CLANG_SOURCEMANAGER_H</code>)
174along with a cached version of the results of the <code>stat()</code>
175system calls performed when building the precompiled header. The
176latter is particularly useful in reducing system time when searching
177for include files.</p>
178
179<h3 name="preprocessor">Preprocessor Block</h3>
180
181<p>The preprocessor block contains the serialized representation of
182the preprocessor. Specifically, it contains all of the macros that
183have been defined by the end of the header used to build the
184precompiled header, along with the token sequences that comprise each
185macro. The macro definitions are only read from the PCH file when the
186name of the macro first occurs in the program. This lazy loading of
187macro definitions is trigged by lookups into the <a
188 href="#idtable">identifier table</a>.</p>
189
190<h3 name="types">Types Block</h3>
191
192<p>The types block contains the serialized representation of all of
193the types referenced in the translation unit. Each Clang type node
194(<code>PointerType</code>, <code>FunctionProtoType</code>, etc.) has a
195corresponding record type in the PCH file. When types are deserialized
196from the precompiled header, the data within the record is used to
197reconstruct the appropriate type node using the AST context.</p>
198
199<p>Each type has a unique type ID, which is an integer that uniquely
200identifies that type. Type ID 0 represents the NULL type, type IDs
201less than <code>NUM_PREDEF_TYPE_IDS</code> represent predefined types
202(<code>void</code>, <code>float</code>, etc.), while other
203"user-defined" type IDs are assigned consecutively from
204<code>NUM_PREDEF_TYPE_IDS</code> upward as the types are encountered.
205The PCH file has an associated mapping from the user-defined types
206block to the location within the types block where the serialized
207representation of that type resides, enabling lazy deserialization of
208types. When a type is referenced from within the PCH file, that
209reference is encoded using the type ID shifted left by 3 bits. The
210lower three bits are used to represent the <code>const</code>,
211<code>volatile</code>, and <code>restrict</code> qualifiers, as in
212Clang's <a
213 href="http://clang.llvm.org/docs/InternalsManual.html#Type">QualType</a>
214class.</p>
215
216<h3 name="decls">Declarations Block</h3>
217
218<p>The declarations block contains the serialized representation of
219all of the declarations referenced in the translation unit. Each Clang
220declaration node (<code>VarDecl</code>, <code>FunctionDecl</code>,
221etc.) has a corresponding record type in the PCH file. When
222declarations are deserialized from the precompiled header, the data
223within the record is used to build and populate a new instance of the
224corresponding <code>Decl</code> node. As with types, each declaration
225node has a numeric ID that is used to refer to that declaration within
226the PCH file. In addition, a lookup table provides a mapping from that
227numeric ID to the offset within the precompiled header where that
228declaration is described.</p>
229
230<p>Declarations in Clang's abstract syntax trees are stored
231hierarchically. At the top of the hierarchy is the translation unit
232(<code>TranslationUnitDecl</code>), which contains all of the
233declarations in the translation unit. These declarations---such as
234functions or struct types---may also contain other declarations inside
235them, and so on. Within Clang, each declaration is stored within a <a
236href="http://clang.llvm.org/docs/InternalsManual.html#DeclContext">declaration
237context</a>, as represented by the <code>DeclContext</code> class.
238Declaration contexts provide the mechanism to perform name lookup
239within a given declaration (e.g., find the member named <code>x</code>
240in a structure) and iterate over the declarations stored within a
241context (e.g., iterate over all of the fields of a structure for
242structure layout).</p>
243
244<p>In Clang's precompiled header format, deserializing a declaration
245that is a <code>DeclContext</code> is a separate operation from
246deserializing all of the declarations stored within that declaration
247context. Therefore, Clang will deserialize the translation unit
248declaration without deserializing the declarations within that
249translation unit. When required, the declarations stored within a
250declaration context will be serialized. There are two representations
251of the declarations within a declaration context, which correspond to
252the name-lookup and iteration behavior described above:</p>
253
254<ul>
255 <li>When the front end performs name lookup to find a name
256 <code>x</code> within a given declaration context (for example,
257 during semantic analysis of the expression <code>p-&gt;x</code>,
258 where <code>p</code>'s type is defined in the precompiled header),
259 Clang deserializes a hash table mapping from the names within that
260 declaration context to the declaration IDs that represent each
261 visible declaration with that name. The entire hash table is
262 deserialized at this point (into the <code>llvm::DenseMap</code>
263 stored within each <code>DeclContext</code> object), but the actual
264 declarations are not yet deserialized. In a second step, those
265 declarations with the name <code>x</code> will be deserialized and
266 will be used as the result of name lookup.</li>
267
268 <li>When the front end performs iteration over all of the
269 declarations within a declaration context, all of those declarations
270 are immediately de-serialized. For large declaration contexts (e.g.,
271 the translation unit), this operation is expensive; however, large
272 declaration contexts are not traversed in normal compilation, since
273 such a traversal is unnecessary. However, it is common for the code
274 generator and semantic analysis to traverse declaration contexts for
275 structs, classes, unions, and enumerations, although those contexts
276 contain relatively few declarations in the common case.</li>
277</ul>
278
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]279<h3 name"stmt">Statements and Expressions</h3>
280
281<p>Statements and expressions are stored in the precompiled header in
282both the <a href="#types">types</a> and the <a
283 href="#decls">declarations</a> blocks, because every statement or
284expression will be associated with either a type or declaration. The
285actual statement and expression records are stored immediately
286following the declaration or type that owns the statement or
287expression. For example, the statement representing the body of a
288function will be stored directly following the declaration of the
289function.</p>
290
291<p>As with types and declarations, each statement and expression kind
292in Clang's abstract syntax tree (<code>ForStmt</code>,
293<code>CallExpr</code>, etc.) has a corresponding record type in the
294precompiled header, which contains the serialized representation of
Douglas Gregorfe3f2232009-06-03 18:26:16 +0000[diff] [blame^]295that statement or expression. Each substatement or subexpression
296within an expression is stored as a separate record (which keeps most
297records to a fixed size). Within the precompiled header, the
298subexpressions of an expression are stored prior to the expression
299that owns those expression, using a form of <a
300href="http://en.wikipedia.org/wiki/Reverse_Polish_notation">Reverse
301Polish Notation</a>. For example, an expression <code>3 - 4 + 5</code>
302would be represented as follows:</p>
303
304<table border="1">
305 <tr><td><code>IntegerLiteral(3)</code></td></tr>
306 <tr><td><code>IntegerLiteral(4)</code></td></tr>
307 <tr><td><code>BinaryOperator(-)</code></td></tr>
308 <tr><td><code>IntegerLiteral(5)</code></td></tr>
309 <tr><td><code>BinaryOperator(+)</code></td></tr>
310 <tr><td>STOP</td></tr>
311</table>
312
313<p>When reading this representation, Clang evaluates each expression
314record it encounters, builds the appropriate abstract synax tree node,
315and then pushes that expression on to a stack. When a record contains <i>N</i>
316subexpressions--<code>BinaryOperator</code> has two of them--those
317expressions are popped from the top of the stack. The special STOP
318code indicates that we have reached the end of a serialized expression
319or statement; other expression or statement records may follow, but
320they are part of a different expression.</p>
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]321
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]322<h3 name="idtable">Identifier Table Block</h3>
323
324<p>The identifier table block contains an on-disk hash table that maps
325each identifier mentioned within the precompiled header to the
326serialized representation of the identifier's information (e.g, the
327<code>IdentifierInfo</code> structure). The serialized representation
328contains:</p>
329
330<ul>
331 <li>The actual identifier string.</li>
332 <li>Flags that describe whether this identifier is the name of a
333 built-in, a poisoned identifier, an extension token, or a
334 macro.</li>
335 <li>If the identifier names a macro, the offset of the macro
336 definition within the <a href="#preprocessor">preprocessor
337 block</a>.</li>
338 <li>If the identifier names one or more declarations visible from
339 translation unit scope, the <a href="#decls">declaration IDs</a> of these
340 declarations.</li>
341</ul>
342
343<p>When a precompiled header is loaded, the precompiled header
344mechanism introduces itself into the identifier table as an external
345lookup source. Thus, when the user program refers to an identifier
346that has not yet been seen, Clang will perform a lookup into the
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]347identifier table. If an identifier is found, its contents---macro definitions, flags, top-level declarations, etc.---will be deserialized, at which point the corresponding <code>IdentifierInfo</code> structure will have the same contents it would have after parsing the headers in the precompiled header.</p>
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]348
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]349<p>Within the PCH file, the identifiers used to name declarations are represented with an integral value. A separate table provides a mapping from this integral value (the identifier ID) to the location within the on-disk
Douglas Gregor2cc390e2009-06-02 22:08:07 +0000[diff] [blame]350hash table where that identifier is stored. This mapping is used when
351deserializing the name of a declaration, the identifier of a token, or
352any other construct in the PCH file that refers to a name.</p>
353
Douglas Gregor5accbb92009-06-03 16:06:22 +0000[diff] [blame]354<h3 name="method-pool">Method Pool Block</h3>
355
356<p>The method pool block is represented as an on-disk hash table that
357serves two purposes: it provides a mapping from the names of
358Objective-C selectors to the set of Objective-C instance and class
359methods that have that particular selector (which is required for
360semantic analysis in Objective-C) and also stores all of the selectors
361used by entities within the precompiled header. The design of the
362method pool is similar to that of the <a href="#idtable">identifier
363table</a>: the first time a particular selector is formed during the
364compilation of the program, Clang will search in the on-disk hash
365table of selectors; if found, Clang will read the Objective-C methods
366associated with that selector into the appropriate front-end data
367structure (<code>Sema::InstanceMethodPool</code> and
368<code>Sema::FactoryMethodPool</code> for instance and class methods,
369respectively).</p>
370
371<p>As with identifiers, selectors are represented by numeric values
372within the PCH file. A separate index maps these numeric selector
373values to the offset of the selector within the on-disk hash table,
374and will be used when de-serializing an Objective-C method declaration
375(or other Objective-C construct) that refers to the selector.</p>
376
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]377</div>
378
Douglas Gregor32110df2009-05-20 00:16:32 +0000[diff] [blame]379</html>