blob: 832d3b0a97884886dbbcc74f22c992f4b7e48b31 [file] [log] [blame]
Ted Kremenek797a2472009-04-08 05:07:30 +00001<html>
2 <head>
3 <title>Pretokenized Headers (PTH)</title>
4 <link type="text/css" rel="stylesheet" href="../menu.css" />
5 <link type="text/css" rel="stylesheet" href="../content.css" />
6 <style type="text/css">
7 td {
8 vertical-align: top;
9 }
10 </style>
11</head>
12<body>
13
14<!--#include virtual="../menu.html.incl"-->
15
16<div id="content">
Ted Kremenek797a2472009-04-08 05:07:30 +000017
Chris Lattner5c3074f2009-04-20 04:37:38 +000018<h1>Pretokenized Headers (PTH)</h1>
Ted Kremenek797a2472009-04-08 05:07:30 +000019
Chris Lattner5c3074f2009-04-20 04:37:38 +000020<p>This document first describes the low-level
21interface for using PTH and then briefly elaborates on its design and
22implementation. If you are interested in the end-user view, please see the
23<a href="UsersManual.html#precompiledheaders">User's Manual</a>.</p>
Ted Kremenek797a2472009-04-08 05:07:30 +000024
25
Ted Kremenekb7fd6b02009-04-09 18:17:39 +000026<h2>Using Pretokenized Headers with <tt>clang-cc</tt> (Low-level Interface)</h2>
Ted Kremenek797a2472009-04-08 05:07:30 +000027
Chris Lattnercdf59da2009-04-08 05:50:25 +000028<p>The low-level Clang compiler tool, <tt>clang-cc</tt>, supports three command
29line options for generating and using PTH files.<p>
Ted Kremenek797a2472009-04-08 05:07:30 +000030
Ted Kremenekb7fd6b02009-04-09 18:17:39 +000031<p>To generate PTH files using <tt>clang-cc</tt>, use the option
32<b><tt>-emit-pth</tt></b>:
33
34<pre> $ clang-cc test.h -emit-pth -o test.h.pth </pre>
Ted Kremenek797a2472009-04-08 05:07:30 +000035
36<p>This option is transparently used by <tt>clang</tt> when generating PTH
Ted Kremenekb7fd6b02009-04-09 18:17:39 +000037files. Similarly, PTH files can be used as prefix headers using the
38<b><tt>-include-pth</tt></b> option:</p>
Ted Kremenek797a2472009-04-08 05:07:30 +000039
40<pre>
41 $ clang-cc -include-pth test.h.pth test.c -o test.s
42</pre>
43
44<p>Alternatively, Clang's PTH files can be used as a raw &quot;token-cache&quot;
45(or &quot;content&quot; cache) of the source included by the original header
46file. This means that the contents of the PTH file are searched as substitutes
47for <em>any</em> source files that are used by <tt>clang-cc</tt> to process a
Ted Kremenekb7fd6b02009-04-09 18:17:39 +000048source file. This is done by specifying the <b><tt>-token-cache</tt></b>
49option:</p>
Ted Kremenek797a2472009-04-08 05:07:30 +000050
51<pre>
52 $ cat test.h
Chris Lattner0a069992009-04-08 06:00:32 +000053 #include &lt;stdio.h&gt;
Ted Kremenek797a2472009-04-08 05:07:30 +000054 $ clang-cc -emit-pth test.h -o test.h.pth
55 $ cat test.c
56 #include "test.h"
57 $ clang-cc test.c -o test -token-cache test.h.pth
58</pre>
59
60<p>In this example the contents of <tt>stdio.h</tt> (and the files it includes)
61will be retrieved from <tt>test.h.pth</tt>, as the PTH file is being used in
62this case as a raw cache of the contents of <tt>test.h</tt>. This is a low-level
63interface used to both implement the high-level PTH interface as well as to
64provide alternative means to use PTH-style caching.</p>
65
66<h2>PTH Design and Implementation</h2>
67
68<p>Unlike GCC's precompiled headers, which cache the full ASTs and preprocessor
69state of a header file, Clang's pretokenized header files mainly cache the raw
70lexer <em>tokens</em> that are needed to segment the stream of characters in a
71source file into keywords, identifiers, and operators. Consequently, PTH serves
72to mainly directly speed up the lexing and preprocessing of a source file, while
73parsing and type-checking must be completely redone every time a PTH file is
74used.</p>
75
76<h3>Basic Design Tradeoffs</h3>
77
78<p>In the long term there are plans to provide an alternate PCH implementation
79for Clang that also caches the work for parsing and type checking the contents
80of header files. The current implementation of PCH in Clang as pretokenized
81header files was motivated by the following factors:<p>
82
83<ul>
Ted Kremenek07f08d22009-04-09 18:03:21 +000084
Ted Kremenek5890c632009-04-09 18:22:40 +000085<li><p><b>Language independence</b>: PTH files work with any language that
Ted Kremenek07f08d22009-04-09 18:03:21 +000086Clang's lexer can handle, including C, Objective-C, and (in the early stages)
87C++. This means development on language features at the parsing level or above
88(which is basically almost all interesting pieces) does not require PTH to be
89modified.</p></li>
Ted Kremenek797a2472009-04-08 05:07:30 +000090
Ted Kremenek5890c632009-04-09 18:22:40 +000091<li><b>Simple design</b>: Relatively speaking, PTH has a simple design and
Ted Kremenek797a2472009-04-08 05:07:30 +000092implementation, making it easy to test. Further, because the machinery for PTH
93resides at the lower-levels of the Clang library stack it is fairly
94straightforward to profile and optimize.</li>
95</ul>
96
97<p>Further, compared to GCC's PCH implementation (which is the dominate
98precompiled header file implementation that Clang can be directly compared
99against) the PTH design in Clang yields several attractive features:</p>
100
101<ul>
102
Ted Kremenek5890c632009-04-09 18:22:40 +0000103<li><p><b>Architecture independence</b>: In contrast to GCC's PCH files (and
Ted Kremenek797a2472009-04-08 05:07:30 +0000104those of several other compilers), Clang's PTH files are architecture
105independent, requiring only a single PTH file when building an program for
106multiple architectures.</p>
107
108<p>For example, on Mac OS X one may wish to
109compile a &quot;universal binary&quot; that runs on PowerPC, 32-bit Intel
110(i386), and 64-bit Intel architectures. In contrast, GCC requires a PCH file for
111each architecture, as the definitions of types in the AST are
112architecture-specific. Since a Clang PTH file essentially represents a lexical
113cache of header files, a single PTH file can be safely used when compiling for
114multiple architectures. This can also reduce compile times because only a single
115PTH file needs to be generated during a build instead of several.</p></li>
116
Ted Kremenek5890c632009-04-09 18:22:40 +0000117<li><p><b>Reduced memory pressure</b>: Similar to GCC,
Ted Kremenek797a2472009-04-08 05:07:30 +0000118Clang reads PTH files via the use of memory mapping (i.e., <tt>mmap</tt>).
119Clang, however, memory maps PTH files as read-only, meaning that multiple
120invocations of <tt>clang-cc</tt> can share the same pages in memory from a
121memory-mapped PTH file. In comparison, GCC also memory maps its PCH files but
122also modifies those pages in memory, incurring the copy-on-write costs. The
123read-only nature of PTH can greatly reduce memory pressure for builds involving
124multiple cores, thus improving overall scalability.</p></li>
125
Ted Kremenek5890c632009-04-09 18:22:40 +0000126<li><p><b>Fast generation</b>: PTH files can be generated in a small fraction
Ted Kremenek07f08d22009-04-09 18:03:21 +0000127of the time needed to generate GCC's PCH files. Since PTH/PCH generation is a
128serial operation that typically blocks progress during a build, faster
129generation time leads to improved processor utilization with parallel builds on
130multicore machines.</p></li>
131
Ted Kremenek797a2472009-04-08 05:07:30 +0000132</ul>
133
134<p>Despite these strengths, PTH's simple design suffers some algorithmic
135handicaps compared to other PCH strategies such as those used by GCC. While PTH
136can greatly speed up the processing time of a header file, the amount of work
137required to process a header file is still roughly linear in the size of the
138header file. In contrast, the amount of work done by GCC to process a
139precompiled header is (theoretically) constant (the ASTs for the header are
140literally memory mapped into the compiler). This means that only the pieces of
141the header file that are referenced by the source file including the header are
142the only ones the compiler needs to process during actual compilation. While
143GCC's particular implementation of PCH mitigates some of these algorithmic
144strengths via the use of copy-on-write pages, the approach itself can
145fundamentally dominate at an algorithmic level, especially when one considers
146header files of arbitrary size.</p>
147
Ted Kremenek07f08d22009-04-09 18:03:21 +0000148<p>There are plans to potentially implement an complementary PCH implementation
149for Clang based on the lazy deserialization of ASTs. This approach would
150theoretically have the same constant-time algorithmic advantages just mentioned
151but would also retain some of the strengths of PTH such as reduced memory
152pressure (ideal for multi-core builds).</p>
Ted Kremenek797a2472009-04-08 05:07:30 +0000153
154<h3>Internal PTH Optimizations</h3>
155
156<p>While the main optimization employed by PTH is to reduce lexing time of
157header files by caching pre-lexed tokens, PTH also employs several other
158optimizations to speed up the processing of header files:</p>
159
160<ul>
161
162<li><p><em><tt>stat</tt> caching</em>: PTH files cache information obtained via
163calls to <tt>stat</tt> that <tt>clang-cc</tt> uses to resolve which files are
164included by <tt>#include</tt> directives. This greatly reduces the overhead
165involved in context-switching to the kernel to resolve included files.</p></li>
166
167<li><p><em>Fasting skipping of <tt>#ifdef</tt>...<tt>#endif</tt> chains</em>:
168PTH files record the basic structure of nested preprocessor blocks. When the
169condition of the preprocessor block is false, all of its tokens are immediately
170skipped instead of requiring them to be handled by Clang's
171preprocessor.</p></li>
172
173</ul>
174
175</div>
176</body>
177</html>