Gitiles
Code Review Sign In
gerrit-public.fairphone.software / fp2-dev / platform / external / clang / f33651c2d79d47d8d05908185613fe33a3e18d89 / . / docs / PTHInternals.html
blob: 2d2e1e3e1ba40c7716270726763db8ebee152eac [file] [log] [blame]
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]1<html>
2 <head>
3 <title>Pretokenized Headers (PTH)</title>
4 <link type="text/css" rel="stylesheet" href="../menu.css" />
5 <link type="text/css" rel="stylesheet" href="../content.css" />
6 <style type="text/css">
7 td {
8 vertical-align: top;
9 }
10 </style>
11</head>
12<body>
13
14<!--#include virtual="../menu.html.incl"-->
15
16<div id="content">
17<h1>Pretokenized Headers</h1>
18
19<p> <a href="http://en.wikipedia.org/wiki/Precompiled_header">Precompiled
Chris Lattnercdf59da2009-04-08 05:50:25 +0000[diff] [blame]20headers</a> are a general approach employed by many compilers to reduce
21compilation time. The underlying motivation of the approach is that it is
22common for the same (and often large) header files to be included by
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]23multiple source files. Consequently, compile times can often be greatly improved
24by caching some of the (redundant) work done by a compiler to process headers.
Chris Lattnercdf59da2009-04-08 05:50:25 +0000[diff] [blame]25Precompiled header files, which represent one of many ways to implement
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]26this optimization, are literally files that represent an on-disk cache that
27contains the vital information necessary to reduce some (or all) of the work
28needed to process a corresponding header file. While details of precompiled
29headers vary between compilers, precompiled headers have been shown to be a
30highly effective at speeding up program compilation on systems with very large
Chris Lattnercdf59da2009-04-08 05:50:25 +0000[diff] [blame]31system headers (e.g., Mac OS/X).</p>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]32
33<p>Clang supports an implementation of precompiled headers known as
34<em>pre-tokenized headers</em> (PTH). Clang's pre-tokenized headers support most
35of same interfaces as GCC's pre-compiled headers (as well as others) but are
Ted Kremenek5890c632009-04-09 18:22:40 +0000[diff] [blame]36completely different in their implementation. This first describes the interface
37for using PTH and then briefly elaborates on its design and implementation.</p>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]38
39
Ted Kremenekb7fd6b02009-04-09 18:17:39 +0000[diff] [blame]40<h2>Using Pretokenized Headers with <tt>clang</tt></h2>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]41
Ted Kremenekb7fd6b02009-04-09 18:17:39 +0000[diff] [blame]42<p>The high-level <tt>clang</tt> driver supports an interface to use PTH files
43that is similar to GCC's interface for precompiled headers.</p>
44
45<h3>Generating a PTH File</h3>
46
47<p>To generate a PTH file using <tt>clang</tt>, one invokes <tt>clang</tt> using
Ted Kremenek8e164082009-04-09 18:20:08 +0000[diff] [blame]48the <b><tt>-x <i>&lt;language&gt;</i>-header</tt></b> option. This mirrors the
49interface in GCC for generating PCH files:</p>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]50
51<pre>
52 $ gcc -x c-header test.h -o test.h.gch
53 $ clang -x c-header test.h -o test.h.pth
54</pre>
55
Ted Kremenekb7fd6b02009-04-09 18:17:39 +0000[diff] [blame]56<h3>Using a PTH File</h3>
57
58<p>A PTH file can then be used as a prefix header when a
59<b><tt>-include</tt></b> option is passed to <tt>clang</tt>:</p>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]60
61<pre>
62 $ clang -include test.h test.c -o test
63</pre>
64
65<p>The <tt>clang</tt> driver will first check if a PTH file for <tt>test.h</tt>
66is available; if so, the contents of <tt>test.h</tt> (and the files it includes)
67will be processed from the PTH file. Otherwise, <tt>clang</tt> falls back to
68directly processing the content of <tt>test.h</tt>. This mirrors the behavior of
69GCC.</p>
70
71<p><b>NOTE:</b> <tt>clang</tt> does <em>not</em> automatically used PTH files
72for headers that are directly included within a source file. For example:</p>
73
74<pre>
75 $ clang -x c-header test.h -o test.h.pth
76 $ cat test.c
77 #include "test.h"
78 $ clang test.c -o test
79</pre>
80
81<p>In this example, <tt>clang</tt> will not automatically use the PTH file for
82<tt>test.h</tt> since <tt>test.h</tt> was included directly in the source file
83and not specified on the command line using <tt>-include</tt>.</p>
84
Ted Kremenekb7fd6b02009-04-09 18:17:39 +0000[diff] [blame]85<h2>Using Pretokenized Headers with <tt>clang-cc</tt> (Low-level Interface)</h2>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]86
Chris Lattnercdf59da2009-04-08 05:50:25 +0000[diff] [blame]87<p>The low-level Clang compiler tool, <tt>clang-cc</tt>, supports three command
88line options for generating and using PTH files.<p>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]89
Ted Kremenekb7fd6b02009-04-09 18:17:39 +0000[diff] [blame]90<p>To generate PTH files using <tt>clang-cc</tt>, use the option
91<b><tt>-emit-pth</tt></b>:
92
93<pre> $ clang-cc test.h -emit-pth -o test.h.pth </pre>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]94
95<p>This option is transparently used by <tt>clang</tt> when generating PTH
Ted Kremenekb7fd6b02009-04-09 18:17:39 +0000[diff] [blame]96files. Similarly, PTH files can be used as prefix headers using the
97<b><tt>-include-pth</tt></b> option:</p>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]98
99<pre>
100 $ clang-cc -include-pth test.h.pth test.c -o test.s
101</pre>
102
103<p>Alternatively, Clang's PTH files can be used as a raw &quot;token-cache&quot;
104(or &quot;content&quot; cache) of the source included by the original header
105file. This means that the contents of the PTH file are searched as substitutes
106for <em>any</em> source files that are used by <tt>clang-cc</tt> to process a
Ted Kremenekb7fd6b02009-04-09 18:17:39 +0000[diff] [blame]107source file. This is done by specifying the <b><tt>-token-cache</tt></b>
108option:</p>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]109
110<pre>
111 $ cat test.h
Chris Lattner0a069992009-04-08 06:00:32 +0000[diff] [blame]112 #include &lt;stdio.h&gt;
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]113 $ clang-cc -emit-pth test.h -o test.h.pth
114 $ cat test.c
115 #include "test.h"
116 $ clang-cc test.c -o test -token-cache test.h.pth
117</pre>
118
119<p>In this example the contents of <tt>stdio.h</tt> (and the files it includes)
120will be retrieved from <tt>test.h.pth</tt>, as the PTH file is being used in
121this case as a raw cache of the contents of <tt>test.h</tt>. This is a low-level
122interface used to both implement the high-level PTH interface as well as to
123provide alternative means to use PTH-style caching.</p>
124
125<h2>PTH Design and Implementation</h2>
126
127<p>Unlike GCC's precompiled headers, which cache the full ASTs and preprocessor
128state of a header file, Clang's pretokenized header files mainly cache the raw
129lexer <em>tokens</em> that are needed to segment the stream of characters in a
130source file into keywords, identifiers, and operators. Consequently, PTH serves
131to mainly directly speed up the lexing and preprocessing of a source file, while
132parsing and type-checking must be completely redone every time a PTH file is
133used.</p>
134
135<h3>Basic Design Tradeoffs</h3>
136
137<p>In the long term there are plans to provide an alternate PCH implementation
138for Clang that also caches the work for parsing and type checking the contents
139of header files. The current implementation of PCH in Clang as pretokenized
140header files was motivated by the following factors:<p>
141
142<ul>
Ted Kremenek07f08d22009-04-09 18:03:21 +0000[diff] [blame]143
Ted Kremenek5890c632009-04-09 18:22:40 +0000[diff] [blame]144<li><p><b>Language independence</b>: PTH files work with any language that
Ted Kremenek07f08d22009-04-09 18:03:21 +0000[diff] [blame]145Clang's lexer can handle, including C, Objective-C, and (in the early stages)
146C++. This means development on language features at the parsing level or above
147(which is basically almost all interesting pieces) does not require PTH to be
148modified.</p></li>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]149
Ted Kremenek5890c632009-04-09 18:22:40 +0000[diff] [blame]150<li><b>Simple design</b>: Relatively speaking, PTH has a simple design and
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]151implementation, making it easy to test. Further, because the machinery for PTH
152resides at the lower-levels of the Clang library stack it is fairly
153straightforward to profile and optimize.</li>
154</ul>
155
156<p>Further, compared to GCC's PCH implementation (which is the dominate
157precompiled header file implementation that Clang can be directly compared
158against) the PTH design in Clang yields several attractive features:</p>
159
160<ul>
161
Ted Kremenek5890c632009-04-09 18:22:40 +0000[diff] [blame]162<li><p><b>Architecture independence</b>: In contrast to GCC's PCH files (and
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]163those of several other compilers), Clang's PTH files are architecture
164independent, requiring only a single PTH file when building an program for
165multiple architectures.</p>
166
167<p>For example, on Mac OS X one may wish to
168compile a &quot;universal binary&quot; that runs on PowerPC, 32-bit Intel
169(i386), and 64-bit Intel architectures. In contrast, GCC requires a PCH file for
170each architecture, as the definitions of types in the AST are
171architecture-specific. Since a Clang PTH file essentially represents a lexical
172cache of header files, a single PTH file can be safely used when compiling for
173multiple architectures. This can also reduce compile times because only a single
174PTH file needs to be generated during a build instead of several.</p></li>
175
Ted Kremenek5890c632009-04-09 18:22:40 +0000[diff] [blame]176<li><p><b>Reduced memory pressure</b>: Similar to GCC,
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]177Clang reads PTH files via the use of memory mapping (i.e., <tt>mmap</tt>).
178Clang, however, memory maps PTH files as read-only, meaning that multiple
179invocations of <tt>clang-cc</tt> can share the same pages in memory from a
180memory-mapped PTH file. In comparison, GCC also memory maps its PCH files but
181also modifies those pages in memory, incurring the copy-on-write costs. The
182read-only nature of PTH can greatly reduce memory pressure for builds involving
183multiple cores, thus improving overall scalability.</p></li>
184
Ted Kremenek5890c632009-04-09 18:22:40 +0000[diff] [blame]185<li><p><b>Fast generation</b>: PTH files can be generated in a small fraction
Ted Kremenek07f08d22009-04-09 18:03:21 +0000[diff] [blame]186of the time needed to generate GCC's PCH files. Since PTH/PCH generation is a
187serial operation that typically blocks progress during a build, faster
188generation time leads to improved processor utilization with parallel builds on
189multicore machines.</p></li>
190
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]191</ul>
192
193<p>Despite these strengths, PTH's simple design suffers some algorithmic
194handicaps compared to other PCH strategies such as those used by GCC. While PTH
195can greatly speed up the processing time of a header file, the amount of work
196required to process a header file is still roughly linear in the size of the
197header file. In contrast, the amount of work done by GCC to process a
198precompiled header is (theoretically) constant (the ASTs for the header are
199literally memory mapped into the compiler). This means that only the pieces of
200the header file that are referenced by the source file including the header are
201the only ones the compiler needs to process during actual compilation. While
202GCC's particular implementation of PCH mitigates some of these algorithmic
203strengths via the use of copy-on-write pages, the approach itself can
204fundamentally dominate at an algorithmic level, especially when one considers
205header files of arbitrary size.</p>
206
Ted Kremenek07f08d22009-04-09 18:03:21 +0000[diff] [blame]207<p>There are plans to potentially implement an complementary PCH implementation
208for Clang based on the lazy deserialization of ASTs. This approach would
209theoretically have the same constant-time algorithmic advantages just mentioned
210but would also retain some of the strengths of PTH such as reduced memory
211pressure (ideal for multi-core builds).</p>
Ted Kremenek797a2472009-04-08 05:07:30 +0000[diff] [blame]212
213<h3>Internal PTH Optimizations</h3>
214
215<p>While the main optimization employed by PTH is to reduce lexing time of
216header files by caching pre-lexed tokens, PTH also employs several other
217optimizations to speed up the processing of header files:</p>
218
219<ul>
220
221<li><p><em><tt>stat</tt> caching</em>: PTH files cache information obtained via
222calls to <tt>stat</tt> that <tt>clang-cc</tt> uses to resolve which files are
223included by <tt>#include</tt> directives. This greatly reduces the overhead
224involved in context-switching to the kernel to resolve included files.</p></li>
225
226<li><p><em>Fasting skipping of <tt>#ifdef</tt>...<tt>#endif</tt> chains</em>:
227PTH files record the basic structure of nested preprocessor blocks. When the
228condition of the preprocessor block is false, all of its tokens are immediately
229skipped instead of requiring them to be handled by Clang's
230preprocessor.</p></li>
231
232</ul>
233
234</div>
235</body>
236</html>