blob: 10dda6117b75542ed6837fea07f02a2e1bb3994f [file] [log] [blame]
Sean Silva93ca0212012-12-13 01:10:46 +00001==========================
2Pretokenized Headers (PTH)
3==========================
4
5This document first describes the low-level interface for using PTH and
6then briefly elaborates on its design and implementation. If you are
Dmitri Gribenko97555a12012-12-15 21:10:51 +00007interested in the end-user view, please see the :ref:`User's Manual
8<usersmanual-precompiled-headers>`.
Sean Silva93ca0212012-12-13 01:10:46 +00009
10Using Pretokenized Headers with ``clang`` (Low-level Interface)
11===============================================================
12
13The Clang compiler frontend, ``clang -cc1``, supports three command line
14options for generating and using PTH files.
15
Dmitri Gribenko97555a12012-12-15 21:10:51 +000016To generate PTH files using ``clang -cc1``, use the option ``-emit-pth``:
Sean Silva93ca0212012-12-13 01:10:46 +000017
Dmitri Gribenko97555a12012-12-15 21:10:51 +000018.. code-block:: console
Sean Silva93ca0212012-12-13 01:10:46 +000019
Dmitri Gribenko97555a12012-12-15 21:10:51 +000020 $ clang -cc1 test.h -emit-pth -o test.h.pth
Sean Silva93ca0212012-12-13 01:10:46 +000021
22This option is transparently used by ``clang`` when generating PTH
23files. Similarly, PTH files can be used as prefix headers using the
24``-include-pth`` option:
25
Dmitri Gribenko97555a12012-12-15 21:10:51 +000026.. code-block:: console
Sean Silva93ca0212012-12-13 01:10:46 +000027
Dmitri Gribenko97555a12012-12-15 21:10:51 +000028 $ clang -cc1 -include-pth test.h.pth test.c -o test.s
Sean Silva93ca0212012-12-13 01:10:46 +000029
30Alternatively, Clang's PTH files can be used as a raw "token-cache" (or
31"content" cache) of the source included by the original header file.
32This means that the contents of the PTH file are searched as substitutes
33for *any* source files that are used by ``clang -cc1`` to process a
34source file. This is done by specifying the ``-token-cache`` option:
35
Dmitri Gribenko97555a12012-12-15 21:10:51 +000036.. code-block:: console
Sean Silva93ca0212012-12-13 01:10:46 +000037
Dmitri Gribenko97555a12012-12-15 21:10:51 +000038 $ cat test.h
39 #include <stdio.h>
40 $ clang -cc1 -emit-pth test.h -o test.h.pth
41 $ cat test.c
42 #include "test.h"
43 $ clang -cc1 test.c -o test -token-cache test.h.pth
Sean Silva93ca0212012-12-13 01:10:46 +000044
45In this example the contents of ``stdio.h`` (and the files it includes)
46will be retrieved from ``test.h.pth``, as the PTH file is being used in
47this case as a raw cache of the contents of ``test.h``. This is a
48low-level interface used to both implement the high-level PTH interface
49as well as to provide alternative means to use PTH-style caching.
50
51PTH Design and Implementation
52=============================
53
54Unlike GCC's precompiled headers, which cache the full ASTs and
55preprocessor state of a header file, Clang's pretokenized header files
56mainly cache the raw lexer *tokens* that are needed to segment the
57stream of characters in a source file into keywords, identifiers, and
58operators. Consequently, PTH serves to mainly directly speed up the
59lexing and preprocessing of a source file, while parsing and
60type-checking must be completely redone every time a PTH file is used.
61
62Basic Design Tradeoffs
Dmitri Gribenko029e70c2012-12-23 18:39:54 +000063----------------------
Sean Silva93ca0212012-12-13 01:10:46 +000064
65In the long term there are plans to provide an alternate PCH
66implementation for Clang that also caches the work for parsing and type
67checking the contents of header files. The current implementation of PCH
68in Clang as pretokenized header files was motivated by the following
69factors:
70
71**Language independence**
72 PTH files work with any language that
73 Clang's lexer can handle, including C, Objective-C, and (in the early
74 stages) C++. This means development on language features at the
75 parsing level or above (which is basically almost all interesting
76 pieces) does not require PTH to be modified.
77
78**Simple design**
79 Relatively speaking, PTH has a simple design and
80 implementation, making it easy to test. Further, because the
81 machinery for PTH resides at the lower-levels of the Clang library
82 stack it is fairly straightforward to profile and optimize.
83
84Further, compared to GCC's PCH implementation (which is the dominate
85precompiled header file implementation that Clang can be directly
86compared against) the PTH design in Clang yields several attractive
87features:
88
89**Architecture independence**
90 In contrast to GCC's PCH files (and
91 those of several other compilers), Clang's PTH files are architecture
Dmitri Gribenko029e70c2012-12-23 18:39:54 +000092 independent, requiring only a single PTH file when building a
Sean Silva93ca0212012-12-13 01:10:46 +000093 program for multiple architectures.
94
95 For example, on Mac OS X one may wish to compile a "universal binary"
96 that runs on PowerPC, 32-bit Intel (i386), and 64-bit Intel
97 architectures. In contrast, GCC requires a PCH file for each
98 architecture, as the definitions of types in the AST are
99 architecture-specific. Since a Clang PTH file essentially represents
100 a lexical cache of header files, a single PTH file can be safely used
101 when compiling for multiple architectures. This can also reduce
102 compile times because only a single PTH file needs to be generated
103 during a build instead of several.
104
105**Reduced memory pressure**
106 Similar to GCC, Clang reads PTH files
107 via the use of memory mapping (i.e., ``mmap``). Clang, however,
108 memory maps PTH files as read-only, meaning that multiple invocations
109 of ``clang -cc1`` can share the same pages in memory from a
110 memory-mapped PTH file. In comparison, GCC also memory maps its PCH
111 files but also modifies those pages in memory, incurring the
112 copy-on-write costs. The read-only nature of PTH can greatly reduce
113 memory pressure for builds involving multiple cores, thus improving
114 overall scalability.
115
116**Fast generation**
117 PTH files can be generated in a small fraction
118 of the time needed to generate GCC's PCH files. Since PTH/PCH
119 generation is a serial operation that typically blocks progress
120 during a build, faster generation time leads to improved processor
121 utilization with parallel builds on multicore machines.
122
123Despite these strengths, PTH's simple design suffers some algorithmic
124handicaps compared to other PCH strategies such as those used by GCC.
125While PTH can greatly speed up the processing time of a header file, the
126amount of work required to process a header file is still roughly linear
127in the size of the header file. In contrast, the amount of work done by
128GCC to process a precompiled header is (theoretically) constant (the
129ASTs for the header are literally memory mapped into the compiler). This
130means that only the pieces of the header file that are referenced by the
131source file including the header are the only ones the compiler needs to
132process during actual compilation. While GCC's particular implementation
133of PCH mitigates some of these algorithmic strengths via the use of
134copy-on-write pages, the approach itself can fundamentally dominate at
135an algorithmic level, especially when one considers header files of
136arbitrary size.
137
138There are plans to potentially implement an complementary PCH
139implementation for Clang based on the lazy deserialization of ASTs. This
140approach would theoretically have the same constant-time algorithmic
141advantages just mentioned but would also retain some of the strengths of
142PTH such as reduced memory pressure (ideal for multi-core builds).
143
144Internal PTH Optimizations
Dmitri Gribenko029e70c2012-12-23 18:39:54 +0000145--------------------------
Sean Silva93ca0212012-12-13 01:10:46 +0000146
147While the main optimization employed by PTH is to reduce lexing time of
148header files by caching pre-lexed tokens, PTH also employs several other
149optimizations to speed up the processing of header files:
150
151- ``stat`` caching: PTH files cache information obtained via calls to
152 ``stat`` that ``clang -cc1`` uses to resolve which files are included
153 by ``#include`` directives. This greatly reduces the overhead
154 involved in context-switching to the kernel to resolve included
155 files.
156
Dmitri Gribenko029e70c2012-12-23 18:39:54 +0000157- Fast skipping of ``#ifdef`` ... ``#endif`` chains: PTH files
Sean Silva93ca0212012-12-13 01:10:46 +0000158 record the basic structure of nested preprocessor blocks. When the
159 condition of the preprocessor block is false, all of its tokens are
160 immediately skipped instead of requiring them to be handled by
161 Clang's preprocessor.
162
163