Blame - docs/PTHInternals.html - platform/external/clang

blob: b15f68160676a6d6f776d15132a8f46f95d2a209 [file] [log] [blame]

Benjamin Kramer	665a8dc	2012-01-15 15:26:07 +0000	[diff] [blame]	1	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
				2	"http://www.w3.org/TR/html4/strict.dtd">
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	3	<html>
				4	<head>
				5	<title>Pretokenized Headers (PTH)</title>
Benjamin Kramer	665a8dc	2012-01-15 15:26:07 +0000	[diff] [blame]	6	<link type="text/css" rel="stylesheet" href="../menu.css">
				7	<link type="text/css" rel="stylesheet" href="../content.css">
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	8	<style type="text/css">
				9	td {
				10	vertical-align: top;
				11	}
				12	</style>
				13	</head>
				14	<body>
				15
				16	<!--#include virtual="../menu.html.incl"-->
				17
				18	<div id="content">
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	19
Chris Lattner	5c3074f	2009-04-20 04:37:38 +0000	[diff] [blame]	20	<h1>Pretokenized Headers (PTH)</h1>
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	21
Chris Lattner	5c3074f	2009-04-20 04:37:38 +0000	[diff] [blame]	22	<p>This document first describes the low-level
				23	interface for using PTH and then briefly elaborates on its design and
				24	implementation. If you are interested in the end-user view, please see the
				25	<a href="UsersManual.html#precompiledheaders">User's Manual</a>.</p>
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	26
				27
Daniel Dunbar	69cfd86	2009-12-11 23:17:03 +0000	[diff] [blame]	28	<h2>Using Pretokenized Headers with <tt>clang</tt> (Low-level Interface)</h2>
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	29
Daniel Dunbar	69cfd86	2009-12-11 23:17:03 +0000	[diff] [blame]	30	<p>The Clang compiler frontend, <tt>clang -cc1</tt>, supports three command line
				31	options for generating and using PTH files.<p>
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	32
Daniel Dunbar	69cfd86	2009-12-11 23:17:03 +0000	[diff] [blame]	33	<p>To generate PTH files using <tt>clang -cc1</tt>, use the option
Ted Kremenek	b7fd6b0	2009-04-09 18:17:39 +0000	[diff] [blame]	34	<b><tt>-emit-pth</tt></b>:
				35
Daniel Dunbar	69cfd86	2009-12-11 23:17:03 +0000	[diff] [blame]	36	<pre> $ clang -cc1 test.h -emit-pth -o test.h.pth </pre>
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	37
				38	<p>This option is transparently used by <tt>clang</tt> when generating PTH
Ted Kremenek	b7fd6b0	2009-04-09 18:17:39 +0000	[diff] [blame]	39	files. Similarly, PTH files can be used as prefix headers using the
				40	<b><tt>-include-pth</tt></b> option:</p>
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	41
				42	<pre>
Daniel Dunbar	69cfd86	2009-12-11 23:17:03 +0000	[diff] [blame]	43	$ clang -cc1 -include-pth test.h.pth test.c -o test.s
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	44	</pre>
				45
				46	<p>Alternatively, Clang's PTH files can be used as a raw "token-cache"
				47	(or "content" cache) of the source included by the original header
				48	file. This means that the contents of the PTH file are searched as substitutes
Daniel Dunbar	69cfd86	2009-12-11 23:17:03 +0000	[diff] [blame]	49	for <em>any</em> source files that are used by <tt>clang -cc1</tt> to process a
Ted Kremenek	b7fd6b0	2009-04-09 18:17:39 +0000	[diff] [blame]	50	source file. This is done by specifying the <b><tt>-token-cache</tt></b>
				51	option:</p>
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	52
				53	<pre>
				54	$ cat test.h
Chris Lattner	0a06999	2009-04-08 06:00:32 +0000	[diff] [blame]	55	#include <stdio.h>
Daniel Dunbar	69cfd86	2009-12-11 23:17:03 +0000	[diff] [blame]	56	$ clang -cc1 -emit-pth test.h -o test.h.pth
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	57	$ cat test.c
				58	#include "test.h"
Daniel Dunbar	69cfd86	2009-12-11 23:17:03 +0000	[diff] [blame]	59	$ clang -cc1 test.c -o test -token-cache test.h.pth
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	60	</pre>
				61
				62	<p>In this example the contents of <tt>stdio.h</tt> (and the files it includes)
				63	will be retrieved from <tt>test.h.pth</tt>, as the PTH file is being used in
				64	this case as a raw cache of the contents of <tt>test.h</tt>. This is a low-level
				65	interface used to both implement the high-level PTH interface as well as to
				66	provide alternative means to use PTH-style caching.</p>
				67
				68	<h2>PTH Design and Implementation</h2>
				69
				70	<p>Unlike GCC's precompiled headers, which cache the full ASTs and preprocessor
				71	state of a header file, Clang's pretokenized header files mainly cache the raw
				72	lexer <em>tokens</em> that are needed to segment the stream of characters in a
				73	source file into keywords, identifiers, and operators. Consequently, PTH serves
				74	to mainly directly speed up the lexing and preprocessing of a source file, while
				75	parsing and type-checking must be completely redone every time a PTH file is
				76	used.</p>
				77
				78	<h3>Basic Design Tradeoffs</h3>
				79
				80	<p>In the long term there are plans to provide an alternate PCH implementation
				81	for Clang that also caches the work for parsing and type checking the contents
				82	of header files. The current implementation of PCH in Clang as pretokenized
				83	header files was motivated by the following factors:<p>
				84
				85	<ul>
Ted Kremenek	07f08d2	2009-04-09 18:03:21 +0000	[diff] [blame]	86
Ted Kremenek	5890c63	2009-04-09 18:22:40 +0000	[diff] [blame]	87	<li><p><b>Language independence</b>: PTH files work with any language that
Ted Kremenek	07f08d2	2009-04-09 18:03:21 +0000	[diff] [blame]	88	Clang's lexer can handle, including C, Objective-C, and (in the early stages)
				89	C++. This means development on language features at the parsing level or above
				90	(which is basically almost all interesting pieces) does not require PTH to be
				91	modified.</p></li>
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	92
Ted Kremenek	5890c63	2009-04-09 18:22:40 +0000	[diff] [blame]	93	<li><b>Simple design</b>: Relatively speaking, PTH has a simple design and
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	94	implementation, making it easy to test. Further, because the machinery for PTH
				95	resides at the lower-levels of the Clang library stack it is fairly
				96	straightforward to profile and optimize.</li>
				97	</ul>
				98
				99	<p>Further, compared to GCC's PCH implementation (which is the dominate
				100	precompiled header file implementation that Clang can be directly compared
				101	against) the PTH design in Clang yields several attractive features:</p>
				102
				103	<ul>
				104
Ted Kremenek	5890c63	2009-04-09 18:22:40 +0000	[diff] [blame]	105	<li><p><b>Architecture independence</b>: In contrast to GCC's PCH files (and
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	106	those of several other compilers), Clang's PTH files are architecture
				107	independent, requiring only a single PTH file when building an program for
				108	multiple architectures.</p>
				109
				110	<p>For example, on Mac OS X one may wish to
				111	compile a "universal binary" that runs on PowerPC, 32-bit Intel
				112	(i386), and 64-bit Intel architectures. In contrast, GCC requires a PCH file for
				113	each architecture, as the definitions of types in the AST are
				114	architecture-specific. Since a Clang PTH file essentially represents a lexical
				115	cache of header files, a single PTH file can be safely used when compiling for
				116	multiple architectures. This can also reduce compile times because only a single
				117	PTH file needs to be generated during a build instead of several.</p></li>
				118
Ted Kremenek	5890c63	2009-04-09 18:22:40 +0000	[diff] [blame]	119	<li><p><b>Reduced memory pressure</b>: Similar to GCC,
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	120	Clang reads PTH files via the use of memory mapping (i.e., <tt>mmap</tt>).
				121	Clang, however, memory maps PTH files as read-only, meaning that multiple
Daniel Dunbar	69cfd86	2009-12-11 23:17:03 +0000	[diff] [blame]	122	invocations of <tt>clang -cc1</tt> can share the same pages in memory from a
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	123	memory-mapped PTH file. In comparison, GCC also memory maps its PCH files but
				124	also modifies those pages in memory, incurring the copy-on-write costs. The
				125	read-only nature of PTH can greatly reduce memory pressure for builds involving
				126	multiple cores, thus improving overall scalability.</p></li>
				127
Ted Kremenek	5890c63	2009-04-09 18:22:40 +0000	[diff] [blame]	128	<li><p><b>Fast generation</b>: PTH files can be generated in a small fraction
Ted Kremenek	07f08d2	2009-04-09 18:03:21 +0000	[diff] [blame]	129	of the time needed to generate GCC's PCH files. Since PTH/PCH generation is a
				130	serial operation that typically blocks progress during a build, faster
				131	generation time leads to improved processor utilization with parallel builds on
				132	multicore machines.</p></li>
				133
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	134	</ul>
				135
				136	<p>Despite these strengths, PTH's simple design suffers some algorithmic
				137	handicaps compared to other PCH strategies such as those used by GCC. While PTH
				138	can greatly speed up the processing time of a header file, the amount of work
				139	required to process a header file is still roughly linear in the size of the
				140	header file. In contrast, the amount of work done by GCC to process a
				141	precompiled header is (theoretically) constant (the ASTs for the header are
				142	literally memory mapped into the compiler). This means that only the pieces of
				143	the header file that are referenced by the source file including the header are
				144	the only ones the compiler needs to process during actual compilation. While
				145	GCC's particular implementation of PCH mitigates some of these algorithmic
				146	strengths via the use of copy-on-write pages, the approach itself can
				147	fundamentally dominate at an algorithmic level, especially when one considers
				148	header files of arbitrary size.</p>
				149
Ted Kremenek	07f08d2	2009-04-09 18:03:21 +0000	[diff] [blame]	150	<p>There are plans to potentially implement an complementary PCH implementation
				151	for Clang based on the lazy deserialization of ASTs. This approach would
				152	theoretically have the same constant-time algorithmic advantages just mentioned
				153	but would also retain some of the strengths of PTH such as reduced memory
				154	pressure (ideal for multi-core builds).</p>
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	155
				156	<h3>Internal PTH Optimizations</h3>
				157
				158	<p>While the main optimization employed by PTH is to reduce lexing time of
				159	header files by caching pre-lexed tokens, PTH also employs several other
				160	optimizations to speed up the processing of header files:</p>
				161
				162	<ul>
				163
				164	<li><p><em><tt>stat</tt> caching</em>: PTH files cache information obtained via
Daniel Dunbar	69cfd86	2009-12-11 23:17:03 +0000	[diff] [blame]	165	calls to <tt>stat</tt> that <tt>clang -cc1</tt> uses to resolve which files are
Ted Kremenek	797a247	2009-04-08 05:07:30 +0000	[diff] [blame]	166	included by <tt>#include</tt> directives. This greatly reduces the overhead
				167	involved in context-switching to the kernel to resolve included files.</p></li>
				168
				169	<li><p><em>Fasting skipping of <tt>#ifdef</tt>...<tt>#endif</tt> chains</em>:
				170	PTH files record the basic structure of nested preprocessor blocks. When the
				171	condition of the preprocessor block is false, all of its tokens are immediately
				172	skipped instead of requiring them to be handled by Clang's
				173	preprocessor.</p></li>
				174
				175	</ul>
				176
				177	</div>
				178	</body>
				179	</html>