Blame - docs/DriverInternals.html - fp2-dev/platform/external/clang

2012-01-15 15:26:07 +0000

[diff] [blame]

1

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"

2

"http://www.w3.org/TR/html4/strict.dtd">

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

3

<html>

4

<head>

5

<title>Clang Driver Manual</title>

Benjamin Kramer

2012-01-15 15:26:07 +0000

[diff] [blame]

6

7

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

td {

vertical-align: top;

}

</style>

</head>

<body>

Chris Lattner

2009-04-19 02:02:45 +0000

[diff] [blame]

20

<h1>Driver Design & Internals</h1>

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

21

22

<ul>

23

<li><a href="#intro">Introduction</a></li>

Benjamin Kramer

2012-01-15 15:26:07 +0000

[diff] [blame]

24

<li><a href="#features">Features and Goals</a>

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

25

<ul>

26

<li><a href="#gcccompat">GCC Compatibility</a></li>

27

<li><a href="#components">Flexible</a></li>

28

<li><a href="#performance">Low Overhead</a></li>

29

<li><a href="#simple">Simple</a></li>

30

</ul>

Benjamin Kramer

2012-01-15 15:26:07 +0000

[diff] [blame]

31

</li>

32

<li><a href="#design">Design</a>

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

33

<ul>

34

<li><a href="#int_intro">Internals Introduction</a></li>

35

<li><a href="#int_overview">Design Overview</a></li>

Benjamin Kramer

2012-01-15 15:26:07 +0000

[diff] [blame]

36

<li><a href="#int_notes">Additional Notes</a>

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

37

<ul>

38

<li><a href="#int_compilation">The Compilation Object</a></li>

39

<li><a href="#int_unified_parsing">Unified Parsing & Pipelining</a></li>

40

<li><a href="#int_toolchain_translation">ToolChain Argument Translation</a></li>

41

<li><a href="#int_unused_warnings">Unused Argument Warnings</a></li>

42

</ul>

Benjamin Kramer

2012-01-15 15:26:07 +0000

[diff] [blame]

43

</li>

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

44

<li><a href="#int_gcc_concepts">Relation to GCC Driver Concepts</a></li>

45

</ul>

Benjamin Kramer

2012-01-15 15:26:07 +0000

[diff] [blame]

46

</li>

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

</ul>

51

<h2 id="intro">Introduction</h2>

52

53

54

<p>This document describes the Clang driver. The purpose of this

55

document is to describe both the motivation and design goals

56

for the driver, as well as details of the internal

57

implementation.</p>

58

59

60

<h2 id="features">Features and Goals</h2>

61

62

63

<p>The Clang driver is intended to be a production quality

64

compiler driver providing access to the Clang compiler and

65

tools, with a command line interface which is compatible with

66

the gcc driver.</p>

67

68

<p>Although the driver is part of and driven by the Clang

69

project, it is logically a separate tool which shares many of

70

the same goals as Clang:</p>

71

72

<p><b>Features</b>:</p>

73

<ul>

74

<li><a href="#gcccompat">GCC Compatibility</a></li>

75

<li><a href="#components">Flexible</a></li>

76

<li><a href="#performance">Low Overhead</a></li>

77

<li><a href="#simple">Simple</a></li>

78

</ul>

79

80

81

<h3 id="gcccompat">GCC Compatibility</h3>

82

83

84

<p>The number one goal of the driver is to ease the adoption of

85

Clang by allowing users to drop Clang into a build system

86

which was designed to call GCC. Although this makes the driver

87

much more complicated than might otherwise be necessary, we

88

decided that being very compatible with the gcc command line

89

interface was worth it in order to allow users to quickly test

90

clang on their projects.</p>

91

92

93

<h3 id="components">Flexible</h3>

94

95

Chris Lattner

fc8f0e1

2011-04-15 05:22:18 +0000

[diff] [blame]

96

<p>The driver was designed to be flexible and easily accommodate

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

97

new uses as we grow the clang and LLVM infrastructure. As one

98

example, the driver can easily support the introduction of

99

tools which have an integrated assembler; something we hope to

100

add to LLVM in the future.</p>

101

102

<p>Similarly, most of the driver functionality is kept in a

103

library which can be used to build other tools which want to

104

implement or accept a gcc like interface. </p>

105

106

107

<h3 id="performance">Low Overhead</h3>

108

109

110

<p>The driver should have as little overhead as possible. In

111

practice, we found that the gcc driver by itself incurred a

112

small but meaningful overhead when compiling many small

113

files. The driver doesn't do much work compared to a

114

compilation, but we have tried to keep it as efficient as

115

possible by following a few simple principles:</p>

116

<ul>

117

<li>Avoid memory allocation and string copying when

118

possible.</li>

119

120

<li>Don't parse arguments more than once.</li>

121

Daniel Dunbar

2009-04-19 02:33:12 +0000

[diff] [blame]

122

<li>Provide a few simple interfaces for efficiently searching

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

arguments.</li>

</ul>

127

<h3 id="simple">Simple</h3>

128

129

130

<p>Finally, the driver was designed to be "as simple as

131

possible", given the other goals. Notably, trying to be

132

completely compatible with the gcc driver adds a significant

133

amount of complexity. However, the design of the driver

134

attempts to mitigate this complexity by dividing the process

135

into a number of independent stages instead of a single

136

monolithic task.</p>

137

138

139

<h2 id="design">Internal Design and Implementation</h2>

140

141

142

<ul>

143

<li><a href="#int_intro">Internals Introduction</a></li>

144

<li><a href="#int_overview">Design Overview</a></li>

145

<li><a href="#int_notes">Additional Notes</a></li>

146

<li><a href="#int_gcc_concepts">Relation to GCC Driver Concepts</a></li>

147

</ul>

148

149

150

<h3><a name="int_intro">Internals Introduction</a></h3>

151

152

153

<p>In order to satisfy the stated goals, the driver was designed

154

to completely subsume the functionality of the gcc executable;

155

that is, the driver should not need to delegate to gcc to

156

perform subtasks. On Darwin, this implies that the Clang

157

driver also subsumes the gcc driver-driver, which is used to

158

implement support for building universal images (binaries and

159

object files). This also implies that the driver should be

160

able to call the language specific compilers (e.g. cc1)

161

directly, which means that it must have enough information to

162

forward command line arguments to child processes

163

correctly.</p>

164

165

166

<h3><a name="int_overview">Design Overview</a></h3>

167

168

169

<p>The diagram below shows the significant components of the

170

driver architecture and how they relate to one another. The

171

orange components represent concrete data structures built by

172

the driver, the green components indicate conceptually

173

distinct stages which manipulate these data structures, and

174

the blue components are important helper classes. </p>

175

Benjamin Kramer

2012-01-15 15:26:07 +0000

[diff] [blame]

176

177

178

<img width=400 src="DriverArchitecture.png"

179

alt="Driver Architecture Diagram">

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

180

</a>

Benjamin Kramer

2012-01-15 15:26:07 +0000

[diff] [blame]

181

</div>

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

182

183

184

<h3><a name="int_stages">Driver Stages</a></h3>

185

186

187

<p>The driver functionality is conceptually divided into five stages:</p>

<ol>

<li>

<b>Parse: Option Parsing</b>

192

193

<p>The command line argument strings are decomposed into

194

arguments (<tt>Arg</tt> instances). The driver expects to

195

understand all available options, although there is some

196

facility for just passing certain classes of options

197

through (like <tt>-Wl,</tt>).</p>

198

199

<p>Each argument corresponds to exactly one

200

abstract <tt>Option</tt> definition, which describes how

201

the option is parsed along with some additional

202

metadata. The Arg instances themselves are lightweight and

203

merely contain enough information for clients to determine

204

which option they correspond to and their values (if they

205

have additional parameters).</p>

206

207

<p>For example, a command line like "-Ifoo -I foo" would

208

parse to two Arg instances (a JoinedArg and a SeparateArg

209

instance), but each would refer to the same Option.</p>

210

211

<p>Options are lazily created in order to avoid populating

212

all Option classes when the driver is loaded. Most of the

213

driver code only needs to deal with options by their

214

unique ID (e.g., <tt>options::OPT_I</tt>),</p>

215

216

<p>Arg instances themselves do not generally store the

Daniel Dunbar

2009-04-19 02:33:12 +0000

[diff] [blame]

217

values of parameters. In many cases, this would

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

218

simply result in creating unnecessary string

219

copies. Instead, Arg instances are always embedded inside

220

an ArgList structure, which contains the original vector

Daniel Dunbar

2009-04-19 02:33:12 +0000

[diff] [blame]

221

of argument strings. Each Arg itself only needs to contain

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

222

an index into this vector instead of storing its values

223

directly.</p>

224

Daniel Dunbar

2009-04-19 02:33:12 +0000

[diff] [blame]

225

<p>The clang driver can dump the results of this

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

226

stage using the <tt>-ccc-print-options</tt> flag (which

Chris Lattner

fc8f0e1

2011-04-15 05:22:18 +0000

[diff] [blame]

227

must precede any actual command line arguments). For

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

228

example:</p>

229

<pre>

230

$ <b>clang -ccc-print-options -Xarch_i386 -fomit-frame-pointer -Wa,-fast -Ifoo -I foo t.c</b>

231

Option 0 - Name: "-Xarch_", Values: {"i386", "-fomit-frame-pointer"}

232

Option 1 - Name: "-Wa,", Values: {"-fast"}

233

Option 2 - Name: "-I", Values: {"foo"}

234

Option 3 - Name: "-I", Values: {"foo"}

235

Option 4 - Name: "<input>", Values: {"t.c"}

236

</pre>

237

238

<p>After this stage is complete the command line should be

239

broken down into well defined option objects with their

240

appropriate parameters. Subsequent stages should rarely,

241

if ever, need to do any string processing.</p>

</li>

<li>

<b>Pipeline: Compilation Job Construction</b>

246

247

<p>Once the arguments are parsed, the tree of subprocess

248

jobs needed for the desired compilation sequence are

Daniel Dunbar

2009-08-14 04:43:04 +0000

[diff] [blame]

249

constructed. This involves determining the input files and

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

250

their types, what work is to be done on them (preprocess,

251

compile, assemble, link, etc.), and constructing a list of

252

Action instances for each task. The result is a list of

253

one or more top-level actions, each of which generally

254

corresponds to a single output (for example, an object or

255

linked executable).</p>

256

257

<p>The majority of Actions correspond to actual tasks,

258

however there are two special Actions. The first is

259

InputAction, which simply serves to adapt an input

260

argument for use as an input to other Actions. The second

261

is BindArchAction, which conceptually alters the

262

architecture to be used for all of its input Actions.</p>

263

Daniel Dunbar

2009-04-19 02:33:12 +0000

[diff] [blame]

264

<p>The clang driver can dump the results of this

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

265

stage using the <tt>-ccc-print-phases</tt> flag. For

266

example:</p>

267

<pre>

268

$ <b>clang -ccc-print-phases -x c t.c -x assembler t.s</b>

269

0: input, "t.c", c

270

1: preprocessor, {0}, cpp-output

271

2: compiler, {1}, assembler

272

3: assembler, {2}, object

273

4: input, "t.s", assembler

274

5: assembler, {4}, object

275

6: linker, {3, 5}, image

276

</pre>

Chris Lattner

715fc92

2009-04-19 02:02:45 +0000

[diff] [blame]

277

<p>Here the driver is constructing seven distinct actions,

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

278

four to compile the "t.c" input into an object file, two to

279

assemble the "t.s" input, and one to link them together.</p>

280

281

<p>A rather different compilation pipeline is shown here; in

282

this example there are two top level actions to compile

283

the input files into two separate object files, where each

284

object file is built using <tt>lipo</tt> to merge results

285

built for two separate architectures.</p>

286

<pre>

287

$ <b>clang -ccc-print-phases -c -arch i386 -arch x86_64 t0.c t1.c</b>

288

0: input, "t0.c", c

289

1: preprocessor, {0}, cpp-output

290

2: compiler, {1}, assembler

291

3: assembler, {2}, object

292

4: bind-arch, "i386", {3}, object

293

5: bind-arch, "x86_64", {3}, object

294

6: lipo, {4, 5}, object

295

7: input, "t1.c", c

296

8: preprocessor, {7}, cpp-output

297

9: compiler, {8}, assembler

298

10: assembler, {9}, object

299

11: bind-arch, "i386", {10}, object

300

12: bind-arch, "x86_64", {10}, object

301

13: lipo, {11, 12}, object

302

</pre>

303

304

<p>After this stage is complete the compilation process is

305

divided into a simple set of actions which need to be

306

performed to produce intermediate or final outputs (in

307

some cases, like <tt>-fsyntax-only</tt>, there is no

308

"real" final output). Phases are well known compilation

309

steps, such as "preprocess", "compile", "assemble",

"link", etc.</p>

</li>

<li>

<b>Bind: Tool & Filename Selection</b>

315

Daniel Dunbar

2009-04-19 02:33:12 +0000

[diff] [blame]

316

<p>This stage (in conjunction with the Translate stage)

317

turns the tree of Actions into a list of actual subprocess

318

to run. Conceptually, the driver performs a top down

319

matching to assign Action(s) to Tools. The ToolChain is

320

responsible for selecting the tool to perform a particular

Daniel Dunbar

2009-08-14 04:43:04 +0000

[diff] [blame]

321

action; once selected the driver interacts with the tool

Daniel Dunbar

2009-04-19 02:33:12 +0000

[diff] [blame]

322

to see if it can match additional actions (for example, by

323

having an integrated preprocessor).

324

325

<p>Once Tools have been selected for all actions, the driver

326

determines how the tools should be connected (for example,

327

using an inprocess module, pipes, temporary files, or user

328

provided filenames). If an output file is required, the

329

driver also computes the appropriate file name (the suffix

330

and file location depend on the input types and options

331

such as <tt>-save-temps</tt>).

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

332

333

<p>The driver interacts with a ToolChain to perform the Tool

334

bindings. Each ToolChain contains information about all

335

the tools needed for compilation for a particular

336

architecture, platform, and operating system. A single

Daniel Dunbar

2009-04-19 02:33:12 +0000

[diff] [blame]

337

driver invocation may query multiple ToolChains during one

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

338

compilation in order to interact with tools for separate

339

architectures.</p>

340

341

<p>The results of this stage are not computed directly, but

342

the driver can print the results via

343

the <tt>-ccc-print-bindings</tt> option. For example:</p>

344

<pre>

345

$ <b>clang -ccc-print-bindings -arch i386 -arch ppc t0.c</b>

Daniel Dunbar

2009-04-19 02:33:12 +0000

[diff] [blame]

346

# "i386-apple-darwin9" - "clang", inputs: ["t0.c"], output: "/tmp/cc-Sn4RKF.s"

347

# "i386-apple-darwin9" - "darwin::Assemble", inputs: ["/tmp/cc-Sn4RKF.s"], output: "/tmp/cc-gvSnbS.o"

348

# "i386-apple-darwin9" - "darwin::Link", inputs: ["/tmp/cc-gvSnbS.o"], output: "/tmp/cc-jgHQxi.out"

349

# "ppc-apple-darwin9" - "gcc::Compile", inputs: ["t0.c"], output: "/tmp/cc-Q0bTox.s"

350

# "ppc-apple-darwin9" - "gcc::Assemble", inputs: ["/tmp/cc-Q0bTox.s"], output: "/tmp/cc-WCdicw.o"

351

# "ppc-apple-darwin9" - "gcc::Link", inputs: ["/tmp/cc-WCdicw.o"], output: "/tmp/cc-HHBEBh.out"

352

# "i386-apple-darwin9" - "darwin::Lipo", inputs: ["/tmp/cc-jgHQxi.out", "/tmp/cc-HHBEBh.out"], output: "a.out"

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

353

</pre>

354

355

<p>This shows the tool chain, tool, inputs and outputs which

356

have been bound for this compilation sequence. Here clang

357

is being used to compile t0.c on the i386 architecture and

358

darwin specific versions of the tools are being used to

359

assemble and link the result, but generic gcc versions of

360

the tools are being used on PowerPC.</p>

</li>

<li>

<b>Translate: Tool Specific Argument Translation</b>

365

366

<p>Once a Tool has been selected to perform a particular

367

Action, the Tool must construct concrete Jobs which will be

368

executed during compilation. The main work is in translating

369

from the gcc style command line options to whatever options

370

the subprocess expects.</p>

371

372

<p>Some tools, such as the assembler, only interact with a

373

handful of arguments and just determine the path of the

374

executable to call and pass on their input and output

375

arguments. Others, like the compiler or the linker, may

376

translate a large number of arguments in addition.</p>

377

378

<p>The ArgList class provides a number of simple helper

379

methods to assist with translating arguments; for example,

380

to pass on only the last of arguments corresponding to some

381

option, or all arguments for an option.</p>

382

383

<p>The result of this stage is a list of Jobs (executable

384

paths and argument strings) to execute.</p>

</li>

<li>

<b>Execute</b>

<p>Finally, the compilation pipeline is executed. This is

390

mostly straightforward, although there is some interaction

391

with options

392

like <tt>-pipe</tt>, <tt>-pass-exit-codes</tt>

393

and <tt>-time</tt>.</p>

</li>

</ol>

399

<h3><a name="int_notes">Additional Notes</a></h3>

400

401

402

<h4 id="int_compilation">The Compilation Object</h4>

403

404

<p>The driver constructs a Compilation object for each set of

405

command line arguments. The Driver itself is intended to be

Daniel Dunbar

2009-08-14 04:43:04 +0000

[diff] [blame]

406

invariant during construction of a Compilation; an IDE should be

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

407

able to construct a single long lived driver instance to use

408

for an entire build, for example.</p>

409

410

<p>The Compilation object holds information that is particular

411

to each compilation sequence. For example, the list of used

412

temporary files (which must be removed once compilation is

413

finished) and result files (which should be removed if

Chris Lattner

de7738c

2011-07-03 02:21:48 +0000

[diff] [blame]

414

compilation fails).</p>

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

415

416

<h4 id="int_unified_parsing">Unified Parsing & Pipelining</h4>

417

Daniel Dunbar

2009-08-14 04:43:04 +0000

[diff] [blame]

418

<p>Parsing and pipelining both occur without reference to a

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

419

Compilation instance. This is by design; the driver expects that

420

both of these phases are platform neutral, with a few very well

421

defined exceptions such as whether the platform uses a driver

422

driver.</p>

423

424

<h4 id="int_toolchain_translation">ToolChain Argument Translation</h4>

425

426

<p>In order to match gcc very closely, the clang driver

427

currently allows tool chains to perform their own translation of

428

the argument list (into a new ArgList data structure). Although

429

this allows the clang driver to match gcc easily, it also makes

430

the driver operation much harder to understand (since the Tools

431

stop seeing some arguments the user provided, and see new ones

432

instead).</p>

433

Daniel Dunbar

2009-08-14 04:43:04 +0000

[diff] [blame]

434

<p>For example, on Darwin <tt>-gfull</tt> gets translated into two

435

separate arguments, <tt>-g</tt>

436

and <tt>-fno-eliminate-unused-debug-symbols</tt>. Trying to write Tool

437

logic to do something with <tt>-gfull</tt> will not work, because Tool

438

argument translation is done after the arguments have been

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

439

translated.</p>

440

441

<p>A long term goal is to remove this tool chain specific

442

translation, and instead force each tool to change its own logic

443

to do the right thing on the untranslated original arguments.</p>

444

445

<h4 id="int_unused_warnings">Unused Argument Warnings</h4>

446

<p>The driver operates by parsing all arguments but giving Tools

447

the opportunity to choose which arguments to pass on. One

448

downside of this infrastructure is that if the user misspells

449

some option, or is confused about which options to use, some

450

command line arguments the user really cared about may go

451

unused. This problem is particularly important when using

452

clang as a compiler, since the clang compiler does not support

Chris Lattner

715fc92

2009-04-19 02:02:45 +0000

[diff] [blame]

453

anywhere near all the options that gcc does, and we want to make

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

454

sure users know which ones are being used.</p>

455

456

<p>To support this, the driver maintains a bit associated with

457

each argument of whether it has been used (at all) during the

458

compilation. This bit usually doesn't need to be set by hand,

459

as the key ArgList accessors will set it automatically.</p>

460

Chris Lattner

715fc92

2009-04-19 02:02:45 +0000

[diff] [blame]

461

<p>When a compilation is successful (there are no errors), the

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

462

driver checks the bit and emits an "unused argument" warning for

463

any arguments which were never accessed. This is conservative

464

(the argument may not have been used to do what the user wanted)

465

but still catches the most obvious cases.</p>

466

467

468

<h3><a name="int_gcc_concepts">Relation to GCC Driver Concepts</a></h3>

469

470

471

<p>For those familiar with the gcc driver, this section provides

472

a brief overview of how things from the gcc driver map to the

clang driver.</p>

<ul>

<li>

<b>Driver Driver</b>

<p>The driver driver is fully integrated into the clang

479

driver. The driver simply constructs additional Actions to

480

bind the architecture during the <i>Pipeline</i>

481

phase. The tool chain specific argument translation is

482

responsible for handling <tt>-Xarch_</tt>.</p>

483

484

<p>The one caveat is that this approach

485

requires <tt>-Xarch_</tt> not be used to alter the

486

compilation itself (for example, one cannot

487

provide <tt>-S</tt> as an <tt>-Xarch_</tt> argument). The

488

driver attempts to reject such invocations, and overall

489

there isn't a good reason to abuse <tt>-Xarch_</tt> to

490

that end in practice.</p>

491

492

<p>The upside is that the clang driver is more efficient and

493

does little extra work to support universal builds. It also

494

provides better error reporting and UI consistency.</p>

</li>

<li>

<b>Specs</b>

Chris Lattner

fc8f0e1

2011-04-15 05:22:18 +0000

[diff] [blame]

499

<p>The clang driver has no direct correspondent for

Daniel Dunbar

2009-03-30 06:50:01 +0000

[diff] [blame]

500

"specs". The majority of the functionality that is

501

embedded in specs is in the Tool specific argument

502

translation routines. The parts of specs which control the

503

compilation pipeline are generally part of

Benjamin Kramer

2012-01-15 15:26:07 +0000

[diff] [blame]

504

the <i>Pipeline</i> stage.</p>

Daniel Dunbar