Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1 | ============================ |
| 2 | "Clang" CFE Internals Manual |
| 3 | ============================ |
| 4 | |
| 5 | .. contents:: |
| 6 | :local: |
| 7 | |
| 8 | Introduction |
| 9 | ============ |
| 10 | |
| 11 | This document describes some of the more important APIs and internal design |
| 12 | decisions made in the Clang C front-end. The purpose of this document is to |
| 13 | both capture some of this high level information and also describe some of the |
| 14 | design decisions behind it. This is meant for people interested in hacking on |
| 15 | Clang, not for end-users. The description below is categorized by libraries, |
| 16 | and does not describe any of the clients of the libraries. |
| 17 | |
| 18 | LLVM Support Library |
| 19 | ==================== |
| 20 | |
| 21 | The LLVM ``libSupport`` library provides many underlying libraries and |
| 22 | `data-structures <http://llvm.org/docs/ProgrammersManual.html>`_, including |
| 23 | command line option processing, various containers and a system abstraction |
| 24 | layer, which is used for file system access. |
| 25 | |
| 26 | The Clang "Basic" Library |
| 27 | ========================= |
| 28 | |
| 29 | This library certainly needs a better name. The "basic" library contains a |
| 30 | number of low-level utilities for tracking and manipulating source buffers, |
| 31 | locations within the source buffers, diagnostics, tokens, target abstraction, |
| 32 | and information about the subset of the language being compiled for. |
| 33 | |
| 34 | Part of this infrastructure is specific to C (such as the ``TargetInfo`` |
| 35 | class), other parts could be reused for other non-C-based languages |
| 36 | (``SourceLocation``, ``SourceManager``, ``Diagnostics``, ``FileManager``). |
| 37 | When and if there is future demand we can figure out if it makes sense to |
| 38 | introduce a new library, move the general classes somewhere else, or introduce |
| 39 | some other solution. |
| 40 | |
| 41 | We describe the roles of these classes in order of their dependencies. |
| 42 | |
| 43 | The Diagnostics Subsystem |
| 44 | ------------------------- |
| 45 | |
| 46 | The Clang Diagnostics subsystem is an important part of how the compiler |
| 47 | communicates with the human. Diagnostics are the warnings and errors produced |
| 48 | when the code is incorrect or dubious. In Clang, each diagnostic produced has |
| 49 | (at the minimum) a unique ID, an English translation associated with it, a |
| 50 | :ref:`SourceLocation <SourceLocation>` to "put the caret", and a severity |
| 51 | (e.g., ``WARNING`` or ``ERROR``). They can also optionally include a number of |
| 52 | arguments to the dianostic (which fill in "%0"'s in the string) as well as a |
| 53 | number of source ranges that related to the diagnostic. |
| 54 | |
| 55 | In this section, we'll be giving examples produced by the Clang command line |
| 56 | driver, but diagnostics can be :ref:`rendered in many different ways |
| 57 | <DiagnosticClient>` depending on how the ``DiagnosticClient`` interface is |
| 58 | implemented. A representative example of a diagnostic is: |
| 59 | |
| 60 | .. code-block:: c++ |
| 61 | |
| 62 | t.c:38:15: error: invalid operands to binary expression ('int *' and '_Complex float') |
| 63 | P = (P-42) + Gamma*4; |
| 64 | ~~~~~~ ^ ~~~~~~~ |
| 65 | |
| 66 | In this example, you can see the English translation, the severity (error), you |
| 67 | can see the source location (the caret ("``^``") and file/line/column info), |
| 68 | the source ranges "``~~~~``", arguments to the diagnostic ("``int*``" and |
| 69 | "``_Complex float``"). You'll have to believe me that there is a unique ID |
| 70 | backing the diagnostic :). |
| 71 | |
| 72 | Getting all of this to happen has several steps and involves many moving |
| 73 | pieces, this section describes them and talks about best practices when adding |
| 74 | a new diagnostic. |
| 75 | |
Dmitri Gribenko | 97555a1 | 2012-12-15 21:10:51 +0000 | [diff] [blame] | 76 | The ``Diagnostic*Kinds.td`` files |
| 77 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 78 | |
| 79 | Diagnostics are created by adding an entry to one of the |
| 80 | ``clang/Basic/Diagnostic*Kinds.td`` files, depending on what library will be |
| 81 | using it. From this file, :program:`tblgen` generates the unique ID of the |
| 82 | diagnostic, the severity of the diagnostic and the English translation + format |
| 83 | string. |
| 84 | |
| 85 | There is little sanity with the naming of the unique ID's right now. Some |
| 86 | start with ``err_``, ``warn_``, ``ext_`` to encode the severity into the name. |
| 87 | Since the enum is referenced in the C++ code that produces the diagnostic, it |
| 88 | is somewhat useful for it to be reasonably short. |
| 89 | |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 90 | The severity of the diagnostic comes from the set {``NOTE``, ``REMARK``, |
| 91 | ``WARNING``, |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 92 | ``EXTENSION``, ``EXTWARN``, ``ERROR``}. The ``ERROR`` severity is used for |
| 93 | diagnostics indicating the program is never acceptable under any circumstances. |
| 94 | When an error is emitted, the AST for the input code may not be fully built. |
| 95 | The ``EXTENSION`` and ``EXTWARN`` severities are used for extensions to the |
| 96 | language that Clang accepts. This means that Clang fully understands and can |
| 97 | represent them in the AST, but we produce diagnostics to tell the user their |
| 98 | code is non-portable. The difference is that the former are ignored by |
| 99 | default, and the later warn by default. The ``WARNING`` severity is used for |
| 100 | constructs that are valid in the currently selected source language but that |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 101 | are dubious in some way. The ``REMARK`` severity provides generic information |
| 102 | about the compilation that is not necessarily related to any dubious code. The |
| 103 | ``NOTE`` level is used to staple more information onto previous diagnostics. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 104 | |
| 105 | These *severities* are mapped into a smaller set (the ``Diagnostic::Level`` |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 106 | enum, {``Ignored``, ``Note``, ``Remark``, ``Warning``, ``Error``, ``Fatal``}) of |
| 107 | output |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 108 | *levels* by the diagnostics subsystem based on various configuration options. |
| 109 | Clang internally supports a fully fine grained mapping mechanism that allows |
| 110 | you to map almost any diagnostic to the output level that you want. The only |
| 111 | diagnostics that cannot be mapped are ``NOTE``\ s, which always follow the |
| 112 | severity of the previously emitted diagnostic and ``ERROR``\ s, which can only |
| 113 | be mapped to ``Fatal`` (it is not possible to turn an error into a warning, for |
| 114 | example). |
| 115 | |
| 116 | Diagnostic mappings are used in many ways. For example, if the user specifies |
| 117 | ``-pedantic``, ``EXTENSION`` maps to ``Warning``, if they specify |
| 118 | ``-pedantic-errors``, it turns into ``Error``. This is used to implement |
| 119 | options like ``-Wunused_macros``, ``-Wundef`` etc. |
| 120 | |
| 121 | Mapping to ``Fatal`` should only be used for diagnostics that are considered so |
| 122 | severe that error recovery won't be able to recover sensibly from them (thus |
| 123 | spewing a ton of bogus errors). One example of this class of error are failure |
| 124 | to ``#include`` a file. |
| 125 | |
| 126 | The Format String |
| 127 | ^^^^^^^^^^^^^^^^^ |
| 128 | |
| 129 | The format string for the diagnostic is very simple, but it has some power. It |
| 130 | takes the form of a string in English with markers that indicate where and how |
| 131 | arguments to the diagnostic are inserted and formatted. For example, here are |
| 132 | some simple format strings: |
| 133 | |
| 134 | .. code-block:: c++ |
| 135 | |
| 136 | "binary integer literals are an extension" |
| 137 | "format string contains '\\0' within the string body" |
| 138 | "more '%%' conversions than data arguments" |
| 139 | "invalid operands to binary expression (%0 and %1)" |
| 140 | "overloaded '%0' must be a %select{unary|binary|unary or binary}2 operator" |
| 141 | " (has %1 parameter%s1)" |
| 142 | |
| 143 | These examples show some important points of format strings. You can use any |
| 144 | plain ASCII character in the diagnostic string except "``%``" without a |
| 145 | problem, but these are C strings, so you have to use and be aware of all the C |
| 146 | escape sequences (as in the second example). If you want to produce a "``%``" |
| 147 | in the output, use the "``%%``" escape sequence, like the third diagnostic. |
| 148 | Finally, Clang uses the "``%...[digit]``" sequences to specify where and how |
| 149 | arguments to the diagnostic are formatted. |
| 150 | |
| 151 | Arguments to the diagnostic are numbered according to how they are specified by |
| 152 | the C++ code that :ref:`produces them <internals-producing-diag>`, and are |
| 153 | referenced by ``%0`` .. ``%9``. If you have more than 10 arguments to your |
| 154 | diagnostic, you are doing something wrong :). Unlike ``printf``, there is no |
| 155 | requirement that arguments to the diagnostic end up in the output in the same |
| 156 | order as they are specified, you could have a format string with "``%1 %0``" |
| 157 | that swaps them, for example. The text in between the percent and digit are |
| 158 | formatting instructions. If there are no instructions, the argument is just |
| 159 | turned into a string and substituted in. |
| 160 | |
| 161 | Here are some "best practices" for writing the English format string: |
| 162 | |
| 163 | * Keep the string short. It should ideally fit in the 80 column limit of the |
| 164 | ``DiagnosticKinds.td`` file. This avoids the diagnostic wrapping when |
| 165 | printed, and forces you to think about the important point you are conveying |
| 166 | with the diagnostic. |
| 167 | * Take advantage of location information. The user will be able to see the |
| 168 | line and location of the caret, so you don't need to tell them that the |
| 169 | problem is with the 4th argument to the function: just point to it. |
| 170 | * Do not capitalize the diagnostic string, and do not end it with a period. |
| 171 | * If you need to quote something in the diagnostic string, use single quotes. |
| 172 | |
| 173 | Diagnostics should never take random English strings as arguments: you |
| 174 | shouldn't use "``you have a problem with %0``" and pass in things like "``your |
| 175 | argument``" or "``your return value``" as arguments. Doing this prevents |
| 176 | :ref:`translating <internals-diag-translation>` the Clang diagnostics to other |
| 177 | languages (because they'll get random English words in their otherwise |
| 178 | localized diagnostic). The exceptions to this are C/C++ language keywords |
| 179 | (e.g., ``auto``, ``const``, ``mutable``, etc) and C/C++ operators (``/=``). |
| 180 | Note that things like "pointer" and "reference" are not keywords. On the other |
| 181 | hand, you *can* include anything that comes from the user's source code, |
| 182 | including variable names, types, labels, etc. The "``select``" format can be |
| 183 | used to achieve this sort of thing in a localizable way, see below. |
| 184 | |
| 185 | Formatting a Diagnostic Argument |
| 186 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 187 | |
| 188 | Arguments to diagnostics are fully typed internally, and come from a couple |
| 189 | different classes: integers, types, names, and random strings. Depending on |
| 190 | the class of the argument, it can be optionally formatted in different ways. |
| 191 | This gives the ``DiagnosticClient`` information about what the argument means |
| 192 | without requiring it to use a specific presentation (consider this MVC for |
| 193 | Clang :). |
| 194 | |
| 195 | Here are the different diagnostic argument formats currently supported by |
| 196 | Clang: |
| 197 | |
| 198 | **"s" format** |
| 199 | |
| 200 | Example: |
| 201 | ``"requires %1 parameter%s1"`` |
| 202 | Class: |
| 203 | Integers |
| 204 | Description: |
| 205 | This is a simple formatter for integers that is useful when producing English |
| 206 | diagnostics. When the integer is 1, it prints as nothing. When the integer |
| 207 | is not 1, it prints as "``s``". This allows some simple grammatical forms to |
| 208 | be to be handled correctly, and eliminates the need to use gross things like |
| 209 | ``"requires %1 parameter(s)"``. |
| 210 | |
| 211 | **"select" format** |
| 212 | |
| 213 | Example: |
| 214 | ``"must be a %select{unary|binary|unary or binary}2 operator"`` |
| 215 | Class: |
| 216 | Integers |
| 217 | Description: |
| 218 | This format specifier is used to merge multiple related diagnostics together |
| 219 | into one common one, without requiring the difference to be specified as an |
| 220 | English string argument. Instead of specifying the string, the diagnostic |
| 221 | gets an integer argument and the format string selects the numbered option. |
| 222 | In this case, the "``%2``" value must be an integer in the range [0..2]. If |
| 223 | it is 0, it prints "unary", if it is 1 it prints "binary" if it is 2, it |
| 224 | prints "unary or binary". This allows other language translations to |
| 225 | substitute reasonable words (or entire phrases) based on the semantics of the |
| 226 | diagnostic instead of having to do things textually. The selected string |
| 227 | does undergo formatting. |
| 228 | |
| 229 | **"plural" format** |
| 230 | |
| 231 | Example: |
| 232 | ``"you have %1 %plural{1:mouse|:mice}1 connected to your computer"`` |
| 233 | Class: |
| 234 | Integers |
| 235 | Description: |
| 236 | This is a formatter for complex plural forms. It is designed to handle even |
| 237 | the requirements of languages with very complex plural forms, as many Baltic |
| 238 | languages have. The argument consists of a series of expression/form pairs, |
| 239 | separated by ":", where the first form whose expression evaluates to true is |
| 240 | the result of the modifier. |
| 241 | |
| 242 | An expression can be empty, in which case it is always true. See the example |
| 243 | at the top. Otherwise, it is a series of one or more numeric conditions, |
| 244 | separated by ",". If any condition matches, the expression matches. Each |
| 245 | numeric condition can take one of three forms. |
| 246 | |
| 247 | * number: A simple decimal number matches if the argument is the same as the |
| 248 | number. Example: ``"%plural{1:mouse|:mice}4"`` |
| 249 | * range: A range in square brackets matches if the argument is within the |
| 250 | range. Then range is inclusive on both ends. Example: |
| 251 | ``"%plural{0:none|1:one|[2,5]:some|:many}2"`` |
| 252 | * modulo: A modulo operator is followed by a number, and equals sign and |
| 253 | either a number or a range. The tests are the same as for plain numbers |
| 254 | and ranges, but the argument is taken modulo the number first. Example: |
| 255 | ``"%plural{%100=0:even hundred|%100=[1,50]:lower half|:everything else}1"`` |
| 256 | |
| 257 | The parser is very unforgiving. A syntax error, even whitespace, will abort, |
| 258 | as will a failure to match the argument against any expression. |
| 259 | |
| 260 | **"ordinal" format** |
| 261 | |
| 262 | Example: |
| 263 | ``"ambiguity in %ordinal0 argument"`` |
| 264 | Class: |
| 265 | Integers |
| 266 | Description: |
| 267 | This is a formatter which represents the argument number as an ordinal: the |
| 268 | value ``1`` becomes ``1st``, ``3`` becomes ``3rd``, and so on. Values less |
| 269 | than ``1`` are not supported. This formatter is currently hard-coded to use |
| 270 | English ordinals. |
| 271 | |
| 272 | **"objcclass" format** |
| 273 | |
| 274 | Example: |
| 275 | ``"method %objcclass0 not found"`` |
| 276 | Class: |
| 277 | ``DeclarationName`` |
| 278 | Description: |
| 279 | This is a simple formatter that indicates the ``DeclarationName`` corresponds |
| 280 | to an Objective-C class method selector. As such, it prints the selector |
| 281 | with a leading "``+``". |
| 282 | |
| 283 | **"objcinstance" format** |
| 284 | |
| 285 | Example: |
| 286 | ``"method %objcinstance0 not found"`` |
| 287 | Class: |
| 288 | ``DeclarationName`` |
| 289 | Description: |
| 290 | This is a simple formatter that indicates the ``DeclarationName`` corresponds |
| 291 | to an Objective-C instance method selector. As such, it prints the selector |
| 292 | with a leading "``-``". |
| 293 | |
| 294 | **"q" format** |
| 295 | |
| 296 | Example: |
| 297 | ``"candidate found by name lookup is %q0"`` |
| 298 | Class: |
| 299 | ``NamedDecl *`` |
| 300 | Description: |
| 301 | This formatter indicates that the fully-qualified name of the declaration |
| 302 | should be printed, e.g., "``std::vector``" rather than "``vector``". |
| 303 | |
| 304 | **"diff" format** |
| 305 | |
| 306 | Example: |
| 307 | ``"no known conversion %diff{from $ to $|from argument type to parameter type}1,2"`` |
| 308 | Class: |
| 309 | ``QualType`` |
| 310 | Description: |
| 311 | This formatter takes two ``QualType``\ s and attempts to print a template |
| 312 | difference between the two. If tree printing is off, the text inside the |
| 313 | braces before the pipe is printed, with the formatted text replacing the $. |
| 314 | If tree printing is on, the text after the pipe is printed and a type tree is |
| 315 | printed after the diagnostic message. |
| 316 | |
| 317 | It is really easy to add format specifiers to the Clang diagnostics system, but |
| 318 | they should be discussed before they are added. If you are creating a lot of |
| 319 | repetitive diagnostics and/or have an idea for a useful formatter, please bring |
| 320 | it up on the cfe-dev mailing list. |
| 321 | |
| 322 | .. _internals-producing-diag: |
| 323 | |
| 324 | Producing the Diagnostic |
| 325 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
| 326 | |
| 327 | Now that you've created the diagnostic in the ``Diagnostic*Kinds.td`` file, you |
| 328 | need to write the code that detects the condition in question and emits the new |
| 329 | diagnostic. Various components of Clang (e.g., the preprocessor, ``Sema``, |
| 330 | etc.) provide a helper function named "``Diag``". It creates a diagnostic and |
| 331 | accepts the arguments, ranges, and other information that goes along with it. |
| 332 | |
| 333 | For example, the binary expression error comes from code like this: |
| 334 | |
| 335 | .. code-block:: c++ |
| 336 | |
| 337 | if (various things that are bad) |
| 338 | Diag(Loc, diag::err_typecheck_invalid_operands) |
| 339 | << lex->getType() << rex->getType() |
| 340 | << lex->getSourceRange() << rex->getSourceRange(); |
| 341 | |
| 342 | This shows that use of the ``Diag`` method: it takes a location (a |
| 343 | :ref:`SourceLocation <SourceLocation>` object) and a diagnostic enum value |
| 344 | (which matches the name from ``Diagnostic*Kinds.td``). If the diagnostic takes |
| 345 | arguments, they are specified with the ``<<`` operator: the first argument |
| 346 | becomes ``%0``, the second becomes ``%1``, etc. The diagnostic interface |
| 347 | allows you to specify arguments of many different types, including ``int`` and |
| 348 | ``unsigned`` for integer arguments, ``const char*`` and ``std::string`` for |
| 349 | string arguments, ``DeclarationName`` and ``const IdentifierInfo *`` for names, |
| 350 | ``QualType`` for types, etc. ``SourceRange``\ s are also specified with the |
| 351 | ``<<`` operator, but do not have a specific ordering requirement. |
| 352 | |
| 353 | As you can see, adding and producing a diagnostic is pretty straightforward. |
| 354 | The hard part is deciding exactly what you need to say to help the user, |
| 355 | picking a suitable wording, and providing the information needed to format it |
| 356 | correctly. The good news is that the call site that issues a diagnostic should |
| 357 | be completely independent of how the diagnostic is formatted and in what |
| 358 | language it is rendered. |
| 359 | |
| 360 | Fix-It Hints |
| 361 | ^^^^^^^^^^^^ |
| 362 | |
| 363 | In some cases, the front end emits diagnostics when it is clear that some small |
| 364 | change to the source code would fix the problem. For example, a missing |
| 365 | semicolon at the end of a statement or a use of deprecated syntax that is |
| 366 | easily rewritten into a more modern form. Clang tries very hard to emit the |
| 367 | diagnostic and recover gracefully in these and other cases. |
| 368 | |
| 369 | However, for these cases where the fix is obvious, the diagnostic can be |
| 370 | annotated with a hint (referred to as a "fix-it hint") that describes how to |
| 371 | change the code referenced by the diagnostic to fix the problem. For example, |
| 372 | it might add the missing semicolon at the end of the statement or rewrite the |
| 373 | use of a deprecated construct into something more palatable. Here is one such |
| 374 | example from the C++ front end, where we warn about the right-shift operator |
| 375 | changing meaning from C++98 to C++11: |
| 376 | |
| 377 | .. code-block:: c++ |
| 378 | |
| 379 | test.cpp:3:7: warning: use of right-shift operator ('>>') in template argument |
| 380 | will require parentheses in C++11 |
| 381 | A<100 >> 2> *a; |
| 382 | ^ |
| 383 | ( ) |
| 384 | |
| 385 | Here, the fix-it hint is suggesting that parentheses be added, and showing |
| 386 | exactly where those parentheses would be inserted into the source code. The |
| 387 | fix-it hints themselves describe what changes to make to the source code in an |
| 388 | abstract manner, which the text diagnostic printer renders as a line of |
| 389 | "insertions" below the caret line. :ref:`Other diagnostic clients |
| 390 | <DiagnosticClient>` might choose to render the code differently (e.g., as |
| 391 | markup inline) or even give the user the ability to automatically fix the |
| 392 | problem. |
| 393 | |
| 394 | Fix-it hints on errors and warnings need to obey these rules: |
| 395 | |
| 396 | * Since they are automatically applied if ``-Xclang -fixit`` is passed to the |
| 397 | driver, they should only be used when it's very likely they match the user's |
| 398 | intent. |
| 399 | * Clang must recover from errors as if the fix-it had been applied. |
| 400 | |
| 401 | If a fix-it can't obey these rules, put the fix-it on a note. Fix-its on notes |
| 402 | are not applied automatically. |
| 403 | |
| 404 | All fix-it hints are described by the ``FixItHint`` class, instances of which |
| 405 | should be attached to the diagnostic using the ``<<`` operator in the same way |
| 406 | that highlighted source ranges and arguments are passed to the diagnostic. |
| 407 | Fix-it hints can be created with one of three constructors: |
| 408 | |
| 409 | * ``FixItHint::CreateInsertion(Loc, Code)`` |
| 410 | |
| 411 | Specifies that the given ``Code`` (a string) should be inserted before the |
| 412 | source location ``Loc``. |
| 413 | |
| 414 | * ``FixItHint::CreateRemoval(Range)`` |
| 415 | |
| 416 | Specifies that the code in the given source ``Range`` should be removed. |
| 417 | |
| 418 | * ``FixItHint::CreateReplacement(Range, Code)`` |
| 419 | |
| 420 | Specifies that the code in the given source ``Range`` should be removed, |
| 421 | and replaced with the given ``Code`` string. |
| 422 | |
| 423 | .. _DiagnosticClient: |
| 424 | |
| 425 | The ``DiagnosticClient`` Interface |
| 426 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 427 | |
| 428 | Once code generates a diagnostic with all of the arguments and the rest of the |
| 429 | relevant information, Clang needs to know what to do with it. As previously |
| 430 | mentioned, the diagnostic machinery goes through some filtering to map a |
| 431 | severity onto a diagnostic level, then (assuming the diagnostic is not mapped |
| 432 | to "``Ignore``") it invokes an object that implements the ``DiagnosticClient`` |
| 433 | interface with the information. |
| 434 | |
| 435 | It is possible to implement this interface in many different ways. For |
| 436 | example, the normal Clang ``DiagnosticClient`` (named |
| 437 | ``TextDiagnosticPrinter``) turns the arguments into strings (according to the |
| 438 | various formatting rules), prints out the file/line/column information and the |
| 439 | string, then prints out the line of code, the source ranges, and the caret. |
| 440 | However, this behavior isn't required. |
| 441 | |
| 442 | Another implementation of the ``DiagnosticClient`` interface is the |
| 443 | ``TextDiagnosticBuffer`` class, which is used when Clang is in ``-verify`` |
| 444 | mode. Instead of formatting and printing out the diagnostics, this |
| 445 | implementation just captures and remembers the diagnostics as they fly by. |
| 446 | Then ``-verify`` compares the list of produced diagnostics to the list of |
| 447 | expected ones. If they disagree, it prints out its own output. Full |
| 448 | documentation for the ``-verify`` mode can be found in the Clang API |
| 449 | documentation for `VerifyDiagnosticConsumer |
| 450 | </doxygen/classclang_1_1VerifyDiagnosticConsumer.html#details>`_. |
| 451 | |
| 452 | There are many other possible implementations of this interface, and this is |
| 453 | why we prefer diagnostics to pass down rich structured information in |
| 454 | arguments. For example, an HTML output might want declaration names be |
| 455 | linkified to where they come from in the source. Another example is that a GUI |
| 456 | might let you click on typedefs to expand them. This application would want to |
| 457 | pass significantly more information about types through to the GUI than a |
| 458 | simple flat string. The interface allows this to happen. |
| 459 | |
| 460 | .. _internals-diag-translation: |
| 461 | |
| 462 | Adding Translations to Clang |
| 463 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 464 | |
| 465 | Not possible yet! Diagnostic strings should be written in UTF-8, the client can |
| 466 | translate to the relevant code page if needed. Each translation completely |
| 467 | replaces the format string for the diagnostic. |
| 468 | |
| 469 | .. _SourceLocation: |
| 470 | .. _SourceManager: |
| 471 | |
| 472 | The ``SourceLocation`` and ``SourceManager`` classes |
| 473 | ---------------------------------------------------- |
| 474 | |
| 475 | Strangely enough, the ``SourceLocation`` class represents a location within the |
| 476 | source code of the program. Important design points include: |
| 477 | |
| 478 | #. ``sizeof(SourceLocation)`` must be extremely small, as these are embedded |
| 479 | into many AST nodes and are passed around often. Currently it is 32 bits. |
| 480 | #. ``SourceLocation`` must be a simple value object that can be efficiently |
| 481 | copied. |
| 482 | #. We should be able to represent a source location for any byte of any input |
| 483 | file. This includes in the middle of tokens, in whitespace, in trigraphs, |
| 484 | etc. |
| 485 | #. A ``SourceLocation`` must encode the current ``#include`` stack that was |
| 486 | active when the location was processed. For example, if the location |
| 487 | corresponds to a token, it should contain the set of ``#include``\ s active |
| 488 | when the token was lexed. This allows us to print the ``#include`` stack |
| 489 | for a diagnostic. |
| 490 | #. ``SourceLocation`` must be able to describe macro expansions, capturing both |
| 491 | the ultimate instantiation point and the source of the original character |
| 492 | data. |
| 493 | |
| 494 | In practice, the ``SourceLocation`` works together with the ``SourceManager`` |
| 495 | class to encode two pieces of information about a location: its spelling |
| 496 | location and its instantiation location. For most tokens, these will be the |
| 497 | same. However, for a macro expansion (or tokens that came from a ``_Pragma`` |
| 498 | directive) these will describe the location of the characters corresponding to |
| 499 | the token and the location where the token was used (i.e., the macro |
| 500 | instantiation point or the location of the ``_Pragma`` itself). |
| 501 | |
| 502 | The Clang front-end inherently depends on the location of a token being tracked |
| 503 | correctly. If it is ever incorrect, the front-end may get confused and die. |
| 504 | The reason for this is that the notion of the "spelling" of a ``Token`` in |
| 505 | Clang depends on being able to find the original input characters for the |
| 506 | token. This concept maps directly to the "spelling location" for the token. |
| 507 | |
| 508 | ``SourceRange`` and ``CharSourceRange`` |
| 509 | --------------------------------------- |
| 510 | |
| 511 | .. mostly taken from http://lists.cs.uiuc.edu/pipermail/cfe-dev/2010-August/010595.html |
| 512 | |
| 513 | Clang represents most source ranges by [first, last], where "first" and "last" |
| 514 | each point to the beginning of their respective tokens. For example consider |
| 515 | the ``SourceRange`` of the following statement: |
| 516 | |
| 517 | .. code-block:: c++ |
| 518 | |
| 519 | x = foo + bar; |
| 520 | ^first ^last |
| 521 | |
| 522 | To map from this representation to a character-based representation, the "last" |
| 523 | location needs to be adjusted to point to (or past) the end of that token with |
| 524 | either ``Lexer::MeasureTokenLength()`` or ``Lexer::getLocForEndOfToken()``. For |
| 525 | the rare cases where character-level source ranges information is needed we use |
| 526 | the ``CharSourceRange`` class. |
| 527 | |
| 528 | The Driver Library |
| 529 | ================== |
| 530 | |
Sean Silva | 159cc9e | 2013-01-02 13:07:47 +0000 | [diff] [blame] | 531 | The clang Driver and library are documented :doc:`here <DriverInternals>`. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 532 | |
| 533 | Precompiled Headers |
| 534 | =================== |
| 535 | |
| 536 | Clang supports two implementations of precompiled headers. The default |
| 537 | implementation, precompiled headers (:doc:`PCH <PCHInternals>`) uses a |
| 538 | serialized representation of Clang's internal data structures, encoded with the |
| 539 | `LLVM bitstream format <http://llvm.org/docs/BitCodeFormat.html>`_. |
Sean Silva | 159cc9e | 2013-01-02 13:07:47 +0000 | [diff] [blame] | 540 | Pretokenized headers (:doc:`PTH <PTHInternals>`), on the other hand, contain a |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 541 | serialized representation of the tokens encountered when preprocessing a header |
| 542 | (and anything that header includes). |
| 543 | |
| 544 | The Frontend Library |
| 545 | ==================== |
| 546 | |
| 547 | The Frontend library contains functionality useful for building tools on top of |
| 548 | the Clang libraries, for example several methods for outputting diagnostics. |
| 549 | |
| 550 | The Lexer and Preprocessor Library |
| 551 | ================================== |
| 552 | |
| 553 | The Lexer library contains several tightly-connected classes that are involved |
| 554 | with the nasty process of lexing and preprocessing C source code. The main |
| 555 | interface to this library for outside clients is the large ``Preprocessor`` |
| 556 | class. It contains the various pieces of state that are required to coherently |
| 557 | read tokens out of a translation unit. |
| 558 | |
| 559 | The core interface to the ``Preprocessor`` object (once it is set up) is the |
| 560 | ``Preprocessor::Lex`` method, which returns the next :ref:`Token <Token>` from |
| 561 | the preprocessor stream. There are two types of token providers that the |
| 562 | preprocessor is capable of reading from: a buffer lexer (provided by the |
| 563 | :ref:`Lexer <Lexer>` class) and a buffered token stream (provided by the |
| 564 | :ref:`TokenLexer <TokenLexer>` class). |
| 565 | |
| 566 | .. _Token: |
| 567 | |
| 568 | The Token class |
| 569 | --------------- |
| 570 | |
| 571 | The ``Token`` class is used to represent a single lexed token. Tokens are |
| 572 | intended to be used by the lexer/preprocess and parser libraries, but are not |
| 573 | intended to live beyond them (for example, they should not live in the ASTs). |
| 574 | |
| 575 | Tokens most often live on the stack (or some other location that is efficient |
| 576 | to access) as the parser is running, but occasionally do get buffered up. For |
| 577 | example, macro definitions are stored as a series of tokens, and the C++ |
| 578 | front-end periodically needs to buffer tokens up for tentative parsing and |
| 579 | various pieces of look-ahead. As such, the size of a ``Token`` matters. On a |
| 580 | 32-bit system, ``sizeof(Token)`` is currently 16 bytes. |
| 581 | |
| 582 | Tokens occur in two forms: :ref:`annotation tokens <AnnotationToken>` and |
| 583 | normal tokens. Normal tokens are those returned by the lexer, annotation |
| 584 | tokens represent semantic information and are produced by the parser, replacing |
| 585 | normal tokens in the token stream. Normal tokens contain the following |
| 586 | information: |
| 587 | |
| 588 | * **A SourceLocation** --- This indicates the location of the start of the |
| 589 | token. |
| 590 | |
| 591 | * **A length** --- This stores the length of the token as stored in the |
| 592 | ``SourceBuffer``. For tokens that include them, this length includes |
| 593 | trigraphs and escaped newlines which are ignored by later phases of the |
| 594 | compiler. By pointing into the original source buffer, it is always possible |
| 595 | to get the original spelling of a token completely accurately. |
| 596 | |
| 597 | * **IdentifierInfo** --- If a token takes the form of an identifier, and if |
| 598 | identifier lookup was enabled when the token was lexed (e.g., the lexer was |
| 599 | not reading in "raw" mode) this contains a pointer to the unique hash value |
| 600 | for the identifier. Because the lookup happens before keyword |
| 601 | identification, this field is set even for language keywords like "``for``". |
| 602 | |
| 603 | * **TokenKind** --- This indicates the kind of token as classified by the |
| 604 | lexer. This includes things like ``tok::starequal`` (for the "``*=``" |
| 605 | operator), ``tok::ampamp`` for the "``&&``" token, and keyword values (e.g., |
| 606 | ``tok::kw_for``) for identifiers that correspond to keywords. Note that |
| 607 | some tokens can be spelled multiple ways. For example, C++ supports |
| 608 | "operator keywords", where things like "``and``" are treated exactly like the |
| 609 | "``&&``" operator. In these cases, the kind value is set to ``tok::ampamp``, |
| 610 | which is good for the parser, which doesn't have to consider both forms. For |
| 611 | something that cares about which form is used (e.g., the preprocessor |
| 612 | "stringize" operator) the spelling indicates the original form. |
| 613 | |
| 614 | * **Flags** --- There are currently four flags tracked by the |
| 615 | lexer/preprocessor system on a per-token basis: |
| 616 | |
| 617 | #. **StartOfLine** --- This was the first token that occurred on its input |
| 618 | source line. |
| 619 | #. **LeadingSpace** --- There was a space character either immediately before |
| 620 | the token or transitively before the token as it was expanded through a |
| 621 | macro. The definition of this flag is very closely defined by the |
| 622 | stringizing requirements of the preprocessor. |
| 623 | #. **DisableExpand** --- This flag is used internally to the preprocessor to |
| 624 | represent identifier tokens which have macro expansion disabled. This |
| 625 | prevents them from being considered as candidates for macro expansion ever |
| 626 | in the future. |
| 627 | #. **NeedsCleaning** --- This flag is set if the original spelling for the |
| 628 | token includes a trigraph or escaped newline. Since this is uncommon, |
| 629 | many pieces of code can fast-path on tokens that did not need cleaning. |
| 630 | |
| 631 | One interesting (and somewhat unusual) aspect of normal tokens is that they |
| 632 | don't contain any semantic information about the lexed value. For example, if |
| 633 | the token was a pp-number token, we do not represent the value of the number |
| 634 | that was lexed (this is left for later pieces of code to decide). |
| 635 | Additionally, the lexer library has no notion of typedef names vs variable |
| 636 | names: both are returned as identifiers, and the parser is left to decide |
| 637 | whether a specific identifier is a typedef or a variable (tracking this |
| 638 | requires scope information among other things). The parser can do this |
| 639 | translation by replacing tokens returned by the preprocessor with "Annotation |
| 640 | Tokens". |
| 641 | |
| 642 | .. _AnnotationToken: |
| 643 | |
| 644 | Annotation Tokens |
| 645 | ----------------- |
| 646 | |
| 647 | Annotation tokens are tokens that are synthesized by the parser and injected |
| 648 | into the preprocessor's token stream (replacing existing tokens) to record |
| 649 | semantic information found by the parser. For example, if "``foo``" is found |
| 650 | to be a typedef, the "``foo``" ``tok::identifier`` token is replaced with an |
| 651 | ``tok::annot_typename``. This is useful for a couple of reasons: 1) this makes |
| 652 | it easy to handle qualified type names (e.g., "``foo::bar::baz<42>::t``") in |
| 653 | C++ as a single "token" in the parser. 2) if the parser backtracks, the |
| 654 | reparse does not need to redo semantic analysis to determine whether a token |
| 655 | sequence is a variable, type, template, etc. |
| 656 | |
| 657 | Annotation tokens are created by the parser and reinjected into the parser's |
| 658 | token stream (when backtracking is enabled). Because they can only exist in |
| 659 | tokens that the preprocessor-proper is done with, it doesn't need to keep |
| 660 | around flags like "start of line" that the preprocessor uses to do its job. |
| 661 | Additionally, an annotation token may "cover" a sequence of preprocessor tokens |
| 662 | (e.g., "``a::b::c``" is five preprocessor tokens). As such, the valid fields |
| 663 | of an annotation token are different than the fields for a normal token (but |
| 664 | they are multiplexed into the normal ``Token`` fields): |
| 665 | |
| 666 | * **SourceLocation "Location"** --- The ``SourceLocation`` for the annotation |
| 667 | token indicates the first token replaced by the annotation token. In the |
| 668 | example above, it would be the location of the "``a``" identifier. |
| 669 | * **SourceLocation "AnnotationEndLoc"** --- This holds the location of the last |
| 670 | token replaced with the annotation token. In the example above, it would be |
| 671 | the location of the "``c``" identifier. |
| 672 | * **void* "AnnotationValue"** --- This contains an opaque object that the |
| 673 | parser gets from ``Sema``. The parser merely preserves the information for |
| 674 | ``Sema`` to later interpret based on the annotation token kind. |
| 675 | * **TokenKind "Kind"** --- This indicates the kind of Annotation token this is. |
| 676 | See below for the different valid kinds. |
| 677 | |
| 678 | Annotation tokens currently come in three kinds: |
| 679 | |
| 680 | #. **tok::annot_typename**: This annotation token represents a resolved |
| 681 | typename token that is potentially qualified. The ``AnnotationValue`` field |
| 682 | contains the ``QualType`` returned by ``Sema::getTypeName()``, possibly with |
| 683 | source location information attached. |
| 684 | #. **tok::annot_cxxscope**: This annotation token represents a C++ scope |
| 685 | specifier, such as "``A::B::``". This corresponds to the grammar |
| 686 | productions "*::*" and "*:: [opt] nested-name-specifier*". The |
| 687 | ``AnnotationValue`` pointer is a ``NestedNameSpecifier *`` returned by the |
| 688 | ``Sema::ActOnCXXGlobalScopeSpecifier`` and |
| 689 | ``Sema::ActOnCXXNestedNameSpecifier`` callbacks. |
| 690 | #. **tok::annot_template_id**: This annotation token represents a C++ |
| 691 | template-id such as "``foo<int, 4>``", where "``foo``" is the name of a |
| 692 | template. The ``AnnotationValue`` pointer is a pointer to a ``malloc``'d |
| 693 | ``TemplateIdAnnotation`` object. Depending on the context, a parsed |
| 694 | template-id that names a type might become a typename annotation token (if |
| 695 | all we care about is the named type, e.g., because it occurs in a type |
| 696 | specifier) or might remain a template-id token (if we want to retain more |
| 697 | source location information or produce a new type, e.g., in a declaration of |
| 698 | a class template specialization). template-id annotation tokens that refer |
| 699 | to a type can be "upgraded" to typename annotation tokens by the parser. |
| 700 | |
| 701 | As mentioned above, annotation tokens are not returned by the preprocessor, |
| 702 | they are formed on demand by the parser. This means that the parser has to be |
| 703 | aware of cases where an annotation could occur and form it where appropriate. |
| 704 | This is somewhat similar to how the parser handles Translation Phase 6 of C99: |
| 705 | String Concatenation (see C99 5.1.1.2). In the case of string concatenation, |
| 706 | the preprocessor just returns distinct ``tok::string_literal`` and |
| 707 | ``tok::wide_string_literal`` tokens and the parser eats a sequence of them |
| 708 | wherever the grammar indicates that a string literal can occur. |
| 709 | |
| 710 | In order to do this, whenever the parser expects a ``tok::identifier`` or |
| 711 | ``tok::coloncolon``, it should call the ``TryAnnotateTypeOrScopeToken`` or |
| 712 | ``TryAnnotateCXXScopeToken`` methods to form the annotation token. These |
| 713 | methods will maximally form the specified annotation tokens and replace the |
| 714 | current token with them, if applicable. If the current tokens is not valid for |
| 715 | an annotation token, it will remain an identifier or "``::``" token. |
| 716 | |
| 717 | .. _Lexer: |
| 718 | |
| 719 | The ``Lexer`` class |
| 720 | ------------------- |
| 721 | |
| 722 | The ``Lexer`` class provides the mechanics of lexing tokens out of a source |
| 723 | buffer and deciding what they mean. The ``Lexer`` is complicated by the fact |
| 724 | that it operates on raw buffers that have not had spelling eliminated (this is |
| 725 | a necessity to get decent performance), but this is countered with careful |
| 726 | coding as well as standard performance techniques (for example, the comment |
| 727 | handling code is vectorized on X86 and PowerPC hosts). |
| 728 | |
| 729 | The lexer has a couple of interesting modal features: |
| 730 | |
| 731 | * The lexer can operate in "raw" mode. This mode has several features that |
| 732 | make it possible to quickly lex the file (e.g., it stops identifier lookup, |
| 733 | doesn't specially handle preprocessor tokens, handles EOF differently, etc). |
| 734 | This mode is used for lexing within an "``#if 0``" block, for example. |
| 735 | * The lexer can capture and return comments as tokens. This is required to |
| 736 | support the ``-C`` preprocessor mode, which passes comments through, and is |
| 737 | used by the diagnostic checker to identifier expect-error annotations. |
| 738 | * The lexer can be in ``ParsingFilename`` mode, which happens when |
| 739 | preprocessing after reading a ``#include`` directive. This mode changes the |
| 740 | parsing of "``<``" to return an "angled string" instead of a bunch of tokens |
| 741 | for each thing within the filename. |
| 742 | * When parsing a preprocessor directive (after "``#``") the |
| 743 | ``ParsingPreprocessorDirective`` mode is entered. This changes the parser to |
| 744 | return EOD at a newline. |
| 745 | * The ``Lexer`` uses a ``LangOptions`` object to know whether trigraphs are |
| 746 | enabled, whether C++ or ObjC keywords are recognized, etc. |
| 747 | |
| 748 | In addition to these modes, the lexer keeps track of a couple of other features |
| 749 | that are local to a lexed buffer, which change as the buffer is lexed: |
| 750 | |
| 751 | * The ``Lexer`` uses ``BufferPtr`` to keep track of the current character being |
| 752 | lexed. |
| 753 | * The ``Lexer`` uses ``IsAtStartOfLine`` to keep track of whether the next |
| 754 | lexed token will start with its "start of line" bit set. |
| 755 | * The ``Lexer`` keeps track of the current "``#if``" directives that are active |
| 756 | (which can be nested). |
| 757 | * The ``Lexer`` keeps track of an :ref:`MultipleIncludeOpt |
| 758 | <MultipleIncludeOpt>` object, which is used to detect whether the buffer uses |
| 759 | the standard "``#ifndef XX`` / ``#define XX``" idiom to prevent multiple |
| 760 | inclusion. If a buffer does, subsequent includes can be ignored if the |
| 761 | "``XX``" macro is defined. |
| 762 | |
| 763 | .. _TokenLexer: |
| 764 | |
| 765 | The ``TokenLexer`` class |
| 766 | ------------------------ |
| 767 | |
| 768 | The ``TokenLexer`` class is a token provider that returns tokens from a list of |
| 769 | tokens that came from somewhere else. It typically used for two things: 1) |
| 770 | returning tokens from a macro definition as it is being expanded 2) returning |
| 771 | tokens from an arbitrary buffer of tokens. The later use is used by |
| 772 | ``_Pragma`` and will most likely be used to handle unbounded look-ahead for the |
| 773 | C++ parser. |
| 774 | |
| 775 | .. _MultipleIncludeOpt: |
| 776 | |
| 777 | The ``MultipleIncludeOpt`` class |
| 778 | -------------------------------- |
| 779 | |
| 780 | The ``MultipleIncludeOpt`` class implements a really simple little state |
| 781 | machine that is used to detect the standard "``#ifndef XX`` / ``#define XX``" |
| 782 | idiom that people typically use to prevent multiple inclusion of headers. If a |
| 783 | buffer uses this idiom and is subsequently ``#include``'d, the preprocessor can |
| 784 | simply check to see whether the guarding condition is defined or not. If so, |
| 785 | the preprocessor can completely ignore the include of the header. |
| 786 | |
| 787 | The Parser Library |
| 788 | ================== |
| 789 | |
| 790 | The AST Library |
| 791 | =============== |
| 792 | |
| 793 | .. _Type: |
| 794 | |
| 795 | The ``Type`` class and its subclasses |
| 796 | ------------------------------------- |
| 797 | |
| 798 | The ``Type`` class (and its subclasses) are an important part of the AST. |
| 799 | Types are accessed through the ``ASTContext`` class, which implicitly creates |
| 800 | and uniques them as they are needed. Types have a couple of non-obvious |
| 801 | features: 1) they do not capture type qualifiers like ``const`` or ``volatile`` |
| 802 | (see :ref:`QualType <QualType>`), and 2) they implicitly capture typedef |
| 803 | information. Once created, types are immutable (unlike decls). |
| 804 | |
| 805 | Typedefs in C make semantic analysis a bit more complex than it would be without |
| 806 | them. The issue is that we want to capture typedef information and represent it |
| 807 | in the AST perfectly, but the semantics of operations need to "see through" |
| 808 | typedefs. For example, consider this code: |
| 809 | |
| 810 | .. code-block:: c++ |
| 811 | |
| 812 | void func() { |
| 813 | typedef int foo; |
| 814 | foo X, *Y; |
| 815 | typedef foo *bar; |
| 816 | bar Z; |
| 817 | *X; // error |
| 818 | **Y; // error |
| 819 | **Z; // error |
| 820 | } |
| 821 | |
| 822 | The code above is illegal, and thus we expect there to be diagnostics emitted |
| 823 | on the annotated lines. In this example, we expect to get: |
| 824 | |
| 825 | .. code-block:: c++ |
| 826 | |
| 827 | test.c:6:1: error: indirection requires pointer operand ('foo' invalid) |
| 828 | *X; // error |
| 829 | ^~ |
| 830 | test.c:7:1: error: indirection requires pointer operand ('foo' invalid) |
| 831 | **Y; // error |
| 832 | ^~~ |
| 833 | test.c:8:1: error: indirection requires pointer operand ('foo' invalid) |
| 834 | **Z; // error |
| 835 | ^~~ |
| 836 | |
| 837 | While this example is somewhat silly, it illustrates the point: we want to |
| 838 | retain typedef information where possible, so that we can emit errors about |
| 839 | "``std::string``" instead of "``std::basic_string<char, std:...``". Doing this |
| 840 | requires properly keeping typedef information (for example, the type of ``X`` |
| 841 | is "``foo``", not "``int``"), and requires properly propagating it through the |
| 842 | various operators (for example, the type of ``*Y`` is "``foo``", not |
| 843 | "``int``"). In order to retain this information, the type of these expressions |
| 844 | is an instance of the ``TypedefType`` class, which indicates that the type of |
| 845 | these expressions is a typedef for "``foo``". |
| 846 | |
| 847 | Representing types like this is great for diagnostics, because the |
| 848 | user-specified type is always immediately available. There are two problems |
| 849 | with this: first, various semantic checks need to make judgements about the |
| 850 | *actual structure* of a type, ignoring typedefs. Second, we need an efficient |
| 851 | way to query whether two types are structurally identical to each other, |
| 852 | ignoring typedefs. The solution to both of these problems is the idea of |
| 853 | canonical types. |
| 854 | |
| 855 | Canonical Types |
| 856 | ^^^^^^^^^^^^^^^ |
| 857 | |
| 858 | Every instance of the ``Type`` class contains a canonical type pointer. For |
| 859 | simple types with no typedefs involved (e.g., "``int``", "``int*``", |
| 860 | "``int**``"), the type just points to itself. For types that have a typedef |
| 861 | somewhere in their structure (e.g., "``foo``", "``foo*``", "``foo**``", |
| 862 | "``bar``"), the canonical type pointer points to their structurally equivalent |
| 863 | type without any typedefs (e.g., "``int``", "``int*``", "``int**``", and |
| 864 | "``int*``" respectively). |
| 865 | |
| 866 | This design provides a constant time operation (dereferencing the canonical type |
| 867 | pointer) that gives us access to the structure of types. For example, we can |
| 868 | trivially tell that "``bar``" and "``foo*``" are the same type by dereferencing |
| 869 | their canonical type pointers and doing a pointer comparison (they both point |
| 870 | to the single "``int*``" type). |
| 871 | |
| 872 | Canonical types and typedef types bring up some complexities that must be |
| 873 | carefully managed. Specifically, the ``isa``/``cast``/``dyn_cast`` operators |
| 874 | generally shouldn't be used in code that is inspecting the AST. For example, |
| 875 | when type checking the indirection operator (unary "``*``" on a pointer), the |
| 876 | type checker must verify that the operand has a pointer type. It would not be |
| 877 | correct to check that with "``isa<PointerType>(SubExpr->getType())``", because |
| 878 | this predicate would fail if the subexpression had a typedef type. |
| 879 | |
| 880 | The solution to this problem are a set of helper methods on ``Type``, used to |
| 881 | check their properties. In this case, it would be correct to use |
| 882 | "``SubExpr->getType()->isPointerType()``" to do the check. This predicate will |
| 883 | return true if the *canonical type is a pointer*, which is true any time the |
| 884 | type is structurally a pointer type. The only hard part here is remembering |
| 885 | not to use the ``isa``/``cast``/``dyn_cast`` operations. |
| 886 | |
| 887 | The second problem we face is how to get access to the pointer type once we |
| 888 | know it exists. To continue the example, the result type of the indirection |
| 889 | operator is the pointee type of the subexpression. In order to determine the |
| 890 | type, we need to get the instance of ``PointerType`` that best captures the |
| 891 | typedef information in the program. If the type of the expression is literally |
| 892 | a ``PointerType``, we can return that, otherwise we have to dig through the |
| 893 | typedefs to find the pointer type. For example, if the subexpression had type |
| 894 | "``foo*``", we could return that type as the result. If the subexpression had |
| 895 | type "``bar``", we want to return "``foo*``" (note that we do *not* want |
| 896 | "``int*``"). In order to provide all of this, ``Type`` has a |
| 897 | ``getAsPointerType()`` method that checks whether the type is structurally a |
| 898 | ``PointerType`` and, if so, returns the best one. If not, it returns a null |
| 899 | pointer. |
| 900 | |
| 901 | This structure is somewhat mystical, but after meditating on it, it will make |
| 902 | sense to you :). |
| 903 | |
| 904 | .. _QualType: |
| 905 | |
| 906 | The ``QualType`` class |
| 907 | ---------------------- |
| 908 | |
| 909 | The ``QualType`` class is designed as a trivial value class that is small, |
| 910 | passed by-value and is efficient to query. The idea of ``QualType`` is that it |
| 911 | stores the type qualifiers (``const``, ``volatile``, ``restrict``, plus some |
| 912 | extended qualifiers required by language extensions) separately from the types |
| 913 | themselves. ``QualType`` is conceptually a pair of "``Type*``" and the bits |
| 914 | for these type qualifiers. |
| 915 | |
| 916 | By storing the type qualifiers as bits in the conceptual pair, it is extremely |
| 917 | efficient to get the set of qualifiers on a ``QualType`` (just return the field |
| 918 | of the pair), add a type qualifier (which is a trivial constant-time operation |
| 919 | that sets a bit), and remove one or more type qualifiers (just return a |
| 920 | ``QualType`` with the bitfield set to empty). |
| 921 | |
| 922 | Further, because the bits are stored outside of the type itself, we do not need |
| 923 | to create duplicates of types with different sets of qualifiers (i.e. there is |
| 924 | only a single heap allocated "``int``" type: "``const int``" and "``volatile |
| 925 | const int``" both point to the same heap allocated "``int``" type). This |
| 926 | reduces the heap size used to represent bits and also means we do not have to |
| 927 | consider qualifiers when uniquing types (:ref:`Type <Type>` does not even |
| 928 | contain qualifiers). |
| 929 | |
| 930 | In practice, the two most common type qualifiers (``const`` and ``restrict``) |
| 931 | are stored in the low bits of the pointer to the ``Type`` object, together with |
| 932 | a flag indicating whether extended qualifiers are present (which must be |
| 933 | heap-allocated). This means that ``QualType`` is exactly the same size as a |
| 934 | pointer. |
| 935 | |
| 936 | .. _DeclarationName: |
| 937 | |
| 938 | Declaration names |
| 939 | ----------------- |
| 940 | |
| 941 | The ``DeclarationName`` class represents the name of a declaration in Clang. |
| 942 | Declarations in the C family of languages can take several different forms. |
| 943 | Most declarations are named by simple identifiers, e.g., "``f``" and "``x``" in |
| 944 | the function declaration ``f(int x)``. In C++, declaration names can also name |
| 945 | class constructors ("``Class``" in ``struct Class { Class(); }``), class |
| 946 | destructors ("``~Class``"), overloaded operator names ("``operator+``"), and |
| 947 | conversion functions ("``operator void const *``"). In Objective-C, |
| 948 | declaration names can refer to the names of Objective-C methods, which involve |
| 949 | the method name and the parameters, collectively called a *selector*, e.g., |
| 950 | "``setWidth:height:``". Since all of these kinds of entities --- variables, |
| 951 | functions, Objective-C methods, C++ constructors, destructors, and operators |
| 952 | --- are represented as subclasses of Clang's common ``NamedDecl`` class, |
| 953 | ``DeclarationName`` is designed to efficiently represent any kind of name. |
| 954 | |
| 955 | Given a ``DeclarationName`` ``N``, ``N.getNameKind()`` will produce a value |
Justin Bogner | 89eb0d6 | 2013-09-27 21:10:54 +0000 | [diff] [blame] | 956 | that describes what kind of name ``N`` stores. There are 10 options (all of |
| 957 | the names are inside the ``DeclarationName`` class). |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 958 | |
| 959 | ``Identifier`` |
| 960 | |
| 961 | The name is a simple identifier. Use ``N.getAsIdentifierInfo()`` to retrieve |
| 962 | the corresponding ``IdentifierInfo*`` pointing to the actual identifier. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 963 | |
| 964 | ``ObjCZeroArgSelector``, ``ObjCOneArgSelector``, ``ObjCMultiArgSelector`` |
| 965 | |
| 966 | The name is an Objective-C selector, which can be retrieved as a ``Selector`` |
| 967 | instance via ``N.getObjCSelector()``. The three possible name kinds for |
| 968 | Objective-C reflect an optimization within the ``DeclarationName`` class: |
| 969 | both zero- and one-argument selectors are stored as a masked |
| 970 | ``IdentifierInfo`` pointer, and therefore require very little space, since |
| 971 | zero- and one-argument selectors are far more common than multi-argument |
| 972 | selectors (which use a different structure). |
| 973 | |
| 974 | ``CXXConstructorName`` |
| 975 | |
| 976 | The name is a C++ constructor name. Use ``N.getCXXNameType()`` to retrieve |
| 977 | the :ref:`type <QualType>` that this constructor is meant to construct. The |
| 978 | type is always the canonical type, since all constructors for a given type |
| 979 | have the same name. |
| 980 | |
| 981 | ``CXXDestructorName`` |
| 982 | |
| 983 | The name is a C++ destructor name. Use ``N.getCXXNameType()`` to retrieve |
| 984 | the :ref:`type <QualType>` whose destructor is being named. This type is |
| 985 | always a canonical type. |
| 986 | |
| 987 | ``CXXConversionFunctionName`` |
| 988 | |
| 989 | The name is a C++ conversion function. Conversion functions are named |
| 990 | according to the type they convert to, e.g., "``operator void const *``". |
| 991 | Use ``N.getCXXNameType()`` to retrieve the type that this conversion function |
| 992 | converts to. This type is always a canonical type. |
| 993 | |
| 994 | ``CXXOperatorName`` |
| 995 | |
| 996 | The name is a C++ overloaded operator name. Overloaded operators are named |
| 997 | according to their spelling, e.g., "``operator+``" or "``operator new []``". |
| 998 | Use ``N.getCXXOverloadedOperator()`` to retrieve the overloaded operator (a |
| 999 | value of type ``OverloadedOperatorKind``). |
| 1000 | |
Justin Bogner | 89eb0d6 | 2013-09-27 21:10:54 +0000 | [diff] [blame] | 1001 | ``CXXLiteralOperatorName`` |
| 1002 | |
| 1003 | The name is a C++11 user defined literal operator. User defined |
| 1004 | Literal operators are named according to the suffix they define, |
| 1005 | e.g., "``_foo``" for "``operator "" _foo``". Use |
| 1006 | ``N.getCXXLiteralIdentifier()`` to retrieve the corresponding |
| 1007 | ``IdentifierInfo*`` pointing to the identifier. |
| 1008 | |
| 1009 | ``CXXUsingDirective`` |
| 1010 | |
| 1011 | The name is a C++ using directive. Using directives are not really |
| 1012 | NamedDecls, in that they all have the same name, but they are |
| 1013 | implemented as such in order to store them in DeclContext |
| 1014 | effectively. |
| 1015 | |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1016 | ``DeclarationName``\ s are cheap to create, copy, and compare. They require |
| 1017 | only a single pointer's worth of storage in the common cases (identifiers, |
| 1018 | zero- and one-argument Objective-C selectors) and use dense, uniqued storage |
| 1019 | for the other kinds of names. Two ``DeclarationName``\ s can be compared for |
| 1020 | equality (``==``, ``!=``) using a simple bitwise comparison, can be ordered |
| 1021 | with ``<``, ``>``, ``<=``, and ``>=`` (which provide a lexicographical ordering |
| 1022 | for normal identifiers but an unspecified ordering for other kinds of names), |
| 1023 | and can be placed into LLVM ``DenseMap``\ s and ``DenseSet``\ s. |
| 1024 | |
| 1025 | ``DeclarationName`` instances can be created in different ways depending on |
| 1026 | what kind of name the instance will store. Normal identifiers |
| 1027 | (``IdentifierInfo`` pointers) and Objective-C selectors (``Selector``) can be |
| 1028 | implicitly converted to ``DeclarationNames``. Names for C++ constructors, |
| 1029 | destructors, conversion functions, and overloaded operators can be retrieved |
| 1030 | from the ``DeclarationNameTable``, an instance of which is available as |
| 1031 | ``ASTContext::DeclarationNames``. The member functions |
| 1032 | ``getCXXConstructorName``, ``getCXXDestructorName``, |
| 1033 | ``getCXXConversionFunctionName``, and ``getCXXOperatorName``, respectively, |
| 1034 | return ``DeclarationName`` instances for the four kinds of C++ special function |
| 1035 | names. |
| 1036 | |
| 1037 | .. _DeclContext: |
| 1038 | |
| 1039 | Declaration contexts |
| 1040 | -------------------- |
| 1041 | |
| 1042 | Every declaration in a program exists within some *declaration context*, such |
| 1043 | as a translation unit, namespace, class, or function. Declaration contexts in |
| 1044 | Clang are represented by the ``DeclContext`` class, from which the various |
| 1045 | declaration-context AST nodes (``TranslationUnitDecl``, ``NamespaceDecl``, |
| 1046 | ``RecordDecl``, ``FunctionDecl``, etc.) will derive. The ``DeclContext`` class |
| 1047 | provides several facilities common to each declaration context: |
| 1048 | |
| 1049 | Source-centric vs. Semantics-centric View of Declarations |
| 1050 | |
| 1051 | ``DeclContext`` provides two views of the declarations stored within a |
| 1052 | declaration context. The source-centric view accurately represents the |
| 1053 | program source code as written, including multiple declarations of entities |
| 1054 | where present (see the section :ref:`Redeclarations and Overloads |
| 1055 | <Redeclarations>`), while the semantics-centric view represents the program |
| 1056 | semantics. The two views are kept synchronized by semantic analysis while |
| 1057 | the ASTs are being constructed. |
| 1058 | |
| 1059 | Storage of declarations within that context |
| 1060 | |
| 1061 | Every declaration context can contain some number of declarations. For |
| 1062 | example, a C++ class (represented by ``RecordDecl``) contains various member |
| 1063 | functions, fields, nested types, and so on. All of these declarations will |
| 1064 | be stored within the ``DeclContext``, and one can iterate over the |
| 1065 | declarations via [``DeclContext::decls_begin()``, |
| 1066 | ``DeclContext::decls_end()``). This mechanism provides the source-centric |
| 1067 | view of declarations in the context. |
| 1068 | |
| 1069 | Lookup of declarations within that context |
| 1070 | |
| 1071 | The ``DeclContext`` structure provides efficient name lookup for names within |
| 1072 | that declaration context. For example, if ``N`` is a namespace we can look |
| 1073 | for the name ``N::f`` using ``DeclContext::lookup``. The lookup itself is |
| 1074 | based on a lazily-constructed array (for declaration contexts with a small |
| 1075 | number of declarations) or hash table (for declaration contexts with more |
| 1076 | declarations). The lookup operation provides the semantics-centric view of |
| 1077 | the declarations in the context. |
| 1078 | |
| 1079 | Ownership of declarations |
| 1080 | |
| 1081 | The ``DeclContext`` owns all of the declarations that were declared within |
| 1082 | its declaration context, and is responsible for the management of their |
| 1083 | memory as well as their (de-)serialization. |
| 1084 | |
| 1085 | All declarations are stored within a declaration context, and one can query |
| 1086 | information about the context in which each declaration lives. One can |
| 1087 | retrieve the ``DeclContext`` that contains a particular ``Decl`` using |
| 1088 | ``Decl::getDeclContext``. However, see the section |
| 1089 | :ref:`LexicalAndSemanticContexts` for more information about how to interpret |
| 1090 | this context information. |
| 1091 | |
| 1092 | .. _Redeclarations: |
| 1093 | |
| 1094 | Redeclarations and Overloads |
| 1095 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1096 | |
| 1097 | Within a translation unit, it is common for an entity to be declared several |
| 1098 | times. For example, we might declare a function "``f``" and then later |
| 1099 | re-declare it as part of an inlined definition: |
| 1100 | |
| 1101 | .. code-block:: c++ |
| 1102 | |
| 1103 | void f(int x, int y, int z = 1); |
| 1104 | |
| 1105 | inline void f(int x, int y, int z) { /* ... */ } |
| 1106 | |
| 1107 | The representation of "``f``" differs in the source-centric and |
| 1108 | semantics-centric views of a declaration context. In the source-centric view, |
| 1109 | all redeclarations will be present, in the order they occurred in the source |
| 1110 | code, making this view suitable for clients that wish to see the structure of |
| 1111 | the source code. In the semantics-centric view, only the most recent "``f``" |
| 1112 | will be found by the lookup, since it effectively replaces the first |
| 1113 | declaration of "``f``". |
| 1114 | |
| 1115 | In the semantics-centric view, overloading of functions is represented |
| 1116 | explicitly. For example, given two declarations of a function "``g``" that are |
| 1117 | overloaded, e.g., |
| 1118 | |
| 1119 | .. code-block:: c++ |
| 1120 | |
| 1121 | void g(); |
| 1122 | void g(int); |
| 1123 | |
| 1124 | the ``DeclContext::lookup`` operation will return a |
| 1125 | ``DeclContext::lookup_result`` that contains a range of iterators over |
| 1126 | declarations of "``g``". Clients that perform semantic analysis on a program |
| 1127 | that is not concerned with the actual source code will primarily use this |
| 1128 | semantics-centric view. |
| 1129 | |
| 1130 | .. _LexicalAndSemanticContexts: |
| 1131 | |
| 1132 | Lexical and Semantic Contexts |
| 1133 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1134 | |
| 1135 | Each declaration has two potentially different declaration contexts: a |
| 1136 | *lexical* context, which corresponds to the source-centric view of the |
| 1137 | declaration context, and a *semantic* context, which corresponds to the |
| 1138 | semantics-centric view. The lexical context is accessible via |
| 1139 | ``Decl::getLexicalDeclContext`` while the semantic context is accessible via |
| 1140 | ``Decl::getDeclContext``, both of which return ``DeclContext`` pointers. For |
| 1141 | most declarations, the two contexts are identical. For example: |
| 1142 | |
| 1143 | .. code-block:: c++ |
| 1144 | |
| 1145 | class X { |
| 1146 | public: |
| 1147 | void f(int x); |
| 1148 | }; |
| 1149 | |
| 1150 | Here, the semantic and lexical contexts of ``X::f`` are the ``DeclContext`` |
| 1151 | associated with the class ``X`` (itself stored as a ``RecordDecl`` AST node). |
| 1152 | However, we can now define ``X::f`` out-of-line: |
| 1153 | |
| 1154 | .. code-block:: c++ |
| 1155 | |
| 1156 | void X::f(int x = 17) { /* ... */ } |
| 1157 | |
| 1158 | This definition of "``f``" has different lexical and semantic contexts. The |
| 1159 | lexical context corresponds to the declaration context in which the actual |
| 1160 | declaration occurred in the source code, e.g., the translation unit containing |
| 1161 | ``X``. Thus, this declaration of ``X::f`` can be found by traversing the |
| 1162 | declarations provided by [``decls_begin()``, ``decls_end()``) in the |
| 1163 | translation unit. |
| 1164 | |
| 1165 | The semantic context of ``X::f`` corresponds to the class ``X``, since this |
| 1166 | member function is (semantically) a member of ``X``. Lookup of the name ``f`` |
| 1167 | into the ``DeclContext`` associated with ``X`` will then return the definition |
| 1168 | of ``X::f`` (including information about the default argument). |
| 1169 | |
| 1170 | Transparent Declaration Contexts |
| 1171 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1172 | |
| 1173 | In C and C++, there are several contexts in which names that are logically |
| 1174 | declared inside another declaration will actually "leak" out into the enclosing |
| 1175 | scope from the perspective of name lookup. The most obvious instance of this |
| 1176 | behavior is in enumeration types, e.g., |
| 1177 | |
| 1178 | .. code-block:: c++ |
| 1179 | |
| 1180 | enum Color { |
| 1181 | Red, |
| 1182 | Green, |
| 1183 | Blue |
| 1184 | }; |
| 1185 | |
| 1186 | Here, ``Color`` is an enumeration, which is a declaration context that contains |
| 1187 | the enumerators ``Red``, ``Green``, and ``Blue``. Thus, traversing the list of |
| 1188 | declarations contained in the enumeration ``Color`` will yield ``Red``, |
| 1189 | ``Green``, and ``Blue``. However, outside of the scope of ``Color`` one can |
| 1190 | name the enumerator ``Red`` without qualifying the name, e.g., |
| 1191 | |
| 1192 | .. code-block:: c++ |
| 1193 | |
| 1194 | Color c = Red; |
| 1195 | |
| 1196 | There are other entities in C++ that provide similar behavior. For example, |
| 1197 | linkage specifications that use curly braces: |
| 1198 | |
| 1199 | .. code-block:: c++ |
| 1200 | |
| 1201 | extern "C" { |
| 1202 | void f(int); |
| 1203 | void g(int); |
| 1204 | } |
| 1205 | // f and g are visible here |
| 1206 | |
| 1207 | For source-level accuracy, we treat the linkage specification and enumeration |
| 1208 | type as a declaration context in which its enclosed declarations ("``Red``", |
| 1209 | "``Green``", and "``Blue``"; "``f``" and "``g``") are declared. However, these |
| 1210 | declarations are visible outside of the scope of the declaration context. |
| 1211 | |
| 1212 | These language features (and several others, described below) have roughly the |
| 1213 | same set of requirements: declarations are declared within a particular lexical |
| 1214 | context, but the declarations are also found via name lookup in scopes |
| 1215 | enclosing the declaration itself. This feature is implemented via |
| 1216 | *transparent* declaration contexts (see |
| 1217 | ``DeclContext::isTransparentContext()``), whose declarations are visible in the |
| 1218 | nearest enclosing non-transparent declaration context. This means that the |
| 1219 | lexical context of the declaration (e.g., an enumerator) will be the |
| 1220 | transparent ``DeclContext`` itself, as will the semantic context, but the |
| 1221 | declaration will be visible in every outer context up to and including the |
| 1222 | first non-transparent declaration context (since transparent declaration |
| 1223 | contexts can be nested). |
| 1224 | |
| 1225 | The transparent ``DeclContext``\ s are: |
| 1226 | |
| 1227 | * Enumerations (but not C++11 "scoped enumerations"): |
| 1228 | |
| 1229 | .. code-block:: c++ |
| 1230 | |
| 1231 | enum Color { |
| 1232 | Red, |
| 1233 | Green, |
| 1234 | Blue |
| 1235 | }; |
| 1236 | // Red, Green, and Blue are in scope |
| 1237 | |
| 1238 | * C++ linkage specifications: |
| 1239 | |
| 1240 | .. code-block:: c++ |
| 1241 | |
| 1242 | extern "C" { |
| 1243 | void f(int); |
| 1244 | void g(int); |
| 1245 | } |
| 1246 | // f and g are in scope |
| 1247 | |
| 1248 | * Anonymous unions and structs: |
| 1249 | |
| 1250 | .. code-block:: c++ |
| 1251 | |
| 1252 | struct LookupTable { |
| 1253 | bool IsVector; |
| 1254 | union { |
| 1255 | std::vector<Item> *Vector; |
| 1256 | std::set<Item> *Set; |
| 1257 | }; |
| 1258 | }; |
| 1259 | |
| 1260 | LookupTable LT; |
| 1261 | LT.Vector = 0; // Okay: finds Vector inside the unnamed union |
| 1262 | |
| 1263 | * C++11 inline namespaces: |
| 1264 | |
| 1265 | .. code-block:: c++ |
| 1266 | |
| 1267 | namespace mylib { |
| 1268 | inline namespace debug { |
| 1269 | class X; |
| 1270 | } |
| 1271 | } |
| 1272 | mylib::X *xp; // okay: mylib::X refers to mylib::debug::X |
| 1273 | |
| 1274 | .. _MultiDeclContext: |
| 1275 | |
| 1276 | Multiply-Defined Declaration Contexts |
| 1277 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1278 | |
| 1279 | C++ namespaces have the interesting --- and, so far, unique --- property that |
| 1280 | the namespace can be defined multiple times, and the declarations provided by |
| 1281 | each namespace definition are effectively merged (from the semantic point of |
| 1282 | view). For example, the following two code snippets are semantically |
| 1283 | indistinguishable: |
| 1284 | |
| 1285 | .. code-block:: c++ |
| 1286 | |
| 1287 | // Snippet #1: |
| 1288 | namespace N { |
| 1289 | void f(); |
| 1290 | } |
| 1291 | namespace N { |
| 1292 | void f(int); |
| 1293 | } |
| 1294 | |
| 1295 | // Snippet #2: |
| 1296 | namespace N { |
| 1297 | void f(); |
| 1298 | void f(int); |
| 1299 | } |
| 1300 | |
| 1301 | In Clang's representation, the source-centric view of declaration contexts will |
| 1302 | actually have two separate ``NamespaceDecl`` nodes in Snippet #1, each of which |
| 1303 | is a declaration context that contains a single declaration of "``f``". |
| 1304 | However, the semantics-centric view provided by name lookup into the namespace |
| 1305 | ``N`` for "``f``" will return a ``DeclContext::lookup_result`` that contains a |
| 1306 | range of iterators over declarations of "``f``". |
| 1307 | |
| 1308 | ``DeclContext`` manages multiply-defined declaration contexts internally. The |
| 1309 | function ``DeclContext::getPrimaryContext`` retrieves the "primary" context for |
| 1310 | a given ``DeclContext`` instance, which is the ``DeclContext`` responsible for |
| 1311 | maintaining the lookup table used for the semantics-centric view. Given the |
| 1312 | primary context, one can follow the chain of ``DeclContext`` nodes that define |
| 1313 | additional declarations via ``DeclContext::getNextContext``. Note that these |
| 1314 | functions are used internally within the lookup and insertion methods of the |
| 1315 | ``DeclContext``, so the vast majority of clients can ignore them. |
| 1316 | |
| 1317 | .. _CFG: |
| 1318 | |
| 1319 | The ``CFG`` class |
| 1320 | ----------------- |
| 1321 | |
| 1322 | The ``CFG`` class is designed to represent a source-level control-flow graph |
| 1323 | for a single statement (``Stmt*``). Typically instances of ``CFG`` are |
| 1324 | constructed for function bodies (usually an instance of ``CompoundStmt``), but |
| 1325 | can also be instantiated to represent the control-flow of any class that |
| 1326 | subclasses ``Stmt``, which includes simple expressions. Control-flow graphs |
| 1327 | are especially useful for performing `flow- or path-sensitive |
| 1328 | <http://en.wikipedia.org/wiki/Data_flow_analysis#Sensitivities>`_ program |
| 1329 | analyses on a given function. |
| 1330 | |
| 1331 | Basic Blocks |
| 1332 | ^^^^^^^^^^^^ |
| 1333 | |
| 1334 | Concretely, an instance of ``CFG`` is a collection of basic blocks. Each basic |
| 1335 | block is an instance of ``CFGBlock``, which simply contains an ordered sequence |
| 1336 | of ``Stmt*`` (each referring to statements in the AST). The ordering of |
| 1337 | statements within a block indicates unconditional flow of control from one |
| 1338 | statement to the next. :ref:`Conditional control-flow |
| 1339 | <ConditionalControlFlow>` is represented using edges between basic blocks. The |
| 1340 | statements within a given ``CFGBlock`` can be traversed using the |
| 1341 | ``CFGBlock::*iterator`` interface. |
| 1342 | |
| 1343 | A ``CFG`` object owns the instances of ``CFGBlock`` within the control-flow |
| 1344 | graph it represents. Each ``CFGBlock`` within a CFG is also uniquely numbered |
| 1345 | (accessible via ``CFGBlock::getBlockID()``). Currently the number is based on |
| 1346 | the ordering the blocks were created, but no assumptions should be made on how |
| 1347 | ``CFGBlocks`` are numbered other than their numbers are unique and that they |
| 1348 | are numbered from 0..N-1 (where N is the number of basic blocks in the CFG). |
| 1349 | |
| 1350 | Entry and Exit Blocks |
| 1351 | ^^^^^^^^^^^^^^^^^^^^^ |
| 1352 | |
| 1353 | Each instance of ``CFG`` contains two special blocks: an *entry* block |
| 1354 | (accessible via ``CFG::getEntry()``), which has no incoming edges, and an |
| 1355 | *exit* block (accessible via ``CFG::getExit()``), which has no outgoing edges. |
| 1356 | Neither block contains any statements, and they serve the role of providing a |
| 1357 | clear entrance and exit for a body of code such as a function body. The |
| 1358 | presence of these empty blocks greatly simplifies the implementation of many |
| 1359 | analyses built on top of CFGs. |
| 1360 | |
| 1361 | .. _ConditionalControlFlow: |
| 1362 | |
| 1363 | Conditional Control-Flow |
| 1364 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1365 | |
| 1366 | Conditional control-flow (such as those induced by if-statements and loops) is |
| 1367 | represented as edges between ``CFGBlocks``. Because different C language |
| 1368 | constructs can induce control-flow, each ``CFGBlock`` also records an extra |
| 1369 | ``Stmt*`` that represents the *terminator* of the block. A terminator is |
| 1370 | simply the statement that caused the control-flow, and is used to identify the |
| 1371 | nature of the conditional control-flow between blocks. For example, in the |
| 1372 | case of an if-statement, the terminator refers to the ``IfStmt`` object in the |
| 1373 | AST that represented the given branch. |
| 1374 | |
| 1375 | To illustrate, consider the following code example: |
| 1376 | |
| 1377 | .. code-block:: c++ |
| 1378 | |
| 1379 | int foo(int x) { |
| 1380 | x = x + 1; |
| 1381 | if (x > 2) |
| 1382 | x++; |
| 1383 | else { |
| 1384 | x += 2; |
| 1385 | x *= 2; |
| 1386 | } |
| 1387 | |
| 1388 | return x; |
| 1389 | } |
| 1390 | |
| 1391 | After invoking the parser+semantic analyzer on this code fragment, the AST of |
| 1392 | the body of ``foo`` is referenced by a single ``Stmt*``. We can then construct |
| 1393 | an instance of ``CFG`` representing the control-flow graph of this function |
| 1394 | body by single call to a static class method: |
| 1395 | |
| 1396 | .. code-block:: c++ |
| 1397 | |
| 1398 | Stmt *FooBody = ... |
Stephen Hines | 176edba | 2014-12-01 14:53:08 -0800 | [diff] [blame] | 1399 | std::unique_ptr<CFG> FooCFG = CFG::buildCFG(FooBody); |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1400 | |
| 1401 | Along with providing an interface to iterate over its ``CFGBlocks``, the |
| 1402 | ``CFG`` class also provides methods that are useful for debugging and |
| 1403 | visualizing CFGs. For example, the method ``CFG::dump()`` dumps a |
| 1404 | pretty-printed version of the CFG to standard error. This is especially useful |
| 1405 | when one is using a debugger such as gdb. For example, here is the output of |
| 1406 | ``FooCFG->dump()``: |
| 1407 | |
| 1408 | .. code-block:: c++ |
| 1409 | |
| 1410 | [ B5 (ENTRY) ] |
| 1411 | Predecessors (0): |
| 1412 | Successors (1): B4 |
| 1413 | |
| 1414 | [ B4 ] |
| 1415 | 1: x = x + 1 |
| 1416 | 2: (x > 2) |
| 1417 | T: if [B4.2] |
| 1418 | Predecessors (1): B5 |
| 1419 | Successors (2): B3 B2 |
| 1420 | |
| 1421 | [ B3 ] |
| 1422 | 1: x++ |
| 1423 | Predecessors (1): B4 |
| 1424 | Successors (1): B1 |
| 1425 | |
| 1426 | [ B2 ] |
| 1427 | 1: x += 2 |
| 1428 | 2: x *= 2 |
| 1429 | Predecessors (1): B4 |
| 1430 | Successors (1): B1 |
| 1431 | |
| 1432 | [ B1 ] |
| 1433 | 1: return x; |
| 1434 | Predecessors (2): B2 B3 |
| 1435 | Successors (1): B0 |
| 1436 | |
| 1437 | [ B0 (EXIT) ] |
| 1438 | Predecessors (1): B1 |
| 1439 | Successors (0): |
| 1440 | |
| 1441 | For each block, the pretty-printed output displays for each block the number of |
| 1442 | *predecessor* blocks (blocks that have outgoing control-flow to the given |
| 1443 | block) and *successor* blocks (blocks that have control-flow that have incoming |
| 1444 | control-flow from the given block). We can also clearly see the special entry |
| 1445 | and exit blocks at the beginning and end of the pretty-printed output. For the |
| 1446 | entry block (block B5), the number of predecessor blocks is 0, while for the |
| 1447 | exit block (block B0) the number of successor blocks is 0. |
| 1448 | |
| 1449 | The most interesting block here is B4, whose outgoing control-flow represents |
| 1450 | the branching caused by the sole if-statement in ``foo``. Of particular |
| 1451 | interest is the second statement in the block, ``(x > 2)``, and the terminator, |
| 1452 | printed as ``if [B4.2]``. The second statement represents the evaluation of |
| 1453 | the condition of the if-statement, which occurs before the actual branching of |
| 1454 | control-flow. Within the ``CFGBlock`` for B4, the ``Stmt*`` for the second |
| 1455 | statement refers to the actual expression in the AST for ``(x > 2)``. Thus |
| 1456 | pointers to subclasses of ``Expr`` can appear in the list of statements in a |
| 1457 | block, and not just subclasses of ``Stmt`` that refer to proper C statements. |
| 1458 | |
| 1459 | The terminator of block B4 is a pointer to the ``IfStmt`` object in the AST. |
| 1460 | The pretty-printer outputs ``if [B4.2]`` because the condition expression of |
| 1461 | the if-statement has an actual place in the basic block, and thus the |
| 1462 | terminator is essentially *referring* to the expression that is the second |
| 1463 | statement of block B4 (i.e., B4.2). In this manner, conditions for |
| 1464 | control-flow (which also includes conditions for loops and switch statements) |
| 1465 | are hoisted into the actual basic block. |
| 1466 | |
| 1467 | .. Implicit Control-Flow |
| 1468 | .. ^^^^^^^^^^^^^^^^^^^^^ |
| 1469 | |
| 1470 | .. A key design principle of the ``CFG`` class was to not require any |
| 1471 | .. transformations to the AST in order to represent control-flow. Thus the |
| 1472 | .. ``CFG`` does not perform any "lowering" of the statements in an AST: loops |
| 1473 | .. are not transformed into guarded gotos, short-circuit operations are not |
| 1474 | .. converted to a set of if-statements, and so on. |
| 1475 | |
| 1476 | Constant Folding in the Clang AST |
| 1477 | --------------------------------- |
| 1478 | |
| 1479 | There are several places where constants and constant folding matter a lot to |
| 1480 | the Clang front-end. First, in general, we prefer the AST to retain the source |
| 1481 | code as close to how the user wrote it as possible. This means that if they |
| 1482 | wrote "``5+4``", we want to keep the addition and two constants in the AST, we |
| 1483 | don't want to fold to "``9``". This means that constant folding in various |
| 1484 | ways turns into a tree walk that needs to handle the various cases. |
| 1485 | |
| 1486 | However, there are places in both C and C++ that require constants to be |
| 1487 | folded. For example, the C standard defines what an "integer constant |
| 1488 | expression" (i-c-e) is with very precise and specific requirements. The |
| 1489 | language then requires i-c-e's in a lot of places (for example, the size of a |
| 1490 | bitfield, the value for a case statement, etc). For these, we have to be able |
| 1491 | to constant fold the constants, to do semantic checks (e.g., verify bitfield |
| 1492 | size is non-negative and that case statements aren't duplicated). We aim for |
| 1493 | Clang to be very pedantic about this, diagnosing cases when the code does not |
| 1494 | use an i-c-e where one is required, but accepting the code unless running with |
| 1495 | ``-pedantic-errors``. |
| 1496 | |
| 1497 | Things get a little bit more tricky when it comes to compatibility with |
| 1498 | real-world source code. Specifically, GCC has historically accepted a huge |
| 1499 | superset of expressions as i-c-e's, and a lot of real world code depends on |
| 1500 | this unfortuate accident of history (including, e.g., the glibc system |
| 1501 | headers). GCC accepts anything its "fold" optimizer is capable of reducing to |
| 1502 | an integer constant, which means that the definition of what it accepts changes |
| 1503 | as its optimizer does. One example is that GCC accepts things like "``case |
| 1504 | X-X:``" even when ``X`` is a variable, because it can fold this to 0. |
| 1505 | |
| 1506 | Another issue are how constants interact with the extensions we support, such |
| 1507 | as ``__builtin_constant_p``, ``__builtin_inf``, ``__extension__`` and many |
| 1508 | others. C99 obviously does not specify the semantics of any of these |
| 1509 | extensions, and the definition of i-c-e does not include them. However, these |
| 1510 | extensions are often used in real code, and we have to have a way to reason |
| 1511 | about them. |
| 1512 | |
| 1513 | Finally, this is not just a problem for semantic analysis. The code generator |
| 1514 | and other clients have to be able to fold constants (e.g., to initialize global |
| 1515 | variables) and has to handle a superset of what C99 allows. Further, these |
| 1516 | clients can benefit from extended information. For example, we know that |
| 1517 | "``foo() || 1``" always evaluates to ``true``, but we can't replace the |
| 1518 | expression with ``true`` because it has side effects. |
| 1519 | |
| 1520 | Implementation Approach |
| 1521 | ^^^^^^^^^^^^^^^^^^^^^^^ |
| 1522 | |
| 1523 | After trying several different approaches, we've finally converged on a design |
| 1524 | (Note, at the time of this writing, not all of this has been implemented, |
| 1525 | consider this a design goal!). Our basic approach is to define a single |
| 1526 | recursive method evaluation method (``Expr::Evaluate``), which is implemented |
| 1527 | in ``AST/ExprConstant.cpp``. Given an expression with "scalar" type (integer, |
| 1528 | fp, complex, or pointer) this method returns the following information: |
| 1529 | |
| 1530 | * Whether the expression is an integer constant expression, a general constant |
| 1531 | that was folded but has no side effects, a general constant that was folded |
| 1532 | but that does have side effects, or an uncomputable/unfoldable value. |
| 1533 | * If the expression was computable in any way, this method returns the |
| 1534 | ``APValue`` for the result of the expression. |
| 1535 | * If the expression is not evaluatable at all, this method returns information |
| 1536 | on one of the problems with the expression. This includes a |
| 1537 | ``SourceLocation`` for where the problem is, and a diagnostic ID that explains |
| 1538 | the problem. The diagnostic should have ``ERROR`` type. |
| 1539 | * If the expression is not an integer constant expression, this method returns |
| 1540 | information on one of the problems with the expression. This includes a |
| 1541 | ``SourceLocation`` for where the problem is, and a diagnostic ID that |
| 1542 | explains the problem. The diagnostic should have ``EXTENSION`` type. |
| 1543 | |
| 1544 | This information gives various clients the flexibility that they want, and we |
| 1545 | will eventually have some helper methods for various extensions. For example, |
| 1546 | ``Sema`` should have a ``Sema::VerifyIntegerConstantExpression`` method, which |
| 1547 | calls ``Evaluate`` on the expression. If the expression is not foldable, the |
| 1548 | error is emitted, and it would return ``true``. If the expression is not an |
| 1549 | i-c-e, the ``EXTENSION`` diagnostic is emitted. Finally it would return |
| 1550 | ``false`` to indicate that the AST is OK. |
| 1551 | |
| 1552 | Other clients can use the information in other ways, for example, codegen can |
| 1553 | just use expressions that are foldable in any way. |
| 1554 | |
| 1555 | Extensions |
| 1556 | ^^^^^^^^^^ |
| 1557 | |
| 1558 | This section describes how some of the various extensions Clang supports |
| 1559 | interacts with constant evaluation: |
| 1560 | |
| 1561 | * ``__extension__``: The expression form of this extension causes any |
| 1562 | evaluatable subexpression to be accepted as an integer constant expression. |
| 1563 | * ``__builtin_constant_p``: This returns true (as an integer constant |
| 1564 | expression) if the operand evaluates to either a numeric value (that is, not |
| 1565 | a pointer cast to integral type) of integral, enumeration, floating or |
| 1566 | complex type, or if it evaluates to the address of the first character of a |
| 1567 | string literal (possibly cast to some other type). As a special case, if |
| 1568 | ``__builtin_constant_p`` is the (potentially parenthesized) condition of a |
| 1569 | conditional operator expression ("``?:``"), only the true side of the |
| 1570 | conditional operator is considered, and it is evaluated with full constant |
| 1571 | folding. |
| 1572 | * ``__builtin_choose_expr``: The condition is required to be an integer |
| 1573 | constant expression, but we accept any constant as an "extension of an |
| 1574 | extension". This only evaluates one operand depending on which way the |
| 1575 | condition evaluates. |
| 1576 | * ``__builtin_classify_type``: This always returns an integer constant |
| 1577 | expression. |
| 1578 | * ``__builtin_inf, nan, ...``: These are treated just like a floating-point |
| 1579 | literal. |
| 1580 | * ``__builtin_abs, copysign, ...``: These are constant folded as general |
| 1581 | constant expressions. |
| 1582 | * ``__builtin_strlen`` and ``strlen``: These are constant folded as integer |
| 1583 | constant expressions if the argument is a string literal. |
| 1584 | |
| 1585 | How to change Clang |
| 1586 | =================== |
| 1587 | |
| 1588 | How to add an attribute |
| 1589 | ----------------------- |
| 1590 | |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1591 | Attribute Basics |
| 1592 | ^^^^^^^^^^^^^^^^ |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1593 | |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1594 | Attributes in clang come in two forms: parsed form, and semantic form. Both |
| 1595 | forms are represented via a tablegen definition of the attribute, specified in |
| 1596 | Attr.td. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1597 | |
| 1598 | |
| 1599 | ``include/clang/Basic/Attr.td`` |
| 1600 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1601 | |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1602 | First, add your attribute to the `include/clang/Basic/Attr.td |
| 1603 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/Attr.td?view=markup>`_ |
| 1604 | file. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1605 | |
| 1606 | Each attribute gets a ``def`` inheriting from ``Attr`` or one of its |
| 1607 | subclasses. ``InheritableAttr`` means that the attribute also applies to |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1608 | subsequent declarations of the same name. ``InheritableParamAttr`` is similar |
| 1609 | to ``InheritableAttr``, except that the attribute is written on a parameter |
| 1610 | instead of a declaration, type or statement. Attributes inheriting from |
| 1611 | ``TypeAttr`` are pure type attributes which generally are not given a |
| 1612 | representation in the AST. Attributes inheriting from ``TargetSpecificAttr`` |
| 1613 | are attributes specific to one or more target architectures. An attribute that |
| 1614 | inherits from ``IgnoredAttr`` is parsed, but will generate an ignored attribute |
| 1615 | diagnostic when used. The attribute type may be useful when an attribute is |
| 1616 | supported by another vendor, but not supported by clang. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1617 | |
| 1618 | ``Spellings`` lists the strings that can appear in ``__attribute__((here))`` or |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1619 | ``[[here]]``. All such strings will be synonymous. Possible ``Spellings`` |
| 1620 | are: ``GNU`` (for use with GNU-style __attribute__ spellings), ``Declspec`` |
| 1621 | (for use with Microsoft Visual Studio-style __declspec spellings), ``CXX11` |
| 1622 | (for use with C++11-style [[foo]] and [[foo::bar]] spellings), and ``Keyword`` |
| 1623 | (for use with attributes that are implemented as keywords, like C++11's |
| 1624 | ``override`` or ``final``). If you want to allow the ``[[]]`` C++11 syntax, you |
| 1625 | have to define a list of ``Namespaces``, which will let users write |
| 1626 | ``[[namespace::spelling]]``. Using the empty string for a namespace will allow |
| 1627 | users to write just the spelling with no "``::``". Attributes which g++-4.8 |
| 1628 | or later accepts should also have a ``CXX11<"gnu", "spelling">`` spelling. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1629 | |
| 1630 | ``Subjects`` restricts what kinds of AST node to which this attribute can |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1631 | appertain (roughly, attach). The subjects are specified via a ``SubjectList``, |
| 1632 | which specify the list of subjects. Additionally, subject-related diagnostics |
| 1633 | can be specified to be warnings or errors, with the default being a warning. |
| 1634 | The diagnostics displayed to the user are automatically determined based on |
| 1635 | the subjects in the list, but a custom diagnostic parameter can also be |
| 1636 | specified in the ``SubjectList``. The diagnostics generated for subject list |
| 1637 | violations are either ``diag::warn_attribute_wrong_decl_type`` or |
| 1638 | ``diag::err_attribute_wrong_decl_type``, and the parameter enumeration is |
| 1639 | found in `include/clang/Sema/AttributeList.h |
| 1640 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Sema/AttributeList.h?view=markup>`_ |
| 1641 | If you add new Decl nodes to the ``SubjectList``, you may need to update the |
| 1642 | logic used to automatically determine the diagnostic parameter in `utils/TableGen/ClangAttrEmitter.cpp |
| 1643 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/utils/TableGen/ClangAttrEmitter.cpp?view=markup>`_. |
| 1644 | |
| 1645 | Diagnostic checking for attribute subject lists is automated except when |
| 1646 | ``HasCustomParsing`` is set to ``1``. |
| 1647 | |
| 1648 | By default, all subjects in the SubjectList must either be a Decl node defined |
| 1649 | in ``DeclNodes.td``, or a statement node defined in ``StmtNodes.td``. However, |
| 1650 | more complex subjects can be created by creating a ``SubsetSubject`` object. |
| 1651 | Each such object has a base subject which it appertains to (which must be a |
| 1652 | Decl or Stmt node, and not a SubsetSubject node), and some custom code which is |
| 1653 | called when determining whether an attribute appertains to the subject. For |
| 1654 | instance, a ``NonBitField`` SubsetSubject appertains to a ``FieldDecl``, and |
| 1655 | tests whether the given FieldDecl is a bit field. When a SubsetSubject is |
| 1656 | specified in a SubjectList, a custom diagnostic parameter must also be provided. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1657 | |
| 1658 | ``Args`` names the arguments the attribute takes, in order. If ``Args`` is |
| 1659 | ``[StringArgument<"Arg1">, IntArgument<"Arg2">]`` then |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1660 | ``__attribute__((myattribute("Hello", 3)))`` will be a valid use. Attribute |
| 1661 | arguments specify both the parsed form and the semantic form of the attribute. |
| 1662 | The previous example shows an attribute which requires two attributes while |
| 1663 | parsing, and the Attr subclass' constructor for the attribute will require a |
| 1664 | string and integer argument. |
| 1665 | |
| 1666 | Diagnostic checking for argument counts is automated except when |
| 1667 | ``HasCustomParsing`` is set to ``1``, or when the attribute uses an optional or |
| 1668 | variadic argument. Diagnostic checking for argument semantics is not automated. |
| 1669 | |
| 1670 | If the parsed form of the attribute is more complex, or differs from the |
| 1671 | semantic form, the ``HasCustomParsing`` bit can be set to ``1`` for the class, |
| 1672 | and the parsing code in `Parser::ParseGNUAttributeArgs |
| 1673 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/ParseDecl.cpp?view=markup>`_ |
| 1674 | can be updated for the special case. Note that this only applies to arguments |
| 1675 | with a GNU spelling -- attributes with a __declspec spelling currently ignore |
| 1676 | this flag and are handled by ``Parser::ParseMicrosoftDeclSpec``. |
| 1677 | |
| 1678 | Custom accessors can be generated for an attribute based on the spelling list |
| 1679 | for that attribute. For instance, if an attribute has two different spellings: |
| 1680 | 'Foo' and 'Bar', accessors can be created: |
| 1681 | ``[Accessor<"isFoo", [GNU<"Foo">]>, Accessor<"isBar", [GNU<"Bar">]>]`` |
| 1682 | These accessors will be generated on the semantic form of the attribute, |
| 1683 | accepting no arguments and returning a Boolean. |
| 1684 | |
| 1685 | Attributes which do not require an AST node should set the ``ASTNode`` field to |
| 1686 | ``0`` to avoid polluting the AST. Note that anything inheriting from |
| 1687 | ``TypeAttr`` or ``IgnoredAttr`` automatically do not generate an AST node. All |
| 1688 | other attributes generate an AST node by default. The AST node is the semantic |
| 1689 | representation of the attribute. |
| 1690 | |
| 1691 | Attributes which do not require custom semantic handling should set the |
| 1692 | ``SemaHandler`` field to ``0``. Note that anything inheriting from |
| 1693 | ``IgnoredAttr`` automatically do not get a semantic handler. All other |
| 1694 | attributes are assumed to use a semantic handler by default. Attributes |
| 1695 | without a semantic handler are not given a parsed attribute Kind enumeration. |
| 1696 | |
| 1697 | The ``LangOpts`` field can be used to specify a list of language options |
| 1698 | required by the attribute. For instance, all of the CUDA-specific attributes |
| 1699 | specify ``[CUDA]`` for the ``LangOpts`` field, and when the CUDA language |
| 1700 | option is not enabled, an "attribute ignored" warning diagnostic is emitted. |
| 1701 | Since language options are not table generated nodes, new language options must |
| 1702 | be created manually and should specify the spelling used by ``LangOptions`` class. |
| 1703 | |
| 1704 | Target-specific attribute sometimes share a spelling with other attributes in |
| 1705 | different targets. For instance, the ARM and MSP430 targets both have an |
| 1706 | attribute spelled ``GNU<"interrupt">``, but with different parsing and semantic |
| 1707 | requirements. To support this feature, an attribute inheriting from |
| 1708 | ``TargetSpecificAttribute`` make specify a ``ParseKind`` field. This field |
| 1709 | should be the same value between all arguments sharing a spelling, and |
| 1710 | corresponds to the parsed attribute's Kind enumeration. This allows attributes |
| 1711 | to share a parsed attribute kind, but have distinct semantic attribute classes. |
| 1712 | For instance, ``AttributeList::AT_Interrupt`` is the shared parsed attribute |
| 1713 | kind, but ARMInterruptAttr and MSP430InterruptAttr are the semantic attributes |
| 1714 | generated. |
| 1715 | |
| 1716 | By default, when declarations are merging attributes, an attribute will not be |
| 1717 | duplicated. However, if an attribute can be duplicated during this merging |
| 1718 | stage, set ``DuplicatesAllowedWhileMerging`` to ``1``, and the attribute will |
| 1719 | be merged. |
| 1720 | |
| 1721 | By default, attribute arguments are parsed in an evaluated context. If the |
| 1722 | arguments for an attribute should be parsed in an unevaluated context (akin to |
| 1723 | the way the argument to a ``sizeof`` expression is parsed), you can set |
| 1724 | ``ParseArgumentsAsUnevaluated`` to ``1``. |
| 1725 | |
| 1726 | If additional functionality is desired for the semantic form of the attribute, |
| 1727 | the ``AdditionalMembers`` field specifies code to be copied verbatim into the |
| 1728 | semantic attribute class object. |
| 1729 | |
| 1730 | All attributes must have one or more form of documentation, which is provided |
| 1731 | in the ``Documentation`` list. Generally, the documentation for an attribute |
| 1732 | is a stand-alone definition in `include/clang/Basic/AttrDocs.td |
| 1733 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/AttdDocs.td?view=markup>`_ |
| 1734 | that is named after the attribute being documented. Each documentation element |
| 1735 | is given a ``Category`` (variable, function, or type) and ``Content``. A single |
| 1736 | attribute may contain multiple documentation elements for distinct categories. |
| 1737 | For instance, an attribute which can appertain to both function and types (such |
| 1738 | as a calling convention attribute), should contain two documentation elements. |
| 1739 | The ``Content`` for an attribute uses reStructuredText (RST) syntax. |
| 1740 | |
| 1741 | If an attribute is used internally by the compiler, but is not written by users |
| 1742 | (such as attributes with an empty spelling list), it can use the |
| 1743 | ``Undocumented`` documentation element. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1744 | |
| 1745 | Boilerplate |
| 1746 | ^^^^^^^^^^^ |
| 1747 | |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1748 | All semantic processing of declaration attributes happens in `lib/Sema/SemaDeclAttr.cpp |
| 1749 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Sema/SemaDeclAttr.cpp?view=markup>`_, |
| 1750 | and generally starts in the ``ProcessDeclAttribute`` function. If your |
| 1751 | attribute is a "simple" attribute -- meaning that it requires no custom |
| 1752 | semantic processing aside from what is automatically provided for you, you can |
| 1753 | add a call to ``handleSimpleAttribute<YourAttr>(S, D, Attr);`` to the switch |
| 1754 | statement. Otherwise, write a new ``handleYourAttr()`` function, and add that |
| 1755 | to the switch statement. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1756 | |
| 1757 | If your attribute causes extra warnings to fire, define a ``DiagGroup`` in |
| 1758 | `include/clang/Basic/DiagnosticGroups.td |
| 1759 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/DiagnosticGroups.td?view=markup>`_ |
| 1760 | named after the attribute's ``Spelling`` with "_"s replaced by "-"s. If you're |
| 1761 | only defining one diagnostic, you can skip ``DiagnosticGroups.td`` and use |
| 1762 | ``InGroup<DiagGroup<"your-attribute">>`` directly in `DiagnosticSemaKinds.td |
| 1763 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/DiagnosticSemaKinds.td?view=markup>`_ |
| 1764 | |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1765 | All semantic diagnostics generated for your attribute, including automatically- |
| 1766 | generated ones (such as subjects and argument counts), should have a |
| 1767 | corresponding test case. |
| 1768 | |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1769 | The meat of your attribute |
| 1770 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1771 | |
| 1772 | Find an appropriate place in Clang to do whatever your attribute needs to do. |
| 1773 | Check for the attribute's presence using ``Decl::getAttr<YourAttr>()``. |
| 1774 | |
| 1775 | Update the :doc:`LanguageExtensions` document to describe your new attribute. |
| 1776 | |
| 1777 | How to add an expression or statement |
| 1778 | ------------------------------------- |
| 1779 | |
| 1780 | Expressions and statements are one of the most fundamental constructs within a |
| 1781 | compiler, because they interact with many different parts of the AST, semantic |
| 1782 | analysis, and IR generation. Therefore, adding a new expression or statement |
| 1783 | kind into Clang requires some care. The following list details the various |
| 1784 | places in Clang where an expression or statement needs to be introduced, along |
| 1785 | with patterns to follow to ensure that the new expression or statement works |
| 1786 | well across all of the C languages. We focus on expressions, but statements |
| 1787 | are similar. |
| 1788 | |
| 1789 | #. Introduce parsing actions into the parser. Recursive-descent parsing is |
| 1790 | mostly self-explanatory, but there are a few things that are worth keeping |
| 1791 | in mind: |
| 1792 | |
| 1793 | * Keep as much source location information as possible! You'll want it later |
| 1794 | to produce great diagnostics and support Clang's various features that map |
| 1795 | between source code and the AST. |
| 1796 | * Write tests for all of the "bad" parsing cases, to make sure your recovery |
| 1797 | is good. If you have matched delimiters (e.g., parentheses, square |
| 1798 | brackets, etc.), use ``Parser::BalancedDelimiterTracker`` to give nice |
| 1799 | diagnostics when things go wrong. |
| 1800 | |
| 1801 | #. Introduce semantic analysis actions into ``Sema``. Semantic analysis should |
| 1802 | always involve two functions: an ``ActOnXXX`` function that will be called |
| 1803 | directly from the parser, and a ``BuildXXX`` function that performs the |
| 1804 | actual semantic analysis and will (eventually!) build the AST node. It's |
| 1805 | fairly common for the ``ActOnCXX`` function to do very little (often just |
| 1806 | some minor translation from the parser's representation to ``Sema``'s |
| 1807 | representation of the same thing), but the separation is still important: |
| 1808 | C++ template instantiation, for example, should always call the ``BuildXXX`` |
| 1809 | variant. Several notes on semantic analysis before we get into construction |
| 1810 | of the AST: |
| 1811 | |
| 1812 | * Your expression probably involves some types and some subexpressions. |
| 1813 | Make sure to fully check that those types, and the types of those |
| 1814 | subexpressions, meet your expectations. Add implicit conversions where |
| 1815 | necessary to make sure that all of the types line up exactly the way you |
| 1816 | want them. Write extensive tests to check that you're getting good |
| 1817 | diagnostics for mistakes and that you can use various forms of |
| 1818 | subexpressions with your expression. |
| 1819 | * When type-checking a type or subexpression, make sure to first check |
| 1820 | whether the type is "dependent" (``Type::isDependentType()``) or whether a |
| 1821 | subexpression is type-dependent (``Expr::isTypeDependent()``). If any of |
| 1822 | these return ``true``, then you're inside a template and you can't do much |
| 1823 | type-checking now. That's normal, and your AST node (when you get there) |
| 1824 | will have to deal with this case. At this point, you can write tests that |
| 1825 | use your expression within templates, but don't try to instantiate the |
| 1826 | templates. |
| 1827 | * For each subexpression, be sure to call ``Sema::CheckPlaceholderExpr()`` |
| 1828 | to deal with "weird" expressions that don't behave well as subexpressions. |
| 1829 | Then, determine whether you need to perform lvalue-to-rvalue conversions |
| 1830 | (``Sema::DefaultLvalueConversions``) or the usual unary conversions |
| 1831 | (``Sema::UsualUnaryConversions``), for places where the subexpression is |
| 1832 | producing a value you intend to use. |
| 1833 | * Your ``BuildXXX`` function will probably just return ``ExprError()`` at |
| 1834 | this point, since you don't have an AST. That's perfectly fine, and |
| 1835 | shouldn't impact your testing. |
| 1836 | |
| 1837 | #. Introduce an AST node for your new expression. This starts with declaring |
| 1838 | the node in ``include/Basic/StmtNodes.td`` and creating a new class for your |
| 1839 | expression in the appropriate ``include/AST/Expr*.h`` header. It's best to |
| 1840 | look at the class for a similar expression to get ideas, and there are some |
| 1841 | specific things to watch for: |
| 1842 | |
| 1843 | * If you need to allocate memory, use the ``ASTContext`` allocator to |
| 1844 | allocate memory. Never use raw ``malloc`` or ``new``, and never hold any |
| 1845 | resources in an AST node, because the destructor of an AST node is never |
| 1846 | called. |
| 1847 | * Make sure that ``getSourceRange()`` covers the exact source range of your |
| 1848 | expression. This is needed for diagnostics and for IDE support. |
| 1849 | * Make sure that ``children()`` visits all of the subexpressions. This is |
| 1850 | important for a number of features (e.g., IDE support, C++ variadic |
| 1851 | templates). If you have sub-types, you'll also need to visit those |
Stephen Hines | c568f1e | 2014-07-21 00:47:37 -0700 | [diff] [blame] | 1852 | sub-types in ``RecursiveASTVisitor`` and ``DataRecursiveASTVisitor``. |
| 1853 | * Add printing support (``StmtPrinter.cpp``) for your expression. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1854 | * Add profiling support (``StmtProfile.cpp``) for your AST node, noting the |
| 1855 | distinguishing (non-source location) characteristics of an instance of |
| 1856 | your expression. Omitting this step will lead to hard-to-diagnose |
| 1857 | failures regarding matching of template declarations. |
Stephen Hines | c568f1e | 2014-07-21 00:47:37 -0700 | [diff] [blame] | 1858 | * Add serialization support (``ASTReaderStmt.cpp``, ``ASTWriterStmt.cpp``) |
| 1859 | for your AST node. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1860 | |
| 1861 | #. Teach semantic analysis to build your AST node. At this point, you can wire |
| 1862 | up your ``Sema::BuildXXX`` function to actually create your AST. A few |
| 1863 | things to check at this point: |
| 1864 | |
| 1865 | * If your expression can construct a new C++ class or return a new |
| 1866 | Objective-C object, be sure to update and then call |
| 1867 | ``Sema::MaybeBindToTemporary`` for your just-created AST node to be sure |
| 1868 | that the object gets properly destructed. An easy way to test this is to |
| 1869 | return a C++ class with a private destructor: semantic analysis should |
| 1870 | flag an error here with the attempt to call the destructor. |
| 1871 | * Inspect the generated AST by printing it using ``clang -cc1 -ast-print``, |
| 1872 | to make sure you're capturing all of the important information about how |
| 1873 | the AST was written. |
| 1874 | * Inspect the generated AST under ``clang -cc1 -ast-dump`` to verify that |
| 1875 | all of the types in the generated AST line up the way you want them. |
| 1876 | Remember that clients of the AST should never have to "think" to |
| 1877 | understand what's going on. For example, all implicit conversions should |
| 1878 | show up explicitly in the AST. |
| 1879 | * Write tests that use your expression as a subexpression of other, |
| 1880 | well-known expressions. Can you call a function using your expression as |
| 1881 | an argument? Can you use the ternary operator? |
| 1882 | |
| 1883 | #. Teach code generation to create IR to your AST node. This step is the first |
| 1884 | (and only) that requires knowledge of LLVM IR. There are several things to |
| 1885 | keep in mind: |
| 1886 | |
| 1887 | * Code generation is separated into scalar/aggregate/complex and |
| 1888 | lvalue/rvalue paths, depending on what kind of result your expression |
| 1889 | produces. On occasion, this requires some careful factoring of code to |
| 1890 | avoid duplication. |
| 1891 | * ``CodeGenFunction`` contains functions ``ConvertType`` and |
| 1892 | ``ConvertTypeForMem`` that convert Clang's types (``clang::Type*`` or |
| 1893 | ``clang::QualType``) to LLVM types. Use the former for values, and the |
| 1894 | later for memory locations: test with the C++ "``bool``" type to check |
| 1895 | this. If you find that you are having to use LLVM bitcasts to make the |
| 1896 | subexpressions of your expression have the type that your expression |
| 1897 | expects, STOP! Go fix semantic analysis and the AST so that you don't |
| 1898 | need these bitcasts. |
| 1899 | * The ``CodeGenFunction`` class has a number of helper functions to make |
| 1900 | certain operations easy, such as generating code to produce an lvalue or |
| 1901 | an rvalue, or to initialize a memory location with a given value. Prefer |
| 1902 | to use these functions rather than directly writing loads and stores, |
| 1903 | because these functions take care of some of the tricky details for you |
| 1904 | (e.g., for exceptions). |
| 1905 | * If your expression requires some special behavior in the event of an |
| 1906 | exception, look at the ``push*Cleanup`` functions in ``CodeGenFunction`` |
| 1907 | to introduce a cleanup. You shouldn't have to deal with |
| 1908 | exception-handling directly. |
| 1909 | * Testing is extremely important in IR generation. Use ``clang -cc1 |
| 1910 | -emit-llvm`` and `FileCheck |
| 1911 | <http://llvm.org/docs/CommandGuide/FileCheck.html>`_ to verify that you're |
| 1912 | generating the right IR. |
| 1913 | |
| 1914 | #. Teach template instantiation how to cope with your AST node, which requires |
| 1915 | some fairly simple code: |
| 1916 | |
| 1917 | * Make sure that your expression's constructor properly computes the flags |
| 1918 | for type dependence (i.e., the type your expression produces can change |
| 1919 | from one instantiation to the next), value dependence (i.e., the constant |
| 1920 | value your expression produces can change from one instantiation to the |
| 1921 | next), instantiation dependence (i.e., a template parameter occurs |
| 1922 | anywhere in your expression), and whether your expression contains a |
| 1923 | parameter pack (for variadic templates). Often, computing these flags |
| 1924 | just means combining the results from the various types and |
| 1925 | subexpressions. |
| 1926 | * Add ``TransformXXX`` and ``RebuildXXX`` functions to the ``TreeTransform`` |
| 1927 | class template in ``Sema``. ``TransformXXX`` should (recursively) |
| 1928 | transform all of the subexpressions and types within your expression, |
| 1929 | using ``getDerived().TransformYYY``. If all of the subexpressions and |
| 1930 | types transform without error, it will then call the ``RebuildXXX`` |
| 1931 | function, which will in turn call ``getSema().BuildXXX`` to perform |
| 1932 | semantic analysis and build your expression. |
| 1933 | * To test template instantiation, take those tests you wrote to make sure |
| 1934 | that you were type checking with type-dependent expressions and dependent |
| 1935 | types (from step #2) and instantiate those templates with various types, |
| 1936 | some of which type-check and some that don't, and test the error messages |
| 1937 | in each case. |
| 1938 | |
| 1939 | #. There are some "extras" that make other features work better. It's worth |
| 1940 | handling these extras to give your expression complete integration into |
| 1941 | Clang: |
| 1942 | |
| 1943 | * Add code completion support for your expression in |
| 1944 | ``SemaCodeComplete.cpp``. |
| 1945 | * If your expression has types in it, or has any "interesting" features |
| 1946 | other than subexpressions, extend libclang's ``CursorVisitor`` to provide |
| 1947 | proper visitation for your expression, enabling various IDE features such |
| 1948 | as syntax highlighting, cross-referencing, and so on. The |
| 1949 | ``c-index-test`` helper program can be used to test these features. |
| 1950 | |