Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1 | ============================ |
| 2 | "Clang" CFE Internals Manual |
| 3 | ============================ |
| 4 | |
| 5 | .. contents:: |
| 6 | :local: |
| 7 | |
| 8 | Introduction |
| 9 | ============ |
| 10 | |
| 11 | This document describes some of the more important APIs and internal design |
| 12 | decisions made in the Clang C front-end. The purpose of this document is to |
| 13 | both capture some of this high level information and also describe some of the |
| 14 | design decisions behind it. This is meant for people interested in hacking on |
| 15 | Clang, not for end-users. The description below is categorized by libraries, |
| 16 | and does not describe any of the clients of the libraries. |
| 17 | |
| 18 | LLVM Support Library |
| 19 | ==================== |
| 20 | |
| 21 | The LLVM ``libSupport`` library provides many underlying libraries and |
| 22 | `data-structures <http://llvm.org/docs/ProgrammersManual.html>`_, including |
| 23 | command line option processing, various containers and a system abstraction |
| 24 | layer, which is used for file system access. |
| 25 | |
| 26 | The Clang "Basic" Library |
| 27 | ========================= |
| 28 | |
| 29 | This library certainly needs a better name. The "basic" library contains a |
| 30 | number of low-level utilities for tracking and manipulating source buffers, |
| 31 | locations within the source buffers, diagnostics, tokens, target abstraction, |
| 32 | and information about the subset of the language being compiled for. |
| 33 | |
| 34 | Part of this infrastructure is specific to C (such as the ``TargetInfo`` |
| 35 | class), other parts could be reused for other non-C-based languages |
| 36 | (``SourceLocation``, ``SourceManager``, ``Diagnostics``, ``FileManager``). |
| 37 | When and if there is future demand we can figure out if it makes sense to |
| 38 | introduce a new library, move the general classes somewhere else, or introduce |
| 39 | some other solution. |
| 40 | |
| 41 | We describe the roles of these classes in order of their dependencies. |
| 42 | |
| 43 | The Diagnostics Subsystem |
| 44 | ------------------------- |
| 45 | |
| 46 | The Clang Diagnostics subsystem is an important part of how the compiler |
| 47 | communicates with the human. Diagnostics are the warnings and errors produced |
| 48 | when the code is incorrect or dubious. In Clang, each diagnostic produced has |
| 49 | (at the minimum) a unique ID, an English translation associated with it, a |
| 50 | :ref:`SourceLocation <SourceLocation>` to "put the caret", and a severity |
| 51 | (e.g., ``WARNING`` or ``ERROR``). They can also optionally include a number of |
| 52 | arguments to the dianostic (which fill in "%0"'s in the string) as well as a |
| 53 | number of source ranges that related to the diagnostic. |
| 54 | |
| 55 | In this section, we'll be giving examples produced by the Clang command line |
| 56 | driver, but diagnostics can be :ref:`rendered in many different ways |
| 57 | <DiagnosticClient>` depending on how the ``DiagnosticClient`` interface is |
| 58 | implemented. A representative example of a diagnostic is: |
| 59 | |
Pirama Arumuga Nainar | 4967a71 | 2016-09-19 22:19:55 -0700 | [diff] [blame] | 60 | .. code-block:: text |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 61 | |
| 62 | t.c:38:15: error: invalid operands to binary expression ('int *' and '_Complex float') |
| 63 | P = (P-42) + Gamma*4; |
| 64 | ~~~~~~ ^ ~~~~~~~ |
| 65 | |
| 66 | In this example, you can see the English translation, the severity (error), you |
| 67 | can see the source location (the caret ("``^``") and file/line/column info), |
| 68 | the source ranges "``~~~~``", arguments to the diagnostic ("``int*``" and |
| 69 | "``_Complex float``"). You'll have to believe me that there is a unique ID |
| 70 | backing the diagnostic :). |
| 71 | |
| 72 | Getting all of this to happen has several steps and involves many moving |
| 73 | pieces, this section describes them and talks about best practices when adding |
| 74 | a new diagnostic. |
| 75 | |
Dmitri Gribenko | 97555a1 | 2012-12-15 21:10:51 +0000 | [diff] [blame] | 76 | The ``Diagnostic*Kinds.td`` files |
| 77 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 78 | |
| 79 | Diagnostics are created by adding an entry to one of the |
| 80 | ``clang/Basic/Diagnostic*Kinds.td`` files, depending on what library will be |
| 81 | using it. From this file, :program:`tblgen` generates the unique ID of the |
| 82 | diagnostic, the severity of the diagnostic and the English translation + format |
| 83 | string. |
| 84 | |
| 85 | There is little sanity with the naming of the unique ID's right now. Some |
| 86 | start with ``err_``, ``warn_``, ``ext_`` to encode the severity into the name. |
| 87 | Since the enum is referenced in the C++ code that produces the diagnostic, it |
| 88 | is somewhat useful for it to be reasonably short. |
| 89 | |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 90 | The severity of the diagnostic comes from the set {``NOTE``, ``REMARK``, |
| 91 | ``WARNING``, |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 92 | ``EXTENSION``, ``EXTWARN``, ``ERROR``}. The ``ERROR`` severity is used for |
| 93 | diagnostics indicating the program is never acceptable under any circumstances. |
| 94 | When an error is emitted, the AST for the input code may not be fully built. |
| 95 | The ``EXTENSION`` and ``EXTWARN`` severities are used for extensions to the |
| 96 | language that Clang accepts. This means that Clang fully understands and can |
| 97 | represent them in the AST, but we produce diagnostics to tell the user their |
| 98 | code is non-portable. The difference is that the former are ignored by |
| 99 | default, and the later warn by default. The ``WARNING`` severity is used for |
| 100 | constructs that are valid in the currently selected source language but that |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 101 | are dubious in some way. The ``REMARK`` severity provides generic information |
| 102 | about the compilation that is not necessarily related to any dubious code. The |
| 103 | ``NOTE`` level is used to staple more information onto previous diagnostics. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 104 | |
| 105 | These *severities* are mapped into a smaller set (the ``Diagnostic::Level`` |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 106 | enum, {``Ignored``, ``Note``, ``Remark``, ``Warning``, ``Error``, ``Fatal``}) of |
| 107 | output |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 108 | *levels* by the diagnostics subsystem based on various configuration options. |
| 109 | Clang internally supports a fully fine grained mapping mechanism that allows |
| 110 | you to map almost any diagnostic to the output level that you want. The only |
| 111 | diagnostics that cannot be mapped are ``NOTE``\ s, which always follow the |
| 112 | severity of the previously emitted diagnostic and ``ERROR``\ s, which can only |
| 113 | be mapped to ``Fatal`` (it is not possible to turn an error into a warning, for |
| 114 | example). |
| 115 | |
| 116 | Diagnostic mappings are used in many ways. For example, if the user specifies |
| 117 | ``-pedantic``, ``EXTENSION`` maps to ``Warning``, if they specify |
| 118 | ``-pedantic-errors``, it turns into ``Error``. This is used to implement |
| 119 | options like ``-Wunused_macros``, ``-Wundef`` etc. |
| 120 | |
| 121 | Mapping to ``Fatal`` should only be used for diagnostics that are considered so |
| 122 | severe that error recovery won't be able to recover sensibly from them (thus |
| 123 | spewing a ton of bogus errors). One example of this class of error are failure |
| 124 | to ``#include`` a file. |
| 125 | |
| 126 | The Format String |
| 127 | ^^^^^^^^^^^^^^^^^ |
| 128 | |
| 129 | The format string for the diagnostic is very simple, but it has some power. It |
| 130 | takes the form of a string in English with markers that indicate where and how |
| 131 | arguments to the diagnostic are inserted and formatted. For example, here are |
| 132 | some simple format strings: |
| 133 | |
| 134 | .. code-block:: c++ |
| 135 | |
| 136 | "binary integer literals are an extension" |
| 137 | "format string contains '\\0' within the string body" |
| 138 | "more '%%' conversions than data arguments" |
| 139 | "invalid operands to binary expression (%0 and %1)" |
| 140 | "overloaded '%0' must be a %select{unary|binary|unary or binary}2 operator" |
| 141 | " (has %1 parameter%s1)" |
| 142 | |
| 143 | These examples show some important points of format strings. You can use any |
| 144 | plain ASCII character in the diagnostic string except "``%``" without a |
| 145 | problem, but these are C strings, so you have to use and be aware of all the C |
| 146 | escape sequences (as in the second example). If you want to produce a "``%``" |
| 147 | in the output, use the "``%%``" escape sequence, like the third diagnostic. |
| 148 | Finally, Clang uses the "``%...[digit]``" sequences to specify where and how |
| 149 | arguments to the diagnostic are formatted. |
| 150 | |
| 151 | Arguments to the diagnostic are numbered according to how they are specified by |
| 152 | the C++ code that :ref:`produces them <internals-producing-diag>`, and are |
| 153 | referenced by ``%0`` .. ``%9``. If you have more than 10 arguments to your |
| 154 | diagnostic, you are doing something wrong :). Unlike ``printf``, there is no |
| 155 | requirement that arguments to the diagnostic end up in the output in the same |
| 156 | order as they are specified, you could have a format string with "``%1 %0``" |
| 157 | that swaps them, for example. The text in between the percent and digit are |
| 158 | formatting instructions. If there are no instructions, the argument is just |
| 159 | turned into a string and substituted in. |
| 160 | |
| 161 | Here are some "best practices" for writing the English format string: |
| 162 | |
| 163 | * Keep the string short. It should ideally fit in the 80 column limit of the |
| 164 | ``DiagnosticKinds.td`` file. This avoids the diagnostic wrapping when |
| 165 | printed, and forces you to think about the important point you are conveying |
| 166 | with the diagnostic. |
| 167 | * Take advantage of location information. The user will be able to see the |
| 168 | line and location of the caret, so you don't need to tell them that the |
| 169 | problem is with the 4th argument to the function: just point to it. |
| 170 | * Do not capitalize the diagnostic string, and do not end it with a period. |
| 171 | * If you need to quote something in the diagnostic string, use single quotes. |
| 172 | |
| 173 | Diagnostics should never take random English strings as arguments: you |
| 174 | shouldn't use "``you have a problem with %0``" and pass in things like "``your |
| 175 | argument``" or "``your return value``" as arguments. Doing this prevents |
| 176 | :ref:`translating <internals-diag-translation>` the Clang diagnostics to other |
| 177 | languages (because they'll get random English words in their otherwise |
| 178 | localized diagnostic). The exceptions to this are C/C++ language keywords |
| 179 | (e.g., ``auto``, ``const``, ``mutable``, etc) and C/C++ operators (``/=``). |
| 180 | Note that things like "pointer" and "reference" are not keywords. On the other |
| 181 | hand, you *can* include anything that comes from the user's source code, |
| 182 | including variable names, types, labels, etc. The "``select``" format can be |
| 183 | used to achieve this sort of thing in a localizable way, see below. |
| 184 | |
| 185 | Formatting a Diagnostic Argument |
| 186 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 187 | |
| 188 | Arguments to diagnostics are fully typed internally, and come from a couple |
| 189 | different classes: integers, types, names, and random strings. Depending on |
| 190 | the class of the argument, it can be optionally formatted in different ways. |
| 191 | This gives the ``DiagnosticClient`` information about what the argument means |
| 192 | without requiring it to use a specific presentation (consider this MVC for |
| 193 | Clang :). |
| 194 | |
| 195 | Here are the different diagnostic argument formats currently supported by |
| 196 | Clang: |
| 197 | |
| 198 | **"s" format** |
| 199 | |
| 200 | Example: |
| 201 | ``"requires %1 parameter%s1"`` |
| 202 | Class: |
| 203 | Integers |
| 204 | Description: |
| 205 | This is a simple formatter for integers that is useful when producing English |
| 206 | diagnostics. When the integer is 1, it prints as nothing. When the integer |
| 207 | is not 1, it prints as "``s``". This allows some simple grammatical forms to |
| 208 | be to be handled correctly, and eliminates the need to use gross things like |
| 209 | ``"requires %1 parameter(s)"``. |
| 210 | |
| 211 | **"select" format** |
| 212 | |
| 213 | Example: |
| 214 | ``"must be a %select{unary|binary|unary or binary}2 operator"`` |
| 215 | Class: |
| 216 | Integers |
| 217 | Description: |
| 218 | This format specifier is used to merge multiple related diagnostics together |
| 219 | into one common one, without requiring the difference to be specified as an |
| 220 | English string argument. Instead of specifying the string, the diagnostic |
| 221 | gets an integer argument and the format string selects the numbered option. |
| 222 | In this case, the "``%2``" value must be an integer in the range [0..2]. If |
| 223 | it is 0, it prints "unary", if it is 1 it prints "binary" if it is 2, it |
| 224 | prints "unary or binary". This allows other language translations to |
| 225 | substitute reasonable words (or entire phrases) based on the semantics of the |
| 226 | diagnostic instead of having to do things textually. The selected string |
| 227 | does undergo formatting. |
| 228 | |
| 229 | **"plural" format** |
| 230 | |
| 231 | Example: |
| 232 | ``"you have %1 %plural{1:mouse|:mice}1 connected to your computer"`` |
| 233 | Class: |
| 234 | Integers |
| 235 | Description: |
| 236 | This is a formatter for complex plural forms. It is designed to handle even |
| 237 | the requirements of languages with very complex plural forms, as many Baltic |
| 238 | languages have. The argument consists of a series of expression/form pairs, |
| 239 | separated by ":", where the first form whose expression evaluates to true is |
| 240 | the result of the modifier. |
| 241 | |
| 242 | An expression can be empty, in which case it is always true. See the example |
| 243 | at the top. Otherwise, it is a series of one or more numeric conditions, |
| 244 | separated by ",". If any condition matches, the expression matches. Each |
| 245 | numeric condition can take one of three forms. |
| 246 | |
| 247 | * number: A simple decimal number matches if the argument is the same as the |
| 248 | number. Example: ``"%plural{1:mouse|:mice}4"`` |
| 249 | * range: A range in square brackets matches if the argument is within the |
| 250 | range. Then range is inclusive on both ends. Example: |
| 251 | ``"%plural{0:none|1:one|[2,5]:some|:many}2"`` |
| 252 | * modulo: A modulo operator is followed by a number, and equals sign and |
| 253 | either a number or a range. The tests are the same as for plain numbers |
| 254 | and ranges, but the argument is taken modulo the number first. Example: |
| 255 | ``"%plural{%100=0:even hundred|%100=[1,50]:lower half|:everything else}1"`` |
| 256 | |
| 257 | The parser is very unforgiving. A syntax error, even whitespace, will abort, |
| 258 | as will a failure to match the argument against any expression. |
| 259 | |
| 260 | **"ordinal" format** |
| 261 | |
| 262 | Example: |
| 263 | ``"ambiguity in %ordinal0 argument"`` |
| 264 | Class: |
| 265 | Integers |
| 266 | Description: |
| 267 | This is a formatter which represents the argument number as an ordinal: the |
| 268 | value ``1`` becomes ``1st``, ``3`` becomes ``3rd``, and so on. Values less |
| 269 | than ``1`` are not supported. This formatter is currently hard-coded to use |
| 270 | English ordinals. |
| 271 | |
| 272 | **"objcclass" format** |
| 273 | |
| 274 | Example: |
| 275 | ``"method %objcclass0 not found"`` |
| 276 | Class: |
| 277 | ``DeclarationName`` |
| 278 | Description: |
| 279 | This is a simple formatter that indicates the ``DeclarationName`` corresponds |
| 280 | to an Objective-C class method selector. As such, it prints the selector |
| 281 | with a leading "``+``". |
| 282 | |
| 283 | **"objcinstance" format** |
| 284 | |
| 285 | Example: |
| 286 | ``"method %objcinstance0 not found"`` |
| 287 | Class: |
| 288 | ``DeclarationName`` |
| 289 | Description: |
| 290 | This is a simple formatter that indicates the ``DeclarationName`` corresponds |
| 291 | to an Objective-C instance method selector. As such, it prints the selector |
| 292 | with a leading "``-``". |
| 293 | |
| 294 | **"q" format** |
| 295 | |
| 296 | Example: |
| 297 | ``"candidate found by name lookup is %q0"`` |
| 298 | Class: |
| 299 | ``NamedDecl *`` |
| 300 | Description: |
| 301 | This formatter indicates that the fully-qualified name of the declaration |
| 302 | should be printed, e.g., "``std::vector``" rather than "``vector``". |
| 303 | |
| 304 | **"diff" format** |
| 305 | |
| 306 | Example: |
| 307 | ``"no known conversion %diff{from $ to $|from argument type to parameter type}1,2"`` |
| 308 | Class: |
| 309 | ``QualType`` |
| 310 | Description: |
| 311 | This formatter takes two ``QualType``\ s and attempts to print a template |
| 312 | difference between the two. If tree printing is off, the text inside the |
| 313 | braces before the pipe is printed, with the formatted text replacing the $. |
| 314 | If tree printing is on, the text after the pipe is printed and a type tree is |
| 315 | printed after the diagnostic message. |
| 316 | |
| 317 | It is really easy to add format specifiers to the Clang diagnostics system, but |
| 318 | they should be discussed before they are added. If you are creating a lot of |
| 319 | repetitive diagnostics and/or have an idea for a useful formatter, please bring |
| 320 | it up on the cfe-dev mailing list. |
| 321 | |
| 322 | .. _internals-producing-diag: |
| 323 | |
| 324 | Producing the Diagnostic |
| 325 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
| 326 | |
| 327 | Now that you've created the diagnostic in the ``Diagnostic*Kinds.td`` file, you |
| 328 | need to write the code that detects the condition in question and emits the new |
| 329 | diagnostic. Various components of Clang (e.g., the preprocessor, ``Sema``, |
| 330 | etc.) provide a helper function named "``Diag``". It creates a diagnostic and |
| 331 | accepts the arguments, ranges, and other information that goes along with it. |
| 332 | |
| 333 | For example, the binary expression error comes from code like this: |
| 334 | |
| 335 | .. code-block:: c++ |
| 336 | |
| 337 | if (various things that are bad) |
| 338 | Diag(Loc, diag::err_typecheck_invalid_operands) |
| 339 | << lex->getType() << rex->getType() |
| 340 | << lex->getSourceRange() << rex->getSourceRange(); |
| 341 | |
| 342 | This shows that use of the ``Diag`` method: it takes a location (a |
| 343 | :ref:`SourceLocation <SourceLocation>` object) and a diagnostic enum value |
| 344 | (which matches the name from ``Diagnostic*Kinds.td``). If the diagnostic takes |
| 345 | arguments, they are specified with the ``<<`` operator: the first argument |
| 346 | becomes ``%0``, the second becomes ``%1``, etc. The diagnostic interface |
| 347 | allows you to specify arguments of many different types, including ``int`` and |
| 348 | ``unsigned`` for integer arguments, ``const char*`` and ``std::string`` for |
| 349 | string arguments, ``DeclarationName`` and ``const IdentifierInfo *`` for names, |
| 350 | ``QualType`` for types, etc. ``SourceRange``\ s are also specified with the |
| 351 | ``<<`` operator, but do not have a specific ordering requirement. |
| 352 | |
| 353 | As you can see, adding and producing a diagnostic is pretty straightforward. |
| 354 | The hard part is deciding exactly what you need to say to help the user, |
| 355 | picking a suitable wording, and providing the information needed to format it |
| 356 | correctly. The good news is that the call site that issues a diagnostic should |
| 357 | be completely independent of how the diagnostic is formatted and in what |
| 358 | language it is rendered. |
| 359 | |
| 360 | Fix-It Hints |
| 361 | ^^^^^^^^^^^^ |
| 362 | |
| 363 | In some cases, the front end emits diagnostics when it is clear that some small |
| 364 | change to the source code would fix the problem. For example, a missing |
| 365 | semicolon at the end of a statement or a use of deprecated syntax that is |
| 366 | easily rewritten into a more modern form. Clang tries very hard to emit the |
| 367 | diagnostic and recover gracefully in these and other cases. |
| 368 | |
| 369 | However, for these cases where the fix is obvious, the diagnostic can be |
| 370 | annotated with a hint (referred to as a "fix-it hint") that describes how to |
| 371 | change the code referenced by the diagnostic to fix the problem. For example, |
| 372 | it might add the missing semicolon at the end of the statement or rewrite the |
| 373 | use of a deprecated construct into something more palatable. Here is one such |
| 374 | example from the C++ front end, where we warn about the right-shift operator |
| 375 | changing meaning from C++98 to C++11: |
| 376 | |
Pirama Arumuga Nainar | 4967a71 | 2016-09-19 22:19:55 -0700 | [diff] [blame] | 377 | .. code-block:: text |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 378 | |
| 379 | test.cpp:3:7: warning: use of right-shift operator ('>>') in template argument |
| 380 | will require parentheses in C++11 |
| 381 | A<100 >> 2> *a; |
| 382 | ^ |
| 383 | ( ) |
| 384 | |
| 385 | Here, the fix-it hint is suggesting that parentheses be added, and showing |
| 386 | exactly where those parentheses would be inserted into the source code. The |
| 387 | fix-it hints themselves describe what changes to make to the source code in an |
| 388 | abstract manner, which the text diagnostic printer renders as a line of |
| 389 | "insertions" below the caret line. :ref:`Other diagnostic clients |
| 390 | <DiagnosticClient>` might choose to render the code differently (e.g., as |
| 391 | markup inline) or even give the user the ability to automatically fix the |
| 392 | problem. |
| 393 | |
| 394 | Fix-it hints on errors and warnings need to obey these rules: |
| 395 | |
| 396 | * Since they are automatically applied if ``-Xclang -fixit`` is passed to the |
| 397 | driver, they should only be used when it's very likely they match the user's |
| 398 | intent. |
| 399 | * Clang must recover from errors as if the fix-it had been applied. |
| 400 | |
| 401 | If a fix-it can't obey these rules, put the fix-it on a note. Fix-its on notes |
| 402 | are not applied automatically. |
| 403 | |
| 404 | All fix-it hints are described by the ``FixItHint`` class, instances of which |
| 405 | should be attached to the diagnostic using the ``<<`` operator in the same way |
| 406 | that highlighted source ranges and arguments are passed to the diagnostic. |
| 407 | Fix-it hints can be created with one of three constructors: |
| 408 | |
| 409 | * ``FixItHint::CreateInsertion(Loc, Code)`` |
| 410 | |
| 411 | Specifies that the given ``Code`` (a string) should be inserted before the |
| 412 | source location ``Loc``. |
| 413 | |
| 414 | * ``FixItHint::CreateRemoval(Range)`` |
| 415 | |
| 416 | Specifies that the code in the given source ``Range`` should be removed. |
| 417 | |
| 418 | * ``FixItHint::CreateReplacement(Range, Code)`` |
| 419 | |
| 420 | Specifies that the code in the given source ``Range`` should be removed, |
| 421 | and replaced with the given ``Code`` string. |
| 422 | |
| 423 | .. _DiagnosticClient: |
| 424 | |
| 425 | The ``DiagnosticClient`` Interface |
| 426 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 427 | |
| 428 | Once code generates a diagnostic with all of the arguments and the rest of the |
| 429 | relevant information, Clang needs to know what to do with it. As previously |
| 430 | mentioned, the diagnostic machinery goes through some filtering to map a |
| 431 | severity onto a diagnostic level, then (assuming the diagnostic is not mapped |
| 432 | to "``Ignore``") it invokes an object that implements the ``DiagnosticClient`` |
| 433 | interface with the information. |
| 434 | |
| 435 | It is possible to implement this interface in many different ways. For |
| 436 | example, the normal Clang ``DiagnosticClient`` (named |
| 437 | ``TextDiagnosticPrinter``) turns the arguments into strings (according to the |
| 438 | various formatting rules), prints out the file/line/column information and the |
| 439 | string, then prints out the line of code, the source ranges, and the caret. |
| 440 | However, this behavior isn't required. |
| 441 | |
| 442 | Another implementation of the ``DiagnosticClient`` interface is the |
| 443 | ``TextDiagnosticBuffer`` class, which is used when Clang is in ``-verify`` |
| 444 | mode. Instead of formatting and printing out the diagnostics, this |
| 445 | implementation just captures and remembers the diagnostics as they fly by. |
| 446 | Then ``-verify`` compares the list of produced diagnostics to the list of |
| 447 | expected ones. If they disagree, it prints out its own output. Full |
| 448 | documentation for the ``-verify`` mode can be found in the Clang API |
| 449 | documentation for `VerifyDiagnosticConsumer |
| 450 | </doxygen/classclang_1_1VerifyDiagnosticConsumer.html#details>`_. |
| 451 | |
| 452 | There are many other possible implementations of this interface, and this is |
| 453 | why we prefer diagnostics to pass down rich structured information in |
| 454 | arguments. For example, an HTML output might want declaration names be |
| 455 | linkified to where they come from in the source. Another example is that a GUI |
| 456 | might let you click on typedefs to expand them. This application would want to |
| 457 | pass significantly more information about types through to the GUI than a |
| 458 | simple flat string. The interface allows this to happen. |
| 459 | |
| 460 | .. _internals-diag-translation: |
| 461 | |
| 462 | Adding Translations to Clang |
| 463 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 464 | |
| 465 | Not possible yet! Diagnostic strings should be written in UTF-8, the client can |
| 466 | translate to the relevant code page if needed. Each translation completely |
| 467 | replaces the format string for the diagnostic. |
| 468 | |
| 469 | .. _SourceLocation: |
| 470 | .. _SourceManager: |
| 471 | |
| 472 | The ``SourceLocation`` and ``SourceManager`` classes |
| 473 | ---------------------------------------------------- |
| 474 | |
| 475 | Strangely enough, the ``SourceLocation`` class represents a location within the |
| 476 | source code of the program. Important design points include: |
| 477 | |
| 478 | #. ``sizeof(SourceLocation)`` must be extremely small, as these are embedded |
| 479 | into many AST nodes and are passed around often. Currently it is 32 bits. |
| 480 | #. ``SourceLocation`` must be a simple value object that can be efficiently |
| 481 | copied. |
| 482 | #. We should be able to represent a source location for any byte of any input |
| 483 | file. This includes in the middle of tokens, in whitespace, in trigraphs, |
| 484 | etc. |
| 485 | #. A ``SourceLocation`` must encode the current ``#include`` stack that was |
| 486 | active when the location was processed. For example, if the location |
| 487 | corresponds to a token, it should contain the set of ``#include``\ s active |
| 488 | when the token was lexed. This allows us to print the ``#include`` stack |
| 489 | for a diagnostic. |
| 490 | #. ``SourceLocation`` must be able to describe macro expansions, capturing both |
| 491 | the ultimate instantiation point and the source of the original character |
| 492 | data. |
| 493 | |
| 494 | In practice, the ``SourceLocation`` works together with the ``SourceManager`` |
| 495 | class to encode two pieces of information about a location: its spelling |
| 496 | location and its instantiation location. For most tokens, these will be the |
| 497 | same. However, for a macro expansion (or tokens that came from a ``_Pragma`` |
| 498 | directive) these will describe the location of the characters corresponding to |
| 499 | the token and the location where the token was used (i.e., the macro |
| 500 | instantiation point or the location of the ``_Pragma`` itself). |
| 501 | |
| 502 | The Clang front-end inherently depends on the location of a token being tracked |
| 503 | correctly. If it is ever incorrect, the front-end may get confused and die. |
| 504 | The reason for this is that the notion of the "spelling" of a ``Token`` in |
| 505 | Clang depends on being able to find the original input characters for the |
| 506 | token. This concept maps directly to the "spelling location" for the token. |
| 507 | |
| 508 | ``SourceRange`` and ``CharSourceRange`` |
| 509 | --------------------------------------- |
| 510 | |
Pirama Arumuga Nainar | 87d948e | 2016-03-03 15:49:35 -0800 | [diff] [blame] | 511 | .. mostly taken from http://lists.llvm.org/pipermail/cfe-dev/2010-August/010595.html |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 512 | |
| 513 | Clang represents most source ranges by [first, last], where "first" and "last" |
| 514 | each point to the beginning of their respective tokens. For example consider |
| 515 | the ``SourceRange`` of the following statement: |
| 516 | |
Pirama Arumuga Nainar | 4967a71 | 2016-09-19 22:19:55 -0700 | [diff] [blame] | 517 | .. code-block:: text |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 518 | |
| 519 | x = foo + bar; |
| 520 | ^first ^last |
| 521 | |
| 522 | To map from this representation to a character-based representation, the "last" |
| 523 | location needs to be adjusted to point to (or past) the end of that token with |
| 524 | either ``Lexer::MeasureTokenLength()`` or ``Lexer::getLocForEndOfToken()``. For |
| 525 | the rare cases where character-level source ranges information is needed we use |
| 526 | the ``CharSourceRange`` class. |
| 527 | |
| 528 | The Driver Library |
| 529 | ================== |
| 530 | |
Sean Silva | 159cc9e | 2013-01-02 13:07:47 +0000 | [diff] [blame] | 531 | The clang Driver and library are documented :doc:`here <DriverInternals>`. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 532 | |
| 533 | Precompiled Headers |
| 534 | =================== |
| 535 | |
| 536 | Clang supports two implementations of precompiled headers. The default |
| 537 | implementation, precompiled headers (:doc:`PCH <PCHInternals>`) uses a |
| 538 | serialized representation of Clang's internal data structures, encoded with the |
| 539 | `LLVM bitstream format <http://llvm.org/docs/BitCodeFormat.html>`_. |
Sean Silva | 159cc9e | 2013-01-02 13:07:47 +0000 | [diff] [blame] | 540 | Pretokenized headers (:doc:`PTH <PTHInternals>`), on the other hand, contain a |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 541 | serialized representation of the tokens encountered when preprocessing a header |
| 542 | (and anything that header includes). |
| 543 | |
| 544 | The Frontend Library |
| 545 | ==================== |
| 546 | |
| 547 | The Frontend library contains functionality useful for building tools on top of |
| 548 | the Clang libraries, for example several methods for outputting diagnostics. |
| 549 | |
| 550 | The Lexer and Preprocessor Library |
| 551 | ================================== |
| 552 | |
| 553 | The Lexer library contains several tightly-connected classes that are involved |
| 554 | with the nasty process of lexing and preprocessing C source code. The main |
| 555 | interface to this library for outside clients is the large ``Preprocessor`` |
| 556 | class. It contains the various pieces of state that are required to coherently |
| 557 | read tokens out of a translation unit. |
| 558 | |
| 559 | The core interface to the ``Preprocessor`` object (once it is set up) is the |
| 560 | ``Preprocessor::Lex`` method, which returns the next :ref:`Token <Token>` from |
| 561 | the preprocessor stream. There are two types of token providers that the |
| 562 | preprocessor is capable of reading from: a buffer lexer (provided by the |
| 563 | :ref:`Lexer <Lexer>` class) and a buffered token stream (provided by the |
| 564 | :ref:`TokenLexer <TokenLexer>` class). |
| 565 | |
| 566 | .. _Token: |
| 567 | |
| 568 | The Token class |
| 569 | --------------- |
| 570 | |
| 571 | The ``Token`` class is used to represent a single lexed token. Tokens are |
| 572 | intended to be used by the lexer/preprocess and parser libraries, but are not |
| 573 | intended to live beyond them (for example, they should not live in the ASTs). |
| 574 | |
| 575 | Tokens most often live on the stack (or some other location that is efficient |
| 576 | to access) as the parser is running, but occasionally do get buffered up. For |
| 577 | example, macro definitions are stored as a series of tokens, and the C++ |
| 578 | front-end periodically needs to buffer tokens up for tentative parsing and |
| 579 | various pieces of look-ahead. As such, the size of a ``Token`` matters. On a |
| 580 | 32-bit system, ``sizeof(Token)`` is currently 16 bytes. |
| 581 | |
| 582 | Tokens occur in two forms: :ref:`annotation tokens <AnnotationToken>` and |
| 583 | normal tokens. Normal tokens are those returned by the lexer, annotation |
| 584 | tokens represent semantic information and are produced by the parser, replacing |
| 585 | normal tokens in the token stream. Normal tokens contain the following |
| 586 | information: |
| 587 | |
| 588 | * **A SourceLocation** --- This indicates the location of the start of the |
| 589 | token. |
| 590 | |
| 591 | * **A length** --- This stores the length of the token as stored in the |
| 592 | ``SourceBuffer``. For tokens that include them, this length includes |
| 593 | trigraphs and escaped newlines which are ignored by later phases of the |
| 594 | compiler. By pointing into the original source buffer, it is always possible |
| 595 | to get the original spelling of a token completely accurately. |
| 596 | |
| 597 | * **IdentifierInfo** --- If a token takes the form of an identifier, and if |
| 598 | identifier lookup was enabled when the token was lexed (e.g., the lexer was |
| 599 | not reading in "raw" mode) this contains a pointer to the unique hash value |
| 600 | for the identifier. Because the lookup happens before keyword |
| 601 | identification, this field is set even for language keywords like "``for``". |
| 602 | |
| 603 | * **TokenKind** --- This indicates the kind of token as classified by the |
| 604 | lexer. This includes things like ``tok::starequal`` (for the "``*=``" |
| 605 | operator), ``tok::ampamp`` for the "``&&``" token, and keyword values (e.g., |
| 606 | ``tok::kw_for``) for identifiers that correspond to keywords. Note that |
| 607 | some tokens can be spelled multiple ways. For example, C++ supports |
| 608 | "operator keywords", where things like "``and``" are treated exactly like the |
| 609 | "``&&``" operator. In these cases, the kind value is set to ``tok::ampamp``, |
| 610 | which is good for the parser, which doesn't have to consider both forms. For |
| 611 | something that cares about which form is used (e.g., the preprocessor |
| 612 | "stringize" operator) the spelling indicates the original form. |
| 613 | |
| 614 | * **Flags** --- There are currently four flags tracked by the |
| 615 | lexer/preprocessor system on a per-token basis: |
| 616 | |
| 617 | #. **StartOfLine** --- This was the first token that occurred on its input |
| 618 | source line. |
| 619 | #. **LeadingSpace** --- There was a space character either immediately before |
| 620 | the token or transitively before the token as it was expanded through a |
| 621 | macro. The definition of this flag is very closely defined by the |
| 622 | stringizing requirements of the preprocessor. |
| 623 | #. **DisableExpand** --- This flag is used internally to the preprocessor to |
| 624 | represent identifier tokens which have macro expansion disabled. This |
| 625 | prevents them from being considered as candidates for macro expansion ever |
| 626 | in the future. |
| 627 | #. **NeedsCleaning** --- This flag is set if the original spelling for the |
| 628 | token includes a trigraph or escaped newline. Since this is uncommon, |
| 629 | many pieces of code can fast-path on tokens that did not need cleaning. |
| 630 | |
| 631 | One interesting (and somewhat unusual) aspect of normal tokens is that they |
| 632 | don't contain any semantic information about the lexed value. For example, if |
| 633 | the token was a pp-number token, we do not represent the value of the number |
| 634 | that was lexed (this is left for later pieces of code to decide). |
| 635 | Additionally, the lexer library has no notion of typedef names vs variable |
| 636 | names: both are returned as identifiers, and the parser is left to decide |
| 637 | whether a specific identifier is a typedef or a variable (tracking this |
| 638 | requires scope information among other things). The parser can do this |
| 639 | translation by replacing tokens returned by the preprocessor with "Annotation |
| 640 | Tokens". |
| 641 | |
| 642 | .. _AnnotationToken: |
| 643 | |
| 644 | Annotation Tokens |
| 645 | ----------------- |
| 646 | |
| 647 | Annotation tokens are tokens that are synthesized by the parser and injected |
| 648 | into the preprocessor's token stream (replacing existing tokens) to record |
| 649 | semantic information found by the parser. For example, if "``foo``" is found |
| 650 | to be a typedef, the "``foo``" ``tok::identifier`` token is replaced with an |
| 651 | ``tok::annot_typename``. This is useful for a couple of reasons: 1) this makes |
| 652 | it easy to handle qualified type names (e.g., "``foo::bar::baz<42>::t``") in |
| 653 | C++ as a single "token" in the parser. 2) if the parser backtracks, the |
| 654 | reparse does not need to redo semantic analysis to determine whether a token |
| 655 | sequence is a variable, type, template, etc. |
| 656 | |
| 657 | Annotation tokens are created by the parser and reinjected into the parser's |
| 658 | token stream (when backtracking is enabled). Because they can only exist in |
| 659 | tokens that the preprocessor-proper is done with, it doesn't need to keep |
| 660 | around flags like "start of line" that the preprocessor uses to do its job. |
| 661 | Additionally, an annotation token may "cover" a sequence of preprocessor tokens |
| 662 | (e.g., "``a::b::c``" is five preprocessor tokens). As such, the valid fields |
| 663 | of an annotation token are different than the fields for a normal token (but |
| 664 | they are multiplexed into the normal ``Token`` fields): |
| 665 | |
| 666 | * **SourceLocation "Location"** --- The ``SourceLocation`` for the annotation |
| 667 | token indicates the first token replaced by the annotation token. In the |
| 668 | example above, it would be the location of the "``a``" identifier. |
| 669 | * **SourceLocation "AnnotationEndLoc"** --- This holds the location of the last |
| 670 | token replaced with the annotation token. In the example above, it would be |
| 671 | the location of the "``c``" identifier. |
| 672 | * **void* "AnnotationValue"** --- This contains an opaque object that the |
| 673 | parser gets from ``Sema``. The parser merely preserves the information for |
| 674 | ``Sema`` to later interpret based on the annotation token kind. |
| 675 | * **TokenKind "Kind"** --- This indicates the kind of Annotation token this is. |
| 676 | See below for the different valid kinds. |
| 677 | |
| 678 | Annotation tokens currently come in three kinds: |
| 679 | |
| 680 | #. **tok::annot_typename**: This annotation token represents a resolved |
| 681 | typename token that is potentially qualified. The ``AnnotationValue`` field |
| 682 | contains the ``QualType`` returned by ``Sema::getTypeName()``, possibly with |
| 683 | source location information attached. |
| 684 | #. **tok::annot_cxxscope**: This annotation token represents a C++ scope |
| 685 | specifier, such as "``A::B::``". This corresponds to the grammar |
| 686 | productions "*::*" and "*:: [opt] nested-name-specifier*". The |
| 687 | ``AnnotationValue`` pointer is a ``NestedNameSpecifier *`` returned by the |
| 688 | ``Sema::ActOnCXXGlobalScopeSpecifier`` and |
| 689 | ``Sema::ActOnCXXNestedNameSpecifier`` callbacks. |
| 690 | #. **tok::annot_template_id**: This annotation token represents a C++ |
| 691 | template-id such as "``foo<int, 4>``", where "``foo``" is the name of a |
| 692 | template. The ``AnnotationValue`` pointer is a pointer to a ``malloc``'d |
| 693 | ``TemplateIdAnnotation`` object. Depending on the context, a parsed |
| 694 | template-id that names a type might become a typename annotation token (if |
| 695 | all we care about is the named type, e.g., because it occurs in a type |
| 696 | specifier) or might remain a template-id token (if we want to retain more |
| 697 | source location information or produce a new type, e.g., in a declaration of |
| 698 | a class template specialization). template-id annotation tokens that refer |
| 699 | to a type can be "upgraded" to typename annotation tokens by the parser. |
| 700 | |
| 701 | As mentioned above, annotation tokens are not returned by the preprocessor, |
| 702 | they are formed on demand by the parser. This means that the parser has to be |
| 703 | aware of cases where an annotation could occur and form it where appropriate. |
| 704 | This is somewhat similar to how the parser handles Translation Phase 6 of C99: |
| 705 | String Concatenation (see C99 5.1.1.2). In the case of string concatenation, |
| 706 | the preprocessor just returns distinct ``tok::string_literal`` and |
| 707 | ``tok::wide_string_literal`` tokens and the parser eats a sequence of them |
| 708 | wherever the grammar indicates that a string literal can occur. |
| 709 | |
| 710 | In order to do this, whenever the parser expects a ``tok::identifier`` or |
| 711 | ``tok::coloncolon``, it should call the ``TryAnnotateTypeOrScopeToken`` or |
| 712 | ``TryAnnotateCXXScopeToken`` methods to form the annotation token. These |
| 713 | methods will maximally form the specified annotation tokens and replace the |
| 714 | current token with them, if applicable. If the current tokens is not valid for |
| 715 | an annotation token, it will remain an identifier or "``::``" token. |
| 716 | |
| 717 | .. _Lexer: |
| 718 | |
| 719 | The ``Lexer`` class |
| 720 | ------------------- |
| 721 | |
| 722 | The ``Lexer`` class provides the mechanics of lexing tokens out of a source |
| 723 | buffer and deciding what they mean. The ``Lexer`` is complicated by the fact |
| 724 | that it operates on raw buffers that have not had spelling eliminated (this is |
| 725 | a necessity to get decent performance), but this is countered with careful |
| 726 | coding as well as standard performance techniques (for example, the comment |
| 727 | handling code is vectorized on X86 and PowerPC hosts). |
| 728 | |
| 729 | The lexer has a couple of interesting modal features: |
| 730 | |
| 731 | * The lexer can operate in "raw" mode. This mode has several features that |
| 732 | make it possible to quickly lex the file (e.g., it stops identifier lookup, |
| 733 | doesn't specially handle preprocessor tokens, handles EOF differently, etc). |
| 734 | This mode is used for lexing within an "``#if 0``" block, for example. |
| 735 | * The lexer can capture and return comments as tokens. This is required to |
| 736 | support the ``-C`` preprocessor mode, which passes comments through, and is |
| 737 | used by the diagnostic checker to identifier expect-error annotations. |
| 738 | * The lexer can be in ``ParsingFilename`` mode, which happens when |
| 739 | preprocessing after reading a ``#include`` directive. This mode changes the |
| 740 | parsing of "``<``" to return an "angled string" instead of a bunch of tokens |
| 741 | for each thing within the filename. |
| 742 | * When parsing a preprocessor directive (after "``#``") the |
| 743 | ``ParsingPreprocessorDirective`` mode is entered. This changes the parser to |
| 744 | return EOD at a newline. |
| 745 | * The ``Lexer`` uses a ``LangOptions`` object to know whether trigraphs are |
| 746 | enabled, whether C++ or ObjC keywords are recognized, etc. |
| 747 | |
| 748 | In addition to these modes, the lexer keeps track of a couple of other features |
| 749 | that are local to a lexed buffer, which change as the buffer is lexed: |
| 750 | |
| 751 | * The ``Lexer`` uses ``BufferPtr`` to keep track of the current character being |
| 752 | lexed. |
| 753 | * The ``Lexer`` uses ``IsAtStartOfLine`` to keep track of whether the next |
| 754 | lexed token will start with its "start of line" bit set. |
| 755 | * The ``Lexer`` keeps track of the current "``#if``" directives that are active |
| 756 | (which can be nested). |
| 757 | * The ``Lexer`` keeps track of an :ref:`MultipleIncludeOpt |
| 758 | <MultipleIncludeOpt>` object, which is used to detect whether the buffer uses |
| 759 | the standard "``#ifndef XX`` / ``#define XX``" idiom to prevent multiple |
| 760 | inclusion. If a buffer does, subsequent includes can be ignored if the |
| 761 | "``XX``" macro is defined. |
| 762 | |
| 763 | .. _TokenLexer: |
| 764 | |
| 765 | The ``TokenLexer`` class |
| 766 | ------------------------ |
| 767 | |
| 768 | The ``TokenLexer`` class is a token provider that returns tokens from a list of |
| 769 | tokens that came from somewhere else. It typically used for two things: 1) |
| 770 | returning tokens from a macro definition as it is being expanded 2) returning |
| 771 | tokens from an arbitrary buffer of tokens. The later use is used by |
| 772 | ``_Pragma`` and will most likely be used to handle unbounded look-ahead for the |
| 773 | C++ parser. |
| 774 | |
| 775 | .. _MultipleIncludeOpt: |
| 776 | |
| 777 | The ``MultipleIncludeOpt`` class |
| 778 | -------------------------------- |
| 779 | |
| 780 | The ``MultipleIncludeOpt`` class implements a really simple little state |
| 781 | machine that is used to detect the standard "``#ifndef XX`` / ``#define XX``" |
| 782 | idiom that people typically use to prevent multiple inclusion of headers. If a |
| 783 | buffer uses this idiom and is subsequently ``#include``'d, the preprocessor can |
| 784 | simply check to see whether the guarding condition is defined or not. If so, |
| 785 | the preprocessor can completely ignore the include of the header. |
| 786 | |
Stephen Hines | 0e2c34f | 2015-03-23 12:09:02 -0700 | [diff] [blame] | 787 | .. _Parser: |
| 788 | |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 789 | The Parser Library |
| 790 | ================== |
| 791 | |
Stephen Hines | 0e2c34f | 2015-03-23 12:09:02 -0700 | [diff] [blame] | 792 | This library contains a recursive-descent parser that polls tokens from the |
| 793 | preprocessor and notifies a client of the parsing progress. |
| 794 | |
| 795 | Historically, the parser used to talk to an abstract ``Action`` interface that |
| 796 | had virtual methods for parse events, for example ``ActOnBinOp()``. When Clang |
| 797 | grew C++ support, the parser stopped supporting general ``Action`` clients -- |
| 798 | it now always talks to the :ref:`Sema libray <Sema>`. However, the Parser |
| 799 | still accesses AST objects only through opaque types like ``ExprResult`` and |
| 800 | ``StmtResult``. Only :ref:`Sema <Sema>` looks at the AST node contents of these |
| 801 | wrappers. |
| 802 | |
| 803 | .. _AST: |
| 804 | |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 805 | The AST Library |
| 806 | =============== |
| 807 | |
| 808 | .. _Type: |
| 809 | |
| 810 | The ``Type`` class and its subclasses |
| 811 | ------------------------------------- |
| 812 | |
| 813 | The ``Type`` class (and its subclasses) are an important part of the AST. |
| 814 | Types are accessed through the ``ASTContext`` class, which implicitly creates |
| 815 | and uniques them as they are needed. Types have a couple of non-obvious |
| 816 | features: 1) they do not capture type qualifiers like ``const`` or ``volatile`` |
| 817 | (see :ref:`QualType <QualType>`), and 2) they implicitly capture typedef |
| 818 | information. Once created, types are immutable (unlike decls). |
| 819 | |
| 820 | Typedefs in C make semantic analysis a bit more complex than it would be without |
| 821 | them. The issue is that we want to capture typedef information and represent it |
| 822 | in the AST perfectly, but the semantics of operations need to "see through" |
| 823 | typedefs. For example, consider this code: |
| 824 | |
| 825 | .. code-block:: c++ |
| 826 | |
| 827 | void func() { |
| 828 | typedef int foo; |
| 829 | foo X, *Y; |
| 830 | typedef foo *bar; |
| 831 | bar Z; |
| 832 | *X; // error |
| 833 | **Y; // error |
| 834 | **Z; // error |
| 835 | } |
| 836 | |
| 837 | The code above is illegal, and thus we expect there to be diagnostics emitted |
| 838 | on the annotated lines. In this example, we expect to get: |
| 839 | |
Pirama Arumuga Nainar | 4967a71 | 2016-09-19 22:19:55 -0700 | [diff] [blame] | 840 | .. code-block:: text |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 841 | |
| 842 | test.c:6:1: error: indirection requires pointer operand ('foo' invalid) |
| 843 | *X; // error |
| 844 | ^~ |
| 845 | test.c:7:1: error: indirection requires pointer operand ('foo' invalid) |
| 846 | **Y; // error |
| 847 | ^~~ |
| 848 | test.c:8:1: error: indirection requires pointer operand ('foo' invalid) |
| 849 | **Z; // error |
| 850 | ^~~ |
| 851 | |
| 852 | While this example is somewhat silly, it illustrates the point: we want to |
| 853 | retain typedef information where possible, so that we can emit errors about |
| 854 | "``std::string``" instead of "``std::basic_string<char, std:...``". Doing this |
| 855 | requires properly keeping typedef information (for example, the type of ``X`` |
| 856 | is "``foo``", not "``int``"), and requires properly propagating it through the |
| 857 | various operators (for example, the type of ``*Y`` is "``foo``", not |
| 858 | "``int``"). In order to retain this information, the type of these expressions |
| 859 | is an instance of the ``TypedefType`` class, which indicates that the type of |
| 860 | these expressions is a typedef for "``foo``". |
| 861 | |
| 862 | Representing types like this is great for diagnostics, because the |
| 863 | user-specified type is always immediately available. There are two problems |
| 864 | with this: first, various semantic checks need to make judgements about the |
| 865 | *actual structure* of a type, ignoring typedefs. Second, we need an efficient |
| 866 | way to query whether two types are structurally identical to each other, |
| 867 | ignoring typedefs. The solution to both of these problems is the idea of |
| 868 | canonical types. |
| 869 | |
| 870 | Canonical Types |
| 871 | ^^^^^^^^^^^^^^^ |
| 872 | |
| 873 | Every instance of the ``Type`` class contains a canonical type pointer. For |
| 874 | simple types with no typedefs involved (e.g., "``int``", "``int*``", |
| 875 | "``int**``"), the type just points to itself. For types that have a typedef |
| 876 | somewhere in their structure (e.g., "``foo``", "``foo*``", "``foo**``", |
| 877 | "``bar``"), the canonical type pointer points to their structurally equivalent |
| 878 | type without any typedefs (e.g., "``int``", "``int*``", "``int**``", and |
| 879 | "``int*``" respectively). |
| 880 | |
| 881 | This design provides a constant time operation (dereferencing the canonical type |
| 882 | pointer) that gives us access to the structure of types. For example, we can |
| 883 | trivially tell that "``bar``" and "``foo*``" are the same type by dereferencing |
| 884 | their canonical type pointers and doing a pointer comparison (they both point |
| 885 | to the single "``int*``" type). |
| 886 | |
| 887 | Canonical types and typedef types bring up some complexities that must be |
| 888 | carefully managed. Specifically, the ``isa``/``cast``/``dyn_cast`` operators |
| 889 | generally shouldn't be used in code that is inspecting the AST. For example, |
| 890 | when type checking the indirection operator (unary "``*``" on a pointer), the |
| 891 | type checker must verify that the operand has a pointer type. It would not be |
| 892 | correct to check that with "``isa<PointerType>(SubExpr->getType())``", because |
| 893 | this predicate would fail if the subexpression had a typedef type. |
| 894 | |
| 895 | The solution to this problem are a set of helper methods on ``Type``, used to |
| 896 | check their properties. In this case, it would be correct to use |
| 897 | "``SubExpr->getType()->isPointerType()``" to do the check. This predicate will |
| 898 | return true if the *canonical type is a pointer*, which is true any time the |
| 899 | type is structurally a pointer type. The only hard part here is remembering |
| 900 | not to use the ``isa``/``cast``/``dyn_cast`` operations. |
| 901 | |
| 902 | The second problem we face is how to get access to the pointer type once we |
| 903 | know it exists. To continue the example, the result type of the indirection |
| 904 | operator is the pointee type of the subexpression. In order to determine the |
| 905 | type, we need to get the instance of ``PointerType`` that best captures the |
| 906 | typedef information in the program. If the type of the expression is literally |
| 907 | a ``PointerType``, we can return that, otherwise we have to dig through the |
| 908 | typedefs to find the pointer type. For example, if the subexpression had type |
| 909 | "``foo*``", we could return that type as the result. If the subexpression had |
| 910 | type "``bar``", we want to return "``foo*``" (note that we do *not* want |
| 911 | "``int*``"). In order to provide all of this, ``Type`` has a |
| 912 | ``getAsPointerType()`` method that checks whether the type is structurally a |
| 913 | ``PointerType`` and, if so, returns the best one. If not, it returns a null |
| 914 | pointer. |
| 915 | |
| 916 | This structure is somewhat mystical, but after meditating on it, it will make |
| 917 | sense to you :). |
| 918 | |
| 919 | .. _QualType: |
| 920 | |
| 921 | The ``QualType`` class |
| 922 | ---------------------- |
| 923 | |
| 924 | The ``QualType`` class is designed as a trivial value class that is small, |
| 925 | passed by-value and is efficient to query. The idea of ``QualType`` is that it |
| 926 | stores the type qualifiers (``const``, ``volatile``, ``restrict``, plus some |
| 927 | extended qualifiers required by language extensions) separately from the types |
| 928 | themselves. ``QualType`` is conceptually a pair of "``Type*``" and the bits |
| 929 | for these type qualifiers. |
| 930 | |
| 931 | By storing the type qualifiers as bits in the conceptual pair, it is extremely |
| 932 | efficient to get the set of qualifiers on a ``QualType`` (just return the field |
| 933 | of the pair), add a type qualifier (which is a trivial constant-time operation |
| 934 | that sets a bit), and remove one or more type qualifiers (just return a |
| 935 | ``QualType`` with the bitfield set to empty). |
| 936 | |
| 937 | Further, because the bits are stored outside of the type itself, we do not need |
| 938 | to create duplicates of types with different sets of qualifiers (i.e. there is |
| 939 | only a single heap allocated "``int``" type: "``const int``" and "``volatile |
| 940 | const int``" both point to the same heap allocated "``int``" type). This |
| 941 | reduces the heap size used to represent bits and also means we do not have to |
| 942 | consider qualifiers when uniquing types (:ref:`Type <Type>` does not even |
| 943 | contain qualifiers). |
| 944 | |
| 945 | In practice, the two most common type qualifiers (``const`` and ``restrict``) |
| 946 | are stored in the low bits of the pointer to the ``Type`` object, together with |
| 947 | a flag indicating whether extended qualifiers are present (which must be |
| 948 | heap-allocated). This means that ``QualType`` is exactly the same size as a |
| 949 | pointer. |
| 950 | |
| 951 | .. _DeclarationName: |
| 952 | |
| 953 | Declaration names |
| 954 | ----------------- |
| 955 | |
| 956 | The ``DeclarationName`` class represents the name of a declaration in Clang. |
| 957 | Declarations in the C family of languages can take several different forms. |
| 958 | Most declarations are named by simple identifiers, e.g., "``f``" and "``x``" in |
| 959 | the function declaration ``f(int x)``. In C++, declaration names can also name |
| 960 | class constructors ("``Class``" in ``struct Class { Class(); }``), class |
| 961 | destructors ("``~Class``"), overloaded operator names ("``operator+``"), and |
| 962 | conversion functions ("``operator void const *``"). In Objective-C, |
| 963 | declaration names can refer to the names of Objective-C methods, which involve |
| 964 | the method name and the parameters, collectively called a *selector*, e.g., |
| 965 | "``setWidth:height:``". Since all of these kinds of entities --- variables, |
| 966 | functions, Objective-C methods, C++ constructors, destructors, and operators |
| 967 | --- are represented as subclasses of Clang's common ``NamedDecl`` class, |
| 968 | ``DeclarationName`` is designed to efficiently represent any kind of name. |
| 969 | |
| 970 | Given a ``DeclarationName`` ``N``, ``N.getNameKind()`` will produce a value |
Justin Bogner | 89eb0d6 | 2013-09-27 21:10:54 +0000 | [diff] [blame] | 971 | that describes what kind of name ``N`` stores. There are 10 options (all of |
| 972 | the names are inside the ``DeclarationName`` class). |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 973 | |
| 974 | ``Identifier`` |
| 975 | |
| 976 | The name is a simple identifier. Use ``N.getAsIdentifierInfo()`` to retrieve |
| 977 | the corresponding ``IdentifierInfo*`` pointing to the actual identifier. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 978 | |
| 979 | ``ObjCZeroArgSelector``, ``ObjCOneArgSelector``, ``ObjCMultiArgSelector`` |
| 980 | |
| 981 | The name is an Objective-C selector, which can be retrieved as a ``Selector`` |
| 982 | instance via ``N.getObjCSelector()``. The three possible name kinds for |
| 983 | Objective-C reflect an optimization within the ``DeclarationName`` class: |
| 984 | both zero- and one-argument selectors are stored as a masked |
| 985 | ``IdentifierInfo`` pointer, and therefore require very little space, since |
| 986 | zero- and one-argument selectors are far more common than multi-argument |
| 987 | selectors (which use a different structure). |
| 988 | |
| 989 | ``CXXConstructorName`` |
| 990 | |
| 991 | The name is a C++ constructor name. Use ``N.getCXXNameType()`` to retrieve |
| 992 | the :ref:`type <QualType>` that this constructor is meant to construct. The |
| 993 | type is always the canonical type, since all constructors for a given type |
| 994 | have the same name. |
| 995 | |
| 996 | ``CXXDestructorName`` |
| 997 | |
| 998 | The name is a C++ destructor name. Use ``N.getCXXNameType()`` to retrieve |
| 999 | the :ref:`type <QualType>` whose destructor is being named. This type is |
| 1000 | always a canonical type. |
| 1001 | |
| 1002 | ``CXXConversionFunctionName`` |
| 1003 | |
| 1004 | The name is a C++ conversion function. Conversion functions are named |
| 1005 | according to the type they convert to, e.g., "``operator void const *``". |
| 1006 | Use ``N.getCXXNameType()`` to retrieve the type that this conversion function |
| 1007 | converts to. This type is always a canonical type. |
| 1008 | |
| 1009 | ``CXXOperatorName`` |
| 1010 | |
| 1011 | The name is a C++ overloaded operator name. Overloaded operators are named |
| 1012 | according to their spelling, e.g., "``operator+``" or "``operator new []``". |
| 1013 | Use ``N.getCXXOverloadedOperator()`` to retrieve the overloaded operator (a |
| 1014 | value of type ``OverloadedOperatorKind``). |
| 1015 | |
Justin Bogner | 89eb0d6 | 2013-09-27 21:10:54 +0000 | [diff] [blame] | 1016 | ``CXXLiteralOperatorName`` |
| 1017 | |
| 1018 | The name is a C++11 user defined literal operator. User defined |
| 1019 | Literal operators are named according to the suffix they define, |
| 1020 | e.g., "``_foo``" for "``operator "" _foo``". Use |
| 1021 | ``N.getCXXLiteralIdentifier()`` to retrieve the corresponding |
| 1022 | ``IdentifierInfo*`` pointing to the identifier. |
| 1023 | |
| 1024 | ``CXXUsingDirective`` |
| 1025 | |
| 1026 | The name is a C++ using directive. Using directives are not really |
| 1027 | NamedDecls, in that they all have the same name, but they are |
| 1028 | implemented as such in order to store them in DeclContext |
| 1029 | effectively. |
| 1030 | |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1031 | ``DeclarationName``\ s are cheap to create, copy, and compare. They require |
| 1032 | only a single pointer's worth of storage in the common cases (identifiers, |
| 1033 | zero- and one-argument Objective-C selectors) and use dense, uniqued storage |
| 1034 | for the other kinds of names. Two ``DeclarationName``\ s can be compared for |
| 1035 | equality (``==``, ``!=``) using a simple bitwise comparison, can be ordered |
| 1036 | with ``<``, ``>``, ``<=``, and ``>=`` (which provide a lexicographical ordering |
| 1037 | for normal identifiers but an unspecified ordering for other kinds of names), |
| 1038 | and can be placed into LLVM ``DenseMap``\ s and ``DenseSet``\ s. |
| 1039 | |
| 1040 | ``DeclarationName`` instances can be created in different ways depending on |
| 1041 | what kind of name the instance will store. Normal identifiers |
| 1042 | (``IdentifierInfo`` pointers) and Objective-C selectors (``Selector``) can be |
| 1043 | implicitly converted to ``DeclarationNames``. Names for C++ constructors, |
| 1044 | destructors, conversion functions, and overloaded operators can be retrieved |
| 1045 | from the ``DeclarationNameTable``, an instance of which is available as |
| 1046 | ``ASTContext::DeclarationNames``. The member functions |
| 1047 | ``getCXXConstructorName``, ``getCXXDestructorName``, |
| 1048 | ``getCXXConversionFunctionName``, and ``getCXXOperatorName``, respectively, |
| 1049 | return ``DeclarationName`` instances for the four kinds of C++ special function |
| 1050 | names. |
| 1051 | |
| 1052 | .. _DeclContext: |
| 1053 | |
| 1054 | Declaration contexts |
| 1055 | -------------------- |
| 1056 | |
| 1057 | Every declaration in a program exists within some *declaration context*, such |
| 1058 | as a translation unit, namespace, class, or function. Declaration contexts in |
| 1059 | Clang are represented by the ``DeclContext`` class, from which the various |
| 1060 | declaration-context AST nodes (``TranslationUnitDecl``, ``NamespaceDecl``, |
| 1061 | ``RecordDecl``, ``FunctionDecl``, etc.) will derive. The ``DeclContext`` class |
| 1062 | provides several facilities common to each declaration context: |
| 1063 | |
| 1064 | Source-centric vs. Semantics-centric View of Declarations |
| 1065 | |
| 1066 | ``DeclContext`` provides two views of the declarations stored within a |
| 1067 | declaration context. The source-centric view accurately represents the |
| 1068 | program source code as written, including multiple declarations of entities |
| 1069 | where present (see the section :ref:`Redeclarations and Overloads |
| 1070 | <Redeclarations>`), while the semantics-centric view represents the program |
| 1071 | semantics. The two views are kept synchronized by semantic analysis while |
| 1072 | the ASTs are being constructed. |
| 1073 | |
| 1074 | Storage of declarations within that context |
| 1075 | |
| 1076 | Every declaration context can contain some number of declarations. For |
| 1077 | example, a C++ class (represented by ``RecordDecl``) contains various member |
| 1078 | functions, fields, nested types, and so on. All of these declarations will |
| 1079 | be stored within the ``DeclContext``, and one can iterate over the |
| 1080 | declarations via [``DeclContext::decls_begin()``, |
| 1081 | ``DeclContext::decls_end()``). This mechanism provides the source-centric |
| 1082 | view of declarations in the context. |
| 1083 | |
| 1084 | Lookup of declarations within that context |
| 1085 | |
| 1086 | The ``DeclContext`` structure provides efficient name lookup for names within |
| 1087 | that declaration context. For example, if ``N`` is a namespace we can look |
| 1088 | for the name ``N::f`` using ``DeclContext::lookup``. The lookup itself is |
| 1089 | based on a lazily-constructed array (for declaration contexts with a small |
| 1090 | number of declarations) or hash table (for declaration contexts with more |
| 1091 | declarations). The lookup operation provides the semantics-centric view of |
| 1092 | the declarations in the context. |
| 1093 | |
| 1094 | Ownership of declarations |
| 1095 | |
| 1096 | The ``DeclContext`` owns all of the declarations that were declared within |
| 1097 | its declaration context, and is responsible for the management of their |
| 1098 | memory as well as their (de-)serialization. |
| 1099 | |
| 1100 | All declarations are stored within a declaration context, and one can query |
| 1101 | information about the context in which each declaration lives. One can |
| 1102 | retrieve the ``DeclContext`` that contains a particular ``Decl`` using |
| 1103 | ``Decl::getDeclContext``. However, see the section |
| 1104 | :ref:`LexicalAndSemanticContexts` for more information about how to interpret |
| 1105 | this context information. |
| 1106 | |
| 1107 | .. _Redeclarations: |
| 1108 | |
| 1109 | Redeclarations and Overloads |
| 1110 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1111 | |
| 1112 | Within a translation unit, it is common for an entity to be declared several |
| 1113 | times. For example, we might declare a function "``f``" and then later |
| 1114 | re-declare it as part of an inlined definition: |
| 1115 | |
| 1116 | .. code-block:: c++ |
| 1117 | |
| 1118 | void f(int x, int y, int z = 1); |
| 1119 | |
| 1120 | inline void f(int x, int y, int z) { /* ... */ } |
| 1121 | |
| 1122 | The representation of "``f``" differs in the source-centric and |
| 1123 | semantics-centric views of a declaration context. In the source-centric view, |
| 1124 | all redeclarations will be present, in the order they occurred in the source |
| 1125 | code, making this view suitable for clients that wish to see the structure of |
| 1126 | the source code. In the semantics-centric view, only the most recent "``f``" |
| 1127 | will be found by the lookup, since it effectively replaces the first |
| 1128 | declaration of "``f``". |
| 1129 | |
| 1130 | In the semantics-centric view, overloading of functions is represented |
| 1131 | explicitly. For example, given two declarations of a function "``g``" that are |
| 1132 | overloaded, e.g., |
| 1133 | |
| 1134 | .. code-block:: c++ |
| 1135 | |
| 1136 | void g(); |
| 1137 | void g(int); |
| 1138 | |
| 1139 | the ``DeclContext::lookup`` operation will return a |
| 1140 | ``DeclContext::lookup_result`` that contains a range of iterators over |
| 1141 | declarations of "``g``". Clients that perform semantic analysis on a program |
| 1142 | that is not concerned with the actual source code will primarily use this |
| 1143 | semantics-centric view. |
| 1144 | |
| 1145 | .. _LexicalAndSemanticContexts: |
| 1146 | |
| 1147 | Lexical and Semantic Contexts |
| 1148 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1149 | |
| 1150 | Each declaration has two potentially different declaration contexts: a |
| 1151 | *lexical* context, which corresponds to the source-centric view of the |
| 1152 | declaration context, and a *semantic* context, which corresponds to the |
| 1153 | semantics-centric view. The lexical context is accessible via |
| 1154 | ``Decl::getLexicalDeclContext`` while the semantic context is accessible via |
| 1155 | ``Decl::getDeclContext``, both of which return ``DeclContext`` pointers. For |
| 1156 | most declarations, the two contexts are identical. For example: |
| 1157 | |
| 1158 | .. code-block:: c++ |
| 1159 | |
| 1160 | class X { |
| 1161 | public: |
| 1162 | void f(int x); |
| 1163 | }; |
| 1164 | |
| 1165 | Here, the semantic and lexical contexts of ``X::f`` are the ``DeclContext`` |
| 1166 | associated with the class ``X`` (itself stored as a ``RecordDecl`` AST node). |
| 1167 | However, we can now define ``X::f`` out-of-line: |
| 1168 | |
| 1169 | .. code-block:: c++ |
| 1170 | |
| 1171 | void X::f(int x = 17) { /* ... */ } |
| 1172 | |
| 1173 | This definition of "``f``" has different lexical and semantic contexts. The |
| 1174 | lexical context corresponds to the declaration context in which the actual |
| 1175 | declaration occurred in the source code, e.g., the translation unit containing |
| 1176 | ``X``. Thus, this declaration of ``X::f`` can be found by traversing the |
| 1177 | declarations provided by [``decls_begin()``, ``decls_end()``) in the |
| 1178 | translation unit. |
| 1179 | |
| 1180 | The semantic context of ``X::f`` corresponds to the class ``X``, since this |
| 1181 | member function is (semantically) a member of ``X``. Lookup of the name ``f`` |
| 1182 | into the ``DeclContext`` associated with ``X`` will then return the definition |
| 1183 | of ``X::f`` (including information about the default argument). |
| 1184 | |
| 1185 | Transparent Declaration Contexts |
| 1186 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1187 | |
| 1188 | In C and C++, there are several contexts in which names that are logically |
| 1189 | declared inside another declaration will actually "leak" out into the enclosing |
| 1190 | scope from the perspective of name lookup. The most obvious instance of this |
| 1191 | behavior is in enumeration types, e.g., |
| 1192 | |
| 1193 | .. code-block:: c++ |
| 1194 | |
| 1195 | enum Color { |
| 1196 | Red, |
| 1197 | Green, |
| 1198 | Blue |
| 1199 | }; |
| 1200 | |
| 1201 | Here, ``Color`` is an enumeration, which is a declaration context that contains |
| 1202 | the enumerators ``Red``, ``Green``, and ``Blue``. Thus, traversing the list of |
| 1203 | declarations contained in the enumeration ``Color`` will yield ``Red``, |
| 1204 | ``Green``, and ``Blue``. However, outside of the scope of ``Color`` one can |
| 1205 | name the enumerator ``Red`` without qualifying the name, e.g., |
| 1206 | |
| 1207 | .. code-block:: c++ |
| 1208 | |
| 1209 | Color c = Red; |
| 1210 | |
| 1211 | There are other entities in C++ that provide similar behavior. For example, |
| 1212 | linkage specifications that use curly braces: |
| 1213 | |
| 1214 | .. code-block:: c++ |
| 1215 | |
| 1216 | extern "C" { |
| 1217 | void f(int); |
| 1218 | void g(int); |
| 1219 | } |
| 1220 | // f and g are visible here |
| 1221 | |
| 1222 | For source-level accuracy, we treat the linkage specification and enumeration |
| 1223 | type as a declaration context in which its enclosed declarations ("``Red``", |
| 1224 | "``Green``", and "``Blue``"; "``f``" and "``g``") are declared. However, these |
| 1225 | declarations are visible outside of the scope of the declaration context. |
| 1226 | |
| 1227 | These language features (and several others, described below) have roughly the |
| 1228 | same set of requirements: declarations are declared within a particular lexical |
| 1229 | context, but the declarations are also found via name lookup in scopes |
| 1230 | enclosing the declaration itself. This feature is implemented via |
| 1231 | *transparent* declaration contexts (see |
| 1232 | ``DeclContext::isTransparentContext()``), whose declarations are visible in the |
| 1233 | nearest enclosing non-transparent declaration context. This means that the |
| 1234 | lexical context of the declaration (e.g., an enumerator) will be the |
| 1235 | transparent ``DeclContext`` itself, as will the semantic context, but the |
| 1236 | declaration will be visible in every outer context up to and including the |
| 1237 | first non-transparent declaration context (since transparent declaration |
| 1238 | contexts can be nested). |
| 1239 | |
| 1240 | The transparent ``DeclContext``\ s are: |
| 1241 | |
| 1242 | * Enumerations (but not C++11 "scoped enumerations"): |
| 1243 | |
| 1244 | .. code-block:: c++ |
| 1245 | |
| 1246 | enum Color { |
| 1247 | Red, |
| 1248 | Green, |
| 1249 | Blue |
| 1250 | }; |
| 1251 | // Red, Green, and Blue are in scope |
| 1252 | |
| 1253 | * C++ linkage specifications: |
| 1254 | |
| 1255 | .. code-block:: c++ |
| 1256 | |
| 1257 | extern "C" { |
| 1258 | void f(int); |
| 1259 | void g(int); |
| 1260 | } |
| 1261 | // f and g are in scope |
| 1262 | |
| 1263 | * Anonymous unions and structs: |
| 1264 | |
| 1265 | .. code-block:: c++ |
| 1266 | |
| 1267 | struct LookupTable { |
| 1268 | bool IsVector; |
| 1269 | union { |
| 1270 | std::vector<Item> *Vector; |
| 1271 | std::set<Item> *Set; |
| 1272 | }; |
| 1273 | }; |
| 1274 | |
| 1275 | LookupTable LT; |
| 1276 | LT.Vector = 0; // Okay: finds Vector inside the unnamed union |
| 1277 | |
| 1278 | * C++11 inline namespaces: |
| 1279 | |
| 1280 | .. code-block:: c++ |
| 1281 | |
| 1282 | namespace mylib { |
| 1283 | inline namespace debug { |
| 1284 | class X; |
| 1285 | } |
| 1286 | } |
| 1287 | mylib::X *xp; // okay: mylib::X refers to mylib::debug::X |
| 1288 | |
| 1289 | .. _MultiDeclContext: |
| 1290 | |
| 1291 | Multiply-Defined Declaration Contexts |
| 1292 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1293 | |
| 1294 | C++ namespaces have the interesting --- and, so far, unique --- property that |
| 1295 | the namespace can be defined multiple times, and the declarations provided by |
| 1296 | each namespace definition are effectively merged (from the semantic point of |
| 1297 | view). For example, the following two code snippets are semantically |
| 1298 | indistinguishable: |
| 1299 | |
| 1300 | .. code-block:: c++ |
| 1301 | |
| 1302 | // Snippet #1: |
| 1303 | namespace N { |
| 1304 | void f(); |
| 1305 | } |
| 1306 | namespace N { |
| 1307 | void f(int); |
| 1308 | } |
| 1309 | |
| 1310 | // Snippet #2: |
| 1311 | namespace N { |
| 1312 | void f(); |
| 1313 | void f(int); |
| 1314 | } |
| 1315 | |
| 1316 | In Clang's representation, the source-centric view of declaration contexts will |
| 1317 | actually have two separate ``NamespaceDecl`` nodes in Snippet #1, each of which |
| 1318 | is a declaration context that contains a single declaration of "``f``". |
| 1319 | However, the semantics-centric view provided by name lookup into the namespace |
| 1320 | ``N`` for "``f``" will return a ``DeclContext::lookup_result`` that contains a |
| 1321 | range of iterators over declarations of "``f``". |
| 1322 | |
| 1323 | ``DeclContext`` manages multiply-defined declaration contexts internally. The |
| 1324 | function ``DeclContext::getPrimaryContext`` retrieves the "primary" context for |
| 1325 | a given ``DeclContext`` instance, which is the ``DeclContext`` responsible for |
Stephen Hines | 0e2c34f | 2015-03-23 12:09:02 -0700 | [diff] [blame] | 1326 | maintaining the lookup table used for the semantics-centric view. Given a |
| 1327 | DeclContext, one can obtain the set of declaration contexts that are semanticaly |
| 1328 | connected to this declaration context, in source order, including this context |
| 1329 | (which will be the only result, for non-namespace contexts) via |
| 1330 | ``DeclContext::collectAllContexts``. Note that these functions are used |
| 1331 | internally within the lookup and insertion methods of the ``DeclContext``, so |
| 1332 | the vast majority of clients can ignore them. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1333 | |
| 1334 | .. _CFG: |
| 1335 | |
| 1336 | The ``CFG`` class |
| 1337 | ----------------- |
| 1338 | |
| 1339 | The ``CFG`` class is designed to represent a source-level control-flow graph |
| 1340 | for a single statement (``Stmt*``). Typically instances of ``CFG`` are |
| 1341 | constructed for function bodies (usually an instance of ``CompoundStmt``), but |
| 1342 | can also be instantiated to represent the control-flow of any class that |
| 1343 | subclasses ``Stmt``, which includes simple expressions. Control-flow graphs |
| 1344 | are especially useful for performing `flow- or path-sensitive |
| 1345 | <http://en.wikipedia.org/wiki/Data_flow_analysis#Sensitivities>`_ program |
| 1346 | analyses on a given function. |
| 1347 | |
| 1348 | Basic Blocks |
| 1349 | ^^^^^^^^^^^^ |
| 1350 | |
| 1351 | Concretely, an instance of ``CFG`` is a collection of basic blocks. Each basic |
| 1352 | block is an instance of ``CFGBlock``, which simply contains an ordered sequence |
| 1353 | of ``Stmt*`` (each referring to statements in the AST). The ordering of |
| 1354 | statements within a block indicates unconditional flow of control from one |
| 1355 | statement to the next. :ref:`Conditional control-flow |
| 1356 | <ConditionalControlFlow>` is represented using edges between basic blocks. The |
| 1357 | statements within a given ``CFGBlock`` can be traversed using the |
| 1358 | ``CFGBlock::*iterator`` interface. |
| 1359 | |
| 1360 | A ``CFG`` object owns the instances of ``CFGBlock`` within the control-flow |
| 1361 | graph it represents. Each ``CFGBlock`` within a CFG is also uniquely numbered |
| 1362 | (accessible via ``CFGBlock::getBlockID()``). Currently the number is based on |
| 1363 | the ordering the blocks were created, but no assumptions should be made on how |
| 1364 | ``CFGBlocks`` are numbered other than their numbers are unique and that they |
| 1365 | are numbered from 0..N-1 (where N is the number of basic blocks in the CFG). |
| 1366 | |
| 1367 | Entry and Exit Blocks |
| 1368 | ^^^^^^^^^^^^^^^^^^^^^ |
| 1369 | |
| 1370 | Each instance of ``CFG`` contains two special blocks: an *entry* block |
| 1371 | (accessible via ``CFG::getEntry()``), which has no incoming edges, and an |
| 1372 | *exit* block (accessible via ``CFG::getExit()``), which has no outgoing edges. |
| 1373 | Neither block contains any statements, and they serve the role of providing a |
| 1374 | clear entrance and exit for a body of code such as a function body. The |
| 1375 | presence of these empty blocks greatly simplifies the implementation of many |
| 1376 | analyses built on top of CFGs. |
| 1377 | |
| 1378 | .. _ConditionalControlFlow: |
| 1379 | |
| 1380 | Conditional Control-Flow |
| 1381 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1382 | |
| 1383 | Conditional control-flow (such as those induced by if-statements and loops) is |
| 1384 | represented as edges between ``CFGBlocks``. Because different C language |
| 1385 | constructs can induce control-flow, each ``CFGBlock`` also records an extra |
| 1386 | ``Stmt*`` that represents the *terminator* of the block. A terminator is |
| 1387 | simply the statement that caused the control-flow, and is used to identify the |
| 1388 | nature of the conditional control-flow between blocks. For example, in the |
| 1389 | case of an if-statement, the terminator refers to the ``IfStmt`` object in the |
| 1390 | AST that represented the given branch. |
| 1391 | |
| 1392 | To illustrate, consider the following code example: |
| 1393 | |
| 1394 | .. code-block:: c++ |
| 1395 | |
| 1396 | int foo(int x) { |
| 1397 | x = x + 1; |
| 1398 | if (x > 2) |
| 1399 | x++; |
| 1400 | else { |
| 1401 | x += 2; |
| 1402 | x *= 2; |
| 1403 | } |
| 1404 | |
| 1405 | return x; |
| 1406 | } |
| 1407 | |
| 1408 | After invoking the parser+semantic analyzer on this code fragment, the AST of |
| 1409 | the body of ``foo`` is referenced by a single ``Stmt*``. We can then construct |
| 1410 | an instance of ``CFG`` representing the control-flow graph of this function |
| 1411 | body by single call to a static class method: |
| 1412 | |
| 1413 | .. code-block:: c++ |
| 1414 | |
| 1415 | Stmt *FooBody = ... |
Stephen Hines | 176edba | 2014-12-01 14:53:08 -0800 | [diff] [blame] | 1416 | std::unique_ptr<CFG> FooCFG = CFG::buildCFG(FooBody); |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1417 | |
| 1418 | Along with providing an interface to iterate over its ``CFGBlocks``, the |
| 1419 | ``CFG`` class also provides methods that are useful for debugging and |
| 1420 | visualizing CFGs. For example, the method ``CFG::dump()`` dumps a |
| 1421 | pretty-printed version of the CFG to standard error. This is especially useful |
| 1422 | when one is using a debugger such as gdb. For example, here is the output of |
| 1423 | ``FooCFG->dump()``: |
| 1424 | |
Pirama Arumuga Nainar | 4967a71 | 2016-09-19 22:19:55 -0700 | [diff] [blame] | 1425 | .. code-block:: text |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1426 | |
| 1427 | [ B5 (ENTRY) ] |
| 1428 | Predecessors (0): |
| 1429 | Successors (1): B4 |
| 1430 | |
| 1431 | [ B4 ] |
| 1432 | 1: x = x + 1 |
| 1433 | 2: (x > 2) |
| 1434 | T: if [B4.2] |
| 1435 | Predecessors (1): B5 |
| 1436 | Successors (2): B3 B2 |
| 1437 | |
| 1438 | [ B3 ] |
| 1439 | 1: x++ |
| 1440 | Predecessors (1): B4 |
| 1441 | Successors (1): B1 |
| 1442 | |
| 1443 | [ B2 ] |
| 1444 | 1: x += 2 |
| 1445 | 2: x *= 2 |
| 1446 | Predecessors (1): B4 |
| 1447 | Successors (1): B1 |
| 1448 | |
| 1449 | [ B1 ] |
| 1450 | 1: return x; |
| 1451 | Predecessors (2): B2 B3 |
| 1452 | Successors (1): B0 |
| 1453 | |
| 1454 | [ B0 (EXIT) ] |
| 1455 | Predecessors (1): B1 |
| 1456 | Successors (0): |
| 1457 | |
| 1458 | For each block, the pretty-printed output displays for each block the number of |
| 1459 | *predecessor* blocks (blocks that have outgoing control-flow to the given |
| 1460 | block) and *successor* blocks (blocks that have control-flow that have incoming |
| 1461 | control-flow from the given block). We can also clearly see the special entry |
| 1462 | and exit blocks at the beginning and end of the pretty-printed output. For the |
| 1463 | entry block (block B5), the number of predecessor blocks is 0, while for the |
| 1464 | exit block (block B0) the number of successor blocks is 0. |
| 1465 | |
| 1466 | The most interesting block here is B4, whose outgoing control-flow represents |
| 1467 | the branching caused by the sole if-statement in ``foo``. Of particular |
| 1468 | interest is the second statement in the block, ``(x > 2)``, and the terminator, |
| 1469 | printed as ``if [B4.2]``. The second statement represents the evaluation of |
| 1470 | the condition of the if-statement, which occurs before the actual branching of |
| 1471 | control-flow. Within the ``CFGBlock`` for B4, the ``Stmt*`` for the second |
| 1472 | statement refers to the actual expression in the AST for ``(x > 2)``. Thus |
| 1473 | pointers to subclasses of ``Expr`` can appear in the list of statements in a |
| 1474 | block, and not just subclasses of ``Stmt`` that refer to proper C statements. |
| 1475 | |
| 1476 | The terminator of block B4 is a pointer to the ``IfStmt`` object in the AST. |
| 1477 | The pretty-printer outputs ``if [B4.2]`` because the condition expression of |
| 1478 | the if-statement has an actual place in the basic block, and thus the |
| 1479 | terminator is essentially *referring* to the expression that is the second |
| 1480 | statement of block B4 (i.e., B4.2). In this manner, conditions for |
| 1481 | control-flow (which also includes conditions for loops and switch statements) |
| 1482 | are hoisted into the actual basic block. |
| 1483 | |
| 1484 | .. Implicit Control-Flow |
| 1485 | .. ^^^^^^^^^^^^^^^^^^^^^ |
| 1486 | |
| 1487 | .. A key design principle of the ``CFG`` class was to not require any |
| 1488 | .. transformations to the AST in order to represent control-flow. Thus the |
| 1489 | .. ``CFG`` does not perform any "lowering" of the statements in an AST: loops |
| 1490 | .. are not transformed into guarded gotos, short-circuit operations are not |
| 1491 | .. converted to a set of if-statements, and so on. |
| 1492 | |
| 1493 | Constant Folding in the Clang AST |
| 1494 | --------------------------------- |
| 1495 | |
| 1496 | There are several places where constants and constant folding matter a lot to |
| 1497 | the Clang front-end. First, in general, we prefer the AST to retain the source |
| 1498 | code as close to how the user wrote it as possible. This means that if they |
| 1499 | wrote "``5+4``", we want to keep the addition and two constants in the AST, we |
| 1500 | don't want to fold to "``9``". This means that constant folding in various |
| 1501 | ways turns into a tree walk that needs to handle the various cases. |
| 1502 | |
| 1503 | However, there are places in both C and C++ that require constants to be |
| 1504 | folded. For example, the C standard defines what an "integer constant |
| 1505 | expression" (i-c-e) is with very precise and specific requirements. The |
| 1506 | language then requires i-c-e's in a lot of places (for example, the size of a |
| 1507 | bitfield, the value for a case statement, etc). For these, we have to be able |
| 1508 | to constant fold the constants, to do semantic checks (e.g., verify bitfield |
| 1509 | size is non-negative and that case statements aren't duplicated). We aim for |
| 1510 | Clang to be very pedantic about this, diagnosing cases when the code does not |
| 1511 | use an i-c-e where one is required, but accepting the code unless running with |
| 1512 | ``-pedantic-errors``. |
| 1513 | |
| 1514 | Things get a little bit more tricky when it comes to compatibility with |
| 1515 | real-world source code. Specifically, GCC has historically accepted a huge |
| 1516 | superset of expressions as i-c-e's, and a lot of real world code depends on |
| 1517 | this unfortuate accident of history (including, e.g., the glibc system |
| 1518 | headers). GCC accepts anything its "fold" optimizer is capable of reducing to |
| 1519 | an integer constant, which means that the definition of what it accepts changes |
| 1520 | as its optimizer does. One example is that GCC accepts things like "``case |
| 1521 | X-X:``" even when ``X`` is a variable, because it can fold this to 0. |
| 1522 | |
| 1523 | Another issue are how constants interact with the extensions we support, such |
| 1524 | as ``__builtin_constant_p``, ``__builtin_inf``, ``__extension__`` and many |
| 1525 | others. C99 obviously does not specify the semantics of any of these |
| 1526 | extensions, and the definition of i-c-e does not include them. However, these |
| 1527 | extensions are often used in real code, and we have to have a way to reason |
| 1528 | about them. |
| 1529 | |
| 1530 | Finally, this is not just a problem for semantic analysis. The code generator |
| 1531 | and other clients have to be able to fold constants (e.g., to initialize global |
| 1532 | variables) and has to handle a superset of what C99 allows. Further, these |
| 1533 | clients can benefit from extended information. For example, we know that |
| 1534 | "``foo() || 1``" always evaluates to ``true``, but we can't replace the |
| 1535 | expression with ``true`` because it has side effects. |
| 1536 | |
| 1537 | Implementation Approach |
| 1538 | ^^^^^^^^^^^^^^^^^^^^^^^ |
| 1539 | |
| 1540 | After trying several different approaches, we've finally converged on a design |
| 1541 | (Note, at the time of this writing, not all of this has been implemented, |
| 1542 | consider this a design goal!). Our basic approach is to define a single |
| 1543 | recursive method evaluation method (``Expr::Evaluate``), which is implemented |
| 1544 | in ``AST/ExprConstant.cpp``. Given an expression with "scalar" type (integer, |
| 1545 | fp, complex, or pointer) this method returns the following information: |
| 1546 | |
| 1547 | * Whether the expression is an integer constant expression, a general constant |
| 1548 | that was folded but has no side effects, a general constant that was folded |
| 1549 | but that does have side effects, or an uncomputable/unfoldable value. |
| 1550 | * If the expression was computable in any way, this method returns the |
| 1551 | ``APValue`` for the result of the expression. |
| 1552 | * If the expression is not evaluatable at all, this method returns information |
| 1553 | on one of the problems with the expression. This includes a |
| 1554 | ``SourceLocation`` for where the problem is, and a diagnostic ID that explains |
| 1555 | the problem. The diagnostic should have ``ERROR`` type. |
| 1556 | * If the expression is not an integer constant expression, this method returns |
| 1557 | information on one of the problems with the expression. This includes a |
| 1558 | ``SourceLocation`` for where the problem is, and a diagnostic ID that |
| 1559 | explains the problem. The diagnostic should have ``EXTENSION`` type. |
| 1560 | |
| 1561 | This information gives various clients the flexibility that they want, and we |
| 1562 | will eventually have some helper methods for various extensions. For example, |
| 1563 | ``Sema`` should have a ``Sema::VerifyIntegerConstantExpression`` method, which |
| 1564 | calls ``Evaluate`` on the expression. If the expression is not foldable, the |
| 1565 | error is emitted, and it would return ``true``. If the expression is not an |
| 1566 | i-c-e, the ``EXTENSION`` diagnostic is emitted. Finally it would return |
| 1567 | ``false`` to indicate that the AST is OK. |
| 1568 | |
| 1569 | Other clients can use the information in other ways, for example, codegen can |
| 1570 | just use expressions that are foldable in any way. |
| 1571 | |
| 1572 | Extensions |
| 1573 | ^^^^^^^^^^ |
| 1574 | |
| 1575 | This section describes how some of the various extensions Clang supports |
| 1576 | interacts with constant evaluation: |
| 1577 | |
| 1578 | * ``__extension__``: The expression form of this extension causes any |
| 1579 | evaluatable subexpression to be accepted as an integer constant expression. |
| 1580 | * ``__builtin_constant_p``: This returns true (as an integer constant |
| 1581 | expression) if the operand evaluates to either a numeric value (that is, not |
| 1582 | a pointer cast to integral type) of integral, enumeration, floating or |
| 1583 | complex type, or if it evaluates to the address of the first character of a |
| 1584 | string literal (possibly cast to some other type). As a special case, if |
| 1585 | ``__builtin_constant_p`` is the (potentially parenthesized) condition of a |
| 1586 | conditional operator expression ("``?:``"), only the true side of the |
| 1587 | conditional operator is considered, and it is evaluated with full constant |
| 1588 | folding. |
| 1589 | * ``__builtin_choose_expr``: The condition is required to be an integer |
| 1590 | constant expression, but we accept any constant as an "extension of an |
| 1591 | extension". This only evaluates one operand depending on which way the |
| 1592 | condition evaluates. |
| 1593 | * ``__builtin_classify_type``: This always returns an integer constant |
| 1594 | expression. |
| 1595 | * ``__builtin_inf, nan, ...``: These are treated just like a floating-point |
| 1596 | literal. |
| 1597 | * ``__builtin_abs, copysign, ...``: These are constant folded as general |
| 1598 | constant expressions. |
| 1599 | * ``__builtin_strlen`` and ``strlen``: These are constant folded as integer |
| 1600 | constant expressions if the argument is a string literal. |
| 1601 | |
Stephen Hines | 0e2c34f | 2015-03-23 12:09:02 -0700 | [diff] [blame] | 1602 | .. _Sema: |
| 1603 | |
| 1604 | The Sema Library |
| 1605 | ================ |
| 1606 | |
| 1607 | This library is called by the :ref:`Parser library <Parser>` during parsing to |
| 1608 | do semantic analysis of the input. For valid programs, Sema builds an AST for |
| 1609 | parsed constructs. |
| 1610 | |
| 1611 | .. _CodeGen: |
| 1612 | |
| 1613 | The CodeGen Library |
| 1614 | =================== |
| 1615 | |
| 1616 | CodeGen takes an :ref:`AST <AST>` as input and produces `LLVM IR code |
| 1617 | <//llvm.org/docs/LangRef.html>`_ from it. |
| 1618 | |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1619 | How to change Clang |
| 1620 | =================== |
| 1621 | |
| 1622 | How to add an attribute |
| 1623 | ----------------------- |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1624 | Attributes are a form of metadata that can be attached to a program construct, |
| 1625 | allowing the programmer to pass semantic information along to the compiler for |
| 1626 | various uses. For example, attributes may be used to alter the code generation |
| 1627 | for a program construct, or to provide extra semantic information for static |
| 1628 | analysis. This document explains how to add a custom attribute to Clang. |
| 1629 | Documentation on existing attributes can be found `here |
| 1630 | <//clang.llvm.org/docs/AttributeReference.html>`_. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1631 | |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1632 | Attribute Basics |
| 1633 | ^^^^^^^^^^^^^^^^ |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1634 | Attributes in Clang are handled in three stages: parsing into a parsed attribute |
| 1635 | representation, conversion from a parsed attribute into a semantic attribute, |
| 1636 | and then the semantic handling of the attribute. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1637 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1638 | Parsing of the attribute is determined by the various syntactic forms attributes |
| 1639 | can take, such as GNU, C++11, and Microsoft style attributes, as well as other |
| 1640 | information provided by the table definition of the attribute. Ultimately, the |
| 1641 | parsed representation of an attribute object is an ``AttributeList`` object. |
| 1642 | These parsed attributes chain together as a list of parsed attributes attached |
| 1643 | to a declarator or declaration specifier. The parsing of attributes is handled |
| 1644 | automatically by Clang, except for attributes spelled as keywords. When |
| 1645 | implementing a keyword attribute, the parsing of the keyword and creation of the |
| 1646 | ``AttributeList`` object must be done manually. |
| 1647 | |
| 1648 | Eventually, ``Sema::ProcessDeclAttributeList()`` is called with a ``Decl`` and |
| 1649 | an ``AttributeList``, at which point the parsed attribute can be transformed |
| 1650 | into a semantic attribute. The process by which a parsed attribute is converted |
| 1651 | into a semantic attribute depends on the attribute definition and semantic |
| 1652 | requirements of the attribute. The end result, however, is that the semantic |
| 1653 | attribute object is attached to the ``Decl`` object, and can be obtained by a |
| 1654 | call to ``Decl::getAttr<T>()``. |
| 1655 | |
| 1656 | The structure of the semantic attribute is also governed by the attribute |
| 1657 | definition given in Attr.td. This definition is used to automatically generate |
| 1658 | functionality used for the implementation of the attribute, such as a class |
| 1659 | derived from ``clang::Attr``, information for the parser to use, automated |
| 1660 | semantic checking for some attributes, etc. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1661 | |
| 1662 | |
| 1663 | ``include/clang/Basic/Attr.td`` |
| 1664 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1665 | The first step to adding a new attribute to Clang is to add its definition to |
| 1666 | `include/clang/Basic/Attr.td |
| 1667 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/Attr.td?view=markup>`_. |
| 1668 | This tablegen definition must derive from the ``Attr`` (tablegen, not |
| 1669 | semantic) type, or one of its derivatives. Most attributes will derive from the |
| 1670 | ``InheritableAttr`` type, which specifies that the attribute can be inherited by |
| 1671 | later redeclarations of the ``Decl`` it is associated with. |
| 1672 | ``InheritableParamAttr`` is similar to ``InheritableAttr``, except that the |
| 1673 | attribute is written on a parameter instead of a declaration. If the attribute |
| 1674 | is intended to apply to a type instead of a declaration, such an attribute |
| 1675 | should derive from ``TypeAttr``, and will generally not be given an AST |
| 1676 | representation. (Note that this document does not cover the creation of type |
| 1677 | attributes.) An attribute that inherits from ``IgnoredAttr`` is parsed, but will |
| 1678 | generate an ignored attribute diagnostic when used, which may be useful when an |
| 1679 | attribute is supported by another vendor but not supported by clang. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1680 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1681 | The definition will specify several key pieces of information, such as the |
| 1682 | semantic name of the attribute, the spellings the attribute supports, the |
| 1683 | arguments the attribute expects, and more. Most members of the ``Attr`` tablegen |
| 1684 | type do not require definitions in the derived definition as the default |
| 1685 | suffice. However, every attribute must specify at least a spelling list, a |
| 1686 | subject list, and a documentation list. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1687 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1688 | Spellings |
| 1689 | ~~~~~~~~~ |
| 1690 | All attributes are required to specify a spelling list that denotes the ways in |
| 1691 | which the attribute can be spelled. For instance, a single semantic attribute |
| 1692 | may have a keyword spelling, as well as a C++11 spelling and a GNU spelling. An |
| 1693 | empty spelling list is also permissible and may be useful for attributes which |
| 1694 | are created implicitly. The following spellings are accepted: |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1695 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1696 | ============ ================================================================ |
| 1697 | Spelling Description |
| 1698 | ============ ================================================================ |
| 1699 | ``GNU`` Spelled with a GNU-style ``__attribute__((attr))`` syntax and |
| 1700 | placement. |
| 1701 | ``CXX11`` Spelled with a C++-style ``[[attr]]`` syntax. If the attribute |
| 1702 | is meant to be used by Clang, it should set the namespace to |
| 1703 | ``"clang"``. |
| 1704 | ``Declspec`` Spelled with a Microsoft-style ``__declspec(attr)`` syntax. |
| 1705 | ``Keyword`` The attribute is spelled as a keyword, and required custom |
| 1706 | parsing. |
| 1707 | ``GCC`` Specifies two spellings: the first is a GNU-style spelling, and |
| 1708 | the second is a C++-style spelling with the ``gnu`` namespace. |
| 1709 | Attributes should only specify this spelling for attributes |
| 1710 | supported by GCC. |
| 1711 | ``Pragma`` The attribute is spelled as a ``#pragma``, and requires custom |
| 1712 | processing within the preprocessor. If the attribute is meant to |
| 1713 | be used by Clang, it should set the namespace to ``"clang"``. |
| 1714 | Note that this spelling is not used for declaration attributes. |
| 1715 | ============ ================================================================ |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1716 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1717 | Subjects |
| 1718 | ~~~~~~~~ |
| 1719 | Attributes appertain to one or more ``Decl`` subjects. If the attribute attempts |
| 1720 | to attach to a subject that is not in the subject list, a diagnostic is issued |
| 1721 | automatically. Whether the diagnostic is a warning or an error depends on how |
| 1722 | the attribute's ``SubjectList`` is defined, but the default behavior is to warn. |
| 1723 | The diagnostics displayed to the user are automatically determined based on the |
| 1724 | subjects in the list, but a custom diagnostic parameter can also be specified in |
| 1725 | the ``SubjectList``. The diagnostics generated for subject list violations are |
| 1726 | either ``diag::warn_attribute_wrong_decl_type`` or |
| 1727 | ``diag::err_attribute_wrong_decl_type``, and the parameter enumeration is found |
| 1728 | in `include/clang/Sema/AttributeList.h |
| 1729 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Sema/AttributeList.h?view=markup>`_ |
| 1730 | If a previously unused Decl node is added to the ``SubjectList``, the logic used |
| 1731 | to automatically determine the diagnostic parameter in `utils/TableGen/ClangAttrEmitter.cpp |
| 1732 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/utils/TableGen/ClangAttrEmitter.cpp?view=markup>`_ |
| 1733 | may need to be updated. |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1734 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1735 | By default, all subjects in the SubjectList must either be a Decl node defined |
| 1736 | in ``DeclNodes.td``, or a statement node defined in ``StmtNodes.td``. However, |
| 1737 | more complex subjects can be created by creating a ``SubsetSubject`` object. |
| 1738 | Each such object has a base subject which it appertains to (which must be a |
| 1739 | Decl or Stmt node, and not a SubsetSubject node), and some custom code which is |
| 1740 | called when determining whether an attribute appertains to the subject. For |
| 1741 | instance, a ``NonBitField`` SubsetSubject appertains to a ``FieldDecl``, and |
| 1742 | tests whether the given FieldDecl is a bit field. When a SubsetSubject is |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1743 | specified in a SubjectList, a custom diagnostic parameter must also be provided. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1744 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1745 | Diagnostic checking for attribute subject lists is automated except when |
| 1746 | ``HasCustomParsing`` is set to ``1``. |
| 1747 | |
| 1748 | Documentation |
| 1749 | ~~~~~~~~~~~~~ |
| 1750 | All attributes must have some form of documentation associated with them. |
| 1751 | Documentation is table generated on the public web server by a server-side |
| 1752 | process that runs daily. Generally, the documentation for an attribute is a |
| 1753 | stand-alone definition in `include/clang/Basic/AttrDocs.td |
| 1754 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/AttdDocs.td?view=markup>`_ |
| 1755 | that is named after the attribute being documented. |
| 1756 | |
| 1757 | If the attribute is not for public consumption, or is an implicitly-created |
| 1758 | attribute that has no visible spelling, the documentation list can specify the |
| 1759 | ``Undocumented`` object. Otherwise, the attribute should have its documentation |
| 1760 | added to AttrDocs.td. |
| 1761 | |
| 1762 | Documentation derives from the ``Documentation`` tablegen type. All derived |
| 1763 | types must specify a documentation category and the actual documentation itself. |
| 1764 | Additionally, it can specify a custom heading for the attribute, though a |
| 1765 | default heading will be chosen when possible. |
| 1766 | |
| 1767 | There are four predefined documentation categories: ``DocCatFunction`` for |
| 1768 | attributes that appertain to function-like subjects, ``DocCatVariable`` for |
| 1769 | attributes that appertain to variable-like subjects, ``DocCatType`` for type |
| 1770 | attributes, and ``DocCatStmt`` for statement attributes. A custom documentation |
| 1771 | category should be used for groups of attributes with similar functionality. |
| 1772 | Custom categories are good for providing overview information for the attributes |
| 1773 | grouped under it. For instance, the consumed annotation attributes define a |
| 1774 | custom category, ``DocCatConsumed``, that explains what consumed annotations are |
| 1775 | at a high level. |
| 1776 | |
| 1777 | Documentation content (whether it is for an attribute or a category) is written |
| 1778 | using reStructuredText (RST) syntax. |
| 1779 | |
| 1780 | After writing the documentation for the attribute, it should be locally tested |
| 1781 | to ensure that there are no issues generating the documentation on the server. |
| 1782 | Local testing requires a fresh build of clang-tblgen. To generate the attribute |
| 1783 | documentation, execute the following command:: |
| 1784 | |
| 1785 | clang-tblgen -gen-attr-docs -I /path/to/clang/include /path/to/clang/include/clang/Basic/Attr.td -o /path/to/clang/docs/AttributeReference.rst |
| 1786 | |
| 1787 | When testing locally, *do not* commit changes to ``AttributeReference.rst``. |
| 1788 | This file is generated by the server automatically, and any changes made to this |
| 1789 | file will be overwritten. |
| 1790 | |
| 1791 | Arguments |
| 1792 | ~~~~~~~~~ |
| 1793 | Attributes may optionally specify a list of arguments that can be passed to the |
| 1794 | attribute. Attribute arguments specify both the parsed form and the semantic |
| 1795 | form of the attribute. For example, if ``Args`` is |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1796 | ``[StringArgument<"Arg1">, IntArgument<"Arg2">]`` then |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1797 | ``__attribute__((myattribute("Hello", 3)))`` will be a valid use; it requires |
| 1798 | two arguments while parsing, and the Attr subclass' constructor for the |
| 1799 | semantic attribute will require a string and integer argument. |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1800 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1801 | All arguments have a name and a flag that specifies whether the argument is |
| 1802 | optional. The associated C++ type of the argument is determined by the argument |
| 1803 | definition type. If the existing argument types are insufficient, new types can |
| 1804 | be created, but it requires modifying `utils/TableGen/ClangAttrEmitter.cpp |
| 1805 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/utils/TableGen/ClangAttrEmitter.cpp?view=markup>`_ |
| 1806 | to properly support the type. |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1807 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1808 | Other Properties |
| 1809 | ~~~~~~~~~~~~~~~~ |
| 1810 | The ``Attr`` definition has other members which control the behavior of the |
| 1811 | attribute. Many of them are special-purpose and beyond the scope of this |
| 1812 | document, however a few deserve mention. |
| 1813 | |
| 1814 | If the parsed form of the attribute is more complex, or differs from the |
| 1815 | semantic form, the ``HasCustomParsing`` bit can be set to ``1`` for the class, |
| 1816 | and the parsing code in `Parser::ParseGNUAttributeArgs() |
| 1817 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/ParseDecl.cpp?view=markup>`_ |
| 1818 | can be updated for the special case. Note that this only applies to arguments |
| 1819 | with a GNU spelling -- attributes with a __declspec spelling currently ignore |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1820 | this flag and are handled by ``Parser::ParseMicrosoftDeclSpec``. |
| 1821 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1822 | Note that setting this member to 1 will opt out of common attribute semantic |
| 1823 | handling, requiring extra implementation efforts to ensure the attribute |
| 1824 | appertains to the appropriate subject, etc. |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1825 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1826 | If the attribute should not be propagated from from a template declaration to an |
| 1827 | instantiation of the template, set the ``Clone`` member to 0. By default, all |
| 1828 | attributes will be cloned to template instantiations. |
| 1829 | |
| 1830 | Attributes that do not require an AST node should set the ``ASTNode`` field to |
| 1831 | ``0`` to avoid polluting the AST. Note that anything inheriting from |
| 1832 | ``TypeAttr`` or ``IgnoredAttr`` automatically do not generate an AST node. All |
| 1833 | other attributes generate an AST node by default. The AST node is the semantic |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1834 | representation of the attribute. |
| 1835 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1836 | The ``LangOpts`` field specifies a list of language options required by the |
| 1837 | attribute. For instance, all of the CUDA-specific attributes specify ``[CUDA]`` |
| 1838 | for the ``LangOpts`` field, and when the CUDA language option is not enabled, an |
| 1839 | "attribute ignored" warning diagnostic is emitted. Since language options are |
| 1840 | not table generated nodes, new language options must be created manually and |
| 1841 | should specify the spelling used by ``LangOptions`` class. |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1842 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1843 | Custom accessors can be generated for an attribute based on the spelling list |
| 1844 | for that attribute. For instance, if an attribute has two different spellings: |
| 1845 | 'Foo' and 'Bar', accessors can be created: |
| 1846 | ``[Accessor<"isFoo", [GNU<"Foo">]>, Accessor<"isBar", [GNU<"Bar">]>]`` |
| 1847 | These accessors will be generated on the semantic form of the attribute, |
| 1848 | accepting no arguments and returning a ``bool``. |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1849 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1850 | Attributes that do not require custom semantic handling should set the |
| 1851 | ``SemaHandler`` field to ``0``. Note that anything inheriting from |
| 1852 | ``IgnoredAttr`` automatically do not get a semantic handler. All other |
| 1853 | attributes are assumed to use a semantic handler by default. Attributes |
| 1854 | without a semantic handler are not given a parsed attribute ``Kind`` enumerator. |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1855 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1856 | Target-specific attributes may share a spelling with other attributes in |
| 1857 | different targets. For instance, the ARM and MSP430 targets both have an |
| 1858 | attribute spelled ``GNU<"interrupt">``, but with different parsing and semantic |
| 1859 | requirements. To support this feature, an attribute inheriting from |
| 1860 | ``TargetSpecificAttribute`` may specify a ``ParseKind`` field. This field |
| 1861 | should be the same value between all arguments sharing a spelling, and |
| 1862 | corresponds to the parsed attribute's ``Kind`` enumerator. This allows |
| 1863 | attributes to share a parsed attribute kind, but have distinct semantic |
| 1864 | attribute classes. For instance, ``AttributeList::AT_Interrupt`` is the shared |
| 1865 | parsed attribute kind, but ARMInterruptAttr and MSP430InterruptAttr are the |
| 1866 | semantic attributes generated. |
| 1867 | |
| 1868 | By default, when declarations are merging attributes, an attribute will not be |
| 1869 | duplicated. However, if an attribute can be duplicated during this merging |
| 1870 | stage, set ``DuplicatesAllowedWhileMerging`` to ``1``, and the attribute will |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1871 | be merged. |
| 1872 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1873 | By default, attribute arguments are parsed in an evaluated context. If the |
| 1874 | arguments for an attribute should be parsed in an unevaluated context (akin to |
| 1875 | the way the argument to a ``sizeof`` expression is parsed), set |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1876 | ``ParseArgumentsAsUnevaluated`` to ``1``. |
| 1877 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1878 | If additional functionality is desired for the semantic form of the attribute, |
| 1879 | the ``AdditionalMembers`` field specifies code to be copied verbatim into the |
| 1880 | semantic attribute class object, with ``public`` access. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1881 | |
| 1882 | Boilerplate |
| 1883 | ^^^^^^^^^^^ |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1884 | All semantic processing of declaration attributes happens in `lib/Sema/SemaDeclAttr.cpp |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1885 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Sema/SemaDeclAttr.cpp?view=markup>`_, |
| 1886 | and generally starts in the ``ProcessDeclAttribute()`` function. If the |
| 1887 | attribute is a "simple" attribute -- meaning that it requires no custom semantic |
| 1888 | processing aside from what is automatically provided, add a call to |
| 1889 | ``handleSimpleAttribute<YourAttr>(S, D, Attr);`` to the switch statement. |
| 1890 | Otherwise, write a new ``handleYourAttr()`` function, and add that to the switch |
| 1891 | statement. Please do not implement handling logic directly in the ``case`` for |
| 1892 | the attribute. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1893 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1894 | Unless otherwise specified by the attribute definition, common semantic checking |
| 1895 | of the parsed attribute is handled automatically. This includes diagnosing |
| 1896 | parsed attributes that do not appertain to the given ``Decl``, ensuring the |
| 1897 | correct minimum number of arguments are passed, etc. |
| 1898 | |
| 1899 | If the attribute adds additional warnings, define a ``DiagGroup`` in |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1900 | `include/clang/Basic/DiagnosticGroups.td |
| 1901 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/DiagnosticGroups.td?view=markup>`_ |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1902 | named after the attribute's ``Spelling`` with "_"s replaced by "-"s. If there |
| 1903 | is only a single diagnostic, it is permissible to use ``InGroup<DiagGroup<"your-attribute">>`` |
| 1904 | directly in `DiagnosticSemaKinds.td |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1905 | <http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/DiagnosticSemaKinds.td?view=markup>`_ |
| 1906 | |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1907 | All semantic diagnostics generated for your attribute, including automatically- |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1908 | generated ones (such as subjects and argument counts), should have a |
Stephen Hines | 651f13c | 2014-04-23 16:59:28 -0700 | [diff] [blame] | 1909 | corresponding test case. |
| 1910 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1911 | Semantic handling |
| 1912 | ^^^^^^^^^^^^^^^^^ |
| 1913 | Most attributes are implemented to have some effect on the compiler. For |
| 1914 | instance, to modify the way code is generated, or to add extra semantic checks |
| 1915 | for an analysis pass, etc. Having added the attribute definition and conversion |
| 1916 | to the semantic representation for the attribute, what remains is to implement |
| 1917 | the custom logic requiring use of the attribute. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1918 | |
Pirama Arumuga Nainar | b6d6993 | 2015-07-01 12:25:36 -0700 | [diff] [blame] | 1919 | The ``clang::Decl`` object can be queried for the presence or absence of an |
| 1920 | attribute using ``hasAttr<T>()``. To obtain a pointer to the semantic |
| 1921 | representation of the attribute, ``getAttr<T>`` may be used. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 1922 | |
| 1923 | How to add an expression or statement |
| 1924 | ------------------------------------- |
| 1925 | |
| 1926 | Expressions and statements are one of the most fundamental constructs within a |
| 1927 | compiler, because they interact with many different parts of the AST, semantic |
| 1928 | analysis, and IR generation. Therefore, adding a new expression or statement |
| 1929 | kind into Clang requires some care. The following list details the various |
| 1930 | places in Clang where an expression or statement needs to be introduced, along |
| 1931 | with patterns to follow to ensure that the new expression or statement works |
| 1932 | well across all of the C languages. We focus on expressions, but statements |
| 1933 | are similar. |
| 1934 | |
| 1935 | #. Introduce parsing actions into the parser. Recursive-descent parsing is |
| 1936 | mostly self-explanatory, but there are a few things that are worth keeping |
| 1937 | in mind: |
| 1938 | |
| 1939 | * Keep as much source location information as possible! You'll want it later |
| 1940 | to produce great diagnostics and support Clang's various features that map |
| 1941 | between source code and the AST. |
| 1942 | * Write tests for all of the "bad" parsing cases, to make sure your recovery |
| 1943 | is good. If you have matched delimiters (e.g., parentheses, square |
| 1944 | brackets, etc.), use ``Parser::BalancedDelimiterTracker`` to give nice |
| 1945 | diagnostics when things go wrong. |
| 1946 | |
| 1947 | #. Introduce semantic analysis actions into ``Sema``. Semantic analysis should |
| 1948 | always involve two functions: an ``ActOnXXX`` function that will be called |
| 1949 | directly from the parser, and a ``BuildXXX`` function that performs the |
| 1950 | actual semantic analysis and will (eventually!) build the AST node. It's |
| 1951 | fairly common for the ``ActOnCXX`` function to do very little (often just |
| 1952 | some minor translation from the parser's representation to ``Sema``'s |
| 1953 | representation of the same thing), but the separation is still important: |
| 1954 | C++ template instantiation, for example, should always call the ``BuildXXX`` |
| 1955 | variant. Several notes on semantic analysis before we get into construction |
| 1956 | of the AST: |
| 1957 | |
| 1958 | * Your expression probably involves some types and some subexpressions. |
| 1959 | Make sure to fully check that those types, and the types of those |
| 1960 | subexpressions, meet your expectations. Add implicit conversions where |
| 1961 | necessary to make sure that all of the types line up exactly the way you |
| 1962 | want them. Write extensive tests to check that you're getting good |
| 1963 | diagnostics for mistakes and that you can use various forms of |
| 1964 | subexpressions with your expression. |
| 1965 | * When type-checking a type or subexpression, make sure to first check |
| 1966 | whether the type is "dependent" (``Type::isDependentType()``) or whether a |
| 1967 | subexpression is type-dependent (``Expr::isTypeDependent()``). If any of |
| 1968 | these return ``true``, then you're inside a template and you can't do much |
| 1969 | type-checking now. That's normal, and your AST node (when you get there) |
| 1970 | will have to deal with this case. At this point, you can write tests that |
| 1971 | use your expression within templates, but don't try to instantiate the |
| 1972 | templates. |
| 1973 | * For each subexpression, be sure to call ``Sema::CheckPlaceholderExpr()`` |
| 1974 | to deal with "weird" expressions that don't behave well as subexpressions. |
| 1975 | Then, determine whether you need to perform lvalue-to-rvalue conversions |
| 1976 | (``Sema::DefaultLvalueConversions``) or the usual unary conversions |
| 1977 | (``Sema::UsualUnaryConversions``), for places where the subexpression is |
| 1978 | producing a value you intend to use. |
| 1979 | * Your ``BuildXXX`` function will probably just return ``ExprError()`` at |
| 1980 | this point, since you don't have an AST. That's perfectly fine, and |
| 1981 | shouldn't impact your testing. |
| 1982 | |
| 1983 | #. Introduce an AST node for your new expression. This starts with declaring |
| 1984 | the node in ``include/Basic/StmtNodes.td`` and creating a new class for your |
| 1985 | expression in the appropriate ``include/AST/Expr*.h`` header. It's best to |
| 1986 | look at the class for a similar expression to get ideas, and there are some |
| 1987 | specific things to watch for: |
| 1988 | |
| 1989 | * If you need to allocate memory, use the ``ASTContext`` allocator to |
| 1990 | allocate memory. Never use raw ``malloc`` or ``new``, and never hold any |
| 1991 | resources in an AST node, because the destructor of an AST node is never |
| 1992 | called. |
| 1993 | * Make sure that ``getSourceRange()`` covers the exact source range of your |
| 1994 | expression. This is needed for diagnostics and for IDE support. |
| 1995 | * Make sure that ``children()`` visits all of the subexpressions. This is |
| 1996 | important for a number of features (e.g., IDE support, C++ variadic |
| 1997 | templates). If you have sub-types, you'll also need to visit those |
Pirama Arumuga Nainar | 87d948e | 2016-03-03 15:49:35 -0800 | [diff] [blame] | 1998 | sub-types in ``RecursiveASTVisitor``. |
Stephen Hines | c568f1e | 2014-07-21 00:47:37 -0700 | [diff] [blame] | 1999 | * Add printing support (``StmtPrinter.cpp``) for your expression. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 2000 | * Add profiling support (``StmtProfile.cpp``) for your AST node, noting the |
| 2001 | distinguishing (non-source location) characteristics of an instance of |
| 2002 | your expression. Omitting this step will lead to hard-to-diagnose |
| 2003 | failures regarding matching of template declarations. |
Stephen Hines | c568f1e | 2014-07-21 00:47:37 -0700 | [diff] [blame] | 2004 | * Add serialization support (``ASTReaderStmt.cpp``, ``ASTWriterStmt.cpp``) |
| 2005 | for your AST node. |
Dmitri Gribenko | 5cc0580 | 2012-12-15 20:41:17 +0000 | [diff] [blame] | 2006 | |
| 2007 | #. Teach semantic analysis to build your AST node. At this point, you can wire |
| 2008 | up your ``Sema::BuildXXX`` function to actually create your AST. A few |
| 2009 | things to check at this point: |
| 2010 | |
| 2011 | * If your expression can construct a new C++ class or return a new |
| 2012 | Objective-C object, be sure to update and then call |
| 2013 | ``Sema::MaybeBindToTemporary`` for your just-created AST node to be sure |
| 2014 | that the object gets properly destructed. An easy way to test this is to |
| 2015 | return a C++ class with a private destructor: semantic analysis should |
| 2016 | flag an error here with the attempt to call the destructor. |
| 2017 | * Inspect the generated AST by printing it using ``clang -cc1 -ast-print``, |
| 2018 | to make sure you're capturing all of the important information about how |
| 2019 | the AST was written. |
| 2020 | * Inspect the generated AST under ``clang -cc1 -ast-dump`` to verify that |
| 2021 | all of the types in the generated AST line up the way you want them. |
| 2022 | Remember that clients of the AST should never have to "think" to |
| 2023 | understand what's going on. For example, all implicit conversions should |
| 2024 | show up explicitly in the AST. |
| 2025 | * Write tests that use your expression as a subexpression of other, |
| 2026 | well-known expressions. Can you call a function using your expression as |
| 2027 | an argument? Can you use the ternary operator? |
| 2028 | |
| 2029 | #. Teach code generation to create IR to your AST node. This step is the first |
| 2030 | (and only) that requires knowledge of LLVM IR. There are several things to |
| 2031 | keep in mind: |
| 2032 | |
| 2033 | * Code generation is separated into scalar/aggregate/complex and |
| 2034 | lvalue/rvalue paths, depending on what kind of result your expression |
| 2035 | produces. On occasion, this requires some careful factoring of code to |
| 2036 | avoid duplication. |
| 2037 | * ``CodeGenFunction`` contains functions ``ConvertType`` and |
| 2038 | ``ConvertTypeForMem`` that convert Clang's types (``clang::Type*`` or |
| 2039 | ``clang::QualType``) to LLVM types. Use the former for values, and the |
| 2040 | later for memory locations: test with the C++ "``bool``" type to check |
| 2041 | this. If you find that you are having to use LLVM bitcasts to make the |
| 2042 | subexpressions of your expression have the type that your expression |
| 2043 | expects, STOP! Go fix semantic analysis and the AST so that you don't |
| 2044 | need these bitcasts. |
| 2045 | * The ``CodeGenFunction`` class has a number of helper functions to make |
| 2046 | certain operations easy, such as generating code to produce an lvalue or |
| 2047 | an rvalue, or to initialize a memory location with a given value. Prefer |
| 2048 | to use these functions rather than directly writing loads and stores, |
| 2049 | because these functions take care of some of the tricky details for you |
| 2050 | (e.g., for exceptions). |
| 2051 | * If your expression requires some special behavior in the event of an |
| 2052 | exception, look at the ``push*Cleanup`` functions in ``CodeGenFunction`` |
| 2053 | to introduce a cleanup. You shouldn't have to deal with |
| 2054 | exception-handling directly. |
| 2055 | * Testing is extremely important in IR generation. Use ``clang -cc1 |
| 2056 | -emit-llvm`` and `FileCheck |
| 2057 | <http://llvm.org/docs/CommandGuide/FileCheck.html>`_ to verify that you're |
| 2058 | generating the right IR. |
| 2059 | |
| 2060 | #. Teach template instantiation how to cope with your AST node, which requires |
| 2061 | some fairly simple code: |
| 2062 | |
| 2063 | * Make sure that your expression's constructor properly computes the flags |
| 2064 | for type dependence (i.e., the type your expression produces can change |
| 2065 | from one instantiation to the next), value dependence (i.e., the constant |
| 2066 | value your expression produces can change from one instantiation to the |
| 2067 | next), instantiation dependence (i.e., a template parameter occurs |
| 2068 | anywhere in your expression), and whether your expression contains a |
| 2069 | parameter pack (for variadic templates). Often, computing these flags |
| 2070 | just means combining the results from the various types and |
| 2071 | subexpressions. |
| 2072 | * Add ``TransformXXX`` and ``RebuildXXX`` functions to the ``TreeTransform`` |
| 2073 | class template in ``Sema``. ``TransformXXX`` should (recursively) |
| 2074 | transform all of the subexpressions and types within your expression, |
| 2075 | using ``getDerived().TransformYYY``. If all of the subexpressions and |
| 2076 | types transform without error, it will then call the ``RebuildXXX`` |
| 2077 | function, which will in turn call ``getSema().BuildXXX`` to perform |
| 2078 | semantic analysis and build your expression. |
| 2079 | * To test template instantiation, take those tests you wrote to make sure |
| 2080 | that you were type checking with type-dependent expressions and dependent |
| 2081 | types (from step #2) and instantiate those templates with various types, |
| 2082 | some of which type-check and some that don't, and test the error messages |
| 2083 | in each case. |
| 2084 | |
| 2085 | #. There are some "extras" that make other features work better. It's worth |
| 2086 | handling these extras to give your expression complete integration into |
| 2087 | Clang: |
| 2088 | |
| 2089 | * Add code completion support for your expression in |
| 2090 | ``SemaCodeComplete.cpp``. |
| 2091 | * If your expression has types in it, or has any "interesting" features |
| 2092 | other than subexpressions, extend libclang's ``CursorVisitor`` to provide |
| 2093 | proper visitation for your expression, enabling various IDE features such |
| 2094 | as syntax highlighting, cross-referencing, and so on. The |
| 2095 | ``c-index-test`` helper program can be used to test these features. |
| 2096 | |