Daniel Dunbar | be7ada7 | 2009-09-08 05:31:18 +0000 | [diff] [blame] | 1 | =pod |
| 2 | |
| 3 | =head1 NAME |
| 4 | |
| 5 | lit - LLVM Integrated Tester |
| 6 | |
| 7 | =head1 SYNOPSIS |
| 8 | |
| 9 | B<lit> [I<options>] [I<tests>] |
| 10 | |
| 11 | =head1 DESCRIPTION |
| 12 | |
| 13 | B<lit> is a portable tool for executing LLVM and Clang style test suites, |
| 14 | summarizing their results, and providing indication of failures. B<lit> is |
| 15 | designed to be a lightweight testing tool with as simple a user interface as |
| 16 | possible. |
| 17 | |
| 18 | B<lit> should be run with one or more I<tests> to run specified on the command |
| 19 | line. Tests can be either individual test files or directories to search for |
| 20 | tests (see L<"TEST DISCOVERY">). |
| 21 | |
| 22 | Each specified test will be executed (potentially in parallel) and once all |
| 23 | tests have been run B<lit> will print summary information on the number of tests |
| 24 | which passed or failed (see L<"TEST STATUS RESULTS">). The B<lit> program will |
| 25 | execute with a non-zero exit code if any tests fail. |
| 26 | |
| 27 | By default B<lit> will use a succinct progress display and will only print |
| 28 | summary information for test failures. See L<"OUTPUT OPTIONS"> for options |
| 29 | controlling the B<lit> progress display and output. |
| 30 | |
Eli Bendersky | 3d6b64a | 2012-02-05 11:17:49 +0000 | [diff] [blame] | 31 | B<lit> also includes a number of options for controlling how tests are executed |
Daniel Dunbar | be7ada7 | 2009-09-08 05:31:18 +0000 | [diff] [blame] | 32 | (specific features may depend on the particular test format). See L<"EXECUTION |
| 33 | OPTIONS"> for more information. |
| 34 | |
| 35 | Finally, B<lit> also supports additional options for only running a subset of |
| 36 | the options specified on the command line, see L<"SELECTION OPTIONS"> for |
| 37 | more information. |
| 38 | |
Daniel Dunbar | 6f479e5 | 2009-11-10 02:41:27 +0000 | [diff] [blame] | 39 | Users interested in the B<lit> architecture or designing a B<lit> testing |
Eli Bendersky | 0ff100d | 2012-01-07 04:11:27 +0000 | [diff] [blame] | 40 | implementation should see L<"LIT INFRASTRUCTURE"> |
Daniel Dunbar | 6f479e5 | 2009-11-10 02:41:27 +0000 | [diff] [blame] | 41 | |
Daniel Dunbar | be7ada7 | 2009-09-08 05:31:18 +0000 | [diff] [blame] | 42 | =head1 GENERAL OPTIONS |
| 43 | |
| 44 | =over |
| 45 | |
| 46 | =item B<-h>, B<--help> |
| 47 | |
| 48 | Show the B<lit> help message. |
| 49 | |
| 50 | =item B<-j> I<N>, B<--threads>=I<N> |
| 51 | |
Duncan Sands | 7c2da96 | 2010-02-08 11:03:31 +0000 | [diff] [blame] | 52 | Run I<N> tests in parallel. By default, this is automatically chosen to match |
| 53 | the number of detected available CPUs. |
Daniel Dunbar | be7ada7 | 2009-09-08 05:31:18 +0000 | [diff] [blame] | 54 | |
Daniel Dunbar | edb8997 | 2009-10-25 03:30:55 +0000 | [diff] [blame] | 55 | =item B<--config-prefix>=I<NAME> |
| 56 | |
| 57 | Search for I<NAME.cfg> and I<NAME.site.cfg> when searching for test suites, |
Duncan Sands | 7c2da96 | 2010-02-08 11:03:31 +0000 | [diff] [blame] | 58 | instead of I<lit.cfg> and I<lit.site.cfg>. |
Daniel Dunbar | edb8997 | 2009-10-25 03:30:55 +0000 | [diff] [blame] | 59 | |
Daniel Dunbar | f626167 | 2009-11-05 16:27:33 +0000 | [diff] [blame] | 60 | =item B<--param> I<NAME>, B<--param> I<NAME>=I<VALUE> |
| 61 | |
| 62 | Add a user defined parameter I<NAME> with the given I<VALUE> (or the empty |
| 63 | string if not given). The meaning and use of these parameters is test suite |
| 64 | dependent. |
| 65 | |
Daniel Dunbar | be7ada7 | 2009-09-08 05:31:18 +0000 | [diff] [blame] | 66 | =back |
| 67 | |
| 68 | =head1 OUTPUT OPTIONS |
| 69 | |
| 70 | =over |
| 71 | |
| 72 | =item B<-q>, B<--quiet> |
| 73 | |
| 74 | Suppress any output except for test failures. |
| 75 | |
| 76 | =item B<-s>, B<--succinct> |
| 77 | |
| 78 | Show less output, for example don't show information on tests that pass. |
| 79 | |
| 80 | =item B<-v>, B<--verbose> |
| 81 | |
| 82 | Show more information on test failures, for example the entire test output |
| 83 | instead of just the test result. |
| 84 | |
| 85 | =item B<--no-progress-bar> |
| 86 | |
| 87 | Do not use curses based progress bar. |
| 88 | |
| 89 | =back |
| 90 | |
| 91 | =head1 EXECUTION OPTIONS |
| 92 | |
| 93 | =over |
| 94 | |
| 95 | =item B<--path>=I<PATH> |
| 96 | |
| 97 | Specify an addition I<PATH> to use when searching for executables in tests. |
| 98 | |
| 99 | =item B<--vg> |
| 100 | |
| 101 | Run individual tests under valgrind (using the memcheck tool). The |
| 102 | I<--error-exitcode> argument for valgrind is used so that valgrind failures will |
| 103 | cause the program to exit with a non-zero status. |
| 104 | |
| 105 | =item B<--vg-arg>=I<ARG> |
| 106 | |
| 107 | When I<--vg> is used, specify an additional argument to pass to valgrind itself. |
| 108 | |
| 109 | =item B<--time-tests> |
| 110 | |
| 111 | Track the wall time individual tests take to execute and includes the results in |
| 112 | the summary output. This is useful for determining which tests in a test suite |
| 113 | take the most time to execute. Note that this option is most useful with I<-j |
| 114 | 1>. |
| 115 | |
| 116 | =back |
| 117 | |
| 118 | =head1 SELECTION OPTIONS |
| 119 | |
| 120 | =over |
| 121 | |
| 122 | =item B<--max-tests>=I<N> |
| 123 | |
| 124 | Run at most I<N> tests and then terminate. |
| 125 | |
| 126 | =item B<--max-time>=I<N> |
| 127 | |
| 128 | Spend at most I<N> seconds (approximately) running tests and then terminate. |
| 129 | |
| 130 | =item B<--shuffle> |
| 131 | |
| 132 | Run the tests in a random order. |
| 133 | |
| 134 | =back |
| 135 | |
| 136 | =head1 ADDITIONAL OPTIONS |
| 137 | |
| 138 | =over |
| 139 | |
| 140 | =item B<--debug> |
| 141 | |
| 142 | Run B<lit> in debug mode, for debugging configuration issues and B<lit> itself. |
| 143 | |
| 144 | =item B<--show-suites> |
| 145 | |
| 146 | List the discovered test suites as part of the standard output. |
| 147 | |
| 148 | =item B<--no-tcl-as-sh> |
| 149 | |
| 150 | Run Tcl scripts internally (instead of converting to shell scripts). |
| 151 | |
Daniel Dunbar | 40c67b5 | 2009-11-15 01:02:09 +0000 | [diff] [blame] | 152 | =item B<--repeat>=I<N> |
| 153 | |
| 154 | Run each test I<N> times. Currently this is primarily useful for timing tests, |
| 155 | other results are not collated in any reasonable fashion. |
| 156 | |
Daniel Dunbar | be7ada7 | 2009-09-08 05:31:18 +0000 | [diff] [blame] | 157 | =back |
| 158 | |
| 159 | =head1 EXIT STATUS |
| 160 | |
| 161 | B<lit> will exit with an exit code of 1 if there are any FAIL or XPASS |
Eli Bendersky | e28e2a8 | 2012-03-14 11:34:19 +0000 | [diff] [blame] | 162 | results. Otherwise, it will exit with the status 0. Other exit codes are used |
| 163 | for non-test related failures (for example a user error or an internal program |
Daniel Dunbar | be7ada7 | 2009-09-08 05:31:18 +0000 | [diff] [blame] | 164 | error). |
| 165 | |
| 166 | =head1 TEST DISCOVERY |
| 167 | |
| 168 | The inputs passed to B<lit> can be either individual tests, or entire |
| 169 | directories or hierarchies of tests to run. When B<lit> starts up, the first |
| 170 | thing it does is convert the inputs into a complete list of tests to run as part |
| 171 | of I<test discovery>. |
| 172 | |
| 173 | In the B<lit> model, every test must exist inside some I<test suite>. B<lit> |
| 174 | resolves the inputs specified on the command line to test suites by searching |
| 175 | upwards from the input path until it finds a I<lit.cfg> or I<lit.site.cfg> |
| 176 | file. These files serve as both a marker of test suites and as configuration |
| 177 | files which B<lit> loads in order to understand how to find and run the tests |
| 178 | inside the test suite. |
| 179 | |
| 180 | Once B<lit> has mapped the inputs into test suites it traverses the list of |
| 181 | inputs adding tests for individual files and recursively searching for tests in |
| 182 | directories. |
| 183 | |
| 184 | This behavior makes it easy to specify a subset of tests to run, while still |
| 185 | allowing the test suite configuration to control exactly how tests are |
| 186 | interpreted. In addition, B<lit> always identifies tests by the test suite they |
| 187 | are in, and their relative path inside the test suite. For appropriately |
| 188 | configured projects, this allows B<lit> to provide convenient and flexible |
| 189 | support for out-of-tree builds. |
| 190 | |
| 191 | =head1 TEST STATUS RESULTS |
| 192 | |
| 193 | Each test ultimately produces one of the following six results: |
| 194 | |
| 195 | =over |
| 196 | |
| 197 | =item B<PASS> |
| 198 | |
| 199 | The test succeeded. |
| 200 | |
| 201 | =item B<XFAIL> |
| 202 | |
| 203 | The test failed, but that is expected. This is used for test formats which allow |
| 204 | specifying that a test does not currently work, but wish to leave it in the test |
| 205 | suite. |
| 206 | |
| 207 | =item B<XPASS> |
| 208 | |
| 209 | The test succeeded, but it was expected to fail. This is used for tests which |
| 210 | were specified as expected to fail, but are now succeeding (generally because |
Duncan Sands | cf6bd5a | 2012-01-02 16:55:01 +0000 | [diff] [blame] | 211 | the feature they test was broken and has been fixed). |
Daniel Dunbar | be7ada7 | 2009-09-08 05:31:18 +0000 | [diff] [blame] | 212 | |
| 213 | =item B<FAIL> |
| 214 | |
| 215 | The test failed. |
| 216 | |
| 217 | =item B<UNRESOLVED> |
| 218 | |
| 219 | The test result could not be determined. For example, this occurs when the test |
| 220 | could not be run, the test itself is invalid, or the test was interrupted. |
| 221 | |
| 222 | =item B<UNSUPPORTED> |
| 223 | |
| 224 | The test is not supported in this environment. This is used by test formats |
| 225 | which can report unsupported tests. |
| 226 | |
| 227 | =back |
| 228 | |
| 229 | Depending on the test format tests may produce additional information about |
Eli Bendersky | 3d6b64a | 2012-02-05 11:17:49 +0000 | [diff] [blame] | 230 | their status (generally only for failures). See the L<Output|"OUTPUT OPTIONS"> |
Daniel Dunbar | be7ada7 | 2009-09-08 05:31:18 +0000 | [diff] [blame] | 231 | section for more information. |
| 232 | |
Daniel Dunbar | 6f479e5 | 2009-11-10 02:41:27 +0000 | [diff] [blame] | 233 | =head1 LIT INFRASTRUCTURE |
| 234 | |
| 235 | This section describes the B<lit> testing architecture for users interested in |
| 236 | creating a new B<lit> testing implementation, or extending an existing one. |
| 237 | |
| 238 | B<lit> proper is primarily an infrastructure for discovering and running |
| 239 | arbitrary tests, and to expose a single convenient interface to these |
Duncan Sands | 7c2da96 | 2010-02-08 11:03:31 +0000 | [diff] [blame] | 240 | tests. B<lit> itself doesn't know how to run tests, rather this logic is |
Daniel Dunbar | 6f479e5 | 2009-11-10 02:41:27 +0000 | [diff] [blame] | 241 | defined by I<test suites>. |
| 242 | |
| 243 | =head2 TEST SUITES |
| 244 | |
| 245 | As described in L<"TEST DISCOVERY">, tests are always located inside a I<test |
| 246 | suite>. Test suites serve to define the format of the tests they contain, the |
| 247 | logic for finding those tests, and any additional information to run the tests. |
| 248 | |
| 249 | B<lit> identifies test suites as directories containing I<lit.cfg> or |
Eli Bendersky | 371d7b0 | 2012-02-05 09:21:25 +0000 | [diff] [blame] | 250 | I<lit.site.cfg> files (see also B<--config-prefix>). Test suites are initially |
Daniel Dunbar | 6f479e5 | 2009-11-10 02:41:27 +0000 | [diff] [blame] | 251 | discovered by recursively searching up the directory hierarchy for all the input |
| 252 | files passed on the command line. You can use B<--show-suites> to display the |
| 253 | discovered test suites at startup. |
| 254 | |
| 255 | Once a test suite is discovered, its config file is loaded. Config files |
Daniel Dunbar | c9d0c4e | 2010-05-11 17:22:45 +0000 | [diff] [blame] | 256 | themselves are Python modules which will be executed. When the config file is |
| 257 | executed, two important global variables are predefined: |
Daniel Dunbar | 6f479e5 | 2009-11-10 02:41:27 +0000 | [diff] [blame] | 258 | |
| 259 | =over |
| 260 | |
| 261 | =item B<lit> |
| 262 | |
| 263 | The global B<lit> configuration object (a I<LitConfig> instance), which defines |
| 264 | the builtin test formats, global configuration parameters, and other helper |
| 265 | routines for implementing test configurations. |
| 266 | |
| 267 | =item B<config> |
| 268 | |
| 269 | This is the config object (a I<TestingConfig> instance) for the test suite, |
| 270 | which the config file is expected to populate. The following variables are also |
| 271 | available on the I<config> object, some of which must be set by the config and |
| 272 | others are optional or predefined: |
| 273 | |
| 274 | B<name> I<[required]> The name of the test suite, for use in reports and |
| 275 | diagnostics. |
| 276 | |
| 277 | B<test_format> I<[required]> The test format object which will be used to |
| 278 | discover and run tests in the test suite. Generally this will be a builtin test |
| 279 | format available from the I<lit.formats> module. |
| 280 | |
| 281 | B<test_src_root> The filesystem path to the test suite root. For out-of-dir |
| 282 | builds this is the directory that will be scanned for tests. |
| 283 | |
| 284 | B<test_exec_root> For out-of-dir builds, the path to the test suite root inside |
| 285 | the object directory. This is where tests will be run and temporary output files |
Eli Bendersky | bf222b9 | 2012-03-14 06:09:20 +0000 | [diff] [blame] | 286 | placed. |
Daniel Dunbar | 6f479e5 | 2009-11-10 02:41:27 +0000 | [diff] [blame] | 287 | |
| 288 | B<environment> A dictionary representing the environment to use when executing |
| 289 | tests in the suite. |
| 290 | |
| 291 | B<suffixes> For B<lit> test formats which scan directories for tests, this |
Eli Bendersky | e28e2a8 | 2012-03-14 11:34:19 +0000 | [diff] [blame] | 292 | variable is a list of suffixes to identify test files. Used by: I<ShTest>, |
Daniel Dunbar | 6f479e5 | 2009-11-10 02:41:27 +0000 | [diff] [blame] | 293 | I<TclTest>. |
| 294 | |
| 295 | B<substitutions> For B<lit> test formats which substitute variables into a test |
| 296 | script, the list of substitutions to perform. Used by: I<ShTest>, I<TclTest>. |
| 297 | |
| 298 | B<unsupported> Mark an unsupported directory, all tests within it will be |
| 299 | reported as unsupported. Used by: I<ShTest>, I<TclTest>. |
| 300 | |
| 301 | B<parent> The parent configuration, this is the config object for the directory |
| 302 | containing the test suite, or None. |
| 303 | |
Eli Bendersky | cc85160 | 2012-03-25 09:02:19 +0000 | [diff] [blame] | 304 | B<root> The root configuration. This is the top-most B<lit> configuration in |
| 305 | the project. |
| 306 | |
Daniel Dunbar | 6f479e5 | 2009-11-10 02:41:27 +0000 | [diff] [blame] | 307 | B<on_clone> The config is actually cloned for every subdirectory inside a test |
| 308 | suite, to allow local configuration on a per-directory basis. The I<on_clone> |
| 309 | variable can be set to a Python function which will be called whenever a |
| 310 | configuration is cloned (for a subdirectory). The function should takes three |
| 311 | arguments: (1) the parent configuration, (2) the new configuration (which the |
| 312 | I<on_clone> function will generally modify), and (3) the test path to the new |
| 313 | directory being scanned. |
| 314 | |
| 315 | =back |
| 316 | |
| 317 | =head2 TEST DISCOVERY |
| 318 | |
| 319 | Once test suites are located, B<lit> recursively traverses the source directory |
| 320 | (following I<test_src_root>) looking for tests. When B<lit> enters a |
Eli Bendersky | e28e2a8 | 2012-03-14 11:34:19 +0000 | [diff] [blame] | 321 | sub-directory, it first checks to see if a nested test suite is defined in that |
Daniel Dunbar | 6f479e5 | 2009-11-10 02:41:27 +0000 | [diff] [blame] | 322 | directory. If so, it loads that test suite recursively, otherwise it |
| 323 | instantiates a local test config for the directory (see L<"LOCAL CONFIGURATION |
| 324 | FILES">). |
| 325 | |
| 326 | Tests are identified by the test suite they are contained within, and the |
| 327 | relative path inside that suite. Note that the relative path may not refer to an |
| 328 | actual file on disk; some test formats (such as I<GoogleTest>) define "virtual |
| 329 | tests" which have a path that contains both the path to the actual test file and |
| 330 | a subpath to identify the virtual test. |
| 331 | |
| 332 | =head2 LOCAL CONFIGURATION FILES |
| 333 | |
| 334 | When B<lit> loads a subdirectory in a test suite, it instantiates a local test |
| 335 | configuration by cloning the configuration for the parent direction -- the root |
| 336 | of this configuration chain will always be a test suite. Once the test |
| 337 | configuration is cloned B<lit> checks for a I<lit.local.cfg> file in the |
| 338 | subdirectory. If present, this file will be loaded and can be used to specialize |
| 339 | the configuration for each individual directory. This facility can be used to |
| 340 | define subdirectories of optional tests, or to change other configuration |
| 341 | parameters -- for example, to change the test format, or the suffixes which |
| 342 | identify test files. |
| 343 | |
Daniel Dunbar | f642821 | 2012-03-26 18:01:14 +0000 | [diff] [blame] | 344 | =head2 TEST RUN OUTPUT FORMAT |
| 345 | |
| 346 | The b<lit> output for a test run conforms to the following schema, in both short |
| 347 | and verbose modes (although in short mode no PASS lines will be shown). This |
| 348 | schema has been chosen to be relatively easy to reliably parse by a machine (for |
| 349 | example in buildbot log scraping), and for other tools to generate. |
| 350 | |
| 351 | Each test result is expected to appear on a line that matches: |
| 352 | |
| 353 | <result code>: <test name> (<progress info>) |
| 354 | |
| 355 | where <result-code> is a standard test result such as PASS, FAIL, XFAIL, XPASS, |
| 356 | UNRESOLVED, or UNSUPPORTED. The performance result codes of IMPROVED and |
| 357 | REGRESSED are also allowed. |
| 358 | |
| 359 | The <test name> field can consist of an arbitrary string containing no newline. |
| 360 | |
| 361 | The <progress info> field can be used to report progress information such as |
| 362 | (1/300) or can be empty, but even when empty the parentheses are required. |
| 363 | |
| 364 | Each test result may include additional (multiline) log information in the |
| 365 | following format. |
| 366 | |
| 367 | <log delineator> TEST '(<test name>)' <trailing delineator> |
| 368 | ... log message ... |
| 369 | <log delineator> |
| 370 | |
| 371 | where <test name> should be the name of a preceeding reported test, <log |
| 372 | delineator> is a string of '*' characters I<at least> four characters long (the |
| 373 | recommended length is 20), and <trailing delineator> is an arbitrary (unparsed) |
| 374 | string. |
| 375 | |
| 376 | The following is an example of a test run output which consists of four tests A, |
| 377 | B, C, and D, and a log message for the failing test C. |
| 378 | |
| 379 | =head3 Example Test Run Output Listing |
| 380 | |
| 381 | PASS: A (1 of 4) |
| 382 | PASS: B (2 of 4) |
| 383 | FAIL: C (3 of 4) |
| 384 | ******************** TEST 'C' FAILED ******************** |
| 385 | Test 'C' failed as a result of exit code 1. |
| 386 | ******************** |
| 387 | PASS: D (4 of 4) |
| 388 | |
| 389 | =back |
| 390 | |
Daniel Dunbar | 6f479e5 | 2009-11-10 02:41:27 +0000 | [diff] [blame] | 391 | =head2 LIT EXAMPLE TESTS |
| 392 | |
| 393 | The B<lit> distribution contains several example implementations of test suites |
| 394 | in the I<ExampleTests> directory. |
| 395 | |
Daniel Dunbar | be7ada7 | 2009-09-08 05:31:18 +0000 | [diff] [blame] | 396 | =head1 SEE ALSO |
| 397 | |
| 398 | L<valgrind(1)> |
| 399 | |
| 400 | =head1 AUTHOR |
| 401 | |
NAKAMURA Takumi | b9a3363 | 2011-04-09 02:13:37 +0000 | [diff] [blame] | 402 | Written by Daniel Dunbar and maintained by the LLVM Team (L<http://llvm.org/>). |
Daniel Dunbar | be7ada7 | 2009-09-08 05:31:18 +0000 | [diff] [blame] | 403 | |
| 404 | =cut |