blob: 6f24c9d297308ebaefbbce5190b2b62e4dfabe53 [file] [log] [blame]
Sean Silvaa89edf62012-11-14 21:09:30 +00001=================================
2LLVM Testing Infrastructure Guide
3=================================
4
Sean Silvaa89edf62012-11-14 21:09:30 +00005.. contents::
6 :local:
7
Sean Silvae0db5192012-11-14 23:11:10 +00008.. toctree::
9 :hidden:
10
Matthias Braun4f340e92018-08-31 21:47:01 +000011 TestSuiteGuide
Sean Silvae0db5192012-11-14 23:11:10 +000012 TestSuiteMakefileGuide
13
Sean Silvaa89edf62012-11-14 21:09:30 +000014Overview
15========
16
17This document is the reference manual for the LLVM testing
18infrastructure. It documents the structure of the LLVM testing
19infrastructure, the tools needed to use it, and how to add and run
20tests.
21
22Requirements
23============
24
Bill Wendling27f96da2013-10-27 04:02:21 +000025In order to use the LLVM testing infrastructure, you will need all of the
Rafael Espindola21a400852014-12-12 15:29:31 +000026software required to build LLVM, as well as `Python <http://python.org>`_ 2.7 or
Bill Wendling27f96da2013-10-27 04:02:21 +000027later.
Sean Silvaa89edf62012-11-14 21:09:30 +000028
Matthias Braun4f340e92018-08-31 21:47:01 +000029LLVM Testing Infrastructure Organization
Sean Silvaa89edf62012-11-14 21:09:30 +000030========================================
31
Michael Platings7e552762019-01-24 15:11:26 +000032The LLVM testing infrastructure contains three major categories of tests:
33unit tests, regression tests and whole programs. The unit tests and regression
34tests are contained inside the LLVM repository itself under ``llvm/unittests``
35and ``llvm/test`` respectively and are expected to always pass -- they should be
36run before every commit.
Sean Silvaa89edf62012-11-14 21:09:30 +000037
38The whole programs tests are referred to as the "LLVM test suite" (or
39"test-suite") and are in the ``test-suite`` module in subversion. For
40historical reasons, these tests are also referred to as the "nightly
41tests" in places, which is less ambiguous than "test-suite" and remains
42in use although we run them much more often than nightly.
43
Michael Platings7e552762019-01-24 15:11:26 +000044Unit tests
45----------
46
47Unit tests are written using `Google Test <https://github.com/google/googletest/blob/master/googletest/docs/primer.md>`_
48and `Google Mock <https://github.com/google/googletest/blob/master/googlemock/docs/ForDummies.md>`_
49and are located in the ``llvm/unittests`` directory.
50
Sean Silvaa89edf62012-11-14 21:09:30 +000051Regression tests
52----------------
53
54The regression tests are small pieces of code that test a specific
Eli Bendersky03551382012-12-03 04:10:58 +000055feature of LLVM or trigger a specific bug in LLVM. The language they are
56written in depends on the part of LLVM being tested. These tests are driven by
57the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and
58are located in the ``llvm/test`` directory.
Sean Silvaa89edf62012-11-14 21:09:30 +000059
60Typically when a bug is found in LLVM, a regression test containing just
61enough code to reproduce the problem should be written and placed
Eli Bendersky03551382012-12-03 04:10:58 +000062somewhere underneath this directory. For example, it can be a small
63piece of LLVM IR distilled from an actual application or benchmark.
Sean Silvaa89edf62012-11-14 21:09:30 +000064
65``test-suite``
66--------------
67
68The test suite contains whole programs, which are pieces of code which
69can be compiled and linked into a stand-alone program that can be
70executed. These programs are generally written in high level languages
71such as C or C++.
72
73These programs are compiled using a user specified compiler and set of
74flags, and then executed to capture the program output and timing
75information. The output of these programs is compared to a reference
76output to ensure that the program is being compiled correctly.
77
78In addition to compiling and executing programs, whole program tests
79serve as a way of benchmarking LLVM performance, both in terms of the
80efficiency of the programs generated as well as the speed with which
81LLVM compiles, optimizes, and generates code.
82
83The test-suite is located in the ``test-suite`` Subversion module.
84
Matthias Braun4f340e92018-08-31 21:47:01 +000085See the :doc:`TestSuiteGuide` for details.
86
Sean Silvaa89edf62012-11-14 21:09:30 +000087Debugging Information tests
88---------------------------
89
90The test suite contains tests to check quality of debugging information.
91The test are written in C based languages or in LLVM assembly language.
92
93These tests are compiled and run under a debugger. The debugger output
94is checked to validate of debugging information. See README.txt in the
95test suite for more information . This test suite is located in the
96``debuginfo-tests`` Subversion module.
97
98Quick start
99===========
100
Michael Platings7e552762019-01-24 15:11:26 +0000101The tests are located in two separate Subversion modules. The unit and
102regression tests are in the main "llvm" module under the directories
103``llvm/unittests`` and ``llvm/test`` (so you get these tests for free with the
104main LLVM tree). Use ``make check-all`` to run the unit and regression tests
105after building LLVM.
Sean Silvaa89edf62012-11-14 21:09:30 +0000106
Matthias Braun4f340e92018-08-31 21:47:01 +0000107The ``test-suite`` module contains more comprehensive tests including whole C
108and C++ programs. See the :doc:`TestSuiteGuide` for details.
Sean Silvaa89edf62012-11-14 21:09:30 +0000109
Michael Platings7e552762019-01-24 15:11:26 +0000110Unit and Regression tests
111-------------------------
112
113To run all of the LLVM unit tests use the check-llvm-unit target:
114
115.. code-block:: bash
116
117 % make check-llvm-unit
Sean Silvaa89edf62012-11-14 21:09:30 +0000118
Chris Bienemanbcc6f192016-01-26 22:53:12 +0000119To run all of the LLVM regression tests use the check-llvm target:
Sean Silvaa89edf62012-11-14 21:09:30 +0000120
121.. code-block:: bash
122
Chris Bienemanbcc6f192016-01-26 22:53:12 +0000123 % make check-llvm
Sean Silvaa89edf62012-11-14 21:09:30 +0000124
Davide Italiano078fb932019-01-22 21:52:50 +0000125In order to get reasonable testing performance, build LLVM and subprojects
126in release mode, i.e.
127
128.. code-block:: bash
129
130 % cmake -DCMAKE_BUILD_TYPE="Release"
131
Sean Silvaa89edf62012-11-14 21:09:30 +0000132If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you
133can run the LLVM and Clang tests simultaneously using:
134
Sean Silvaa89edf62012-11-14 21:09:30 +0000135.. code-block:: bash
136
Eli Bendersky03551382012-12-03 04:10:58 +0000137 % make check-all
Sean Silvaa89edf62012-11-14 21:09:30 +0000138
Daniel Dunbar04388af2013-08-09 19:39:48 +0000139To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make
140variable to pass the required options to lit. For example, you can use:
Sean Silvaa89edf62012-11-14 21:09:30 +0000141
142.. code-block:: bash
143
Daniel Dunbar04388af2013-08-09 19:39:48 +0000144 % make check LIT_ARGS="-v --vg --vg-leak"
145
146to enable testing with valgrind and with leak checking enabled.
Sean Silvaa89edf62012-11-14 21:09:30 +0000147
Eli Bendersky03551382012-12-03 04:10:58 +0000148To run individual tests or subsets of tests, you can use the ``llvm-lit``
Sean Silvaa89edf62012-11-14 21:09:30 +0000149script which is built as part of LLVM. For example, to run the
Eli Bendersky03551382012-12-03 04:10:58 +0000150``Integer/BitPacked.ll`` test by itself you can run:
Sean Silvaa89edf62012-11-14 21:09:30 +0000151
152.. code-block:: bash
153
154 % llvm-lit ~/llvm/test/Integer/BitPacked.ll
155
156or to run all of the ARM CodeGen tests:
157
158.. code-block:: bash
159
160 % llvm-lit ~/llvm/test/CodeGen/ARM
161
Eli Bendersky03551382012-12-03 04:10:58 +0000162For more information on using the :program:`lit` tool, see ``llvm-lit --help``
163or the :doc:`lit man page <CommandGuide/lit>`.
Sean Silvaa89edf62012-11-14 21:09:30 +0000164
165Debugging Information tests
166---------------------------
167
168To run debugging information tests simply checkout the tests inside
169clang/test directory.
170
171.. code-block:: bash
172
173 % cd clang/test
174 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
175
176These tests are already set up to run as part of clang regression tests.
177
178Regression test structure
179=========================
180
Eli Bendersky03551382012-12-03 04:10:58 +0000181The LLVM regression tests are driven by :program:`lit` and are located in the
Sean Silvaa89edf62012-11-14 21:09:30 +0000182``llvm/test`` directory.
183
184This directory contains a large array of small tests that exercise
185various features of LLVM and to ensure that regressions do not occur.
186The directory is broken into several sub-directories, each focused on a
Eli Bendersky42e10732012-12-04 13:55:17 +0000187particular area of LLVM.
Sean Silvaa89edf62012-11-14 21:09:30 +0000188
189Writing new regression tests
190----------------------------
191
192The regression test structure is very simple, but does require some
193information to be set. This information is gathered via ``configure``
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000194and is written to a file, ``test/lit.site.cfg`` in the build directory.
195The ``llvm/test`` Makefile does this work for you.
Sean Silvaa89edf62012-11-14 21:09:30 +0000196
197In order for the regression tests to work, each directory of tests must
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000198have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine
199how to run the tests. This file is just Python code and thus is very
Sean Silvaa89edf62012-11-14 21:09:30 +0000200flexible, but we've standardized it for the LLVM regression tests. If
201you're adding a directory of tests, just copy ``lit.local.cfg`` from
202another directory to get running. The standard ``lit.local.cfg`` simply
203specifies which files to look in for tests. Any directory that contains
Dmitri Gribenko42c31d22012-11-18 10:35:18 +0000204only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit
205documentation <CommandGuide/lit>` for more information.
Sean Silvaa89edf62012-11-14 21:09:30 +0000206
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000207Each test file must contain lines starting with "RUN:" that tell :program:`lit`
208how to run it. If there are no RUN lines, :program:`lit` will issue an error
209while running a test.
Sean Silvaa89edf62012-11-14 21:09:30 +0000210
211RUN lines are specified in the comments of the test program using the
212keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000213to execute. Together, these lines form the "script" that :program:`lit`
214executes to run the test case. The syntax of the RUN lines is similar to a
215shell's syntax for pipelines including I/O redirection and variable
216substitution. However, even though these lines may *look* like a shell
217script, they are not. RUN lines are interpreted by :program:`lit`.
218Consequently, the syntax differs from shell in a few ways. You can specify
219as many RUN lines as needed.
Sean Silvaa89edf62012-11-14 21:09:30 +0000220
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000221:program:`lit` performs substitution on each RUN line to replace LLVM tool names
Sean Silvaa89edf62012-11-14 21:09:30 +0000222with the full paths to the executable built for each tool (in
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000223``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does
224not invoke any stray LLVM tools in the user's path during testing.
Sean Silvaa89edf62012-11-14 21:09:30 +0000225
226Each RUN line is executed on its own, distinct from other lines unless
227its last character is ``\``. This continuation character causes the RUN
228line to be concatenated with the next one. In this way you can build up
229long pipelines of commands without making huge line lengths. The lines
230ending in ``\`` are concatenated until a RUN line that doesn't end in
231``\`` is found. This concatenated set of RUN lines then constitutes one
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000232execution. :program:`lit` will substitute variables and arrange for the pipeline
233to be executed. If any process in the pipeline fails, the entire line (and
Sean Silvaa89edf62012-11-14 21:09:30 +0000234test case) fails too.
235
236Below is an example of legal RUN lines in a ``.ll`` file:
237
238.. code-block:: llvm
239
240 ; RUN: llvm-as < %s | llvm-dis > %t1
241 ; RUN: llvm-dis < %s.bc-13 > %t2
242 ; RUN: diff %t1 %t2
243
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000244As with a Unix shell, the RUN lines permit pipelines and I/O
Sean Silva8eaf3ca2013-03-19 15:22:02 +0000245redirection to be used.
Sean Silvaa89edf62012-11-14 21:09:30 +0000246
247There are some quoting rules that you must pay attention to when writing
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000248your RUN lines. In general nothing needs to be quoted. :program:`lit` won't
249strip off any quote characters so they will get passed to the invoked program.
Eli Benderskyf747bd62013-01-18 19:01:34 +0000250To avoid this use curly braces to tell :program:`lit` that it should treat
251everything enclosed as one value.
Sean Silvaa89edf62012-11-14 21:09:30 +0000252
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000253In general, you should strive to keep your RUN lines as simple as possible,
Eli Benderskyf747bd62013-01-18 19:01:34 +0000254using them only to run tools that generate textual output you can then examine.
Eli Bendersky6f6cbdb2013-03-22 16:09:06 +0000255The recommended way to examine output to figure out if the test passes is using
Eli Benderskyf747bd62013-01-18 19:01:34 +0000256the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN
257lines is deprecated - please do not send or commit patches that use it.]*
Sean Silvaa89edf62012-11-14 21:09:30 +0000258
Davide Italiano4efa3952015-11-17 02:17:35 +0000259Put related tests into a single file rather than having a separate file per
260test. Check if there are files already covering your feature and consider
261adding your code there instead of creating a new file.
262
Sean Silva15ee4082014-11-05 22:17:18 +0000263Extra files
264-----------
265
266If your test requires extra files besides the file containing the ``RUN:``
267lines, the idiomatic place to put them is in a subdirectory ``Inputs``.
268You can then refer to the extra files as ``%S/Inputs/foo.bar``.
269
270For example, consider ``test/Linker/ident.ll``. The directory structure is
271as follows::
272
273 test/
274 Linker/
275 ident.ll
276 Inputs/
277 ident.a.ll
278 ident.b.ll
279
280For convenience, these are the contents:
281
282.. code-block:: llvm
283
284 ;;;;; ident.ll:
285
286 ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s
287
288 ; Verify that multiple input llvm.ident metadata are linked together.
289
290 ; CHECK-DAG: !llvm.ident = !{!0, !1, !2}
291 ; CHECK-DAG: "Compiler V1"
292 ; CHECK-DAG: "Compiler V2"
293 ; CHECK-DAG: "Compiler V3"
294
295 ;;;;; Inputs/ident.a.ll:
296
297 !llvm.ident = !{!0, !1}
298 !0 = metadata !{metadata !"Compiler V1"}
299 !1 = metadata !{metadata !"Compiler V2"}
300
301 ;;;;; Inputs/ident.b.ll:
302
303 !llvm.ident = !{!0}
304 !0 = metadata !{metadata !"Compiler V3"}
305
306For symmetry reasons, ``ident.ll`` is just a dummy file that doesn't
307actually participate in the test besides holding the ``RUN:`` lines.
308
309.. note::
310
311 Some existing tests use ``RUN: true`` in extra files instead of just
312 putting the extra files in an ``Inputs/`` directory. This pattern is
313 deprecated.
314
Dmitri Gribenko12be9282012-12-30 14:51:03 +0000315Fragile tests
316-------------
317
318It is easy to write a fragile test that would fail spuriously if the tool being
319tested outputs a full path to the input file. For example, :program:`opt` by
320default outputs a ``ModuleID``:
321
322.. code-block:: console
323
324 $ cat example.ll
325 define i32 @main() nounwind {
326 ret i32 0
327 }
328
329 $ opt -S /path/to/example.ll
330 ; ModuleID = '/path/to/example.ll'
331
332 define i32 @main() nounwind {
333 ret i32 0
334 }
335
Sylvestre Ledrue6ec4412017-01-14 11:37:01 +0000336``ModuleID`` can unexpectedly match against ``CHECK`` lines. For example:
Dmitri Gribenko12be9282012-12-30 14:51:03 +0000337
338.. code-block:: llvm
339
340 ; RUN: opt -S %s | FileCheck
341
342 define i32 @main() nounwind {
343 ; CHECK-NOT: load
344 ret i32 0
345 }
346
347This test will fail if placed into a ``download`` directory.
348
349To make your tests robust, always use ``opt ... < %s`` in the RUN line.
350:program:`opt` does not output a ``ModuleID`` when input comes from stdin.
351
Renato Golin98c60812013-07-03 20:56:33 +0000352Platform-Specific Tests
353-----------------------
354
355Whenever adding tests that require the knowledge of a specific platform,
356either related to code generated, specific output or back-end features,
357you must make sure to isolate the features, so that buildbots that
358run on different architectures (and don't even compile all back-ends),
359don't fail.
360
361The first problem is to check for target-specific output, for example sizes
362of structures, paths and architecture names, for example:
363
364* Tests containing Windows paths will fail on Linux and vice-versa.
365* Tests that check for ``x86_64`` somewhere in the text will fail anywhere else.
366* Tests where the debug information calculates the size of types and structures.
367
368Also, if the test rely on any behaviour that is coded in any back-end, it must
369go in its own directory. So, for instance, code generator tests for ARM go
370into ``test/CodeGen/ARM`` and so on. Those directories contain a special
371``lit`` configuration file that ensure all tests in that directory will
372only run if a specific back-end is compiled and available.
373
374For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is:
375
376.. code-block:: python
377
378 config.suffixes = ['.ll', '.c', '.cpp', '.test']
Alp Tokerd3d017c2014-06-09 22:42:55 +0000379 if not 'ARM' in config.root.targets:
Renato Golin98c60812013-07-03 20:56:33 +0000380 config.unsupported = True
381
382Other platform-specific tests are those that depend on a specific feature
383of a specific sub-architecture, for example only to Intel chips that support ``AVX2``.
384
385For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture
386variants:
387
388.. code-block:: llvm
389
390 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2
391 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1
392 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2
393
394And the checks are different:
395
396.. code-block:: llvm
397
398 ; SSE2: @test1
399 ; SSE2: psubusw LCPI0_0(%rip), %xmm0
400 ; AVX1: @test1
401 ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0
402 ; AVX2: @test1
403 ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0
404
405So, if you're testing for a behaviour that you know is platform-specific or
406depends on special features of sub-architectures, you must add the specific
407triple, test with the specific FileCheck and put it into the specific
408directory that will filter out all other architectures.
409
Piotr Padlewski7a298c12016-07-08 23:47:29 +0000410
Greg Parker17db7702017-01-25 02:26:03 +0000411Constraining test execution
412---------------------------
413
414Some tests can be run only in specific configurations, such as
415with debug builds or on particular platforms. Use ``REQUIRES``
416and ``UNSUPPORTED`` to control when the test is enabled.
417
418Some tests are expected to fail. For example, there may be a known bug
419that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
420An ``XFAIL`` test will be successful if its execution fails, and
421will be a failure if its execution succeeds.
Piotr Padlewski7a298c12016-07-08 23:47:29 +0000422
423.. code-block:: llvm
424
Greg Parker17db7702017-01-25 02:26:03 +0000425 ; This test will be only enabled in the build with asserts.
Piotr Padlewski7a298c12016-07-08 23:47:29 +0000426 ; REQUIRES: asserts
Greg Parker17db7702017-01-25 02:26:03 +0000427 ; This test is disabled on Linux.
428 ; UNSUPPORTED: -linux-
429 ; This test is expected to fail on PowerPC.
430 ; XFAIL: powerpc
Piotr Padlewski7a298c12016-07-08 23:47:29 +0000431
Greg Parker17db7702017-01-25 02:26:03 +0000432``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
433list of boolean expressions. The values in each expression may be:
Piotr Padlewski7a298c12016-07-08 23:47:29 +0000434
Greg Parker17db7702017-01-25 02:26:03 +0000435- Features added to ``config.available_features`` by
436 configuration files such as ``lit.cfg``.
437- Substrings of the target triple (``UNSUPPORTED`` and ``XFAIL`` only).
438
439| ``REQUIRES`` enables the test if all expressions are true.
440| ``UNSUPPORTED`` disables the test if any expression is true.
441| ``XFAIL`` expects the test to fail if any expression is true.
442
443As a special case, ``XFAIL: *`` is expected to fail everywhere.
444
445.. code-block:: llvm
446
447 ; This test is disabled on Windows,
448 ; and is disabled on Linux, except for Android Linux.
449 ; UNSUPPORTED: windows, linux && !android
450 ; This test is expected to fail on both PowerPC and ARM.
451 ; XFAIL: powerpc || arm
452
Renato Golin98c60812013-07-03 20:56:33 +0000453
Nico Rieckc4e7f302014-02-15 08:35:56 +0000454Substitutions
455-------------
Sean Silvaa89edf62012-11-14 21:09:30 +0000456
Nico Rieckc4e7f302014-02-15 08:35:56 +0000457Besides replacing LLVM tool names the following substitutions are performed in
458RUN lines:
Sean Silvaa89edf62012-11-14 21:09:30 +0000459
Nico Rieckc4e7f302014-02-15 08:35:56 +0000460``%%``
461 Replaced by a single ``%``. This allows escaping other substitutions.
Sean Silvaa89edf62012-11-14 21:09:30 +0000462
Nico Rieckc4e7f302014-02-15 08:35:56 +0000463``%s``
464 File path to the test case's source. This is suitable for passing on the
465 command line as the input to an LLVM tool.
Sean Silvaa89edf62012-11-14 21:09:30 +0000466
Nico Rieckc4e7f302014-02-15 08:35:56 +0000467 Example: ``/home/user/llvm/test/MC/ELF/foo_test.s``
Sean Silvaa89edf62012-11-14 21:09:30 +0000468
Nico Rieckc4e7f302014-02-15 08:35:56 +0000469``%S``
470 Directory path to the test case's source.
Sean Silvaa89edf62012-11-14 21:09:30 +0000471
Nico Rieckc4e7f302014-02-15 08:35:56 +0000472 Example: ``/home/user/llvm/test/MC/ELF``
Sean Silvaa89edf62012-11-14 21:09:30 +0000473
Nico Rieckc4e7f302014-02-15 08:35:56 +0000474``%t``
475 File path to a temporary file name that could be used for this test case.
Sean Silvaa89edf62012-11-14 21:09:30 +0000476 The file name won't conflict with other test cases. You can append to it
477 if you need multiple temporaries. This is useful as the destination of
478 some redirected output.
479
Nico Rieckc4e7f302014-02-15 08:35:56 +0000480 Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp``
Sean Silvaa89edf62012-11-14 21:09:30 +0000481
Nico Rieckc4e7f302014-02-15 08:35:56 +0000482``%T``
Kuba Mracek77920a42018-06-19 22:22:48 +0000483 Directory of ``%t``. Deprecated. Shouldn't be used, because it can be easily
484 misused and cause race conditions between tests.
485
486 Use ``rm -rf %t && mkdir %t`` instead if a temporary directory is necessary.
Sean Silvaa89edf62012-11-14 21:09:30 +0000487
Nico Rieckc4e7f302014-02-15 08:35:56 +0000488 Example: ``/home/user/llvm.build/test/MC/ELF/Output``
Sean Silvaa89edf62012-11-14 21:09:30 +0000489
Nico Rieckc4e7f302014-02-15 08:35:56 +0000490``%{pathsep}``
491
492 Expands to the path separator, i.e. ``:`` (or ``;`` on Windows).
493
David Bozier9126f542017-02-09 14:12:30 +0000494``%/s, %/S, %/t, %/T:``
495
496 Act like the corresponding substitution above but replace any ``\``
497 character with a ``/``. This is useful to normalize path separators.
498
499 Example: ``%s: C:\Desktop Files/foo_test.s.tmp``
500
501 Example: ``%/s: C:/Desktop Files/foo_test.s.tmp``
502
503``%:s, %:S, %:t, %:T:``
504
505 Act like the corresponding substitution above but remove colons at
506 the beginning of Windows paths. This is useful to allow concatenation
507 of absolute paths on Windows to produce a legal path.
508
509 Example: ``%s: C:\Desktop Files\foo_test.s.tmp``
510
511 Example: ``%:s: C\Desktop Files\foo_test.s.tmp``
512
Nico Rieckc4e7f302014-02-15 08:35:56 +0000513
514**LLVM-specific substitutions:**
515
516``%shlibext``
517 The suffix for the host platforms shared library files. This includes the
518 period as the first character.
519
520 Example: ``.so`` (Linux), ``.dylib`` (OS X), ``.dll`` (Windows)
521
522``%exeext``
523 The suffix for the host platforms executable files. This includes the
524 period as the first character.
525
526 Example: ``.exe`` (Windows), empty on Linux.
527
528``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
529 The number of the line where this substitution is used, with an optional
530 integer offset. This can be used in tests with multiple RUN lines, which
531 reference test file's line numbers.
532
533
534**Clang-specific substitutions:**
535
536``%clang``
537 Invokes the Clang driver.
538
539``%clang_cpp``
540 Invokes the Clang driver for C++.
541
542``%clang_cl``
543 Invokes the CL-compatible Clang driver.
544
545``%clangxx``
546 Invokes the G++-compatible Clang driver.
547
548``%clang_cc1``
549 Invokes the Clang frontend.
550
551``%itanium_abi_triple``, ``%ms_abi_triple``
552 These substitutions can be used to get the current target triple adjusted to
553 the desired ABI. For example, if the test suite is running with the
554 ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to
555 ``i686-pc-mingw32``. This allows a test to run with a specific ABI without
556 constraining it to a specific triple.
557
558To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``.
559
Sean Silvaa89edf62012-11-14 21:09:30 +0000560
Matthias Braun29f3f112015-05-04 21:37:00 +0000561Options
562-------
563
564The llvm lit configuration allows to customize some things with user options:
565
566``llc``, ``opt``, ...
567 Substitute the respective llvm tool name with a custom command line. This
568 allows to specify custom paths and default arguments for these tools.
569 Example:
570
571 % llvm-lit "-Dllc=llc -verify-machineinstrs"
572
573``run_long_tests``
574 Enable the execution of long running tests.
575
576``llvm_site_config``
577 Load the specified lit configuration instead of the default one.
578
579
Sean Silvaa89edf62012-11-14 21:09:30 +0000580Other Features
581--------------
582
Nico Rieckea623c62014-01-08 16:30:03 +0000583To make RUN line writing easier, there are several helper programs. These
584helpers are in the PATH when running tests, so you can just call them using
585their name. For example:
Sean Silvaa89edf62012-11-14 21:09:30 +0000586
Sean Silvaa89edf62012-11-14 21:09:30 +0000587``not``
Nico Rieckea623c62014-01-08 16:30:03 +0000588 This program runs its arguments and then inverts the result code from it.
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000589 Zero result codes become 1. Non-zero result codes become 0.
Sean Silvaa89edf62012-11-14 21:09:30 +0000590
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000591To make the output more useful, :program:`lit` will scan
Sean Silvaa89edf62012-11-14 21:09:30 +0000592the lines of the test case for ones that contain a pattern that matches
593``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
594that is related to the test case. The number after "PR" specifies the
595LLVM bugzilla number. When a PR number is specified, it will be used in
596the pass/fail reporting. This is useful to quickly get some context when
597a test fails.
598
599Finally, any line that contains "END." will cause the special
600interpretation of lines to terminate. This is generally done right after
601the last RUN: line. This has two side effects:
602
603(a) it prevents special interpretation of lines that are part of the test
604 program, not the instructions to the test case, and
605
606(b) it speeds things up for really big test cases by avoiding
607 interpretation of the remainder of the file.