blob: f9222372c2afd6c5bc9bd622d74f024d60a06643 [file] [log] [blame]
Sean Silvaa89edf62012-11-14 21:09:30 +00001=================================
2LLVM Testing Infrastructure Guide
3=================================
4
Sean Silvaa89edf62012-11-14 21:09:30 +00005.. contents::
6 :local:
7
Sean Silvae0db5192012-11-14 23:11:10 +00008.. toctree::
9 :hidden:
10
11 TestSuiteMakefileGuide
12
Sean Silvaa89edf62012-11-14 21:09:30 +000013Overview
14========
15
16This document is the reference manual for the LLVM testing
17infrastructure. It documents the structure of the LLVM testing
18infrastructure, the tools needed to use it, and how to add and run
19tests.
20
21Requirements
22============
23
Bill Wendling27f96da2013-10-27 04:02:21 +000024In order to use the LLVM testing infrastructure, you will need all of the
25software required to build LLVM, as well as `Python <http://python.org>`_ 2.5 or
26later.
Sean Silvaa89edf62012-11-14 21:09:30 +000027
28LLVM testing infrastructure organization
29========================================
30
31The LLVM testing infrastructure contains two major categories of tests:
32regression tests and whole programs. The regression tests are contained
33inside the LLVM repository itself under ``llvm/test`` and are expected
34to always pass -- they should be run before every commit.
35
36The whole programs tests are referred to as the "LLVM test suite" (or
37"test-suite") and are in the ``test-suite`` module in subversion. For
38historical reasons, these tests are also referred to as the "nightly
39tests" in places, which is less ambiguous than "test-suite" and remains
40in use although we run them much more often than nightly.
41
42Regression tests
43----------------
44
45The regression tests are small pieces of code that test a specific
Eli Bendersky03551382012-12-03 04:10:58 +000046feature of LLVM or trigger a specific bug in LLVM. The language they are
47written in depends on the part of LLVM being tested. These tests are driven by
48the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and
49are located in the ``llvm/test`` directory.
Sean Silvaa89edf62012-11-14 21:09:30 +000050
51Typically when a bug is found in LLVM, a regression test containing just
52enough code to reproduce the problem should be written and placed
Eli Bendersky03551382012-12-03 04:10:58 +000053somewhere underneath this directory. For example, it can be a small
54piece of LLVM IR distilled from an actual application or benchmark.
Sean Silvaa89edf62012-11-14 21:09:30 +000055
56``test-suite``
57--------------
58
59The test suite contains whole programs, which are pieces of code which
60can be compiled and linked into a stand-alone program that can be
61executed. These programs are generally written in high level languages
62such as C or C++.
63
64These programs are compiled using a user specified compiler and set of
65flags, and then executed to capture the program output and timing
66information. The output of these programs is compared to a reference
67output to ensure that the program is being compiled correctly.
68
69In addition to compiling and executing programs, whole program tests
70serve as a way of benchmarking LLVM performance, both in terms of the
71efficiency of the programs generated as well as the speed with which
72LLVM compiles, optimizes, and generates code.
73
74The test-suite is located in the ``test-suite`` Subversion module.
75
76Debugging Information tests
77---------------------------
78
79The test suite contains tests to check quality of debugging information.
80The test are written in C based languages or in LLVM assembly language.
81
82These tests are compiled and run under a debugger. The debugger output
83is checked to validate of debugging information. See README.txt in the
84test suite for more information . This test suite is located in the
85``debuginfo-tests`` Subversion module.
86
87Quick start
88===========
89
90The tests are located in two separate Subversion modules. The
91regressions tests are in the main "llvm" module under the directory
Eli Bendersky03551382012-12-03 04:10:58 +000092``llvm/test`` (so you get these tests for free with the main LLVM tree).
93Use ``make check-all`` to run the regression tests after building LLVM.
Sean Silvaa89edf62012-11-14 21:09:30 +000094
Sean Silvae0db5192012-11-14 23:11:10 +000095The more comprehensive test suite that includes whole programs in C and C++
96is in the ``test-suite`` module. See :ref:`test-suite Quickstart
97<test-suite-quickstart>` for more information on running these tests.
Sean Silvaa89edf62012-11-14 21:09:30 +000098
99Regression tests
100----------------
101
Eli Bendersky03551382012-12-03 04:10:58 +0000102To run all of the LLVM regression tests, use the master Makefile in the
103``llvm/test`` directory. LLVM Makefiles require GNU Make (read the :doc:`LLVM
104Makefile Guide <MakefileGuide>` for more details):
Sean Silvaa89edf62012-11-14 21:09:30 +0000105
106.. code-block:: bash
107
Eli Bendersky03551382012-12-03 04:10:58 +0000108 % make -C llvm/test
Sean Silvaa89edf62012-11-14 21:09:30 +0000109
Eli Bendersky03551382012-12-03 04:10:58 +0000110or:
Sean Silvaa89edf62012-11-14 21:09:30 +0000111
112.. code-block:: bash
113
Eli Bendersky03551382012-12-03 04:10:58 +0000114 % make check
Sean Silvaa89edf62012-11-14 21:09:30 +0000115
116If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you
117can run the LLVM and Clang tests simultaneously using:
118
Sean Silvaa89edf62012-11-14 21:09:30 +0000119.. code-block:: bash
120
Eli Bendersky03551382012-12-03 04:10:58 +0000121 % make check-all
Sean Silvaa89edf62012-11-14 21:09:30 +0000122
Daniel Dunbar04388af2013-08-09 19:39:48 +0000123To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make
124variable to pass the required options to lit. For example, you can use:
Sean Silvaa89edf62012-11-14 21:09:30 +0000125
126.. code-block:: bash
127
Daniel Dunbar04388af2013-08-09 19:39:48 +0000128 % make check LIT_ARGS="-v --vg --vg-leak"
129
130to enable testing with valgrind and with leak checking enabled.
Sean Silvaa89edf62012-11-14 21:09:30 +0000131
Eli Bendersky03551382012-12-03 04:10:58 +0000132To run individual tests or subsets of tests, you can use the ``llvm-lit``
Sean Silvaa89edf62012-11-14 21:09:30 +0000133script which is built as part of LLVM. For example, to run the
Eli Bendersky03551382012-12-03 04:10:58 +0000134``Integer/BitPacked.ll`` test by itself you can run:
Sean Silvaa89edf62012-11-14 21:09:30 +0000135
136.. code-block:: bash
137
138 % llvm-lit ~/llvm/test/Integer/BitPacked.ll
139
140or to run all of the ARM CodeGen tests:
141
142.. code-block:: bash
143
144 % llvm-lit ~/llvm/test/CodeGen/ARM
145
Eli Bendersky03551382012-12-03 04:10:58 +0000146For more information on using the :program:`lit` tool, see ``llvm-lit --help``
147or the :doc:`lit man page <CommandGuide/lit>`.
Sean Silvaa89edf62012-11-14 21:09:30 +0000148
149Debugging Information tests
150---------------------------
151
152To run debugging information tests simply checkout the tests inside
153clang/test directory.
154
155.. code-block:: bash
156
157 % cd clang/test
158 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
159
160These tests are already set up to run as part of clang regression tests.
161
162Regression test structure
163=========================
164
Eli Bendersky03551382012-12-03 04:10:58 +0000165The LLVM regression tests are driven by :program:`lit` and are located in the
Sean Silvaa89edf62012-11-14 21:09:30 +0000166``llvm/test`` directory.
167
168This directory contains a large array of small tests that exercise
169various features of LLVM and to ensure that regressions do not occur.
170The directory is broken into several sub-directories, each focused on a
Eli Bendersky42e10732012-12-04 13:55:17 +0000171particular area of LLVM.
Sean Silvaa89edf62012-11-14 21:09:30 +0000172
173Writing new regression tests
174----------------------------
175
176The regression test structure is very simple, but does require some
177information to be set. This information is gathered via ``configure``
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000178and is written to a file, ``test/lit.site.cfg`` in the build directory.
179The ``llvm/test`` Makefile does this work for you.
Sean Silvaa89edf62012-11-14 21:09:30 +0000180
181In order for the regression tests to work, each directory of tests must
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000182have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine
183how to run the tests. This file is just Python code and thus is very
Sean Silvaa89edf62012-11-14 21:09:30 +0000184flexible, but we've standardized it for the LLVM regression tests. If
185you're adding a directory of tests, just copy ``lit.local.cfg`` from
186another directory to get running. The standard ``lit.local.cfg`` simply
187specifies which files to look in for tests. Any directory that contains
Dmitri Gribenko42c31d22012-11-18 10:35:18 +0000188only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit
189documentation <CommandGuide/lit>` for more information.
Sean Silvaa89edf62012-11-14 21:09:30 +0000190
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000191Each test file must contain lines starting with "RUN:" that tell :program:`lit`
192how to run it. If there are no RUN lines, :program:`lit` will issue an error
193while running a test.
Sean Silvaa89edf62012-11-14 21:09:30 +0000194
195RUN lines are specified in the comments of the test program using the
196keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000197to execute. Together, these lines form the "script" that :program:`lit`
198executes to run the test case. The syntax of the RUN lines is similar to a
199shell's syntax for pipelines including I/O redirection and variable
200substitution. However, even though these lines may *look* like a shell
201script, they are not. RUN lines are interpreted by :program:`lit`.
202Consequently, the syntax differs from shell in a few ways. You can specify
203as many RUN lines as needed.
Sean Silvaa89edf62012-11-14 21:09:30 +0000204
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000205:program:`lit` performs substitution on each RUN line to replace LLVM tool names
Sean Silvaa89edf62012-11-14 21:09:30 +0000206with the full paths to the executable built for each tool (in
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000207``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does
208not invoke any stray LLVM tools in the user's path during testing.
Sean Silvaa89edf62012-11-14 21:09:30 +0000209
210Each RUN line is executed on its own, distinct from other lines unless
211its last character is ``\``. This continuation character causes the RUN
212line to be concatenated with the next one. In this way you can build up
213long pipelines of commands without making huge line lengths. The lines
214ending in ``\`` are concatenated until a RUN line that doesn't end in
215``\`` is found. This concatenated set of RUN lines then constitutes one
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000216execution. :program:`lit` will substitute variables and arrange for the pipeline
217to be executed. If any process in the pipeline fails, the entire line (and
Sean Silvaa89edf62012-11-14 21:09:30 +0000218test case) fails too.
219
220Below is an example of legal RUN lines in a ``.ll`` file:
221
222.. code-block:: llvm
223
224 ; RUN: llvm-as < %s | llvm-dis > %t1
225 ; RUN: llvm-dis < %s.bc-13 > %t2
226 ; RUN: diff %t1 %t2
227
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000228As with a Unix shell, the RUN lines permit pipelines and I/O
Sean Silva8eaf3ca2013-03-19 15:22:02 +0000229redirection to be used.
Sean Silvaa89edf62012-11-14 21:09:30 +0000230
231There are some quoting rules that you must pay attention to when writing
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000232your RUN lines. In general nothing needs to be quoted. :program:`lit` won't
233strip off any quote characters so they will get passed to the invoked program.
Eli Benderskyf747bd62013-01-18 19:01:34 +0000234To avoid this use curly braces to tell :program:`lit` that it should treat
235everything enclosed as one value.
Sean Silvaa89edf62012-11-14 21:09:30 +0000236
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000237In general, you should strive to keep your RUN lines as simple as possible,
Eli Benderskyf747bd62013-01-18 19:01:34 +0000238using them only to run tools that generate textual output you can then examine.
Eli Bendersky6f6cbdb2013-03-22 16:09:06 +0000239The recommended way to examine output to figure out if the test passes is using
Eli Benderskyf747bd62013-01-18 19:01:34 +0000240the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN
241lines is deprecated - please do not send or commit patches that use it.]*
Sean Silvaa89edf62012-11-14 21:09:30 +0000242
Dmitri Gribenko12be9282012-12-30 14:51:03 +0000243Fragile tests
244-------------
245
246It is easy to write a fragile test that would fail spuriously if the tool being
247tested outputs a full path to the input file. For example, :program:`opt` by
248default outputs a ``ModuleID``:
249
250.. code-block:: console
251
252 $ cat example.ll
253 define i32 @main() nounwind {
254 ret i32 0
255 }
256
257 $ opt -S /path/to/example.ll
258 ; ModuleID = '/path/to/example.ll'
259
260 define i32 @main() nounwind {
261 ret i32 0
262 }
263
264``ModuleID`` can unexpetedly match against ``CHECK`` lines. For example:
265
266.. code-block:: llvm
267
268 ; RUN: opt -S %s | FileCheck
269
270 define i32 @main() nounwind {
271 ; CHECK-NOT: load
272 ret i32 0
273 }
274
275This test will fail if placed into a ``download`` directory.
276
277To make your tests robust, always use ``opt ... < %s`` in the RUN line.
278:program:`opt` does not output a ``ModuleID`` when input comes from stdin.
279
Renato Golin98c60812013-07-03 20:56:33 +0000280Platform-Specific Tests
281-----------------------
282
283Whenever adding tests that require the knowledge of a specific platform,
284either related to code generated, specific output or back-end features,
285you must make sure to isolate the features, so that buildbots that
286run on different architectures (and don't even compile all back-ends),
287don't fail.
288
289The first problem is to check for target-specific output, for example sizes
290of structures, paths and architecture names, for example:
291
292* Tests containing Windows paths will fail on Linux and vice-versa.
293* Tests that check for ``x86_64`` somewhere in the text will fail anywhere else.
294* Tests where the debug information calculates the size of types and structures.
295
296Also, if the test rely on any behaviour that is coded in any back-end, it must
297go in its own directory. So, for instance, code generator tests for ARM go
298into ``test/CodeGen/ARM`` and so on. Those directories contain a special
299``lit`` configuration file that ensure all tests in that directory will
300only run if a specific back-end is compiled and available.
301
302For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is:
303
304.. code-block:: python
305
306 config.suffixes = ['.ll', '.c', '.cpp', '.test']
307 targets = set(config.root.targets_to_build.split())
308 if not 'ARM' in targets:
309 config.unsupported = True
310
311Other platform-specific tests are those that depend on a specific feature
312of a specific sub-architecture, for example only to Intel chips that support ``AVX2``.
313
314For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture
315variants:
316
317.. code-block:: llvm
318
319 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2
320 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1
321 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2
322
323And the checks are different:
324
325.. code-block:: llvm
326
327 ; SSE2: @test1
328 ; SSE2: psubusw LCPI0_0(%rip), %xmm0
329 ; AVX1: @test1
330 ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0
331 ; AVX2: @test1
332 ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0
333
334So, if you're testing for a behaviour that you know is platform-specific or
335depends on special features of sub-architectures, you must add the specific
336triple, test with the specific FileCheck and put it into the specific
337directory that will filter out all other architectures.
338
339
Nico Rieckc4e7f302014-02-15 08:35:56 +0000340Substitutions
341-------------
Sean Silvaa89edf62012-11-14 21:09:30 +0000342
Nico Rieckc4e7f302014-02-15 08:35:56 +0000343Besides replacing LLVM tool names the following substitutions are performed in
344RUN lines:
Sean Silvaa89edf62012-11-14 21:09:30 +0000345
Nico Rieckc4e7f302014-02-15 08:35:56 +0000346``%%``
347 Replaced by a single ``%``. This allows escaping other substitutions.
Sean Silvaa89edf62012-11-14 21:09:30 +0000348
Nico Rieckc4e7f302014-02-15 08:35:56 +0000349``%s``
350 File path to the test case's source. This is suitable for passing on the
351 command line as the input to an LLVM tool.
Sean Silvaa89edf62012-11-14 21:09:30 +0000352
Nico Rieckc4e7f302014-02-15 08:35:56 +0000353 Example: ``/home/user/llvm/test/MC/ELF/foo_test.s``
Sean Silvaa89edf62012-11-14 21:09:30 +0000354
Nico Rieckc4e7f302014-02-15 08:35:56 +0000355``%S``
356 Directory path to the test case's source.
Sean Silvaa89edf62012-11-14 21:09:30 +0000357
Nico Rieckc4e7f302014-02-15 08:35:56 +0000358 Example: ``/home/user/llvm/test/MC/ELF``
Sean Silvaa89edf62012-11-14 21:09:30 +0000359
Nico Rieckc4e7f302014-02-15 08:35:56 +0000360``%t``
361 File path to a temporary file name that could be used for this test case.
Sean Silvaa89edf62012-11-14 21:09:30 +0000362 The file name won't conflict with other test cases. You can append to it
363 if you need multiple temporaries. This is useful as the destination of
364 some redirected output.
365
Nico Rieckc4e7f302014-02-15 08:35:56 +0000366 Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp``
Sean Silvaa89edf62012-11-14 21:09:30 +0000367
Nico Rieckc4e7f302014-02-15 08:35:56 +0000368``%T``
369 Directory of ``%t``.
Sean Silvaa89edf62012-11-14 21:09:30 +0000370
Nico Rieckc4e7f302014-02-15 08:35:56 +0000371 Example: ``/home/user/llvm.build/test/MC/ELF/Output``
Sean Silvaa89edf62012-11-14 21:09:30 +0000372
Nico Rieckc4e7f302014-02-15 08:35:56 +0000373``%{pathsep}``
374
375 Expands to the path separator, i.e. ``:`` (or ``;`` on Windows).
376
377
378**LLVM-specific substitutions:**
379
380``%shlibext``
381 The suffix for the host platforms shared library files. This includes the
382 period as the first character.
383
384 Example: ``.so`` (Linux), ``.dylib`` (OS X), ``.dll`` (Windows)
385
386``%exeext``
387 The suffix for the host platforms executable files. This includes the
388 period as the first character.
389
390 Example: ``.exe`` (Windows), empty on Linux.
391
392``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
393 The number of the line where this substitution is used, with an optional
394 integer offset. This can be used in tests with multiple RUN lines, which
395 reference test file's line numbers.
396
397
398**Clang-specific substitutions:**
399
400``%clang``
401 Invokes the Clang driver.
402
403``%clang_cpp``
404 Invokes the Clang driver for C++.
405
406``%clang_cl``
407 Invokes the CL-compatible Clang driver.
408
409``%clangxx``
410 Invokes the G++-compatible Clang driver.
411
412``%clang_cc1``
413 Invokes the Clang frontend.
414
415``%itanium_abi_triple``, ``%ms_abi_triple``
416 These substitutions can be used to get the current target triple adjusted to
417 the desired ABI. For example, if the test suite is running with the
418 ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to
419 ``i686-pc-mingw32``. This allows a test to run with a specific ABI without
420 constraining it to a specific triple.
421
422To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``.
423
Sean Silvaa89edf62012-11-14 21:09:30 +0000424
425Other Features
426--------------
427
Nico Rieckea623c62014-01-08 16:30:03 +0000428To make RUN line writing easier, there are several helper programs. These
429helpers are in the PATH when running tests, so you can just call them using
430their name. For example:
Sean Silvaa89edf62012-11-14 21:09:30 +0000431
Sean Silvaa89edf62012-11-14 21:09:30 +0000432``not``
Nico Rieckea623c62014-01-08 16:30:03 +0000433 This program runs its arguments and then inverts the result code from it.
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000434 Zero result codes become 1. Non-zero result codes become 0.
Sean Silvaa89edf62012-11-14 21:09:30 +0000435
436Sometimes it is necessary to mark a test case as "expected fail" or
437XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
438on a line near the top of the file. This signals that the test case
439should succeed if the test fails. Such test cases are counted separately
440by the testing tool. To specify an expected fail, use the XFAIL keyword
441in the comments of the test program followed by a colon and one or more
442failure patterns. Each failure pattern can be either ``*`` (to specify
443fail everywhere), or a part of a target triple (indicating the test
444should fail on that platform), or the name of a configurable feature
445(for example, ``loadable_module``). If there is a match, the test is
446expected to fail. If not, the test is expected to succeed. To XFAIL
447everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
448line:
449
450.. code-block:: llvm
451
452 ; XFAIL: darwin,sun
453
Eli Bendersky0ffc0d42012-12-04 14:34:00 +0000454To make the output more useful, :program:`lit` will scan
Sean Silvaa89edf62012-11-14 21:09:30 +0000455the lines of the test case for ones that contain a pattern that matches
456``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
457that is related to the test case. The number after "PR" specifies the
458LLVM bugzilla number. When a PR number is specified, it will be used in
459the pass/fail reporting. This is useful to quickly get some context when
460a test fails.
461
462Finally, any line that contains "END." will cause the special
463interpretation of lines to terminate. This is generally done right after
464the last RUN: line. This has two side effects:
465
466(a) it prevents special interpretation of lines that are part of the test
467 program, not the instructions to the test case, and
468
469(b) it speeds things up for really big test cases by avoiding
470 interpretation of the remainder of the file.
471
472``test-suite`` Overview
473=======================
474
475The ``test-suite`` module contains a number of programs that can be
476compiled and executed. The ``test-suite`` includes reference outputs for
477all of the programs, so that the output of the executed program can be
478checked for correctness.
479
480``test-suite`` tests are divided into three types of tests: MultiSource,
481SingleSource, and External.
482
483- ``test-suite/SingleSource``
484
485 The SingleSource directory contains test programs that are only a
486 single source file in size. These are usually small benchmark
487 programs or small programs that calculate a particular value. Several
488 such programs are grouped together in each directory.
489
490- ``test-suite/MultiSource``
491
492 The MultiSource directory contains subdirectories which contain
493 entire programs with multiple source files. Large benchmarks and
494 whole applications go here.
495
496- ``test-suite/External``
497
498 The External directory contains Makefiles for building code that is
499 external to (i.e., not distributed with) LLVM. The most prominent
500 members of this directory are the SPEC 95 and SPEC 2000 benchmark
501 suites. The ``External`` directory does not contain these actual
502 tests, but only the Makefiles that know how to properly compile these
503 programs from somewhere else. When using ``LNT``, use the
504 ``--test-externals`` option to include these tests in the results.
505
Sean Silvae0db5192012-11-14 23:11:10 +0000506.. _test-suite-quickstart:
507
Sean Silvaa89edf62012-11-14 21:09:30 +0000508``test-suite`` Quickstart
509-------------------------
510
511The modern way of running the ``test-suite`` is focused on testing and
512benchmarking complete compilers using the
513`LNT <http://llvm.org/docs/lnt>`_ testing infrastructure.
514
515For more information on using LNT to execute the ``test-suite``, please
516see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_
517documentation.
518
519``test-suite`` Makefiles
520------------------------
521
522Historically, the ``test-suite`` was executed using a complicated setup
523of Makefiles. The LNT based approach above is recommended for most
524users, but there are some testing scenarios which are not supported by
525the LNT approach. In addition, LNT currently uses the Makefile setup
526under the covers and so developers who are interested in how LNT works
527under the hood may want to understand the Makefile based setup.
528
529For more information on the ``test-suite`` Makefile setup, please see
Sean Silvae0db5192012-11-14 23:11:10 +0000530the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`.