Gitiles
Code Review Sign In
gerrit-public.fairphone.software / platform / external / llvm / a1f7cac0273ce871e14bba597a9ad0838fff143d / . / docs / TestingGuide.html
blob: 05af6bcecbbae061c8f7b9f81ffcc78b07764de6 [file] [log] [blame]
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2<html>
3 <head>
4 <title>LLVM Test Suite Guide</title>
5 </head>
6
7 <body bgcolor=white>
8
9 <center><h1>LLVM Test Suite Guide<br></h1></center>
10
11 <!--===============================================================-->
12 <h2><a name="overview">Overview</a><hr></h2>
13 <!--===============================================================-->
14
15 This document is the reference manual for the LLVM test suite. It
16 documents the structure of the LLVM test suite, the tools needed to
17 use it, and how to add and run tests.
18
19 <!--===============================================================-->
20 <h2><a name="Requirements">Requirements</a><hr></h2>
21 <!--===============================================================-->
22
23 In order to use the LLVM test suite, you will need all of the software
24 required to build LLVM, plus the following:
25 <dl compact>
26 <dt><A HREF="http://www.qmtest.com">QMTest</A>
27 <dd>
28 The LLVM test suite uses QMTest to organize and run tests.
29 <p>
30
31 <dt><A HREF="http://www.python.org">Python</A>
32 <dd>
33 You will need a python interpreter that works with QMTest.
34 Python will need zlib and SAX support enabled.
35 <p>
36 </dl>
37
38 <!--===============================================================-->
39 <h2><a name="quick">Quick Start</a><hr></h2>
40 <!--===============================================================-->
41
42 To run all of the tests in LLVM, use the Master Makefile in llvm/test:
43 <p>
44 <tt>
45 cd test
46 <br>
47 make
48 </tt>
49
50 <p>
51
52 To run only the code fragment tests (i.e. those that do basic testing of
53 LLVM), run the tests organized by QMTest:
54 <p>
55
56 <tt>
57 cd test
58 <br>
59 make qmtest
60 </tt>
61
62 <p>
63
64 To run only the tests that compile and execute whole programs, run the
65 Programs tests:
66 <p>
67
68 <tt>
69 cd test/Programs
70 <br>
71 make
72 </tt>
73 <p>
74
75 <!--===============================================================-->
76 <h2><a name="org">LLVM Test Suite Organization</a><hr></h2>
77 <!--===============================================================-->
78
79 The LLVM test suite contains two major types of tests:
80 <ul>
81 <li>Code Fragments<br>
82 Code fragments are small pieces of code that test a specific
83 feature of LLVM or trigger a specific bug in LLVM. They are
84 usually written in LLVM assembly language, but can be
85 written in other languages if the test targets a particular language
86 front end.
87 <p>
88 Code fragments are not complete programs, and they are never executed
89 to determine correct behavior.
90 <p>
91 The tests in the llvm/test/Features and llvm/test/Regression directories
92 contain code fragments.
93
94 <li>Whole Programs<br>
95 Whole Programs are pieces of code which can be compiled and
96 linked into a stand-alone program that can be executed. These programs
97 are generally written in high level languages such as C or C++, but
98 sometimes they are written straight in LLVM assembly.
99 <p>
100 These programs are compiled and then executed using several different
101 methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code
102 generation, etc). The output of these programs is compared to ensure
103 that LLVM is compiling the program correctly.
104
105 <p>
106 In addition to compiling and executing programs, whole program tests
107 serve as a way of benchmarking LLVM performance, both in terms of the
108 efficiency of the programs generated as well as the speed with which
109 LLVM compiles, optimizes, and generates code.
110
111 <p>
112 The test/Programs directory contains all tests which compile and
113 benchmark whole programs.
114 </ul>
115
116 <!--===============================================================-->
117 <h2><a name="tree">LLVM Test Suite Tree</a><hr></h2>
118 <!--===============================================================-->
119
120 The LLVM test suite is broken up into the following directory
121 hierarchy:
122
123 <ul>
124 <li> Features<br>
125 This directory contains sample codes that test various features
126 of the LLVM language. These pieces of sample code are run
127 through various assembler, disassembler, and optimizer passes.
128 <p>
129
130 <li>Regression<br>
131 This directory contains regression tests for LLVM. When a bug
132 is found in LLVM, a regression test containing just enough
133 code to reproduce the problem should be written and placed
134 somewhere underneath this directory. In most cases, this
135 will be a small piece of LLVM assembly language code, often
136 distilled from an actual application or benchmark.
137 <p>
138
139 <li>Programs<br>
140 The Programs directory contains programs that can be compiled
141 with LLVM and executed. These programs are compiled using the
142 native compiler and various LLVM backends. The output from the
143 program compiled with the native compiler is assumed correct;
144 the results from the other programs are compared to the native
145 program output and pass if they match.
146 <p>
147 In addition for testing correctness, the Programs directory
148 also performs timing tests of various LLVM optimizations.
149 It also records compilation times for the compilers and the
150 JIT. This information can be used to compare the
151 effectiveness of LLVM's optimizations and code generation.
152 <p>
153 The Programs directory is subdivided into several smaller
154 subdirectories:
155 <ul>
156 <li>SingleSource<br>
157 The SingleSource directory contains test programs that
158 are only a single source file in size. These are
159 usually small benchmark programs or small programs that
160 calculate a particular value. Several such programs are grouped
161 together in each directory.
162 <p>
163
164 <li>MultiSource<br>
165 The MultiSource directory contains subdirectories which contain
166 entire programs with multiple source files. Large benchmarks and
167 whole applications go here.
168 <p>
169
170 <li>External<br>
171 The External directory contains Makefiles for building
172 code that is external to (i.e. not distributed with)
173 LLVM. The most prominent member of this directory is
174 the SPEC 2000 benchmark suite. The presence and location
175 of these external programs is configured by the LLVM
176 <tt>configure</tt> script.
177 </ul>
178
179 <p>
180
181 <li>QMTest<br>
182 This directory contains the QMTest information files. Inside this
183 directory are QMTest administration files and the Python code that
184 implements the LLVM test and database classes.
185 </ul>
186
187 <!--===============================================================-->
188 <h2><a name="qmstructure">QMTest Structure</a><hr></h2>
189 <!--===============================================================-->
190
191 The LLVM test suite is partially driven by QMTest and partially
192 driven by GNU Make. Specifically, the Features and Regression tests
193 are all driven by QMTest. The Programs directory is currently
194 driven by a set of Makefiles.
195 <p>
196
197 The QMTest system needs to have several pieces of information
198 available; these pieces of configuration information are known
199 collectively as the "context" in QMTest parlance. Since the context
200 for LLVM is relatively large, the master Makefile in llvm/test
201 sets it for you.
202
203 <p>
204
205 The LLVM database class makes the directory tree underneath llvm/test a
206 QMTest test database. For each directory that contains tests driven by
207 QMTest, it knows what type of test the source file is and how to run it.
208
209 <p>
210
211 Hence, the QMTest namespace is essentially what you see in
212 llvm/test/Feature and llvm/test/Regression, but there is some magic that
213 the database class performs (as described below).
214
215 <p>
216
217 The QMTest namespace is currently composed of the following tests and
218 test suites:
219
220 <ul>
221 <li>Feature<br>
222 These are the feature tests found in llvm/test/Feature. They are broken
223 up into the following categories:
224 <ul>
John Criswella1f7cac2003-10-10 19:50:53 +0000[diff] [blame^]225 <li>ad<br>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]226 Assembler/Disassembler tests. These tests verify that a piece of
227 LLVM assembly language can be assembled into bytecode and then
228 disassembled into the original assembly language code.
229 It does this several times to ensure that assembled
230 output can be disassembled and disassembler output can
231 be assembled. It also verifies that the give assembly language file
232 can be assembled correctly.
233 <p>
234
John Criswella1f7cac2003-10-10 19:50:53 +0000[diff] [blame^]235 <li>opt<br>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]236 Optimizer tests. These tests verify that two of the
237 optimizer passes completely optimize a program (i.e.
238 after a single pass, they cannot optimize a program
239 any further).
240 <p>
241
John Criswella1f7cac2003-10-10 19:50:53 +0000[diff] [blame^]242 <li>mc<br>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]243 Machine code tests. These tests verify that the LLVM assembly
244 language file can be translated into native assembly code.
245 <p>
246
John Criswella1f7cac2003-10-10 19:50:53 +0000[diff] [blame^]247 <li>cc<br>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]248 C code tests. These tests verify that the specified LLVM assembly
249 code can be converted into C source code using the C backend.
250 </ul>
251
252 <p>
253
254 The LLVM database class looks at every file in llvm/test/Feature and
255 creates a fake test hierarchy containing
256 Feature.&lt;testtype&gt;.&lt;testname&gt;.
257 So, if you add an LLVM assembly language file to llvm/test/Feature, it
258 actually creates 5 news test: assembler/disassembler, assembler,
259 optimizer, machine code, and C code.
260
261 <li>Regression<br>
262 These are the regression tests. There is one suite for each directory
263 in llvm/test/Regression.
264 <p>
265
266 If you add a new directory to llvm/test/Regression, you will need to
267 modify llvm/test/QMTest/llvmdb.py so that it knows what sorts of tests
268 are in it and how to run them.
269 </ul>
270
271 <!--===============================================================-->
272 <h2><a name="progstructure">Programs Structure</a><hr></h2>
273 <!--===============================================================-->
274 As mentioned previously, the Programs tree in llvm/test provides three types
275 of tests: MultiSource, SingleSource, and External. Each tree is then
276 subdivided into several categories, including applications, benchmarks,
277 regression tests, code that is strange grammatically, etc. These
278 organizations should be relatively self explanatory.
279 <p>
280 In addition to the regular Programs tests, the Programs tree also provides a
281 mechanism for compiling the programs in different ways. If the variable TEST
282 is defined on the gmake command line, the test system will include a Makefile
283 named TEST.&lt;value of TEST variable&gt;.Makefile. This Makefile can modify
284 build rules that yield different results.
285 <p>
286 For example, the LLVM nightly tester uses TEST.nightly.Makefile to create the
287 nightly test reports. To run the nightly tests, run <tt>gmake
288 TEST=nightly</tt>.
289 <p>
290 There are several TEST Makefiles available in the tree. Some of them are
291 designed for internal LLVM research and will not work outside of the LLVM
292 research group. They may still be valuable, however, as a guide to writing
293 your own TEST Makefile for any optimization or analysis passes that you
294 develop with LLVM.
295
296 <!--===============================================================-->
297 <h2><a name="run">Running the LLVM Tests</a><hr></h2>
298 <!--===============================================================-->
299
300 First, all tests are executed within the LLVM object directory tree. They
301 <i>are not</i> executed inside of the LLVM source tree. This is because
302 the test suite creates temporary files during execution.
303
304 <p>
305
306 The master Makefile in llvm/test is capable of running both the
307 QMTest driven tests and the Programs tests. By default, it will run
308 all of the tests.
309 <p>
310 To run only the QMTest driven tests, run <tt>make qmtest</tt> at the
311 command line in llvm/tests. To run a specific qmtest, suffix the test name
312 with ".t" when running make.
313 <p>
314 For example, to run the Regression.LLC tests, type
315 <tt>make Regression.LLC.t</tt> in llvm/tests.
316 <p>
317 Note that the Makefiles in llvm/test/Features and llvm/test/Regression
318 are gone. You must now use QMTest from the llvm/test directory to run them.
319 <p>
320
321 To run the Programs test, cd into the llvm/test/Programs directory
322 and type <tt>make</tt>. Alternatively, you can type <tt>make
323 TEST=&lt;type&gt; test</tt> to run one of the specialized tests in
324 llvm/test/Programs/TEST.&lt;type&gt;.Makefile. For example, you could run
325 the nightly tester tests using the following commands:
326 <p>
327 <tt>
328 cd llvm/test/Programs
329 <br>
330 make TEST=nightly test
331 </tt>
332
333 <p>
334 Regardless of which test you're running, the results are printed on standard
335 output and standard error. You can redirect these results to a file if you
336 choose.
337 <p>
338 Some tests are known to fail. Some are bugs that we have not fixed yet;
339 others are features that we haven't added yet (or may never add). In QMTest,
340 the result for such tests will be XFAIL (eXpected FAILure). In this way, you
341 can tell the difference between an expected and unexpected failure.
342 <p>
343 The Programs tests have no such feature as of this time. If the test passes,
344 only warnings and other miscellaneous output will be generated. If a test
345 fails, a large &lt;program&gt; FAILED message will be displayed. This will
346 help you separate benign warnings from actual test failures.
347
348 <hr>
349
350 </body>
351</html>