Gitiles
Code Review Sign In
gerrit-public.fairphone.software / fp2-dev / platform / external / llvm / 020cbd88588435f13983e1ab53b918fcec4bbf95 / . / docs / TestingGuide.html
blob: bf740cf9da3dec5a8a3d60fc10650a28cd2120fe [file] [log] [blame]
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame^]1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2<html>
3 <head>
4 <title>LLVM Test Suite Guide</title>
5 </head>
6
7 <body bgcolor=white>
8
9 <center><h1>LLVM Test Suite Guide<br></h1></center>
10
11 <!--===============================================================-->
12 <h2><a name="overview">Overview</a><hr></h2>
13 <!--===============================================================-->
14
15 This document is the reference manual for the LLVM test suite. It
16 documents the structure of the LLVM test suite, the tools needed to
17 use it, and how to add and run tests.
18
19 <!--===============================================================-->
20 <h2><a name="Requirements">Requirements</a><hr></h2>
21 <!--===============================================================-->
22
23 In order to use the LLVM test suite, you will need all of the software
24 required to build LLVM, plus the following:
25 <dl compact>
26 <dt><A HREF="http://www.qmtest.com">QMTest</A>
27 <dd>
28 The LLVM test suite uses QMTest to organize and run tests.
29 <p>
30
31 <dt><A HREF="http://www.python.org">Python</A>
32 <dd>
33 You will need a python interpreter that works with QMTest.
34 Python will need zlib and SAX support enabled.
35 <p>
36 </dl>
37
38 <!--===============================================================-->
39 <h2><a name="quick">Quick Start</a><hr></h2>
40 <!--===============================================================-->
41
42 To run all of the tests in LLVM, use the Master Makefile in llvm/test:
43 <p>
44 <tt>
45 cd test
46 <br>
47 make
48 </tt>
49
50 <p>
51
52 To run only the code fragment tests (i.e. those that do basic testing of
53 LLVM), run the tests organized by QMTest:
54 <p>
55
56 <tt>
57 cd test
58 <br>
59 make qmtest
60 </tt>
61
62 <p>
63
64 To run only the tests that compile and execute whole programs, run the
65 Programs tests:
66 <p>
67
68 <tt>
69 cd test/Programs
70 <br>
71 make
72 </tt>
73 <p>
74
75 <!--===============================================================-->
76 <h2><a name="org">LLVM Test Suite Organization</a><hr></h2>
77 <!--===============================================================-->
78
79 The LLVM test suite contains two major types of tests:
80 <ul>
81 <li>Code Fragments<br>
82 Code fragments are small pieces of code that test a specific
83 feature of LLVM or trigger a specific bug in LLVM. They are
84 usually written in LLVM assembly language, but can be
85 written in other languages if the test targets a particular language
86 front end.
87 <p>
88 Code fragments are not complete programs, and they are never executed
89 to determine correct behavior.
90 <p>
91 The tests in the llvm/test/Features and llvm/test/Regression directories
92 contain code fragments.
93
94 <li>Whole Programs<br>
95 Whole Programs are pieces of code which can be compiled and
96 linked into a stand-alone program that can be executed. These programs
97 are generally written in high level languages such as C or C++, but
98 sometimes they are written straight in LLVM assembly.
99 <p>
100 These programs are compiled and then executed using several different
101 methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code
102 generation, etc). The output of these programs is compared to ensure
103 that LLVM is compiling the program correctly.
104
105 <p>
106 In addition to compiling and executing programs, whole program tests
107 serve as a way of benchmarking LLVM performance, both in terms of the
108 efficiency of the programs generated as well as the speed with which
109 LLVM compiles, optimizes, and generates code.
110
111 <p>
112 The test/Programs directory contains all tests which compile and
113 benchmark whole programs.
114 </ul>
115
116 <!--===============================================================-->
117 <h2><a name="tree">LLVM Test Suite Tree</a><hr></h2>
118 <!--===============================================================-->
119
120 The LLVM test suite is broken up into the following directory
121 hierarchy:
122
123 <ul>
124 <li> Features<br>
125 This directory contains sample codes that test various features
126 of the LLVM language. These pieces of sample code are run
127 through various assembler, disassembler, and optimizer passes.
128 <p>
129
130 <li>Regression<br>
131 This directory contains regression tests for LLVM. When a bug
132 is found in LLVM, a regression test containing just enough
133 code to reproduce the problem should be written and placed
134 somewhere underneath this directory. In most cases, this
135 will be a small piece of LLVM assembly language code, often
136 distilled from an actual application or benchmark.
137 <p>
138
139 <li>Programs<br>
140 The Programs directory contains programs that can be compiled
141 with LLVM and executed. These programs are compiled using the
142 native compiler and various LLVM backends. The output from the
143 program compiled with the native compiler is assumed correct;
144 the results from the other programs are compared to the native
145 program output and pass if they match.
146 <p>
147 In addition for testing correctness, the Programs directory
148 also performs timing tests of various LLVM optimizations.
149 It also records compilation times for the compilers and the
150 JIT. This information can be used to compare the
151 effectiveness of LLVM's optimizations and code generation.
152 <p>
153 The Programs directory is subdivided into several smaller
154 subdirectories:
155 <ul>
156 <li>SingleSource<br>
157 The SingleSource directory contains test programs that
158 are only a single source file in size. These are
159 usually small benchmark programs or small programs that
160 calculate a particular value. Several such programs are grouped
161 together in each directory.
162 <p>
163
164 <li>MultiSource<br>
165 The MultiSource directory contains subdirectories which contain
166 entire programs with multiple source files. Large benchmarks and
167 whole applications go here.
168 <p>
169
170 <li>External<br>
171 The External directory contains Makefiles for building
172 code that is external to (i.e. not distributed with)
173 LLVM. The most prominent member of this directory is
174 the SPEC 2000 benchmark suite. The presence and location
175 of these external programs is configured by the LLVM
176 <tt>configure</tt> script.
177 </ul>
178
179 <p>
180
181 <li>QMTest<br>
182 This directory contains the QMTest information files. Inside this
183 directory are QMTest administration files and the Python code that
184 implements the LLVM test and database classes.
185 </ul>
186
187 <!--===============================================================-->
188 <h2><a name="qmstructure">QMTest Structure</a><hr></h2>
189 <!--===============================================================-->
190
191 The LLVM test suite is partially driven by QMTest and partially
192 driven by GNU Make. Specifically, the Features and Regression tests
193 are all driven by QMTest. The Programs directory is currently
194 driven by a set of Makefiles.
195 <p>
196
197 The QMTest system needs to have several pieces of information
198 available; these pieces of configuration information are known
199 collectively as the "context" in QMTest parlance. Since the context
200 for LLVM is relatively large, the master Makefile in llvm/test
201 sets it for you.
202
203 <p>
204
205 The LLVM database class makes the directory tree underneath llvm/test a
206 QMTest test database. For each directory that contains tests driven by
207 QMTest, it knows what type of test the source file is and how to run it.
208
209 <p>
210
211 Hence, the QMTest namespace is essentially what you see in
212 llvm/test/Feature and llvm/test/Regression, but there is some magic that
213 the database class performs (as described below).
214
215 <p>
216
217 The QMTest namespace is currently composed of the following tests and
218 test suites:
219
220 <ul>
221 <li>Feature<br>
222 These are the feature tests found in llvm/test/Feature. They are broken
223 up into the following categories:
224 <ul>
225 <li>fad<br>
226 Assembler/Disassembler tests. These tests verify that a piece of
227 LLVM assembly language can be assembled into bytecode and then
228 disassembled into the original assembly language code.
229 It does this several times to ensure that assembled
230 output can be disassembled and disassembler output can
231 be assembled. It also verifies that the give assembly language file
232 can be assembled correctly.
233 <p>
234
235 <li>fasm<br>
236 Assembler tests. These tests verify that the code can be translated
237 into native assembly code.
238 <p>
239
240 <li>fopt<br>
241 Optimizer tests. These tests verify that two of the
242 optimizer passes completely optimize a program (i.e.
243 after a single pass, they cannot optimize a program
244 any further).
245 <p>
246
247 <li>fmc<br>
248 Machine code tests. These tests verify that the LLVM assembly
249 language file can be translated into native assembly code.
250 <p>
251
252 <li>fcc<br>
253 C code tests. These tests verify that the specified LLVM assembly
254 code can be converted into C source code using the C backend.
255 </ul>
256
257 <p>
258
259 The LLVM database class looks at every file in llvm/test/Feature and
260 creates a fake test hierarchy containing
261 Feature.&lt;testtype&gt;.&lt;testname&gt;.
262 So, if you add an LLVM assembly language file to llvm/test/Feature, it
263 actually creates 5 news test: assembler/disassembler, assembler,
264 optimizer, machine code, and C code.
265
266 <li>Regression<br>
267 These are the regression tests. There is one suite for each directory
268 in llvm/test/Regression.
269 <p>
270
271 If you add a new directory to llvm/test/Regression, you will need to
272 modify llvm/test/QMTest/llvmdb.py so that it knows what sorts of tests
273 are in it and how to run them.
274 </ul>
275
276 <!--===============================================================-->
277 <h2><a name="progstructure">Programs Structure</a><hr></h2>
278 <!--===============================================================-->
279 As mentioned previously, the Programs tree in llvm/test provides three types
280 of tests: MultiSource, SingleSource, and External. Each tree is then
281 subdivided into several categories, including applications, benchmarks,
282 regression tests, code that is strange grammatically, etc. These
283 organizations should be relatively self explanatory.
284 <p>
285 In addition to the regular Programs tests, the Programs tree also provides a
286 mechanism for compiling the programs in different ways. If the variable TEST
287 is defined on the gmake command line, the test system will include a Makefile
288 named TEST.&lt;value of TEST variable&gt;.Makefile. This Makefile can modify
289 build rules that yield different results.
290 <p>
291 For example, the LLVM nightly tester uses TEST.nightly.Makefile to create the
292 nightly test reports. To run the nightly tests, run <tt>gmake
293 TEST=nightly</tt>.
294 <p>
295 There are several TEST Makefiles available in the tree. Some of them are
296 designed for internal LLVM research and will not work outside of the LLVM
297 research group. They may still be valuable, however, as a guide to writing
298 your own TEST Makefile for any optimization or analysis passes that you
299 develop with LLVM.
300
301 <!--===============================================================-->
302 <h2><a name="run">Running the LLVM Tests</a><hr></h2>
303 <!--===============================================================-->
304
305 First, all tests are executed within the LLVM object directory tree. They
306 <i>are not</i> executed inside of the LLVM source tree. This is because
307 the test suite creates temporary files during execution.
308
309 <p>
310
311 The master Makefile in llvm/test is capable of running both the
312 QMTest driven tests and the Programs tests. By default, it will run
313 all of the tests.
314 <p>
315 To run only the QMTest driven tests, run <tt>make qmtest</tt> at the
316 command line in llvm/tests. To run a specific qmtest, suffix the test name
317 with ".t" when running make.
318 <p>
319 For example, to run the Regression.LLC tests, type
320 <tt>make Regression.LLC.t</tt> in llvm/tests.
321 <p>
322 Note that the Makefiles in llvm/test/Features and llvm/test/Regression
323 are gone. You must now use QMTest from the llvm/test directory to run them.
324 <p>
325
326 To run the Programs test, cd into the llvm/test/Programs directory
327 and type <tt>make</tt>. Alternatively, you can type <tt>make
328 TEST=&lt;type&gt; test</tt> to run one of the specialized tests in
329 llvm/test/Programs/TEST.&lt;type&gt;.Makefile. For example, you could run
330 the nightly tester tests using the following commands:
331 <p>
332 <tt>
333 cd llvm/test/Programs
334 <br>
335 make TEST=nightly test
336 </tt>
337
338 <p>
339 Regardless of which test you're running, the results are printed on standard
340 output and standard error. You can redirect these results to a file if you
341 choose.
342 <p>
343 Some tests are known to fail. Some are bugs that we have not fixed yet;
344 others are features that we haven't added yet (or may never add). In QMTest,
345 the result for such tests will be XFAIL (eXpected FAILure). In this way, you
346 can tell the difference between an expected and unexpected failure.
347 <p>
348 The Programs tests have no such feature as of this time. If the test passes,
349 only warnings and other miscellaneous output will be generated. If a test
350 fails, a large &lt;program&gt; FAILED message will be displayed. This will
351 help you separate benign warnings from actual test failures.
352
353 <hr>
354
355 </body>
356</html>