Gitiles
Code Review Sign In
gerrit-public.fairphone.software / fp2-dev / platform / external / llvm / af19f2e52c17a9bd91130d5dd0451fef477575e0 / . / docs / TestingGuide.html
blob: 71354e29570bd6016113fa1a60f3b8bd8d10660c [file] [log] [blame]
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2<html>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]3<head>
4 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
5 <link rel="stylesheet" href="llvm.css" type="text/css" media="screen" />
6 <title>LLVM Test Suite Guide</title>
7</head>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]8
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]9<body>
10
11<div class="doc_title">
12 LLVM Test Suite Guide
13</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]14
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]15<ol>
16<li><a href="#overview">Overview</a></li>
17<li><a href="#Requirements">Requirements</a></li>
18<li><a href="#quick">Quick Start</a></li>
19<li><a href="#org">LLVM Test Suite Organization</a></li>
20<ul>
21 <li><a href="#codefragments">Code Fragments</a></li>
22 <li><a href="#wholeprograms">Whole Programs</a></li>
23</ul>
24<li><a href="#tree">LLVM Test Suite Tree</a></li>
25<li><a href="#qmstructure">QMTest Structure</a></li>
26<li><a href="#progstructure">Programs Structure</a></li>
27<li><a href="#run">Running the LLVM Tests</a></li>
28<p><b>Written by John T. Criswell</b></p>
29</ol>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]30
31 <!--===============================================================-->
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]32 <div class="doc_section"><a name="overview">Overview</a></div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]33 <!--===============================================================-->
34
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]35 <div class="doc_text">
36 <p>
37 This document is the reference manual for the LLVM test suite. It
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]38 documents the structure of the LLVM test suite, the tools needed to
39 use it, and how to add and run tests.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]40 </p>
41 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]42
43 <!--===============================================================-->
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]44 <div class="doc_section"><a name="Requirements">Requirements</a></div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]45 <!--===============================================================-->
46
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]47 <div class="doc_text">
48 <p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]49 In order to use the LLVM test suite, you will need all of the software
50 required to build LLVM, plus the following:
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]51 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]52 <dl compact>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]53 <dt><A HREF="http://www.qmtest.com">QMTest</A></dt>
54 <dd>The LLVM test suite uses QMTest to organize and
55 run tests.</dd>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]56
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]57 <dt><A HREF="http://www.python.org">Python</A></dt>
58 <dd>You will need a Python interpreter that works with
59 QMTest. Python will need zlib and SAX support
60 enabled.</dd>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]61 </dl>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]62 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]63
64 <!--===============================================================-->
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]65 <div class="doc_section"><a name="quick">Quick Start</a></div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]66 <!--===============================================================-->
67
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]68 <div class="doc_text">
69 <p>
70 The tests are located in the LLVM source tree under the directory
71 <tt>llvm/test</tt>. To run all of the tests in LLVM, use the Master
72 Makefile in that directory:
73 </p>
74 <pre>
75 % make -C llvm/test
76 </pre>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]77
78 <p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]79 To run only the code fragment tests (i.e. those that do basic testing of
80 LLVM), run the tests organized by QMTest:
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]81 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]82
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]83 <pre>
84 % make -C llvm/test qmtest
85 </pre>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]86
87 <p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]88 To run only the tests that compile and execute whole programs, run the
89 Programs tests:
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]90 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]91
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]92 <pre>
93 % make -C llvm/test/Programs
94 </pre>
95 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]96
97 <!--===============================================================-->
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]98 <div class="doc_section"><h2><a name="org">LLVM Test Suite
99 Organization </a></div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]100 <!--===============================================================-->
101
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]102 <div class="doc_text">
103 <p>The LLVM test suite contains two major categories of tests: code
104 fragments and whole programs.</p>
105 </div>
106
107 <div class="doc_subsection"><a name="codefragments">Code Fragments</a>
108 </div>
109
110 <div class="doc_text">
111 <p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]112 Code fragments are small pieces of code that test a specific
113 feature of LLVM or trigger a specific bug in LLVM. They are
114 usually written in LLVM assembly language, but can be
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]115 written in other languages if the test targets a
116 particular language front end.
117 </p><p>
118 Code fragments are not complete programs, and they are
119 never executed to determine correct behavior.
120 </p><p>
121 The tests in the Features and
122 Regression directories contain code fragments.
123 </p>
124 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]125
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]126 <div class="doc_subsection"><a name="wholeprograms">Whole Programs</a>
127 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]128
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]129 <div class="doc_text">
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]130 <p>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]131 Whole Programs are pieces of code which can be compiled and
132 linked into a stand-alone program that can be executed. These
133 programs are generally written in high level languages such as C
134 or C++, but sometimes they are written straight in LLVM
135 assembly.
136 </p><p>
137 These programs are compiled and then executed using several
138 different methods (native compiler, LLVM C backend, LLVM JIT,
139 LLVM native code generation, etc). The output of these programs
140 is compared to ensure that LLVM is compiling the program
141 correctly.
142 </p><p>
143 In addition to compiling and executing programs, whole program
144 tests serve as a way of benchmarking LLVM performance, both in
145 terms of the efficiency of the programs generated as well as the
146 speed with which LLVM compiles, optimizes, and generates code.
147 </p><p>
148 The Programs directory contains all tests which compile and
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]149 benchmark whole programs.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]150 </p>
151 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]152
153 <!--===============================================================-->
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]154 <div class="doc_section"><h2><a name="tree">LLVM Test Suite Tree</a>
155 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]156 <!--===============================================================-->
157
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]158 <div class="doc_text">
159 <p>Each type of test in the LLVM test suite has its own directory. The
160 major subtrees of the test suite directory tree are as follows:</p>
161
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]162 <ul>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]163 <li>Features
164 <p>
165 This directory contains sample codes that test various features
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]166 of the LLVM language. These pieces of sample code are run
167 through various assembler, disassembler, and optimizer passes.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]168 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]169
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]170 <li>Regression
171 <p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]172 This directory contains regression tests for LLVM. When a bug
173 is found in LLVM, a regression test containing just enough
174 code to reproduce the problem should be written and placed
175 somewhere underneath this directory. In most cases, this
176 will be a small piece of LLVM assembly language code, often
177 distilled from an actual application or benchmark.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]178 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]179
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]180 <li>Programs
181 <p>
182 The Programs directory contains programs that can be compiled
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]183 with LLVM and executed. These programs are compiled using the
184 native compiler and various LLVM backends. The output from the
185 program compiled with the native compiler is assumed correct;
186 the results from the other programs are compared to the native
187 program output and pass if they match.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]188 </p><p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]189 In addition for testing correctness, the Programs directory
190 also performs timing tests of various LLVM optimizations.
191 It also records compilation times for the compilers and the
192 JIT. This information can be used to compare the
193 effectiveness of LLVM's optimizations and code generation.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]194 </p><p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]195 The Programs directory is subdivided into several smaller
196 subdirectories:
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]197 </p>
198
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]199 <ul>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]200 <li>Programs/SingleSource
201 <p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]202 The SingleSource directory contains test programs that
203 are only a single source file in size. These are
204 usually small benchmark programs or small programs that
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]205 calculate a particular value. Several such programs are
206 grouped together in each directory.
207 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]208
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]209 <li>Programs/MultiSource
210 <p>
211 The MultiSource directory contains subdirectories which
212 contain entire programs with multiple source files.
213 Large benchmarks and whole applications go here.
214 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]215
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]216 <li>Programs/External
217 <p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]218 The External directory contains Makefiles for building
219 code that is external to (i.e. not distributed with)
220 LLVM. The most prominent member of this directory is
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]221 the SPEC 2000 benchmark suite. The presence and
222 location of these external programs is configured by the
223 LLVM <tt>configure</tt> script.
224 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]225 </ul>
226
227 <p>
228
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]229 <li>QMTest
230 <p>
231 This directory contains the QMTest information files. Inside
232 this directory are QMTest administration files and the Python
233 code that implements the LLVM test and database classes.
234 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]235 </ul>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]236 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]237
238 <!--===============================================================-->
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]239 <div class="doc_section"><h2><a name="qmstructure">QMTest Structure</a>
240 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]241 <!--===============================================================-->
242
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]243 <div class="doc_text">
244 <p>
245 The LLVM test suite is partially driven by QMTest and partially
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]246 driven by GNU Make. Specifically, the Features and Regression tests
247 are all driven by QMTest. The Programs directory is currently
248 driven by a set of Makefiles.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]249 </p><p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]250 The QMTest system needs to have several pieces of information
251 available; these pieces of configuration information are known
252 collectively as the "context" in QMTest parlance. Since the context
253 for LLVM is relatively large, the master Makefile in llvm/test
254 sets it for you.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]255 </p><p>
256 The LLVM database class makes the subdirectories of llvm/test a
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]257 QMTest test database. For each directory that contains tests driven by
258 QMTest, it knows what type of test the source file is and how to run it.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]259 </p><p>
260 Hence, the QMTest namespace is essentially what you see in the
261 Feature and Regression directories, but there is some magic that
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]262 the database class performs (as described below).
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]263 </p><p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]264 The QMTest namespace is currently composed of the following tests and
265 test suites:
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]266 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]267
268 <ul>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]269 <li>Feature
270 <p>
271 These are the feature tests found in the Feature directory.
272 They are broken up into the following categories:
273 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]274 <ul>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]275 <li>ad
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]276 <p>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]277 Assembler/Disassembler tests. These tests verify that a
278 piece of LLVM assembly language can be assembled into
279 bytecode and then disassembled into the original
280 assembly language code. It does this several times to
281 ensure that assembled output can be disassembled and
282 disassembler output can be assembled. It also verifies
283 that the give assembly language file can be assembled
284 correctly.
285 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]286
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]287 <li>opt
288 <p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]289 Optimizer tests. These tests verify that two of the
290 optimizer passes completely optimize a program (i.e.
291 after a single pass, they cannot optimize a program
292 any further).
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]293 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]294
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]295 <li>mc
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]296 <p>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]297 Machine code tests. These tests verify that the LLVM
298 assembly language file can be translated into native
299 assembly code.
300 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]301
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]302 <li>cc
303 <p>
304 C code tests. These tests verify that the specified
305 LLVM assembly code can be converted into C source code
306 using the C backend.
307 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]308 </ul>
309
310 <p>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]311 The LLVM database class looks at every file in the Feature
312 directory and creates a fake test hierarchy containing
313 <tt>Feature.&lt;testtype&gt;.&lt;testname&gt;</tt>. So, if you
314 add an LLVM assembly language file to the Feature directory, it
315 actually creates 5 new tests: assembler/disassembler, assembler,
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]316 optimizer, machine code, and C code.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]317 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]318
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]319 <li>Regression
320 <p>
321 These are the regression tests. There is one suite for each
322 subdirectory of the Regression directory. If you add a new
323 subdirectory there, you will need to modify, at least, the
324 <tt>RegressionMap</tt> variable in <tt>QMTest/llvmdb.py</tt> so
325 that QMTest knows how to run the tests in the new subdirectory.
326 </p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]327 </ul>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]328 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]329
330 <!--===============================================================-->
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]331 <div class="doc_section"><h2><a name="progstructure">Programs
332 Structure</a></div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]333 <!--===============================================================-->
334
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]335 <div class="doc_text">
336 <p>
337 As mentioned previously, the Programs tree in llvm/test provides three
338 types of tests: MultiSource, SingleSource, and External. Each tree is
339 then subdivided into several categories, including applications,
340 benchmarks, regression tests, code that is strange grammatically, etc.
341 These organizations should be relatively self explanatory.
342 </p><p>
343 In addition to the regular Programs tests, the Programs tree also
344 provides a mechanism for compiling the programs in different ways. If
345 the variable TEST is defined on the gmake command line, the test system
346 will include a Makefile named <tt>TEST.&lt;value of TEST
347 variable&gt;.Makefile</tt>. This Makefile can modify build rules to
348 yield different results.
349 </p><p>
350 For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt>
351 to create the nightly test reports. To run the nightly tests, run
352 <tt>gmake TEST=nightly</tt>.
353 </p><p>
354 There are several TEST Makefiles available in the tree. Some of them
355 are designed for internal LLVM research and will not work outside of the
356 LLVM research group. They may still be valuable, however, as a guide to
357 writing your own TEST Makefile for any optimization or analysis passes
358 that you develop with LLVM.
359 </p>
360 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]361
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]362 <!--===============================================================-->
363 <div class="doc_section"><h2><a name="run">Running the LLVM Tests</a>
364 </div>
365 <!--===============================================================-->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]366
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]367 <div class="doc_text">
368 <p>
369 First, all tests are executed within the LLVM object directory tree.
370 They <i>are not</i> executed inside of the LLVM source tree. This is
371 because the test suite creates temporary files during execution.
372 </p><p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]373 The master Makefile in llvm/test is capable of running both the
374 QMTest driven tests and the Programs tests. By default, it will run
375 all of the tests.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]376 </p><p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]377 To run only the QMTest driven tests, run <tt>make qmtest</tt> at the
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]378 command line in llvm/tests. To run a specific qmtest, suffix the test
379 name with ".t" when running make.
380 </p><p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]381 For example, to run the Regression.LLC tests, type
382 <tt>make Regression.LLC.t</tt> in llvm/tests.
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]383 </p><p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]384 Note that the Makefiles in llvm/test/Features and llvm/test/Regression
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]385 are gone. You must now use QMTest from the llvm/test directory to run
386 them.
387 </p><p>
388 To run the Programs test, cd into the llvm/test/Programs directory and
389 type <tt>make</tt>. Alternatively, you can type <tt>make
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]390 TEST=&lt;type&gt; test</tt> to run one of the specialized tests in
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]391 llvm/test/Programs/TEST.&lt;type&gt;.Makefile. For example, you could
392 run the nightly tester tests using the following commands:
393 </p>
394
395 <pre>
396 % cd llvm/test/Programs
397 % make TEST=nightly test
398 </pre>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]399
400 <p>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]401 Regardless of which test you're running, the results are printed on
402 standard output and standard error. You can redirect these results to a
403 file if you choose.
404 </p><p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]405 Some tests are known to fail. Some are bugs that we have not fixed yet;
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]406 others are features that we haven't added yet (or may never add). In
407 QMTest, the result for such tests will be XFAIL (eXpected FAILure). In
408 this way, you can tell the difference between an expected and unexpected
409 failure.
410 </p><p>
411 The Programs tests have no such feature as of this time. If the test
412 passes, only warnings and other miscellaneous output will be generated.
413 If a test fails, a large &lt;program&gt; FAILED message will be
414 displayed. This will help you separate benign warnings from actual test
415 failures.
416 </p>
417 </div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]418
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]419<!-- *********************************************************************** -->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]420
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame^]421<hr><font size="-1">
422<address>John T. Criswell</address>
423Last modified: $Date$
424</font>
425
426</body>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]427</html>