Gitiles
Code Review Sign In
gerrit-public.fairphone.software / fp2-dev / platform / external / llvm / d10bc6e0bb1f41ed9cf356f59c2d17721ad49719 / . / docs / TestingGuide.html
blob: e57c39e200bb311cbf77449f275db04f43605e5a [file] [log] [blame]
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
2 "http://www.w3.org/TR/html4/strict.dtd">
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]3<html>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]4<head>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]5 <title>LLVM Test Suite Guide</title>
6 <link rel="stylesheet" href="llvm.css" type="text/css">
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]7</head>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]8<body>
9
10<div class="doc_title">
11 LLVM Test Suite Guide
12</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]13
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]14<ol>
Reid Spencer820e2472004-11-01 08:30:14 +0000[diff] [blame]15 <li><a href="#overview">Overview</a></li>
16 <li><a href="#Requirements">Requirements</a></li>
17 <li><a href="#quick">Quick Start</a></li>
18 <li><a href="#org">LLVM Test Suite Organization</a>
19 <ul>
20 <li><a href="#codefragments">Code Fragments</a></li>
21 <li><a href="#wholeprograms">Whole Programs</a></li>
22 </ul>
23 </li>
24 <li><a href="#tree">LLVM Test Suite Tree</a></li>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]25 <li><a href="#dgstructure">DejaGNU Structure</a></li>
Reid Spencer820e2472004-11-01 08:30:14 +0000[diff] [blame]26 <li><a href="#qmstructure">QMTest Structure</a></li>
27 <li><a href="#progstructure"><tt>llvm-test</tt> Structure</a></li>
28 <li><a href="#run">Running the LLVM Tests</a></li>
29 <li><a href="#nightly">Running the nightly tester</a></li>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]30</ol>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]31
Chris Lattner7911ce22004-05-23 21:07:27 +0000[diff] [blame]32<div class="doc_author">
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]33 <p>Written by John T. Criswell, <a
34 href="http://llvm.x10sys.com/rspencer">Reid Spencer</a>, and Tanya Lattner</p>
Chris Lattner7911ce22004-05-23 21:07:27 +0000[diff] [blame]35</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]36
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]37<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]38<div class="doc_section"><a name="overview">Overview</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]39<!--=========================================================================-->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]40
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]41<div class="doc_text">
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]42
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]43<p>This document is the reference manual for the LLVM test suite. It documents
44the structure of the LLVM test suite, the tools needed to use it, and how to add
45and run tests.</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]46
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]47</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]48
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]49<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]50<div class="doc_section"><a name="Requirements">Requirements</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]51<!--=========================================================================-->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]52
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]53<div class="doc_text">
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]54
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]55<p>In order to use the LLVM test suite, you will need all of the software
56required to build LLVM, plus the following:</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]57
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]58<dl>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]59<dt><a href="http://www.gnu.org/software/dejagnu/">DejaGNU</a></dt>
60<dd>The Feature and Regressions tests are organized and run by DejaGNU.</dd>
61<dt><a href="http://expect.nist.gov/">Expect</a></dt>
62<dd>Expect is required by DejaGNU.</dd>
63<dt>tclsh</dt>
64<dd>Tclsh is required by DejaGNU. </dd>
65
Reid Spencer376d7ac2004-11-01 08:26:25 +0000[diff] [blame]66<dt><a href="http://www.qmtest.com">QMTest</a></dt>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]67<dd>The LLVM test suite uses QMTest to organize and run tests. <b>Note:
68you will need <a href="http://llvm.cs.uiuc.edu/qm-2.0.3.tar.gz">QMTest
692.0.3 (source tar.gz file)</a> to be successful. The tests do not run with
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]70any other version.</b> (optional, required only for QMTest)</dd>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]71
Reid Spencer376d7ac2004-11-01 08:26:25 +0000[diff] [blame]72<dt><a href="http://www.python.org">Python</a></dt>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]73<dd>You will need a Python interpreter that works with QMTest. Python will
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]74need zlib and SAX support enabled. (optional, required only for QMTest) </dd>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]75
Reid Spencer376d7ac2004-11-01 08:26:25 +0000[diff] [blame]76<dt><a href="http://www.netlib.org/f2c">F2C</a></dt>
Misha Brukman0adfeed2004-10-08 00:55:43 +0000[diff] [blame]77<dd>For now, LLVM does not have a Fortran front-end, but using F2C, we can run
78Fortran benchmarks. F2C support must be enabled via <tt>configure</tt> if not
79installed in a standard place. F2C requires three items: the <tt>f2c</tt>
80executable, <tt>f2c.h</tt> to compile the generated code, and <tt>libf2c.a</tt>
81to link generated code. By default, given an F2C directory <tt>$DIR</tt>, the
82configure script will search <tt>$DIR/bin</tt> for <tt>f2c</tt>,
83<tt>$DIR/include</tt> for <tt>f2c.h</tt>, and <tt>$DIR/lib</tt> for
84<tt>libf2c.a</tt>. The default <tt>$DIR</tt> values are: <tt>/usr</tt>,
85<tt>/usr/local</tt>, <tt>/sw</tt>, and <tt>/opt</tt>. If you installed F2C in a
86different location, you must tell <tt>configure</tt>:
87
88<ul>
89<li><tt>./configure --with-f2c=$DIR</tt><br>
90This will specify a new <tt>$DIR</tt> for the above-described search
91process. This will only work if the binary, header, and library are in their
92respective subdirectories of <tt>$DIR</tt>.</li>
93
94<li><tt>./configure --with-f2c-bin=/binary/path --with-f2c-inc=/include/path
95--with-f2c-lib=/lib/path</tt><br>
96This allows you to specify the F2C components separately. Note: if you choose
97this route, you MUST specify all three components, and you need to only specify
98<em>directories</em> where the files are located; do NOT include the
99filenames themselves on the <tt>configure</tt> line.</li>
100</ul></dd>
101</dl>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]102</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]103
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]104<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]105<div class="doc_section"><a name="quick">Quick Start</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]106<!--=========================================================================-->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]107
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]108<div class="doc_text">
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]109
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]110<p>The tests are located in two separate CVS modules. The basic feature and
111regression tests are in the main "llvm" module under the directory
112<tt>llvm/test</tt>. A more comprehensive test suite that includes whole
113programs in C and C++ is in the <tt>llvm-test</tt> module. This module should
114be checked out to the <tt>llvm/projects</tt> directory. When you
115<tt>configure</tt> the <tt>llvm</tt> module, the <tt>llvm-test</tt> module
116will be automatically configured. Or you can do it manually.</p>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]117<p>To run all of the simple tests in LLVM using DejaGNU, use the master Makefile in the
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]118<tt>llvm/test</tt> directory:</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]119<pre>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]120% gmake -C llvm/test
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]121</pre>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]122or<br>
123<pre>
124% gmake check
125</pre>
126<p>To run only a subdirectory of tests in llvm/test using DejaGNU (ie. Regression/Transforms). Just substitute the path to the subdirectory:</p>
127<pre>
128% gmake -C llvm/test TESTSUITE=Regression/Transforms
129</pre>
130<dd><b>Note: If you are running the tests with <tt>objdir != subdir</tt> you must have run the complete testsuite before you can specify a subdirectory.</b></dd>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]131
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]132<p>To run the simple tests (i.e. those that do basic testing of
133LLVM), using QMTest:</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]134<pre>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]135% gmake -C llvm/test qmtest
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]136</pre>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]137
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]138<p>To run only the basic feature tests, QMTest supports the following
139target:</p>
140<pre>
141% gmake -C llvm/test Feature.t
142</pre>
143
144<p>To run only the regression tests, QMTest supports the following
145target:</p>
146<pre>
147% gmake -C llvm/test Regression.t
148</pre>
149
150<p>To run the comprehensive test suite (tests that compile and execute whole
151programs), run the <tt>llvm-test</tt> tests:</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]152
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]153<pre>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]154% cd llvm/projects
155% cvs co llvm-test
156% cd llvm-test
157% ./configure --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT
158% gmake
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]159</pre>
160
161</div>
162
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]163<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]164<div class="doc_section"><a name="org">LLVM Test Suite Organization</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]165<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]166
167<div class="doc_text">
168
169<p>The LLVM test suite contains two major categories of tests: code
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]170fragments and whole programs. Code fragments are in the <tt>llvm</tt> module
171under the directory under the <tt>llvm/test</tt> directory. The whole programs
172test suite are n the <tt>llvm-test</tt> module under the main directory.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]173
174</div>
175
176<div class="doc_subsection"><a name="codefragments">Code Fragments</a>
177</div>
178
179<div class="doc_text">
180
181<p>Code fragments are small pieces of code that test a specific feature of LLVM
182or trigger a specific bug in LLVM. They are usually written in LLVM assembly
183language, but can be written in other languages if the test targets a particular
184language front end.</p>
185
186<p>Code fragments are not complete programs, and they are never executed to
187determine correct behavior.</p>
188
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]189<p>These code fragment tests are located in the <tt>llvm/test/Features</tt> and
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]190<tt>llvm/test/Regression</tt> directories.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]191
192</div>
193
194<div class="doc_subsection"><a name="wholeprograms">Whole Programs</a></div>
195
196<div class="doc_text">
197
198<p>Whole Programs are pieces of code which can be compiled and linked into a
199stand-alone program that can be executed. These programs are generally written
200in high level languages such as C or C++, but sometimes they are written
201straight in LLVM assembly.</p>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]202
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]203<p>These programs are compiled and then executed using several different
204methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation,
205etc). The output of these programs is compared to ensure that LLVM is compiling
206the program correctly.</p>
207
208<p>In addition to compiling and executing programs, whole program tests serve as
209a way of benchmarking LLVM performance, both in terms of the efficiency of the
210programs generated as well as the speed with which LLVM compiles, optimizes, and
211generates code.</p>
212
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]213<p>All "whole program" tests are located in the <tt>llvm-test</tt> CVS
214module.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]215
216</div>
217
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]218<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]219<div class="doc_section"><a name="tree">LLVM Test Suite Tree</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]220<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]221
222<div class="doc_text">
223
224<p>Each type of test in the LLVM test suite has its own directory. The major
225subtrees of the test suite directory tree are as follows:</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]226
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]227<ul>
228<li><tt>llvm/test/Features</tt>
229<p>This directory contains sample codes that test various features of the
230LLVM language. These pieces of sample code are run through various
231assembler, disassembler, and optimizer passes.</p>
232</li>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]233
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]234<li><tt>llvm/test/Regression</tt>
235<p>This directory contains regression tests for LLVM. When a bug is found
236in LLVM, a regression test containing just enough code to reproduce the
237problem should be written and placed somewhere underneath this directory.
238In most cases, this will be a small piece of LLVM assembly language code,
239often distilled from an actual application or benchmark.</p>
240</li>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]241
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]242<li><tt>llvm-test</tt>
243<p>The <tt>llvm-test</tt> CVS module contains programs that can be compiled
244with LLVM and executed. These programs are compiled using the native compiler
245and various LLVM backends. The output from the program compiled with the
246native compiler is assumed correct; the results from the other programs are
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]247compared to the native program output and pass if they match.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]248
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]249<p>In addition for testing correctness, the <tt>llvm-test</tt> directory also
250performs timing tests of various LLVM optimizations. It also records
251compilation times for the compilers and the JIT. This information can be
252used to compare the effectiveness of LLVM's optimizations and code
253generation.</p></li>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]254
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]255<li><tt>llvm-test/SingleSource</tt>
256<p>The SingleSource directory contains test programs that are only a single
257source file in size. These are usually small benchmark programs or small
258programs that calculate a particular value. Several such programs are grouped
259together in each directory.</p></li>
260
261<li><tt>llvm-test/MultiSource</tt>
262<p>The MultiSource directory contains subdirectories which contain entire
263programs with multiple source files. Large benchmarks and whole applications
264go here.</p></li>
265
266<li><tt>llvm-test/External</tt>
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]267<p>The External directory contains Makefiles for building code that is external
268to (i.e., not distributed with) LLVM. The most prominent members of this
269directory are the SPEC 95 and SPEC 2000 benchmark suites. The presence and
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]270location of these external programs is configured by the llvm-test
271<tt>configure</tt> script.</p></li>
272
273<li><tt>llvm/test/QMTest</tt>
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]274<p>This directory contains the QMTest information files. Inside this directory
275are QMTest administration files and the Python code that implements the LLVM
276test and database classes.</p></li>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]277
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]278</ul>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]279
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]280</div>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]281<!--=========================================================================-->
282<div class="doc_section"><a name="dgstructure">DejaGNU Structure</a></div>
283<!--=========================================================================-->
284
285<div class="doc_text">
286<p>The LLVM test suite is partially driven by DejaGNU and partially
287driven by GNU Make. Specifically, the Features and Regression tests
288are all driven by DejaGNU (and optionally QMTest). The llvm-test
289module is currently driven by a set of Makefiles.</p>
290
291<p>The DejaGNU structure is very simple, but does require some
292information to be set. This information is gathered via configure and
293is written to a file, <tt>site.exp</tt> in llvm/test. The llvm/test
294Makefile does this work for you.</p>
295
296<p>In order for DejaGNU to work, each directory of tests must have a
297<tt>dg.exp</tt> file. This file is a program written in tcl that calls
298the <tt>llvm-runtests</tt> procedure on each test file. The
299llvm-runtests procedure is defined in
300<tt>llvm/test/lib/llvm-dg.exp</tt>. Any directory that contains only
301directories does not need the <tt>dg.exp</tt> file.</p>
302
303<p>In order for a test to be run, it must contain information within
304the test file on how to run the test. These are called <tt>RUN</tt>
305lines. Run lines are specified in the comments of the test program
306using the keyword <tt>RUN</tt> followed by a colon, and lastly the
307commands to execute. These commands will be executed in a bash script,
308so any bash syntax is acceptable. You can specify as many RUN lines as
309necessary. Each RUN line translates to one line in the resulting bash
310script. Below is an example of legal RUN lines in a <tt>.ll</tt>
311file:</p>
312<pre>
313; RUN: llvm-as < %s | llvm-dis > %t1
314; RUN: llvm-dis < %s.bc-13 > %t2
315; RUN: diff %t1 %t2
316</pre>
317<p>There are a couple patterns within a <tt>RUN</tt> line that the
318llvm-runtest procedure looks for and replaces with the appropriate
319syntax:</p>
320<ul>
321<dt>%p</dt>
322<dd>The path to the source directory. This is for locating
323any supporting files that are not generated by the test, but used by
324the test.</dd>
325<dt>%s</dt>
326<dd>The test file.</dd>
327
328<dt>$t</dt>
329<dd>Temporary filename: testscript.test_filename.tmp, where
330test_filename is the name of the test file. All temporary files are
331placed in the Output directory within the directory the test is
332located.</dd>
333
334<dt>%prcontext</dt>
335<dd>Path to a script that performs grep -C. Use this since not all
336platforms support grep -C.</dd>
337
338<dt>%llvmgcc</dt> <dd>Full path to the llvmgcc executable.</dd>
339<dt>%llvmgxx</dt> <dd>Full path to the llvmg++ executable.</dd>
340</ul>
341
342<p>There are also several scripts in the llvm/test/Scripts directory
343that you might find useful when writing <tt>RUN</tt> lines.</p>
344
345<p>Lastly, you can easily mark a test that is expected to fail on a
346specific platform by using the <tt>XFAIL</tt> keyword. Xfail lines are
347specified in the comments of the test program using <tt>XFAIL</tt>,
348followed by a colon, and one or more regular expressions (separated by
349a comma) that will match against the target triplet for the
350machine. You can use * to match all targets. Here is an example of an
351<tt>XFAIL</tt> line:</p>
352<pre>
353; XFAIL: darwin,sun
354</pre>
355
356</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]357
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]358<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]359<div class="doc_section"><a name="qmstructure">QMTest Structure</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]360<!--=========================================================================-->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]361
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]362<div class="doc_text">
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]363
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]364<p>The Feature and Regression tests can also be run using QMTest.</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]365
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]366<p>The QMTest system needs to have several pieces of information available;
367these pieces of configuration information are known collectively as the
368"context" in QMTest parlance. Since the context for LLVM is relatively large,
369the master Makefile in llvm/test sets it for you.</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]370
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]371<p>The LLVM database class makes the subdirectories of llvm/test a QMTest test
372database. For each directory that contains tests driven by QMTest, it knows
373what type of test the source file is and how to run it.</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]374
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]375<p>Hence, the QMTest namespace is essentially what you see in the Feature and
376Regression directories, but there is some magic that the database class performs
377(as described below).</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]378
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]379<p>The QMTest namespace is currently composed of the following tests and test
380suites:</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]381
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]382<ul>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]383<li>Feature
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]384
385<p>These are the feature tests found in the Feature directory.
386They are broken up into the following categories:</p>
387
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]388<ul>
389 <li>ad
390 <p>Assembler/Disassembler tests. These tests verify that a piece of LLVM
391 assembly language can be assembled into bytecode and then disassembled
392 into the original assembly language code. It does this several times to
393 ensure that assembled output can be disassembled and disassembler output
394 can be assembled. It also verifies that the give assembly language file
395 can be assembled correctly.</p></li>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]396
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]397 <li>opt
398 <p>Optimizer tests. These tests verify that two of the optimizer passes
399 completely optimize a program (i.e. after a single pass, they cannot
400 optimize a program any further).</p></li>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]401
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]402 <li>mc
403 <p> Machine code tests. These tests verify that the LLVM assembly
404 language file can be translated into native assembly code.</p></li>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]405
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]406 <li>cc
407 <p>C code tests. These tests verify that the specified LLVM assembly
408 code can be converted into C source code using the C backend.</p></li>
409</ul>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]410
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]411<p>The LLVM database class looks at every file in the Feature directory and
412creates a fake test hierarchy containing
413<tt>Feature.&lt;testtype&gt;.&lt;testname&gt;</tt>. So, if you add an LLVM
414assembly language file to the Feature directory, it actually creates 5 new
415tests: assembler/disassembler, assembler, optimizer, machine code, and C code.
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]416</p></li>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]417
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]418<li>Regression
419 <p>These are the regression tests. There is one suite for each
420 subdirectory of the Regression directory. If you add a new subdirectory
421 there, you will need to modify, at least, the <tt>RegressionMap</tt>
422 variable in <tt>QMTest/llvmdb.py</tt> so that QMTest knows how to run the
423 tests in the new subdirectory.</p>
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]424</li>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]425
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]426</ul>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]427
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]428</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]429
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]430<!--=========================================================================-->
431<div class="doc_section"><a name="progstructure"><tt>llvm-test</tt>
432Structure</a></div>
433<!--=========================================================================-->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]434
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]435<div class="doc_text">
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]436
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]437<p>As mentioned previously, the <tt>llvm-test</tt> module provides three types
438of tests: MultiSource, SingleSource, and External. Each tree is then subdivided
439into several categories, including applications, benchmarks, regression tests,
440code that is strange grammatically, etc. These organizations should be
441relatively self explanatory.</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]442
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]443<p>In addition to the regular "whole program" tests, the <tt>llvm-test</tt>
444module also provides a mechanism for compiling the programs in different ways.
445If the variable TEST is defined on the gmake command line, the test system will
446include a Makefile named <tt>TEST.&lt;value of TEST variable&gt;.Makefile</tt>.
447This Makefile can modify build rules to yield different results.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]448
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]449<p>For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt> to
450create the nightly test reports. To run the nightly tests, run <tt>gmake
451TEST=nightly</tt>.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]452
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]453<p>There are several TEST Makefiles available in the tree. Some of them are
454designed for internal LLVM research and will not work outside of the LLVM
455research group. They may still be valuable, however, as a guide to writing your
456own TEST Makefile for any optimization or analysis passes that you develop with
457LLVM.</p>
Reid Spencer3281ead2004-12-08 16:52:51 +0000[diff] [blame]458
459<p>Note, when configuring the <tt>llvm-test</tt> module, you might want to
460specify the following configuration options:</p>
461<dl>
462 <dt><i>--enable-spec2000</i>
463 <dt><i>--enable-spec2000=&lt;<tt>directory</tt>&gt;</i>
464 <dd>
465 Enable the use of SPEC2000 when testing LLVM. This is disabled by default
466 (unless <tt>configure</tt> finds SPEC2000 installed). By specifying
467 <tt>directory</tt>, you can tell configure where to find the SPEC2000
468 benchmarks. If <tt>directory</tt> is left unspecified, <tt>configure</tt>
469 uses the default value
470 <tt>/home/vadve/shared/benchmarks/speccpu2000/benchspec</tt>.
471 <p>
472 <dt><i>--enable-spec95</i>
473 <dt><i>--enable-spec95=&lt;<tt>directory</tt>&gt;</i>
474 <dd>
475 Enable the use of SPEC95 when testing LLVM. It is similar to the
476 <i>--enable-spec2000</i> option.
477 <p>
478 <dt><i>--enable-povray</i>
479 <dt><i>--enable-povray=&lt;<tt>directory</tt>&gt;</i>
480 <dd>
481 Enable the use of Povray as an external test. Versions of Povray written
482 in C should work. This option is similar to the <i>--enable-spec2000</i>
483 option.
484</dl>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]485</div>
486
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]487<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]488<div class="doc_section"><a name="run">Running the LLVM Tests</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]489<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]490
491<div class="doc_text">
492
493<p>First, all tests are executed within the LLVM object directory tree. They
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]494<i>are not</i> executed inside of the LLVM source tree. This is because the
495test suite creates temporary files during execution.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]496
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]497<p>The master Makefile in llvm/test is capable of running only the QMTest driven
498tests. By default, it will run all of these tests.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]499
500<p>To run only the QMTest driven tests, run <tt>gmake qmtest</tt> at the
501command line in llvm/tests. To run a specific qmtest, suffix the test name with
502".t" when running gmake.</p>
503
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]504<p>For example, to run the Regression.LLC tests, type
505<tt>gmake Regression.LLC.t</tt> in <tt>llvm/tests</tt>.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]506
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]507<p>Note that there are no Makefiles in <tt>llvm/test/Features</tt> and
508<tt>llvm/test/Regression</tt>. You must use QMTest from the <tt>llvm/test</tt>
509directory to run them.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]510
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]511<p>To run the <tt>llvm-test</tt> suite, you need to use the following steps:
512</p>
513<ol>
514 <li>cd into the llvm/projects directory</li>
515 <li>check out the <tt>llvm-test</tt> module with:<br/>
516 <tt>cvs -d :pserver:anon@llvm.cs.uiuc.edu:/var/cvs/llvm co -PR llvm-test</tt><br>
517 This will get the test suite into <tt>llvm/projects/llvm-test</tt></li>
518 <li>configure the test suite. You can do this one of two ways:
519 <ol>
520 <li>Use the regular llvm configure:<br/>
521 <tt>cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure</tt><br/>
522 This will ensure that the <tt>projects/llvm-test</tt> directory is also
523 properly configured.</li>
524 <li>Use the <tt>configure</tt> script found in the <tt>llvm-test</tt> source
525 directory:<br/>
526 <tt>$BUILD_SRC_DIR/configure --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT</tt>
527 </li>
528 </ol>
529 <li>gmake</li>
530</ol>
531<p>Note that the second and third steps only need to be done once. After you
532have the suite checked out and configured, you don't need to do it again (unless
533the test code or configure script changes).</p>
534
535<p>To make a specialized test (use one of the
536<tt>llvm-test/TEST.&lt;type&gt;.Makefile</tt>s), just run:<br/>
537<tt>gmake TEST=&lt;type&gt; test</tt><br/>For example, you could run the
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]538nightly tester tests using the following commands:</p>
539
540<pre>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]541 % cd llvm/projects/llvm-test
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]542 % gmake TEST=nightly test
543</pre>
544
545<p>Regardless of which test you're running, the results are printed on standard
546output and standard error. You can redirect these results to a file if you
547choose.</p>
548
549<p>Some tests are known to fail. Some are bugs that we have not fixed yet;
550others are features that we haven't added yet (or may never add). In QMTest,
551the result for such tests will be XFAIL (eXpected FAILure). In this way, you
552can tell the difference between an expected and unexpected failure.</p>
553
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]554<p>The tests in <tt>llvm-test</tt> have no such feature as of this time. If the
555test passes, only warnings and other miscellaneous output will be generated. If
556a test fails, a large &lt;program&gt; FAILED message will be displayed. This
557will help you separate benign warnings from actual test failures.</p>
Chris Lattnerfd9d1b32004-06-24 20:53:09 +0000[diff] [blame]558
559</div>
560
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]561<!--=========================================================================-->
Chris Lattnerfd9d1b32004-06-24 20:53:09 +0000[diff] [blame]562<div class="doc_section"><a name="nightly">Running the nightly tester</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]563<!--=========================================================================-->
Chris Lattnerfd9d1b32004-06-24 20:53:09 +0000[diff] [blame]564
565<div class="doc_text">
566
567<p>
568The <a href="http://llvm.cs.uiuc.edu/testresults/">LLVM Nightly Testers</a>
569automatically check out an LLVM tree, build it, run the "nightly"
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]570program test (described above), run all of the feature and regression tests,
571and then delete the checked out tree. This tester is designed to ensure that
Chris Lattnerfd9d1b32004-06-24 20:53:09 +0000[diff] [blame]572programs don't break as well as keep track of LLVM's progress over time.</p>
573
574<p>
575If you'd like to set up an instance of the nightly tester to run on your
576machine, take a look at the comments at the top of the utils/NightlyTester.pl
577file. We usually run it from a crontab entry that looks ilke this:
578</p>
579
580<pre>
5815 3 * * * LLVM_LIB_SEARCH_PATH=.../llvm-gcc/bytecode-libs $HOME/llvm/utils/NightlyTest.pl -parallel -enable-linscan ...CVSREPOSTRING... $HOME/buildtest-X86 $HOME/cvs/testresults-X86
582</pre>
583
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]584<p>Or, you can create a shell script to encapsulate the running of the script.
585The optimized x86 Linux nightly test is run from just such a script:
586<pre>
587#!/bin/bash
588BASE=/proj/work/llvm/nightlytest
589export CVSROOT=:pserver:anon@llvm.cs.uiuc.edu:/var/cvs/llvm
590export BUILDDIR=$BASE/build
591export WEBDIR=$BASE/testresults
592export LLVMGCCDIR=/proj/work/llvm/cfrontend/install
593export PATH=/proj/install/bin:$LLVMGCCDIR/bin:$PATH
594export LD_LIBRARY_PATH=/proj/install/lib
595export LLVM_LIB_SEARCH_PATH=/proj/work/llvm/cfrontend/install/bytecode-libs
596cd $BASE
597cp /proj/work/llvm/llvm/utils/NightlyTest.pl .
598nice ./NightlyTest.pl -nice -release -verbose -parallel -enable-linscan -noexternals
599</pre>
600
Chris Lattnerfd9d1b32004-06-24 20:53:09 +0000[diff] [blame]601<p>
602Take a look at the NightlyTest.pl file to see what all of the flags and
603strings do. If you start running the nightly tests, please let us know and
604we'll link your page to the global tester page. Thanks!
605</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]606
607</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]608
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]609<!-- *********************************************************************** -->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]610
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]611<hr>
612<address>
613 <a href="http://jigsaw.w3.org/css-validator/check/referer"><img
614 src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!"></a>
615 <a href="http://validator.w3.org/check/referer"><img
616 src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!" /></a>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]617
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]618 John T. Criswell<br>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]619 <a href="http://llvm.cs.uiuc.edu">The LLVM Compiler Infrastructure</a><br/>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]620 Last modified: $Date$
621</address>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]622</body>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]623</html>