Gitiles
Code Review Sign In
gerrit-public.fairphone.software / fp2-dev / platform / external / llvm / eb6aaceb9437872036c7864587f603453091c380 / . / docs / TestingGuide.html
blob: ab4628a080723bf080300d00407670fea3cf2796 [file] [log] [blame]
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
2 "http://www.w3.org/TR/html4/strict.dtd">
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]3<html>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]4<head>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]5 <title>LLVM Test Suite Guide</title>
6 <link rel="stylesheet" href="llvm.css" type="text/css">
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]7</head>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]8<body>
9
10<div class="doc_title">
11 LLVM Test Suite Guide
12</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]13
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]14<ol>
Reid Spencer820e2472004-11-01 08:30:14 +0000[diff] [blame]15 <li><a href="#overview">Overview</a></li>
16 <li><a href="#Requirements">Requirements</a></li>
17 <li><a href="#quick">Quick Start</a></li>
18 <li><a href="#org">LLVM Test Suite Organization</a>
19 <ul>
20 <li><a href="#codefragments">Code Fragments</a></li>
21 <li><a href="#wholeprograms">Whole Programs</a></li>
22 </ul>
23 </li>
24 <li><a href="#tree">LLVM Test Suite Tree</a></li>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]25 <li><a href="#dgstructure">DejaGNU Structure</a></li>
Reid Spencer820e2472004-11-01 08:30:14 +0000[diff] [blame]26 <li><a href="#progstructure"><tt>llvm-test</tt> Structure</a></li>
Chris Lattnereb82da82006-05-23 01:40:20 +0000[diff] [blame]27 <li><a href="#run">Running the LLVM Tests</a>
28 <ul>
29 <li><a href="#customtest">Writing custom tests for llvm-test</a></li>
30 </ul>
31 </li>
Reid Spencer820e2472004-11-01 08:30:14 +0000[diff] [blame]32 <li><a href="#nightly">Running the nightly tester</a></li>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]33</ol>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]34
Chris Lattner7911ce22004-05-23 21:07:27 +0000[diff] [blame]35<div class="doc_author">
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]36 <p>Written by John T. Criswell, <a
37 href="http://llvm.x10sys.com/rspencer">Reid Spencer</a>, and Tanya Lattner</p>
Chris Lattner7911ce22004-05-23 21:07:27 +0000[diff] [blame]38</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]39
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]40<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]41<div class="doc_section"><a name="overview">Overview</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]42<!--=========================================================================-->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]43
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]44<div class="doc_text">
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]45
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]46<p>This document is the reference manual for the LLVM test suite. It documents
47the structure of the LLVM test suite, the tools needed to use it, and how to add
48and run tests.</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]49
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]50</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]51
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]52<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]53<div class="doc_section"><a name="Requirements">Requirements</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]54<!--=========================================================================-->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]55
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]56<div class="doc_text">
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]57
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]58<p>In order to use the LLVM test suite, you will need all of the software
59required to build LLVM, plus the following:</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]60
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]61<dl>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]62<dt><a href="http://www.gnu.org/software/dejagnu/">DejaGNU</a></dt>
63<dd>The Feature and Regressions tests are organized and run by DejaGNU.</dd>
64<dt><a href="http://expect.nist.gov/">Expect</a></dt>
65<dd>Expect is required by DejaGNU.</dd>
Tanya Lattnercc85da02004-12-08 17:35:31 +0000[diff] [blame]66<dt><a href="http://www.tcl.tk/software/tcltk/">tcl</a></dt>
67<dd>Tcl is required by DejaGNU. </dd>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]68
Reid Spencer376d7ac2004-11-01 08:26:25 +0000[diff] [blame]69<dt><a href="http://www.netlib.org/f2c">F2C</a></dt>
Misha Brukman0adfeed2004-10-08 00:55:43 +0000[diff] [blame]70<dd>For now, LLVM does not have a Fortran front-end, but using F2C, we can run
71Fortran benchmarks. F2C support must be enabled via <tt>configure</tt> if not
72installed in a standard place. F2C requires three items: the <tt>f2c</tt>
73executable, <tt>f2c.h</tt> to compile the generated code, and <tt>libf2c.a</tt>
74to link generated code. By default, given an F2C directory <tt>$DIR</tt>, the
75configure script will search <tt>$DIR/bin</tt> for <tt>f2c</tt>,
76<tt>$DIR/include</tt> for <tt>f2c.h</tt>, and <tt>$DIR/lib</tt> for
77<tt>libf2c.a</tt>. The default <tt>$DIR</tt> values are: <tt>/usr</tt>,
78<tt>/usr/local</tt>, <tt>/sw</tt>, and <tt>/opt</tt>. If you installed F2C in a
79different location, you must tell <tt>configure</tt>:
80
81<ul>
82<li><tt>./configure --with-f2c=$DIR</tt><br>
83This will specify a new <tt>$DIR</tt> for the above-described search
84process. This will only work if the binary, header, and library are in their
85respective subdirectories of <tt>$DIR</tt>.</li>
86
87<li><tt>./configure --with-f2c-bin=/binary/path --with-f2c-inc=/include/path
88--with-f2c-lib=/lib/path</tt><br>
89This allows you to specify the F2C components separately. Note: if you choose
90this route, you MUST specify all three components, and you need to only specify
91<em>directories</em> where the files are located; do NOT include the
92filenames themselves on the <tt>configure</tt> line.</li>
93</ul></dd>
94</dl>
Jim Laskeya2dee012006-03-27 18:41:06 +0000[diff] [blame]95
Jim Laskey5f4eaa52006-03-27 19:33:35 +0000[diff] [blame]96<p>Darwin (Mac OS X) developers can simplify the installation of Expect and tcl
97by using fink. <tt>fink install expect</tt> will install both. Alternatively,
Jim Laskey30c495e2006-03-27 19:46:38 +0000[diff] [blame]98Darwinports users can use <tt>sudo port install expect</tt> to install Expect
99and tcl.</p>
Jim Laskeyf8e78f02006-03-27 19:09:44 +0000[diff] [blame]100
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]101</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]102
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]103<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]104<div class="doc_section"><a name="quick">Quick Start</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]105<!--=========================================================================-->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]106
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]107<div class="doc_text">
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]108
Reid Spencer669ed452007-07-09 08:04:31 +0000[diff] [blame]109 <p>The tests are located in two separate Subversion modules. The basic feature
110 and regression tests are in the main "llvm" module under the directory
111 <tt>llvm/test</tt>. A more comprehensive test suite that includes whole
112programs in C and C++ is in the <tt>test-suite</tt> module. This module should
113be checked out to the <tt>llvm/projects</tt> directory as llvm-test (for
114historical purpose). When you <tt>configure</tt> the <tt>llvm</tt> module,
115the <tt>llvm-test</tt> directory will be automatically configured.
116Alternatively, you can configure the <tt>test-suite</tt> module manually.</p>
Chris Lattner792321a2006-05-23 01:25:11 +0000[diff] [blame]117<p>To run all of the simple tests in LLVM using DejaGNU, use the master Makefile
118 in the <tt>llvm/test</tt> directory:</p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]119
120<div class="doc_code">
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]121<pre>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]122% gmake -C llvm/test
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]123</pre>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]124</div>
125
126<p>or</p>
127
128<div class="doc_code">
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]129<pre>
130% gmake check
131</pre>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]132</div>
John Criswell0c8a9a12005-05-13 20:25:49 +0000[diff] [blame]133
134<p>To run only a subdirectory of tests in llvm/test using DejaGNU (ie.
135Regression/Transforms), just set the TESTSUITE variable to the path of the
136subdirectory (relative to <tt>llvm/test</tt>):</p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]137
138<div class="doc_code">
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]139<pre>
140% gmake -C llvm/test TESTSUITE=Regression/Transforms
141</pre>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]142</div>
Misha Brukmanfc1a27b2005-03-10 22:51:59 +0000[diff] [blame]143
John Criswell0c8a9a12005-05-13 20:25:49 +0000[diff] [blame]144<p><b>Note: If you are running the tests with <tt>objdir != subdir</tt>, you
145must have run the complete testsuite before you can specify a
146subdirectory.</b></p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]147
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]148<p>To run the comprehensive test suite (tests that compile and execute whole
149programs), run the <tt>llvm-test</tt> tests:</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]150
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]151<div class="doc_code">
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]152<pre>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]153% cd llvm/projects
Reid Spencer669ed452007-07-09 08:04:31 +0000[diff] [blame]154% svn co http://llvm.org/svn/llvm-project/test-suite/trunk llvm-test
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]155% cd llvm-test
156% ./configure --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT
157% gmake
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]158</pre>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]159</div>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]160
161</div>
162
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]163<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]164<div class="doc_section"><a name="org">LLVM Test Suite Organization</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]165<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]166
167<div class="doc_text">
168
169<p>The LLVM test suite contains two major categories of tests: code
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]170fragments and whole programs. Code fragments are in the <tt>llvm</tt> module
John Criswell0c8a9a12005-05-13 20:25:49 +0000[diff] [blame]171under the <tt>llvm/test</tt> directory. The whole programs
172test suite is in the <tt>llvm-test</tt> module under the main directory.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]173
174</div>
175
Chris Lattnereb82da82006-05-23 01:40:20 +0000[diff] [blame]176<!-- _______________________________________________________________________ -->
177<div class="doc_subsection"><a name="codefragments">Code Fragments</a></div>
178<!-- _______________________________________________________________________ -->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]179
180<div class="doc_text">
181
182<p>Code fragments are small pieces of code that test a specific feature of LLVM
183or trigger a specific bug in LLVM. They are usually written in LLVM assembly
184language, but can be written in other languages if the test targets a particular
185language front end.</p>
186
187<p>Code fragments are not complete programs, and they are never executed to
188determine correct behavior.</p>
189
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]190<p>These code fragment tests are located in the <tt>llvm/test/Features</tt> and
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]191<tt>llvm/test/Regression</tt> directories.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]192
193</div>
194
Chris Lattnereb82da82006-05-23 01:40:20 +0000[diff] [blame]195<!-- _______________________________________________________________________ -->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]196<div class="doc_subsection"><a name="wholeprograms">Whole Programs</a></div>
Chris Lattnereb82da82006-05-23 01:40:20 +0000[diff] [blame]197<!-- _______________________________________________________________________ -->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]198
199<div class="doc_text">
200
201<p>Whole Programs are pieces of code which can be compiled and linked into a
202stand-alone program that can be executed. These programs are generally written
203in high level languages such as C or C++, but sometimes they are written
204straight in LLVM assembly.</p>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]205
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]206<p>These programs are compiled and then executed using several different
207methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation,
208etc). The output of these programs is compared to ensure that LLVM is compiling
209the program correctly.</p>
210
211<p>In addition to compiling and executing programs, whole program tests serve as
212a way of benchmarking LLVM performance, both in terms of the efficiency of the
213programs generated as well as the speed with which LLVM compiles, optimizes, and
214generates code.</p>
215
Reid Spencer669ed452007-07-09 08:04:31 +0000[diff] [blame]216<p>All "whole program" tests are located in the <tt>test-suite</tt> Subversion
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]217module.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]218
219</div>
220
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]221<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]222<div class="doc_section"><a name="tree">LLVM Test Suite Tree</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]223<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]224
225<div class="doc_text">
226
227<p>Each type of test in the LLVM test suite has its own directory. The major
228subtrees of the test suite directory tree are as follows:</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]229
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]230<ul>
Reid Spencera99256e2007-02-08 17:00:55 +0000[diff] [blame]231 <li><tt>llvm/test</tt>
232 <p>This directory contains a large array of small tests
233 that exercise various features of LLVM and to ensure that regressions do not
234 occur. The directory is broken into several sub-directories, each focused on
Bill Wendlingeb6aace2007-09-22 09:20:07 +0000[diff] [blame^]235 a particular area of LLVM. A few of the important ones are:</p>
236 <ul>
Reid Spencera99256e2007-02-08 17:00:55 +0000[diff] [blame]237 <li><tt>Analysis</tt>: checks Analysis passes.</li>
238 <li><tt>Archive</tt>: checks the Archive library.</li>
239 <li><tt>Assembler</tt>: checks Assembly reader/writer functionality.</li>
Gabor Greif04367bf2007-07-06 22:07:22 +0000[diff] [blame]240 <li><tt>Bitcode</tt>: checks Bitcode reader/writer functionality.</li>
Reid Spencera99256e2007-02-08 17:00:55 +0000[diff] [blame]241 <li><tt>CodeGen</tt>: checks code generation and each target.</li>
242 <li><tt>Features</tt>: checks various features of the LLVM language.</li>
Gabor Greif04367bf2007-07-06 22:07:22 +0000[diff] [blame]243 <li><tt>Linker</tt>: tests bitcode linking.</li>
Reid Spencera99256e2007-02-08 17:00:55 +0000[diff] [blame]244 <li><tt>Transforms</tt>: tests each of the scalar, IPO, and utility
245 transforms to ensure they make the right transformations.</li>
246 <li><tt>Verifier</tt>: tests the IR verifier.</li>
Bill Wendlingeb6aace2007-09-22 09:20:07 +0000[diff] [blame^]247 </ul>
Reid Spencera99256e2007-02-08 17:00:55 +0000[diff] [blame]248 <p>Typically when a bug is found in LLVM, a regression test containing
249 just enough code to reproduce the problem should be written and placed
250 somewhere underneath this directory. In most cases, this will be a small
251 piece of LLVM assembly language code, often distilled from an actual
252 application or benchmark.</p></li>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]253
Reid Spencer669ed452007-07-09 08:04:31 +0000[diff] [blame]254<li><tt>test-suite</tt>
255<p>The <tt>test-suite</tt> module contains programs that can be compiled
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]256with LLVM and executed. These programs are compiled using the native compiler
257and various LLVM backends. The output from the program compiled with the
258native compiler is assumed correct; the results from the other programs are
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]259compared to the native program output and pass if they match.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]260
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]261<p>In addition for testing correctness, the <tt>llvm-test</tt> directory also
262performs timing tests of various LLVM optimizations. It also records
263compilation times for the compilers and the JIT. This information can be
264used to compare the effectiveness of LLVM's optimizations and code
265generation.</p></li>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]266
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]267<li><tt>llvm-test/SingleSource</tt>
268<p>The SingleSource directory contains test programs that are only a single
269source file in size. These are usually small benchmark programs or small
270programs that calculate a particular value. Several such programs are grouped
271together in each directory.</p></li>
272
273<li><tt>llvm-test/MultiSource</tt>
274<p>The MultiSource directory contains subdirectories which contain entire
275programs with multiple source files. Large benchmarks and whole applications
276go here.</p></li>
277
278<li><tt>llvm-test/External</tt>
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]279<p>The External directory contains Makefiles for building code that is external
280to (i.e., not distributed with) LLVM. The most prominent members of this
281directory are the SPEC 95 and SPEC 2000 benchmark suites. The presence and
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]282location of these external programs is configured by the llvm-test
283<tt>configure</tt> script.</p></li>
284
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]285</ul>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]286
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]287</div>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]288<!--=========================================================================-->
289<div class="doc_section"><a name="dgstructure">DejaGNU Structure</a></div>
290<!--=========================================================================-->
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]291<div class="doc_text">
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]292 <p>The LLVM test suite is partially driven by DejaGNU and partially driven by
293 GNU Make. Specifically, the Features and Regression tests are all driven by
294 DejaGNU. The <tt>llvm-test</tt> module is currently driven by a set of
295 Makefiles.</p>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]296
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]297 <p>The DejaGNU structure is very simple, but does require some information to
298 be set. This information is gathered via <tt>configure</tt> and is written
299 to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The <tt>llvm/test</tt>
300 Makefile does this work for you.</p>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]301
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]302 <p>In order for DejaGNU to work, each directory of tests must have a
303 <tt>dg.exp</tt> file. DejaGNU looks for this file to determine how to run the
304 tests. This file is just a Tcl script and it can do anything you want, but
305 we've standardized it for the LLVM regression tests. It simply loads a Tcl
306 library (<tt>test/lib/llvm.exp</tt>) and calls the <tt>llvm_runtests</tt>
307 function defined in that library with a list of file names to run. The names
308 are obtained by using Tcl's glob command. Any directory that contains only
309 directories does not need the <tt>dg.exp</tt> file.</p>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]310
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]311 <p>The <tt>llvm-runtests</tt> function lookas at each file that is passed to
312 it and gathers any lines together that match "RUN:". This are the "RUN" lines
313 that specify how the test is to be run. So, each test script must contain
314 RUN lines if it is to do anything. If there are no RUN lines, the
315 <tt>llvm-runtests</tt> function will issue an error and the test will
316 fail.</p>
Misha Brukmanfc1a27b2005-03-10 22:51:59 +0000[diff] [blame]317
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]318 <p>RUN lines are specified in the comments of the test program using the
319 keyword <tt>RUN</tt> followed by a colon, and lastly the command (pipeline)
320 to execute. Together, these lines form the "script" that
321 <tt>llvm-runtests</tt> executes to run the test case. The syntax of the
322 RUN lines is similar to a shell's syntax for pipelines including I/O
323 redirection and variable substitution. However, even though these lines
324 may <i>look</i> like a shell script, they are not. RUN lines are interpreted
325 directly by the Tcl <tt>exec</tt> command. They are never executed by a
326 shell. Consequently the syntax differs from normal shell script syntax in a
327 few ways. You can specify as many RUN lines as needed.</p>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]328
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]329 <p>Each RUN line is executed on its own, distinct from other lines unless
330 its last character is <tt>\</tt>. This continuation character causes the RUN
331 line to be concatenated with the next one. In this way you can build up long
332 pipelines of commands without making huge line lengths. The lines ending in
333 <tt>\</tt> are concatenated until a RUN line that doesn't end in <tt>\</tt> is
334 found. This concatenated set or RUN lines then constitutes one execution.
335 Tcl will substitute variables and arrange for the pipeline to be executed. If
336 any process in the pipeline fails, the entire line (and test case) fails too.
337 </p>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]338
Reid Spencerf1902772007-04-15 08:01:04 +0000[diff] [blame]339 <p> Below is an example of legal RUN lines in a <tt>.ll</tt> file:</p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]340
341<div class="doc_code">
342<pre>
343; RUN: llvm-as &lt; %s | llvm-dis &gt; %t1
344; RUN: llvm-dis &lt; %s.bc-13 &gt; %t2
345; RUN: diff %t1 %t2
346</pre>
347</div>
Reid Spencerf1902772007-04-15 08:01:04 +0000[diff] [blame]348
Reid Spencer024a1262007-04-14 23:27:06 +0000[diff] [blame]349 <p>As with a Unix shell, the RUN: lines permit pipelines and I/O redirection
350 to be used. However, the usage is slightly different than for Bash. To check
351 what's legal, see the documentation for the
352 <a href="http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M2">Tcl exec</a>
353 command and the
354 <a href="http://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html">tutorial</a>.
355 The major differences are:</p>
356 <ul>
357 <li>You can't do <tt>2&gt;&amp;1</tt>. That will cause Tcl to write to a
358 file named <tt>&amp;1</tt>. Usually this is done to get stderr to go through
359 a pipe. You can do that in tcl with <tt>|&amp;</tt> so replace this idiom:
360 <tt>... 2&gt;&amp;1 | grep</tt> with <tt>... |&amp; grep</tt></li>
361 <li>You can only redirect to a file, not to another descriptor and not from
362 a here document.</li>
363 <li>tcl supports redirecting to open files with the @ syntax but you
364 shouldn't use that here.</li>
365 </ul>
366
Reid Spencerf1902772007-04-15 08:01:04 +0000[diff] [blame]367 <p>There are some quoting rules that you must pay attention to when writing
368 your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any
369 ' or " so they will get passed to the invoked program. For example:</p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]370
371<div class="doc_code">
372<pre>
373... | grep 'find this string'
374</pre>
375</div>
376
Reid Spencerf1902772007-04-15 08:01:04 +0000[diff] [blame]377 <p>This will fail because the ' characters are passed to grep. This would
378 instruction grep to look for <tt>'find</tt> in the files <tt>this</tt> and
379 <tt>string'</tt>. To avoid this use curly braces to tell Tcl that it should
380 treat everything enclosed as one value. So our example would become:</p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]381
382<div class="doc_code">
383<pre>
384... | grep {find this string}
385</pre>
386</div>
387
Reid Spencerf1902772007-04-15 08:01:04 +0000[diff] [blame]388 <p>Additionally, the characters <tt>[</tt> and <tt>]</tt> are treated
389 specially by Tcl. They tell Tcl to interpret the content as a command to
390 execute. Since these characters are often used in regular expressions this can
391 have disastrous results and cause the entire test run in a directory to fail.
392 For example, a common idiom is to look for some basicblock number:</p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]393
394<div class="doc_code">
395<pre>
396... | grep bb[2-8]
397</pre>
398</div>
399
Reid Spencerf1902772007-04-15 08:01:04 +0000[diff] [blame]400 <p>This, however, will cause Tcl to fail because its going to try to execute
401 a program named "2-8". Instead, what you want is this:</p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]402
403<div class="doc_code">
404<pre>
405... | grep {bb\[2-8\]}
406</pre>
407</div>
408
Reid Spencerf1902772007-04-15 08:01:04 +0000[diff] [blame]409 <p>Finally, if you need to pass the <tt>\</tt> character down to a program,
410 then it must be doubled. This is another Tcl special character. So, suppose
411 you had:
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]412
413<div class="doc_code">
414<pre>
415... | grep 'i32\*'
416</pre>
417</div>
418
Reid Spencerf1902772007-04-15 08:01:04 +0000[diff] [blame]419 <p>This will fail to match what you want (a pointer to i32). First, the
420 <tt>'</tt> do not get stripped off. Second, the <tt>\</tt> gets stripped off
421 by Tcl so what grep sees is: <tt>'i32*'</tt>. That's not likely to match
422 anything. To resolve this you must use <tt>\\</tt> and the <tt>{}</tt>, like
423 this:</p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]424
425<div class="doc_code">
426<pre>
427... | grep {i32\\*}
428</pre>
429</div>
Reid Spencerf1902772007-04-15 08:01:04 +0000[diff] [blame]430
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]431</div>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]432
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]433<!-- _______________________________________________________________________ -->
434<div class="doc_subsection"><a name="dgvars">Vars And Substitutions</a></div>
435<div class="doc_text">
436 <p>With a RUN line there are a number of substitutions that are permitted. In
437 general, any Tcl variable that is available in the <tt>substitute</tt>
438 function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line.
439 To make a substitution just write the variable's name preceded by a $.
440 Additionally, for compatibility reasons with previous versions of the test
441 library, certain names can be accessed with an alternate syntax: a % prefix.
442 These alternates are deprecated and may go away in a future version.
443 </p>
Bill Wendlingeb6aace2007-09-22 09:20:07 +0000[diff] [blame^]444 <p>Here are the available variable names. The alternate syntax is listed in
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]445 parentheses.</p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]446
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]447 <dl style="margin-left: 25px">
448 <dt><b>$test</b> (%s)</dt>
449 <dd>The full path to the test case's source. This is suitable for passing
450 on the command line as the input to an llvm tool.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]451
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]452 <dt><b>$srcdir</b></dt>
453 <dd>The source directory from where the "<tt>make check</tt>" was run.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]454
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]455 <dt><b>objdir</b></dt>
Bill Wendlingeb6aace2007-09-22 09:20:07 +0000[diff] [blame^]456 <dd>The object directory that corresponds to the <tt>$srcdir</tt>.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]457
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]458 <dt><b>subdir</b></dt>
459 <dd>A partial path from the <tt>test</tt> directory that contains the
460 sub-directory that contains the test source being executed.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]461
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]462 <dt><b>srcroot</b></dt>
463 <dd>The root directory of the LLVM src tree.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]464
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]465 <dt><b>objroot</b></dt>
466 <dd>The root directory of the LLVM object tree. This could be the same
467 as the srcroot.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]468
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]469 <dt><b>path</b><dt>
470 <dd>The path to the directory that contains the test case source. This is
471 for locating any supporting files that are not generated by the test, but
472 used by the test.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]473
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]474 <dt><b>tmp</b></dt>
475 <dd>The path to a temporary file name that could be used for this test case.
476 The file name won't conflict with other test cases. You can append to it if
477 you need multiple temporaries. This is useful as the destination of some
478 redirected output.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]479
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]480 <dt><b>llvmlibsdir</b> (%llvmlibsdir)</dt>
481 <dd>The directory where the LLVM libraries are located.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]482
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]483 <dt><b>target_triplet</b> (%target_triplet)</dt>
484 <dd>The target triplet that corresponds to the current host machine (the one
485 running the test cases). This should probably be called "host".<dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]486
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]487 <dt><b>prcontext</b> (%prcontext)</dt>
488 <dd>Path to the prcontext tcl script that prints some context around a
489 line that matches a pattern. This isn't strictly necessary as the test suite
490 is run with its PATH altered to include the test/Scripts directory where
491 the prcontext script is located. Note that this script is similar to
492 <tt>grep -C</tt> but you should use the <tt>prcontext</tt> script because
493 not all platforms support <tt>grep -C</tt>.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]494
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]495 <dt><b>llvmgcc</b> (%llvmgcc)</dt>
496 <dd>The full path to the <tt>llvm-gcc</tt> executable as specified in the
497 configured LLVM environment</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]498
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]499 <dt><b>llvmgxx</b> (%llvmgxx)</dt>
500 <dd>The full path to the <tt>llvm-gxx</tt> executable as specified in the
501 configured LLVM environment</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]502
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]503 <dt><b>llvmgcc_version</b> (%llvmgcc_version)</dt>
504 <dd>The full version number of the <tt>llvm-gcc</tt> executable.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]505
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]506 <dt><b>llvmgccmajvers</b> (%llvmgccmajvers)</dt>
507 <dd>The major version number of the <tt>llvm-gcc</tt> executable.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]508
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]509 <dt><b>gccpath</b></dt>
510 <dd>The full path to the C compiler used to <i>build </i> LLVM. Note that
511 this might not be gcc.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]512
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]513 <dt><b>gxxpath</b></dt>
514 <dd>The full path to the C++ compiler used to <i>build </i> LLVM. Note that
515 this might not be g++.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]516
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]517 <dt><b>compile_c</b> (%compile_c)</dt>
518 <dd>The full command line used to compile LLVM C source code. This has all
519 the configured -I, -D and optimization options.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]520
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]521 <dt><b>compile_cxx</b> (%compile_cxx)</dt>
522 <dd>The full command used to compile LLVM C++ source code. This has
523 all the configured -I, -D and optimization options.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]524
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]525 <dt><b>link</b> (%link)</dt>
526 <dd>This full link command used to link LLVM executables. This has all the
527 configured -I, -L and -l options.</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]528
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]529 <dt><b>shlibext</b> (%shlibext)</dt>
530 <dd>The suffix for the host platforms share library (dll) files. This
531 includes the period as the first character.</dd>
532 </dl>
533 <p>To add more variables, two things need to be changed. First, add a line in
534 the <tt>test/Makefile</tt> that creates the <tt>site.exp</tt> file. This will
535 "set" the variable as a global in the site.exp file. Second, in the
536 <tt>test/lib/llvm.exp</tt> file, in the substitute proc, add the variable name
537 to the list of "global" declarations at the beginning of the proc. That's it,
538 the variable can then be used in test scripts.</p>
539</div>
540
541<!-- _______________________________________________________________________ -->
542<div class="doc_subsection"><a name="dgfeatures">Other Features</a></div>
543<div class="doc_text">
544 <p>To make RUN line writing easier, there are several shell scripts located
545 in the <tt>llvm/test/Scripts</tt> directory. For example:</p>
546 <dl>
547 <dt><b>ignore</b></dt>
548 <dd>This script runs its arguments and then always returns 0. This is useful
549 in cases where the test needs to cause a tool to generate an error (e.g. to
550 check the error output). However, any program in a pipeline that returns a
551 non-zero result will cause the test to fail. This script overcomes that
552 issue and nicely documents that the test case is purposefully ignoring the
553 result code of the tool</dd>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]554
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]555 <dt><b>not</b></dt>
556 <dd>This script runs its arguments and then inverts the result code from
557 it. Zero result codes become 1. Non-zero result codes become 0. This is
558 useful to invert the result of a grep. For example "not grep X" means
559 succeed only if you don't find X in the input.</dd>
560 </dl>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]561
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]562 <p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL.
563 You can easily mark a test as XFAIL just by including <tt>XFAIL: </tt> on a
564 line near the top of the file. This signals that the test case should succeed
565 if the test fails. Such test cases are counted separately by DejaGnu. To
566 specify an expected fail, use the XFAIL keyword in the comments of the test
567 program followed by a colon and one or more regular expressions (separated by
568 a comma). The regular expressions allow you to XFAIL the test conditionally
569 by host platform. The regular expressions following the : are matched against
570 the target triplet or llvmgcc version number for the host machine. If there is
571 a match, the test is expected to fail. If not, the test is expected to
572 succeed. To XFAIL everywhere just specify <tt>XFAIL: *</tt>. When matching
573 the llvm-gcc version, you can specify the major (e.g. 3) or full version
574 (i.e. 3.4) number. Here is an example of an <tt>XFAIL</tt> line:</p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]575
576<div class="doc_code">
577<pre>
578; XFAIL: darwin,sun,llvmgcc4
579</pre>
580</div>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]581
Reid Spencerbbb2a7a2007-04-14 21:46:15 +0000[diff] [blame]582 <p>To make the output more useful, the <tt>llvm_runtest</tt> function wil
583 scan the lines of the test case for ones that contain a pattern that matches
584 PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that
585 is related to the test case. The numer after "PR" specifies the LLVM bugzilla
586 number. When a PR number is specified, it will be used in the pass/fail
587 reporting. This is useful to quickly get some context when a test fails.</p>
588
589 <p>Finally, any line that contains "END." will cause the special
590 interpretation of lines to terminate. This is generally done right after the
591 last RUN: line. This has two side effects: (a) it prevents special
592 interpretation of lines that are part of the test program, not the
593 instructions to the test case, and (b) it speeds things up for really big test
594 cases by avoiding interpretation of the remainder of the file.</p>
Tanya Lattner5026c7f2004-12-06 02:11:52 +0000[diff] [blame]595
596</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]597
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]598<!--=========================================================================-->
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]599<div class="doc_section"><a name="progstructure"><tt>llvm-test</tt>
600Structure</a></div>
601<!--=========================================================================-->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]602
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]603<div class="doc_text">
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]604
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]605<p>As mentioned previously, the <tt>llvm-test</tt> module provides three types
606of tests: MultiSource, SingleSource, and External. Each tree is then subdivided
607into several categories, including applications, benchmarks, regression tests,
608code that is strange grammatically, etc. These organizations should be
609relatively self explanatory.</p>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]610
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]611<p>In addition to the regular "whole program" tests, the <tt>llvm-test</tt>
612module also provides a mechanism for compiling the programs in different ways.
613If the variable TEST is defined on the gmake command line, the test system will
614include a Makefile named <tt>TEST.&lt;value of TEST variable&gt;.Makefile</tt>.
615This Makefile can modify build rules to yield different results.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]616
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]617<p>For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt> to
618create the nightly test reports. To run the nightly tests, run <tt>gmake
619TEST=nightly</tt>.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]620
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]621<p>There are several TEST Makefiles available in the tree. Some of them are
622designed for internal LLVM research and will not work outside of the LLVM
623research group. They may still be valuable, however, as a guide to writing your
624own TEST Makefile for any optimization or analysis passes that you develop with
625LLVM.</p>
Reid Spencer3281ead2004-12-08 16:52:51 +0000[diff] [blame]626
627<p>Note, when configuring the <tt>llvm-test</tt> module, you might want to
628specify the following configuration options:</p>
629<dl>
630 <dt><i>--enable-spec2000</i>
631 <dt><i>--enable-spec2000=&lt;<tt>directory</tt>&gt;</i>
632 <dd>
633 Enable the use of SPEC2000 when testing LLVM. This is disabled by default
634 (unless <tt>configure</tt> finds SPEC2000 installed). By specifying
635 <tt>directory</tt>, you can tell configure where to find the SPEC2000
636 benchmarks. If <tt>directory</tt> is left unspecified, <tt>configure</tt>
637 uses the default value
638 <tt>/home/vadve/shared/benchmarks/speccpu2000/benchspec</tt>.
639 <p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]640
Reid Spencer3281ead2004-12-08 16:52:51 +0000[diff] [blame]641 <dt><i>--enable-spec95</i>
642 <dt><i>--enable-spec95=&lt;<tt>directory</tt>&gt;</i>
643 <dd>
644 Enable the use of SPEC95 when testing LLVM. It is similar to the
645 <i>--enable-spec2000</i> option.
646 <p>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]647
Reid Spencer3281ead2004-12-08 16:52:51 +0000[diff] [blame]648 <dt><i>--enable-povray</i>
649 <dt><i>--enable-povray=&lt;<tt>directory</tt>&gt;</i>
650 <dd>
651 Enable the use of Povray as an external test. Versions of Povray written
652 in C should work. This option is similar to the <i>--enable-spec2000</i>
653 option.
654</dl>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]655</div>
656
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]657<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]658<div class="doc_section"><a name="run">Running the LLVM Tests</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]659<!--=========================================================================-->
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]660
661<div class="doc_text">
662
663<p>First, all tests are executed within the LLVM object directory tree. They
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]664<i>are not</i> executed inside of the LLVM source tree. This is because the
665test suite creates temporary files during execution.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]666
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]667<p>The master Makefile in <tt>llvm/test</tt> is capable of running only the
668DejaGNU driven tests. By default, it will run all of these tests.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]669
Tanya Lattnercc85da02004-12-08 17:35:31 +0000[diff] [blame]670<p>To run only the DejaGNU driven tests, run <tt>gmake</tt> at the
John Criswellc6e32eb2005-05-13 19:48:07 +0000[diff] [blame]671command line in <tt>llvm/test</tt>. To run a specific directory of tests, use
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]672the <tt>TESTSUITE</tt> variable.
Tanya Lattnercc85da02004-12-08 17:35:31 +0000[diff] [blame]673</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]674
Tanya Lattnercc85da02004-12-08 17:35:31 +0000[diff] [blame]675<p>For example, to run the Regression tests, type
676<tt>gmake TESTSUITE=Regression</tt> in <tt>llvm/tests</tt>.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]677
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]678<p>Note that there are no Makefiles in <tt>llvm/test/Features</tt> and
Tanya Lattnercc85da02004-12-08 17:35:31 +0000[diff] [blame]679<tt>llvm/test/Regression</tt>. You must use DejaGNU from the <tt>llvm/test</tt>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]680directory to run them.</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]681
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]682<p>To run the <tt>llvm-test</tt> suite, you need to use the following steps:</p>
683
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]684<ol>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]685 <li><tt>cd</tt> into the <tt>llvm/projects</tt> directory</li>
686
687 <li><p>Check out the <tt>test-suite</tt> module with:</p>
688
689<div class="doc_code">
690<pre>
691% svn co http://llvm.org/svn/llvm-project/test-suite/trunk llvm-test
692</pre>
693</div>
694
695 <p>This will get the test suite into <tt>llvm/projects/llvm-test</tt></p>
696
697 <li><p>Configure the test suite. You can do this one of two ways:</p>
698
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]699 <ol>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]700 <li>Use the regular llvm configure:<br/><br/>
701
702<div class="doc_code">
703<pre>
704% cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure
705</pre>
706</div>
707
708 <p>This will ensure that the <tt>projects/llvm-test</tt> directory is
709 also properly configured.</p></li>
710
711 <li><p>Use the <tt>configure</tt> script found in the <tt>llvm-test</tt>
712 source directory:</p>
713
714<div class="doc_code">
715<pre>
716% $LLVM_SRC_ROOT/projects/llvm-test/configure \
717 --with-llvmsrc=$LLVM_SRC_ROOT \
718 --with-llvmobj=$LLVM_OBJ_ROOT
719</pre>
720</div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]721 </li>
722 </ol>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]723 <li><tt>gmake</tt></li>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]724</ol>
725<p>Note that the second and third steps only need to be done once. After you
726have the suite checked out and configured, you don't need to do it again (unless
727the test code or configure script changes).</p>
728
729<p>To make a specialized test (use one of the
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]730<tt>llvm-test/TEST.&lt;type&gt;.Makefile</tt>s), just run:</p>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]731
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]732<div class="doc_code">
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]733<pre>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]734% gmake TEST=&lt;type&gt; test
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]735</pre>
Bill Wendlingda51c4c2007-09-22 09:16:44 +0000[diff] [blame]736</div>
737
738<p>For example, you could run the nightly tester tests using the following
739commands:</p>
740
741<div class="doc_code">
742<pre>
743% cd llvm/projects/llvm-test
744% gmake TEST=nightly test
745</pre>
746</div>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]747
748<p>Regardless of which test you're running, the results are printed on standard
749output and standard error. You can redirect these results to a file if you
750choose.</p>
751
752<p>Some tests are known to fail. Some are bugs that we have not fixed yet;
Tanya Lattnercc85da02004-12-08 17:35:31 +0000[diff] [blame]753others are features that we haven't added yet (or may never add). In DejaGNU,
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]754the result for such tests will be XFAIL (eXpected FAILure). In this way, you
755can tell the difference between an expected and unexpected failure.</p>
756
John Criswellc6e32eb2005-05-13 19:48:07 +0000[diff] [blame]757<p>The tests in <tt>llvm-test</tt> have no such feature at this time. If the
Misha Brukman0904f092004-10-08 00:41:27 +0000[diff] [blame]758test passes, only warnings and other miscellaneous output will be generated. If
759a test fails, a large &lt;program&gt; FAILED message will be displayed. This
760will help you separate benign warnings from actual test failures.</p>
Chris Lattnerfd9d1b32004-06-24 20:53:09 +0000[diff] [blame]761
762</div>
763
Chris Lattnereb82da82006-05-23 01:40:20 +0000[diff] [blame]764<!-- _______________________________________________________________________ -->
765<div class="doc_subsection">
766<a name="customtest">Writing custom tests for llvm-test</a></div>
767<!-- _______________________________________________________________________ -->
768
769<div class="doc_text">
770
771<p>Assuming you can run llvm-test, (e.g. "<tt>gmake TEST=nightly report</tt>"
772should work), it is really easy to run optimizations or code generator
773components against every program in the tree, collecting statistics or running
774custom checks for correctness. At base, this is how the nightly tester works,
775it's just one example of a general framework.</p>
776
777<p>Lets say that you have an LLVM optimization pass, and you want to see how
778many times it triggers. First thing you should do is add an LLVM
779<a href="ProgrammersManual.html#Statistic">statistic</a> to your pass, which
780will tally counts of things you care about.</p>
781
782<p>Following this, you can set up a test and a report that collects these and
783formats them for easy viewing. This consists of two files, an
784"<tt>llvm-test/TEST.XXX.Makefile</tt>" fragment (where XXX is the name of your
785test) and an "<tt>llvm-test/TEST.XXX.report</tt>" file that indicates how to
786format the output into a table. There are many example reports of various
787levels of sophistication included with llvm-test, and the framework is very
788general.</p>
789
790<p>If you are interested in testing an optimization pass, check out the
791"libcalls" test as an example. It can be run like this:<p>
792
793<div class="doc_code">
794<pre>
795% cd llvm/projects/llvm-test/MultiSource/Benchmarks # or some other level
796% make TEST=libcalls report
797</pre>
798</div>
799
800<p>This will do a bunch of stuff, then eventually print a table like this:</p>
801
802<div class="doc_code">
803<pre>
804Name | total | #exit |
805...
806FreeBench/analyzer/analyzer | 51 | 6 |
807FreeBench/fourinarow/fourinarow | 1 | 1 |
808FreeBench/neural/neural | 19 | 9 |
809FreeBench/pifft/pifft | 5 | 3 |
810MallocBench/cfrac/cfrac | 1 | * |
811MallocBench/espresso/espresso | 52 | 12 |
812MallocBench/gs/gs | 4 | * |
813Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * |
814Prolangs-C/agrep/agrep | 33 | 12 |
815Prolangs-C/allroots/allroots | * | * |
816Prolangs-C/assembler/assembler | 47 | * |
817Prolangs-C/bison/mybison | 74 | * |
818...
819</pre>
820</div>
821
822<p>This basically is grepping the -stats output and displaying it in a table.
823You can also use the "TEST=libcalls report.html" target to get the table in HTML
824form, similarly for report.csv and report.tex.</p>
825
826<p>The source for this is in llvm-test/TEST.libcalls.*. The format is pretty
827simple: the Makefile indicates how to run the test (in this case,
828"<tt>opt -simplify-libcalls -stats</tt>"), and the report contains one line for
829each column of the output. The first value is the header for the column and the
830second is the regex to grep the output of the command for. There are lots of
831example reports that can do fancy stuff.</p>
832
833</div>
834
835
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]836<!--=========================================================================-->
Chris Lattnerfd9d1b32004-06-24 20:53:09 +0000[diff] [blame]837<div class="doc_section"><a name="nightly">Running the nightly tester</a></div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]838<!--=========================================================================-->
Chris Lattnerfd9d1b32004-06-24 20:53:09 +0000[diff] [blame]839
840<div class="doc_text">
841
842<p>
Patrick Jenkins825c2a72006-08-11 23:27:02 +0000[diff] [blame]843The <a href="http://llvm.org/nightlytest/">LLVM Nightly Testers</a>
Chris Lattnerfd9d1b32004-06-24 20:53:09 +0000[diff] [blame]844automatically check out an LLVM tree, build it, run the "nightly"
Patrick Jenkins825c2a72006-08-11 23:27:02 +0000[diff] [blame]845program test (described above), run all of the feature and regression tests,
846delete the checked out tree, and then submit the results to
847<a href="http://llvm.org/nightlytest/">http://llvm.org/nightlytest/</a>.
848After test results are submitted to
849<a href="http://llvm.org/nightlytest/">http://llvm.org/nightlytest/</a>,
850they are processed and displayed on the tests page. An email to
851<a href="http://lists.cs.uiuc.edu/pipermail/llvm-testresults/">
852llvm-testresults@cs.uiuc.edu</a> summarizing the results is also generated.
853This testing scheme is designed to ensure that programs don't break as well
854as keep track of LLVM's progress over time.</p>
Chris Lattnerfd9d1b32004-06-24 20:53:09 +0000[diff] [blame]855
Patrick Jenkins825c2a72006-08-11 23:27:02 +0000[diff] [blame]856<p>If you'd like to set up an instance of the nightly tester to run on your
857machine, take a look at the comments at the top of the
858<tt>utils/NewNightlyTest.pl</tt> file. If you decide to set up a nightly tester
859please choose a unique nickname and invoke <tt>utils/NewNightlyTest.pl</tt>
Reid Spencer669ed452007-07-09 08:04:31 +0000[diff] [blame]860with the "-nickname [yournickname]" command line option.
Chris Lattnerfd9d1b32004-06-24 20:53:09 +0000[diff] [blame]861
Reid Spencer669ed452007-07-09 08:04:31 +0000[diff] [blame]862<p>You can create a shell script to encapsulate the running of the script.
Misha Brukmanfc1a27b2005-03-10 22:51:59 +0000[diff] [blame]863The optimized x86 Linux nightly test is run from just such a script:</p>
864
865<div class="doc_code">
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]866<pre>
867#!/bin/bash
868BASE=/proj/work/llvm/nightlytest
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]869export BUILDDIR=$BASE/build
870export WEBDIR=$BASE/testresults
871export LLVMGCCDIR=/proj/work/llvm/cfrontend/install
872export PATH=/proj/install/bin:$LLVMGCCDIR/bin:$PATH
873export LD_LIBRARY_PATH=/proj/install/lib
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]874cd $BASE
Patrick Jenkins825c2a72006-08-11 23:27:02 +0000[diff] [blame]875cp /proj/work/llvm/llvm/utils/NewNightlyTest.pl .
876nice ./NewNightlyTest.pl -nice -release -verbose -parallel -enable-linscan \
Reid Spencer669ed452007-07-09 08:04:31 +0000[diff] [blame]877 -nickname NightlyTester -noexternals &gt; output.log 2&gt;&amp;1
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]878</pre>
Misha Brukmanfc1a27b2005-03-10 22:51:59 +0000[diff] [blame]879</div>
Reid Spencer8284f1f2004-09-05 20:07:26 +0000[diff] [blame]880
Patrick Jenkins825c2a72006-08-11 23:27:02 +0000[diff] [blame]881<p>It is also possible to specify the the location your nightly test results
882are submitted. You can do this by passing the command line option
883"-submit-server [server_address]" and "-submit-script [script_on_server]" to
884<tt>utils/NewNightlyTest.pl</tt>. For example, to submit to the llvm.org
885nightly test results page, you would invoke the nightly test script with
886"-submit-server llvm.org -submit-script /nightlytest/NightlyTestAccept.cgi".
887If these options are not specified, the nightly test script sends the results
888to the llvm.org nightly test results page.</p>
889
890<p>Take a look at the <tt>NewNightlyTest.pl</tt> file to see what all of the
891flags and strings do. If you start running the nightly tests, please let us
892know. Thanks!</p>
Misha Brukmanfc1a27b2005-03-10 22:51:59 +0000[diff] [blame]893
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]894</div>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]895
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]896<!-- *********************************************************************** -->
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]897
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]898<hr>
899<address>
900 <a href="http://jigsaw.w3.org/css-validator/check/referer"><img
901 src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!"></a>
902 <a href="http://validator.w3.org/check/referer"><img
903 src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!" /></a>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]904
John Criswellc6e32eb2005-05-13 19:48:07 +0000[diff] [blame]905 John T. Criswell, Reid Spencer, and Tanya Lattner<br>
Reid Spencer05fe4b02006-03-14 05:39:39 +0000[diff] [blame]906 <a href="http://llvm.org">The LLVM Compiler Infrastructure</a><br/>
Misha Brukman1d83e112004-03-01 18:21:04 +0000[diff] [blame]907 Last modified: $Date$
908</address>
Brian Gaekeaf19f2e2003-10-23 18:10:28 +0000[diff] [blame]909</body>
John Criswell020cbd82003-10-10 18:42:49 +0000[diff] [blame]910</html>