Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 1 | <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" |
| 2 | "http://www.w3.org/TR/html4/strict.dtd"> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 3 | <html> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 4 | <head> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 5 | <title>LLVM Test Suite Guide</title> |
| 6 | <link rel="stylesheet" href="llvm.css" type="text/css"> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 7 | </head> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 8 | <body> |
| 9 | |
| 10 | <div class="doc_title"> |
| 11 | LLVM Test Suite Guide |
| 12 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 13 | |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 14 | <ol> |
| 15 | <li><a href="#overview">Overview</a></li> |
| 16 | <li><a href="#Requirements">Requirements</a></li> |
| 17 | <li><a href="#quick">Quick Start</a></li> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 18 | <li><a href="#org">LLVM Test Suite Organization</a> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 19 | <ul> |
| 20 | <li><a href="#codefragments">Code Fragments</a></li> |
| 21 | <li><a href="#wholeprograms">Whole Programs</a></li> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 22 | </ul></li> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 23 | <li><a href="#tree">LLVM Test Suite Tree</a></li> |
| 24 | <li><a href="#qmstructure">QMTest Structure</a></li> |
| 25 | <li><a href="#progstructure">Programs Structure</a></li> |
| 26 | <li><a href="#run">Running the LLVM Tests</a></li> |
Chris Lattner | fd9d1b3 | 2004-06-24 20:53:09 +0000 | [diff] [blame^] | 27 | <li><a href="#nightly">Running the nightly tester</a></li> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 28 | </ol> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 29 | |
Chris Lattner | 7911ce2 | 2004-05-23 21:07:27 +0000 | [diff] [blame] | 30 | <div class="doc_author"> |
| 31 | <p>Written by John T. Criswell</p> |
| 32 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 33 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 34 | <!--===============================================================--> |
| 35 | <div class="doc_section"><a name="overview">Overview</a></div> |
| 36 | <!--===============================================================--> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 37 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 38 | <div class="doc_text"> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 39 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 40 | <p>This document is the reference manual for the LLVM test suite. It documents |
| 41 | the structure of the LLVM test suite, the tools needed to use it, and how to add |
| 42 | and run tests.</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 43 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 44 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 45 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 46 | <!--===============================================================--> |
| 47 | <div class="doc_section"><a name="Requirements">Requirements</a></div> |
| 48 | <!--===============================================================--> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 49 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 50 | <div class="doc_text"> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 51 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 52 | <p>In order to use the LLVM test suite, you will need all of the software |
| 53 | required to build LLVM, plus the following:</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 54 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 55 | <dl> |
| 56 | <dt><a href="http://www.qmtest.com">QMTest</A></dt> |
Reid Spencer | ec86a7a | 2004-06-20 19:21:11 +0000 | [diff] [blame] | 57 | <dd>The LLVM test suite uses QMTest to organize and run tests. <b>Note: |
Brian Gaeke | 826c5e8 | 2004-06-22 19:06:49 +0000 | [diff] [blame] | 58 | you will need <a href="http://llvm.cs.uiuc.edu/qm-2.0.3.tar.gz">QMTest |
| 59 | 2.0.3 (source tar.gz file)</a> to be successful. The tests do not run with |
Reid Spencer | ec86a7a | 2004-06-20 19:21:11 +0000 | [diff] [blame] | 60 | any other version.</b></dd> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 61 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 62 | <dt><a href="http://www.python.org">Python</A></dt> |
| 63 | <dd>You will need a Python interpreter that works with QMTest. Python will |
| 64 | need zlib and SAX support enabled.</dd> |
| 65 | </dl> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 66 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 67 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 68 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 69 | <!--===============================================================--> |
| 70 | <div class="doc_section"><a name="quick">Quick Start</a></div> |
| 71 | <!--===============================================================--> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 72 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 73 | <div class="doc_text"> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 74 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 75 | <p> The tests are located in the LLVM source tree under the directory |
| 76 | <tt>llvm/test</tt>. To run all of the tests in LLVM, use the Master Makefile in |
| 77 | that directory:</p> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 78 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 79 | <pre> |
| 80 | % gmake -C llvm/test |
| 81 | </pre> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 82 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 83 | <p>To run only the code fragment tests (i.e. those that do basic testing of |
| 84 | LLVM), run the tests organized by QMTest:</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 85 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 86 | <pre> |
| 87 | % gmake -C llvm/test qmtest |
| 88 | </pre> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 89 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 90 | <p>To run only the tests that compile and execute whole programs, run the |
| 91 | Programs tests:</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 92 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 93 | <pre> |
| 94 | % gmake -C llvm/test/Programs |
| 95 | </pre> |
| 96 | |
| 97 | </div> |
| 98 | |
| 99 | <!--===============================================================--> |
| 100 | <div class="doc_section"><a name="org">LLVM Test Suite Organization</a></div> |
| 101 | <!--===============================================================--> |
| 102 | |
| 103 | <div class="doc_text"> |
| 104 | |
| 105 | <p>The LLVM test suite contains two major categories of tests: code |
| 106 | fragments and whole programs.</p> |
| 107 | |
| 108 | </div> |
| 109 | |
| 110 | <div class="doc_subsection"><a name="codefragments">Code Fragments</a> |
| 111 | </div> |
| 112 | |
| 113 | <div class="doc_text"> |
| 114 | |
| 115 | <p>Code fragments are small pieces of code that test a specific feature of LLVM |
| 116 | or trigger a specific bug in LLVM. They are usually written in LLVM assembly |
| 117 | language, but can be written in other languages if the test targets a particular |
| 118 | language front end.</p> |
| 119 | |
| 120 | <p>Code fragments are not complete programs, and they are never executed to |
| 121 | determine correct behavior.</p> |
| 122 | |
| 123 | <p>The tests in the Features and Regression directories contain code |
| 124 | fragments.</p> |
| 125 | |
| 126 | </div> |
| 127 | |
| 128 | <div class="doc_subsection"><a name="wholeprograms">Whole Programs</a></div> |
| 129 | |
| 130 | <div class="doc_text"> |
| 131 | |
| 132 | <p>Whole Programs are pieces of code which can be compiled and linked into a |
| 133 | stand-alone program that can be executed. These programs are generally written |
| 134 | in high level languages such as C or C++, but sometimes they are written |
| 135 | straight in LLVM assembly.</p> |
| 136 | |
| 137 | <p>These programs are compiled and then executed using several different |
| 138 | methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation, |
| 139 | etc). The output of these programs is compared to ensure that LLVM is compiling |
| 140 | the program correctly.</p> |
| 141 | |
| 142 | <p>In addition to compiling and executing programs, whole program tests serve as |
| 143 | a way of benchmarking LLVM performance, both in terms of the efficiency of the |
| 144 | programs generated as well as the speed with which LLVM compiles, optimizes, and |
| 145 | generates code.</p> |
| 146 | |
| 147 | <p>The Programs directory contains all tests which compile and benchmark whole |
| 148 | programs.</p> |
| 149 | |
| 150 | </div> |
| 151 | |
| 152 | <!--===============================================================--> |
| 153 | <div class="doc_section"><a name="tree">LLVM Test Suite Tree</a></div> |
| 154 | <!--===============================================================--> |
| 155 | |
| 156 | <div class="doc_text"> |
| 157 | |
| 158 | <p>Each type of test in the LLVM test suite has its own directory. The major |
| 159 | subtrees of the test suite directory tree are as follows:</p> |
| 160 | |
| 161 | <ul> |
| 162 | <li>Features |
| 163 | <p>This directory contains sample codes that test various features of the |
| 164 | LLVM language. These pieces of sample code are run through various |
| 165 | assembler, disassembler, and optimizer passes.</p> |
| 166 | |
| 167 | <li>Regression |
| 168 | <p>This directory contains regression tests for LLVM. When a bug is found |
| 169 | in LLVM, a regression test containing just enough code to reproduce the |
| 170 | problem should be written and placed somewhere underneath this directory. |
| 171 | In most cases, this will be a small piece of LLVM assembly language code, |
| 172 | often distilled from an actual application or benchmark.</p> |
| 173 | |
| 174 | <li>Programs |
| 175 | <p>The Programs directory contains programs that can be compiled with LLVM |
| 176 | and executed. These programs are compiled using the native compiler and |
| 177 | various LLVM backends. The output from the program compiled with the native |
| 178 | compiler is assumed correct; the results from the other programs are |
| 179 | compared to the native program output and pass if they match. </p> |
| 180 | |
| 181 | <p> In addition for testing correctness, the Programs directory also |
| 182 | performs timing tests of various LLVM optimizations. It also records |
| 183 | compilation times for the compilers and the JIT. This information can be |
| 184 | used to compare the effectiveness of LLVM's optimizations and code |
| 185 | generation.</p> |
| 186 | |
| 187 | <p>The Programs directory is subdivided into several smaller subdirectories: |
| 188 | </p> |
| 189 | |
| 190 | <ul> |
| 191 | <li>Programs/SingleSource |
| 192 | <p>The SingleSource directory contains test programs that are only a |
| 193 | single source file in size. These are usually small benchmark programs |
| 194 | or small programs that calculate a particular value. Several such |
| 195 | programs are grouped together in each directory.</p></li> |
| 196 | |
| 197 | <li>Programs/MultiSource |
| 198 | <p>The MultiSource directory contains subdirectories which contain |
| 199 | entire programs with multiple source files. Large benchmarks and whole |
| 200 | applications go here.</p></li> |
| 201 | |
| 202 | <li>Programs/External |
| 203 | <p>The External directory contains Makefiles for building code that is |
| 204 | external to (i.e. not distributed with) LLVM. The most prominent member |
| 205 | of this directory is the SPEC 2000 benchmark suite. The presence and |
| 206 | location of these external programs is configured by the LLVM |
| 207 | <tt>configure</tt> script.</p></li> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 208 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 209 | </ul></li> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 210 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 211 | <li>QMTest |
| 212 | <p>This directory contains the QMTest information files. Inside this |
| 213 | directory are QMTest administration files and the Python code that |
| 214 | implements the LLVM test and database classes.</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 215 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 216 | </ul> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 217 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 218 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 219 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 220 | <!--===============================================================--> |
| 221 | <div class="doc_section"><a name="qmstructure">QMTest Structure</a></div> |
| 222 | <!--===============================================================--> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 223 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 224 | <div class="doc_text"> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 225 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 226 | <p>The LLVM test suite is partially driven by QMTest and partially |
| 227 | driven by GNU Make. Specifically, the Features and Regression tests |
| 228 | are all driven by QMTest. The Programs directory is currently |
| 229 | driven by a set of Makefiles.</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 230 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 231 | <p>The QMTest system needs to have several pieces of information |
| 232 | available; these pieces of configuration information are known |
| 233 | collectively as the "context" in QMTest parlance. Since the context |
| 234 | for LLVM is relatively large, the master Makefile in llvm/test |
| 235 | sets it for you.</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 236 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 237 | <p>The LLVM database class makes the subdirectories of llvm/test a |
| 238 | QMTest test database. For each directory that contains tests driven by |
| 239 | QMTest, it knows what type of test the source file is and how to run it.</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 240 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 241 | <p>Hence, the QMTest namespace is essentially what you see in the |
| 242 | Feature and Regression directories, but there is some magic that |
| 243 | the database class performs (as described below).</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 244 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 245 | <p>The QMTest namespace is currently composed of the following tests and test |
| 246 | suites:</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 247 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 248 | <ul> |
| 249 | <li>Feature |
| 250 | <p> |
| 251 | These are the feature tests found in the Feature directory. |
| 252 | They are broken up into the following categories: |
| 253 | </p> |
| 254 | <ul> |
| 255 | <li>ad |
| 256 | <p>Assembler/Disassembler tests. These tests verify that a piece of LLVM |
| 257 | assembly language can be assembled into bytecode and then disassembled |
| 258 | into the original assembly language code. It does this several times to |
| 259 | ensure that assembled output can be disassembled and disassembler output |
| 260 | can be assembled. It also verifies that the give assembly language file |
| 261 | can be assembled correctly.</p></li> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 262 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 263 | <li>opt |
| 264 | <p>Optimizer tests. These tests verify that two of the optimizer passes |
| 265 | completely optimize a program (i.e. after a single pass, they cannot |
| 266 | optimize a program any further).</p></li> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 267 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 268 | <li>mc |
| 269 | <p> Machine code tests. These tests verify that the LLVM assembly |
| 270 | language file can be translated into native assembly code.</p></li> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 271 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 272 | <li>cc |
| 273 | <p>C code tests. These tests verify that the specified LLVM assembly |
| 274 | code can be converted into C source code using the C backend.</p></li> |
| 275 | </ul> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 276 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 277 | <p>The LLVM database class looks at every file in the Feature directory and |
| 278 | creates a fake test hierarchy containing |
| 279 | <tt>Feature.<testtype>.<testname></tt>. So, if you add an LLVM |
| 280 | assembly language file to the Feature directory, it actually creates 5 new |
| 281 | tests: assembler/disassembler, assembler, optimizer, machine code, and C code. |
| 282 | </p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 283 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 284 | <li>Regression |
| 285 | <p>These are the regression tests. There is one suite for each |
| 286 | subdirectory of the Regression directory. If you add a new subdirectory |
| 287 | there, you will need to modify, at least, the <tt>RegressionMap</tt> |
| 288 | variable in <tt>QMTest/llvmdb.py</tt> so that QMTest knows how to run the |
| 289 | tests in the new subdirectory.</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 290 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 291 | </ul> |
| 292 | |
| 293 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 294 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 295 | <!--===============================================================--> |
| 296 | <div class="doc_section"><a name="progstructure">Programs Structure</a></div> |
| 297 | <!--===============================================================--> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 298 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 299 | <div class="doc_text"> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 300 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 301 | <p>As mentioned previously, the Programs tree in llvm/test provides three types |
| 302 | of tests: MultiSource, SingleSource, and External. Each tree is then subdivided |
| 303 | into several categories, including applications, benchmarks, regression tests, |
| 304 | code that is strange grammatically, etc. These organizations should be |
| 305 | relatively self explanatory.</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 306 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 307 | <p>In addition to the regular Programs tests, the Programs tree also provides a |
| 308 | mechanism for compiling the programs in different ways. If the variable TEST is |
| 309 | defined on the gmake command line, the test system will include a Makefile named |
| 310 | <tt>TEST.<value of TEST variable>.Makefile</tt>. This Makefile can modify |
| 311 | build rules to yield different results.</p> |
| 312 | |
| 313 | <p>For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt> to |
| 314 | create the nightly test reports. To run the nightly tests, run <tt>gmake |
| 315 | TEST=nightly</tt>.</p> |
| 316 | |
| 317 | <p>There are several TEST Makefiles available in the tree. Some of them are |
| 318 | designed for internal LLVM research and will not work outside of the LLVM |
| 319 | research group. They may still be valuable, however, as a guide to writing your |
| 320 | own TEST Makefile for any optimization or analysis passes that you develop with |
| 321 | LLVM.</p> |
| 322 | |
| 323 | </div> |
| 324 | |
| 325 | <!--===============================================================--> |
| 326 | <div class="doc_section"><a name="run">Running the LLVM Tests</a></div> |
| 327 | <!--===============================================================--> |
| 328 | |
| 329 | <div class="doc_text"> |
| 330 | |
| 331 | <p>First, all tests are executed within the LLVM object directory tree. They |
| 332 | <i>are not</i> executed inside of the LLVM source tree. This is because the |
| 333 | test suite creates temporary files during execution. </p> |
| 334 | |
| 335 | <p>The master Makefile in llvm/test is capable of running both the QMTest driven |
| 336 | tests and the Programs tests. By default, it will run all of the tests.</p> |
| 337 | |
| 338 | <p>To run only the QMTest driven tests, run <tt>gmake qmtest</tt> at the |
| 339 | command line in llvm/tests. To run a specific qmtest, suffix the test name with |
| 340 | ".t" when running gmake.</p> |
| 341 | |
| 342 | <p>For example, to run the Regression.LLC tests, type <tt>gmake |
| 343 | Regression.LLC.t</tt> in llvm/tests.</p> |
| 344 | |
| 345 | <p>Note that the Makefiles in llvm/test/Features and llvm/test/Regression are |
| 346 | gone. You must now use QMTest from the llvm/test directory to run them.</p> |
| 347 | |
| 348 | <p>To run the Programs test, cd into the llvm/test/Programs directory and type |
| 349 | <tt>gmake</tt>. Alternatively, you can type <tt>gmake TEST=<type> |
| 350 | test</tt> to run one of the specialized tests in |
| 351 | llvm/test/Programs/TEST.<type>.Makefile. For example, you could run the |
| 352 | nightly tester tests using the following commands:</p> |
| 353 | |
| 354 | <pre> |
| 355 | % cd llvm/test/Programs |
| 356 | % gmake TEST=nightly test |
| 357 | </pre> |
| 358 | |
| 359 | <p>Regardless of which test you're running, the results are printed on standard |
| 360 | output and standard error. You can redirect these results to a file if you |
| 361 | choose.</p> |
| 362 | |
| 363 | <p>Some tests are known to fail. Some are bugs that we have not fixed yet; |
| 364 | others are features that we haven't added yet (or may never add). In QMTest, |
| 365 | the result for such tests will be XFAIL (eXpected FAILure). In this way, you |
| 366 | can tell the difference between an expected and unexpected failure.</p> |
| 367 | |
| 368 | <p>The Programs tests have no such feature as of this time. If the test passes, |
| 369 | only warnings and other miscellaneous output will be generated. If a test |
| 370 | fails, a large <program> FAILED message will be displayed. This will help |
| 371 | you separate benign warnings from actual test failures.</p> |
Chris Lattner | fd9d1b3 | 2004-06-24 20:53:09 +0000 | [diff] [blame^] | 372 | |
| 373 | </div> |
| 374 | |
| 375 | <!--===============================================================--> |
| 376 | <div class="doc_section"><a name="nightly">Running the nightly tester</a></div> |
| 377 | <!--===============================================================--> |
| 378 | |
| 379 | <div class="doc_text"> |
| 380 | |
| 381 | <p> |
| 382 | The <a href="http://llvm.cs.uiuc.edu/testresults/">LLVM Nightly Testers</a> |
| 383 | automatically check out an LLVM tree, build it, run the "nightly" |
| 384 | program test (described above) and all of the regression tests, then |
| 385 | delete the checked out tree. This tester is designed to ensure that |
| 386 | programs don't break as well as keep track of LLVM's progress over time.</p> |
| 387 | |
| 388 | <p> |
| 389 | If you'd like to set up an instance of the nightly tester to run on your |
| 390 | machine, take a look at the comments at the top of the utils/NightlyTester.pl |
| 391 | file. We usually run it from a crontab entry that looks ilke this: |
| 392 | </p> |
| 393 | |
| 394 | <pre> |
| 395 | 5 3 * * * LLVM_LIB_SEARCH_PATH=.../llvm-gcc/bytecode-libs $HOME/llvm/utils/NightlyTest.pl -parallel -enable-linscan ...CVSREPOSTRING... $HOME/buildtest-X86 $HOME/cvs/testresults-X86 |
| 396 | </pre> |
| 397 | |
| 398 | <p> |
| 399 | Take a look at the NightlyTest.pl file to see what all of the flags and |
| 400 | strings do. If you start running the nightly tests, please let us know and |
| 401 | we'll link your page to the global tester page. Thanks! |
| 402 | </p> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 403 | |
| 404 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 405 | |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 406 | <!-- *********************************************************************** --> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 407 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 408 | <hr> |
| 409 | <address> |
| 410 | <a href="http://jigsaw.w3.org/css-validator/check/referer"><img |
| 411 | src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!"></a> |
| 412 | <a href="http://validator.w3.org/check/referer"><img |
| 413 | src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!" /></a> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 414 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 415 | John T. Criswell<br> |
| 416 | <a href="http://llvm.cs.uiuc.edu">The LLVM Compiler Infrastructure</a><br> |
| 417 | Last modified: $Date$ |
| 418 | </address> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 419 | </body> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 420 | </html> |