Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 1 | <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" |
| 2 | "http://www.w3.org/TR/html4/strict.dtd"> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 3 | <html> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 4 | <head> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 5 | <title>LLVM Testing Infrastructure Guide</title> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 6 | <link rel="stylesheet" href="llvm.css" type="text/css"> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 7 | </head> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 8 | <body> |
| 9 | |
| 10 | <div class="doc_title"> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 11 | LLVM Testing Infrastructure Guide |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 12 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 13 | |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 14 | <ol> |
Reid Spencer | 820e247 | 2004-11-01 08:30:14 +0000 | [diff] [blame] | 15 | <li><a href="#overview">Overview</a></li> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 16 | <li><a href="#requirements">Requirements</a></li> |
| 17 | <li><a href="#org">LLVM testing infrastructure organization</a> |
Reid Spencer | 820e247 | 2004-11-01 08:30:14 +0000 | [diff] [blame] | 18 | <ul> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 19 | <li><a href="#dejagnu">DejaGNU tests</a></li> |
| 20 | <li><a href="#testsuite">Test suite</a></li> |
Reid Spencer | 820e247 | 2004-11-01 08:30:14 +0000 | [diff] [blame] | 21 | </ul> |
| 22 | </li> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 23 | <li><a href="#quick">Quick start</a> |
Chris Lattner | eb82da8 | 2006-05-23 01:40:20 +0000 | [diff] [blame] | 24 | <ul> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 25 | <li><a href="#quickdejagnu">DejaGNU tests</a></li> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 26 | <li><a href="#quicktestsuite">Test suite</a></li> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 27 | </ul> |
| 28 | </li> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 29 | <li><a href="#dgstructure">DejaGNU structure</a> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 30 | <ul> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 31 | <li><a href="#dgcustom">Writing new DejaGNU tests</a></li> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 32 | <li><a href="#dgvars">Variables and substitutions</a></li> |
| 33 | <li><a href="#dgfeatures">Other features</a></li> |
| 34 | </ul> |
| 35 | </li> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 36 | <li><a href="#testsuitestructure">Test suite structure</a></li> |
| 37 | <li><a href="#testsuiterun">Running the test suite</a> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 38 | <ul> |
Stuart Hastings | c4c268b | 2009-05-21 20:23:59 +0000 | [diff] [blame] | 39 | <li><a href="#testsuiteexternal">Configuring External Tests</a></li> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 40 | <li><a href="#testsuitetests">Running different tests</a></li> |
| 41 | <li><a href="#testsuiteoutput">Generating test output</a></li> |
| 42 | <li><a href="#testsuitecustom">Writing custom tests for llvm-test</a></li> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 43 | </ul> |
Chris Lattner | eb82da8 | 2006-05-23 01:40:20 +0000 | [diff] [blame] | 44 | </li> |
Reid Spencer | 820e247 | 2004-11-01 08:30:14 +0000 | [diff] [blame] | 45 | <li><a href="#nightly">Running the nightly tester</a></li> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 46 | </ol> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 47 | |
Chris Lattner | 7911ce2 | 2004-05-23 21:07:27 +0000 | [diff] [blame] | 48 | <div class="doc_author"> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 49 | <p>Written by John T. Criswell, <a |
| 50 | href="http://llvm.x10sys.com/rspencer">Reid Spencer</a>, and Tanya Lattner</p> |
Chris Lattner | 7911ce2 | 2004-05-23 21:07:27 +0000 | [diff] [blame] | 51 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 52 | |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 53 | <!--=========================================================================--> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 54 | <div class="doc_section"><a name="overview">Overview</a></div> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 55 | <!--=========================================================================--> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 56 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 57 | <div class="doc_text"> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 58 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 59 | <p>This document is the reference manual for the LLVM testing infrastructure. It documents |
| 60 | the structure of the LLVM testing infrastructure, the tools needed to use it, |
| 61 | and how to add and run tests.</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 62 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 63 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 64 | |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 65 | <!--=========================================================================--> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 66 | <div class="doc_section"><a name="requirements">Requirements</a></div> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 67 | <!--=========================================================================--> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 68 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 69 | <div class="doc_text"> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 70 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 71 | <p>In order to use the LLVM testing infrastructure, you will need all of the software |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 72 | required to build LLVM, plus the following:</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 73 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 74 | <dl> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 75 | <dt><a href="http://www.gnu.org/software/dejagnu/">DejaGNU</a></dt> |
| 76 | <dd>The Feature and Regressions tests are organized and run by DejaGNU.</dd> |
| 77 | <dt><a href="http://expect.nist.gov/">Expect</a></dt> |
| 78 | <dd>Expect is required by DejaGNU.</dd> |
Tanya Lattner | cc85da0 | 2004-12-08 17:35:31 +0000 | [diff] [blame] | 79 | <dt><a href="http://www.tcl.tk/software/tcltk/">tcl</a></dt> |
| 80 | <dd>Tcl is required by DejaGNU. </dd> |
Misha Brukman | 0adfeed | 2004-10-08 00:55:43 +0000 | [diff] [blame] | 81 | </dl> |
Jim Laskey | a2dee01 | 2006-03-27 18:41:06 +0000 | [diff] [blame] | 82 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 83 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 84 | |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 85 | <!--=========================================================================--> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 86 | <div class="doc_section"><a name="org">LLVM testing infrastructure organization</a></div> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 87 | <!--=========================================================================--> |
| 88 | |
| 89 | <div class="doc_text"> |
| 90 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 91 | <p>The LLVM testing infrastructure contains two major categories of tests: code |
| 92 | fragments and whole programs. Code fragments are referred to as the "DejaGNU |
| 93 | tests" and are in the <tt>llvm</tt> module in subversion under the |
| 94 | <tt>llvm/test</tt> directory. The whole programs tests are referred to as the |
| 95 | "Test suite" and are in the <tt>test-suite</tt> module in subversion. |
| 96 | </p> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 97 | |
| 98 | </div> |
| 99 | |
| 100 | <!-- _______________________________________________________________________ --> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 101 | <div class="doc_subsection"><a name="dejagnu">DejaGNU tests</a></div> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 102 | <!-- _______________________________________________________________________ --> |
| 103 | |
| 104 | <div class="doc_text"> |
| 105 | |
Shantonu Sen | 1b6d3da | 2009-06-26 05:44:53 +0000 | [diff] [blame^] | 106 | <p>Code fragments are small pieces of code that test a specific |
| 107 | feature of LLVM or trigger a specific bug in LLVM. They are usually |
| 108 | written in LLVM assembly language, but can be written in other |
| 109 | languages if the test targets a particular language front end (and the |
| 110 | appropriate <tt>--with-llvmgcc</tt> options were used |
| 111 | at <tt>configure</tt> time of the <tt>llvm</tt> module). These tests |
| 112 | are driven by the DejaGNU testing framework, which is hidden behind a |
| 113 | few simple makefiles.</p> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 114 | |
Shantonu Sen | 1b6d3da | 2009-06-26 05:44:53 +0000 | [diff] [blame^] | 115 | <p>These code fragments are not complete programs. The code generated |
| 116 | from them is never executed to determine correct behavior.</p> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 117 | |
| 118 | <p>These code fragment tests are located in the <tt>llvm/test</tt> |
| 119 | directory.</p> |
| 120 | |
| 121 | <p>Typically when a bug is found in LLVM, a regression test containing |
| 122 | just enough code to reproduce the problem should be written and placed |
| 123 | somewhere underneath this directory. In most cases, this will be a small |
| 124 | piece of LLVM assembly language code, often distilled from an actual |
| 125 | application or benchmark.</p> |
| 126 | |
| 127 | </div> |
| 128 | |
| 129 | <!-- _______________________________________________________________________ --> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 130 | <div class="doc_subsection"><a name="testsuite">Test suite</a></div> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 131 | <!-- _______________________________________________________________________ --> |
| 132 | |
| 133 | <div class="doc_text"> |
| 134 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 135 | <p>The test suite contains whole programs, which are pieces of |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 136 | code which can be compiled and linked into a stand-alone program that can be |
| 137 | executed. These programs are generally written in high level languages such as |
| 138 | C or C++, but sometimes they are written straight in LLVM assembly.</p> |
| 139 | |
| 140 | <p>These programs are compiled and then executed using several different |
| 141 | methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation, |
| 142 | etc). The output of these programs is compared to ensure that LLVM is compiling |
| 143 | the program correctly.</p> |
| 144 | |
| 145 | <p>In addition to compiling and executing programs, whole program tests serve as |
| 146 | a way of benchmarking LLVM performance, both in terms of the efficiency of the |
| 147 | programs generated as well as the speed with which LLVM compiles, optimizes, and |
| 148 | generates code.</p> |
| 149 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 150 | <p>The test-suite is located in the <tt>test-suite</tt> Subversion module.</p> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 151 | |
| 152 | </div> |
| 153 | |
| 154 | <!--=========================================================================--> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 155 | <div class="doc_section"><a name="quick">Quick start</a></div> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 156 | <!--=========================================================================--> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 157 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 158 | <div class="doc_text"> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 159 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 160 | <p>The tests are located in two separate Subversion modules. The |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 161 | DejaGNU tests are in the main "llvm" module under the directory |
| 162 | <tt>llvm/test</tt> (so you get these tests for free with the main llvm tree). |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 163 | The more comprehensive test suite that includes whole |
| 164 | programs in C and C++ is in the <tt>test-suite</tt> module. This module should |
| 165 | be checked out to the <tt>llvm/projects</tt> directory (don't use another name |
| 166 | then the default "test-suite", for then the test suite will be run every time |
| 167 | you run <tt>make</tt> in the main <tt>llvm</tt> directory). |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 168 | When you <tt>configure</tt> the <tt>llvm</tt> module, |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 169 | the <tt>test-suite</tt> directory will be automatically configured. |
Reid Spencer | 669ed45 | 2007-07-09 08:04:31 +0000 | [diff] [blame] | 170 | Alternatively, you can configure the <tt>test-suite</tt> module manually.</p> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 171 | |
| 172 | <!-- _______________________________________________________________________ --> |
| 173 | <div class="doc_subsection"><a name="quickdejagnu">DejaGNU tests</a></div> |
| 174 | <!-- _______________________________________________________________________ --> |
Chris Lattner | 792321a | 2006-05-23 01:25:11 +0000 | [diff] [blame] | 175 | <p>To run all of the simple tests in LLVM using DejaGNU, use the master Makefile |
| 176 | in the <tt>llvm/test</tt> directory:</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 177 | |
| 178 | <div class="doc_code"> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 179 | <pre> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 180 | % gmake -C llvm/test |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 181 | </pre> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 182 | </div> |
| 183 | |
| 184 | <p>or</p> |
| 185 | |
| 186 | <div class="doc_code"> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 187 | <pre> |
| 188 | % gmake check |
| 189 | </pre> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 190 | </div> |
John Criswell | 0c8a9a1 | 2005-05-13 20:25:49 +0000 | [diff] [blame] | 191 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 192 | <p>To run only a subdirectory of tests in <tt>llvm/test</tt> using DejaGNU (ie. |
| 193 | Transforms), just set the TESTSUITE variable to the path of the |
John Criswell | 0c8a9a1 | 2005-05-13 20:25:49 +0000 | [diff] [blame] | 194 | subdirectory (relative to <tt>llvm/test</tt>):</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 195 | |
| 196 | <div class="doc_code"> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 197 | <pre> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 198 | % gmake TESTSUITE=Transforms check |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 199 | </pre> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 200 | </div> |
Misha Brukman | fc1a27b | 2005-03-10 22:51:59 +0000 | [diff] [blame] | 201 | |
John Criswell | 0c8a9a1 | 2005-05-13 20:25:49 +0000 | [diff] [blame] | 202 | <p><b>Note: If you are running the tests with <tt>objdir != subdir</tt>, you |
| 203 | must have run the complete testsuite before you can specify a |
| 204 | subdirectory.</b></p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 205 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 206 | <p>To run only a single test, set <tt>TESTONE</tt> to its path (relative to |
| 207 | <tt>llvm/test</tt>) and make the <tt>check-one</tt> target:</p> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 208 | |
| 209 | <div class="doc_code"> |
| 210 | <pre> |
| 211 | % gmake TESTONE=Feature/basictest.ll check-one |
| 212 | </pre> |
| 213 | </div> |
| 214 | |
Nuno Lopes | 21bfe0b | 2008-11-25 15:57:52 +0000 | [diff] [blame] | 215 | <p>To run the tests with Valgrind (Memcheck by default), just append |
| 216 | <tt>VG=1</tt> to the commands above, e.g.:</p> |
| 217 | |
| 218 | <div class="doc_code"> |
| 219 | <pre> |
| 220 | % gmake check VG=1 |
| 221 | </pre> |
| 222 | </div> |
| 223 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 224 | <!-- _______________________________________________________________________ --> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 225 | <div class="doc_subsection"><a name="quicktestsuite">Test suite</a></div> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 226 | <!-- _______________________________________________________________________ --> |
| 227 | |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 228 | <p>To run the comprehensive test suite (tests that compile and execute whole |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 229 | programs), first checkout and setup the <tt>test-suite</tt> module:</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 230 | |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 231 | <div class="doc_code"> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 232 | <pre> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 233 | % cd llvm/projects |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 234 | % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite |
Tanya Lattner | 1de4849 | 2007-11-28 05:13:45 +0000 | [diff] [blame] | 235 | % cd .. |
| 236 | % ./configure --with-llvmgccdir=$LLVM_GCC_DIR |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 237 | </pre> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 238 | </div> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 239 | |
Shantonu Sen | 1b6d3da | 2009-06-26 05:44:53 +0000 | [diff] [blame^] | 240 | <p>where <tt>$LLVM_GCC_DIR</tt> is the directory where |
| 241 | you <em>installed</em> llvm-gcc, not it's src or obj |
| 242 | dir. The <tt>--with-llvmgccdir</tt> option assumes that |
| 243 | the <tt>llvm-gcc-4.2</tt> module was configured with |
| 244 | <tt>--program-prefix=llvm-</tt>, and therefore that the C and C++ |
| 245 | compiler drivers are called <tt>llvm-gcc</tt> and <tt>llvm-g++</tt> |
| 246 | respectively. If this is not the case, |
| 247 | use <tt>--with-llvmgcc</tt>/<tt>--with-llvmgxx</tt> to specify each |
| 248 | executable's location.</p> |
| 249 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 250 | <p>Then, run the entire test suite by running make in the <tt>test-suite</tt> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 251 | directory:</p> |
| 252 | |
| 253 | <div class="doc_code"> |
| 254 | <pre> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 255 | % cd projects/test-suite |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 256 | % gmake |
| 257 | </pre> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 258 | </div> |
| 259 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 260 | <p>Usually, running the "nightly" set of tests is a good idea, and you can also |
| 261 | let it generate a report by running:</p> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 262 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 263 | <div class="doc_code"> |
| 264 | <pre> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 265 | % cd projects/test-suite |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 266 | % gmake TEST=nightly report report.html |
| 267 | </pre> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 268 | </div> |
| 269 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 270 | <p>Any of the above commands can also be run in a subdirectory of |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 271 | <tt>projects/test-suite</tt> to run the specified test only on the programs in |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 272 | that subdirectory.</p> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 273 | |
| 274 | </div> |
| 275 | |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 276 | <!--=========================================================================--> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 277 | <div class="doc_section"><a name="dgstructure">DejaGNU structure</a></div> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 278 | <!--=========================================================================--> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 279 | <div class="doc_text"> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 280 | <p>The LLVM DejaGNU tests are driven by DejaGNU together with GNU Make and are |
| 281 | located in the <tt>llvm/test</tt> directory. |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 282 | |
Reid Spencer | a99256e | 2007-02-08 17:00:55 +0000 | [diff] [blame] | 283 | <p>This directory contains a large array of small tests |
| 284 | that exercise various features of LLVM and to ensure that regressions do not |
| 285 | occur. The directory is broken into several sub-directories, each focused on |
Bill Wendling | eb6aace | 2007-09-22 09:20:07 +0000 | [diff] [blame] | 286 | a particular area of LLVM. A few of the important ones are:</p> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 287 | |
Bill Wendling | eb6aace | 2007-09-22 09:20:07 +0000 | [diff] [blame] | 288 | <ul> |
Reid Spencer | a99256e | 2007-02-08 17:00:55 +0000 | [diff] [blame] | 289 | <li><tt>Analysis</tt>: checks Analysis passes.</li> |
| 290 | <li><tt>Archive</tt>: checks the Archive library.</li> |
| 291 | <li><tt>Assembler</tt>: checks Assembly reader/writer functionality.</li> |
Gabor Greif | 04367bf | 2007-07-06 22:07:22 +0000 | [diff] [blame] | 292 | <li><tt>Bitcode</tt>: checks Bitcode reader/writer functionality.</li> |
Reid Spencer | a99256e | 2007-02-08 17:00:55 +0000 | [diff] [blame] | 293 | <li><tt>CodeGen</tt>: checks code generation and each target.</li> |
| 294 | <li><tt>Features</tt>: checks various features of the LLVM language.</li> |
Gabor Greif | 04367bf | 2007-07-06 22:07:22 +0000 | [diff] [blame] | 295 | <li><tt>Linker</tt>: tests bitcode linking.</li> |
Reid Spencer | a99256e | 2007-02-08 17:00:55 +0000 | [diff] [blame] | 296 | <li><tt>Transforms</tt>: tests each of the scalar, IPO, and utility |
| 297 | transforms to ensure they make the right transformations.</li> |
| 298 | <li><tt>Verifier</tt>: tests the IR verifier.</li> |
Bill Wendling | eb6aace | 2007-09-22 09:20:07 +0000 | [diff] [blame] | 299 | </ul> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 300 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 301 | </div> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 302 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 303 | <!-- _______________________________________________________________________ --> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 304 | <div class="doc_subsection"><a name="dgcustom">Writing new DejaGNU tests</a></div> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 305 | <!-- _______________________________________________________________________ --> |
| 306 | <div class="doc_text"> |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 307 | <p>The DejaGNU structure is very simple, but does require some information to |
| 308 | be set. This information is gathered via <tt>configure</tt> and is written |
| 309 | to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The <tt>llvm/test</tt> |
| 310 | Makefile does this work for you.</p> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 311 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 312 | <p>In order for DejaGNU to work, each directory of tests must have a |
| 313 | <tt>dg.exp</tt> file. DejaGNU looks for this file to determine how to run the |
| 314 | tests. This file is just a Tcl script and it can do anything you want, but |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 315 | we've standardized it for the LLVM regression tests. If you're adding a |
| 316 | directory of tests, just copy <tt>dg.exp</tt> from another directory to get |
| 317 | running. The standard <tt>dg.exp</tt> simply loads a Tcl |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 318 | library (<tt>test/lib/llvm.exp</tt>) and calls the <tt>llvm_runtests</tt> |
| 319 | function defined in that library with a list of file names to run. The names |
| 320 | are obtained by using Tcl's glob command. Any directory that contains only |
| 321 | directories does not need the <tt>dg.exp</tt> file.</p> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 322 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 323 | <p>The <tt>llvm-runtests</tt> function lookas at each file that is passed to |
| 324 | it and gathers any lines together that match "RUN:". This are the "RUN" lines |
| 325 | that specify how the test is to be run. So, each test script must contain |
| 326 | RUN lines if it is to do anything. If there are no RUN lines, the |
| 327 | <tt>llvm-runtests</tt> function will issue an error and the test will |
| 328 | fail.</p> |
Misha Brukman | fc1a27b | 2005-03-10 22:51:59 +0000 | [diff] [blame] | 329 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 330 | <p>RUN lines are specified in the comments of the test program using the |
| 331 | keyword <tt>RUN</tt> followed by a colon, and lastly the command (pipeline) |
| 332 | to execute. Together, these lines form the "script" that |
| 333 | <tt>llvm-runtests</tt> executes to run the test case. The syntax of the |
| 334 | RUN lines is similar to a shell's syntax for pipelines including I/O |
| 335 | redirection and variable substitution. However, even though these lines |
| 336 | may <i>look</i> like a shell script, they are not. RUN lines are interpreted |
| 337 | directly by the Tcl <tt>exec</tt> command. They are never executed by a |
| 338 | shell. Consequently the syntax differs from normal shell script syntax in a |
| 339 | few ways. You can specify as many RUN lines as needed.</p> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 340 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 341 | <p>Each RUN line is executed on its own, distinct from other lines unless |
| 342 | its last character is <tt>\</tt>. This continuation character causes the RUN |
| 343 | line to be concatenated with the next one. In this way you can build up long |
| 344 | pipelines of commands without making huge line lengths. The lines ending in |
| 345 | <tt>\</tt> are concatenated until a RUN line that doesn't end in <tt>\</tt> is |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 346 | found. This concatenated set of RUN lines then constitutes one execution. |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 347 | Tcl will substitute variables and arrange for the pipeline to be executed. If |
| 348 | any process in the pipeline fails, the entire line (and test case) fails too. |
| 349 | </p> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 350 | |
Reid Spencer | f190277 | 2007-04-15 08:01:04 +0000 | [diff] [blame] | 351 | <p> Below is an example of legal RUN lines in a <tt>.ll</tt> file:</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 352 | |
| 353 | <div class="doc_code"> |
| 354 | <pre> |
| 355 | ; RUN: llvm-as < %s | llvm-dis > %t1 |
| 356 | ; RUN: llvm-dis < %s.bc-13 > %t2 |
| 357 | ; RUN: diff %t1 %t2 |
| 358 | </pre> |
| 359 | </div> |
Reid Spencer | f190277 | 2007-04-15 08:01:04 +0000 | [diff] [blame] | 360 | |
Reid Spencer | 024a126 | 2007-04-14 23:27:06 +0000 | [diff] [blame] | 361 | <p>As with a Unix shell, the RUN: lines permit pipelines and I/O redirection |
| 362 | to be used. However, the usage is slightly different than for Bash. To check |
| 363 | what's legal, see the documentation for the |
| 364 | <a href="http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M2">Tcl exec</a> |
| 365 | command and the |
| 366 | <a href="http://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html">tutorial</a>. |
| 367 | The major differences are:</p> |
| 368 | <ul> |
| 369 | <li>You can't do <tt>2>&1</tt>. That will cause Tcl to write to a |
| 370 | file named <tt>&1</tt>. Usually this is done to get stderr to go through |
| 371 | a pipe. You can do that in tcl with <tt>|&</tt> so replace this idiom: |
| 372 | <tt>... 2>&1 | grep</tt> with <tt>... |& grep</tt></li> |
| 373 | <li>You can only redirect to a file, not to another descriptor and not from |
| 374 | a here document.</li> |
| 375 | <li>tcl supports redirecting to open files with the @ syntax but you |
| 376 | shouldn't use that here.</li> |
| 377 | </ul> |
| 378 | |
Reid Spencer | f190277 | 2007-04-15 08:01:04 +0000 | [diff] [blame] | 379 | <p>There are some quoting rules that you must pay attention to when writing |
| 380 | your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any |
| 381 | ' or " so they will get passed to the invoked program. For example:</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 382 | |
| 383 | <div class="doc_code"> |
| 384 | <pre> |
| 385 | ... | grep 'find this string' |
| 386 | </pre> |
| 387 | </div> |
| 388 | |
Reid Spencer | f190277 | 2007-04-15 08:01:04 +0000 | [diff] [blame] | 389 | <p>This will fail because the ' characters are passed to grep. This would |
| 390 | instruction grep to look for <tt>'find</tt> in the files <tt>this</tt> and |
| 391 | <tt>string'</tt>. To avoid this use curly braces to tell Tcl that it should |
| 392 | treat everything enclosed as one value. So our example would become:</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 393 | |
| 394 | <div class="doc_code"> |
| 395 | <pre> |
| 396 | ... | grep {find this string} |
| 397 | </pre> |
| 398 | </div> |
| 399 | |
Reid Spencer | f190277 | 2007-04-15 08:01:04 +0000 | [diff] [blame] | 400 | <p>Additionally, the characters <tt>[</tt> and <tt>]</tt> are treated |
| 401 | specially by Tcl. They tell Tcl to interpret the content as a command to |
| 402 | execute. Since these characters are often used in regular expressions this can |
| 403 | have disastrous results and cause the entire test run in a directory to fail. |
| 404 | For example, a common idiom is to look for some basicblock number:</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 405 | |
| 406 | <div class="doc_code"> |
| 407 | <pre> |
| 408 | ... | grep bb[2-8] |
| 409 | </pre> |
| 410 | </div> |
| 411 | |
Reid Spencer | f190277 | 2007-04-15 08:01:04 +0000 | [diff] [blame] | 412 | <p>This, however, will cause Tcl to fail because its going to try to execute |
| 413 | a program named "2-8". Instead, what you want is this:</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 414 | |
| 415 | <div class="doc_code"> |
| 416 | <pre> |
| 417 | ... | grep {bb\[2-8\]} |
| 418 | </pre> |
| 419 | </div> |
| 420 | |
Reid Spencer | f190277 | 2007-04-15 08:01:04 +0000 | [diff] [blame] | 421 | <p>Finally, if you need to pass the <tt>\</tt> character down to a program, |
| 422 | then it must be doubled. This is another Tcl special character. So, suppose |
| 423 | you had: |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 424 | |
| 425 | <div class="doc_code"> |
| 426 | <pre> |
| 427 | ... | grep 'i32\*' |
| 428 | </pre> |
| 429 | </div> |
| 430 | |
Reid Spencer | f190277 | 2007-04-15 08:01:04 +0000 | [diff] [blame] | 431 | <p>This will fail to match what you want (a pointer to i32). First, the |
| 432 | <tt>'</tt> do not get stripped off. Second, the <tt>\</tt> gets stripped off |
| 433 | by Tcl so what grep sees is: <tt>'i32*'</tt>. That's not likely to match |
| 434 | anything. To resolve this you must use <tt>\\</tt> and the <tt>{}</tt>, like |
| 435 | this:</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 436 | |
| 437 | <div class="doc_code"> |
| 438 | <pre> |
| 439 | ... | grep {i32\\*} |
| 440 | </pre> |
| 441 | </div> |
Reid Spencer | f190277 | 2007-04-15 08:01:04 +0000 | [diff] [blame] | 442 | |
Shantonu Sen | 1b6d3da | 2009-06-26 05:44:53 +0000 | [diff] [blame^] | 443 | <p>If your system includes GNU <tt>grep</tt>, make sure |
| 444 | that <tt>GREP_OPTIONS</tt> is not set in your environment. Otherwise, |
| 445 | you may get invalid results (both false positives and false |
| 446 | negatives).</p> |
| 447 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 448 | </div> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 449 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 450 | <!-- _______________________________________________________________________ --> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 451 | <div class="doc_subsection"><a name="dgvars">Variables and substitutions</a></div> |
| 452 | <!-- _______________________________________________________________________ --> |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 453 | <div class="doc_text"> |
| 454 | <p>With a RUN line there are a number of substitutions that are permitted. In |
| 455 | general, any Tcl variable that is available in the <tt>substitute</tt> |
| 456 | function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line. |
| 457 | To make a substitution just write the variable's name preceded by a $. |
| 458 | Additionally, for compatibility reasons with previous versions of the test |
| 459 | library, certain names can be accessed with an alternate syntax: a % prefix. |
| 460 | These alternates are deprecated and may go away in a future version. |
| 461 | </p> |
Bill Wendling | eb6aace | 2007-09-22 09:20:07 +0000 | [diff] [blame] | 462 | <p>Here are the available variable names. The alternate syntax is listed in |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 463 | parentheses.</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 464 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 465 | <dl style="margin-left: 25px"> |
| 466 | <dt><b>$test</b> (%s)</dt> |
| 467 | <dd>The full path to the test case's source. This is suitable for passing |
| 468 | on the command line as the input to an llvm tool.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 469 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 470 | <dt><b>$srcdir</b></dt> |
| 471 | <dd>The source directory from where the "<tt>make check</tt>" was run.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 472 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 473 | <dt><b>objdir</b></dt> |
Bill Wendling | eb6aace | 2007-09-22 09:20:07 +0000 | [diff] [blame] | 474 | <dd>The object directory that corresponds to the <tt>$srcdir</tt>.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 475 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 476 | <dt><b>subdir</b></dt> |
| 477 | <dd>A partial path from the <tt>test</tt> directory that contains the |
| 478 | sub-directory that contains the test source being executed.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 479 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 480 | <dt><b>srcroot</b></dt> |
| 481 | <dd>The root directory of the LLVM src tree.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 482 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 483 | <dt><b>objroot</b></dt> |
| 484 | <dd>The root directory of the LLVM object tree. This could be the same |
| 485 | as the srcroot.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 486 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 487 | <dt><b>path</b><dt> |
| 488 | <dd>The path to the directory that contains the test case source. This is |
| 489 | for locating any supporting files that are not generated by the test, but |
| 490 | used by the test.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 491 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 492 | <dt><b>tmp</b></dt> |
| 493 | <dd>The path to a temporary file name that could be used for this test case. |
| 494 | The file name won't conflict with other test cases. You can append to it if |
| 495 | you need multiple temporaries. This is useful as the destination of some |
| 496 | redirected output.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 497 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 498 | <dt><b>llvmlibsdir</b> (%llvmlibsdir)</dt> |
| 499 | <dd>The directory where the LLVM libraries are located.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 500 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 501 | <dt><b>target_triplet</b> (%target_triplet)</dt> |
| 502 | <dd>The target triplet that corresponds to the current host machine (the one |
| 503 | running the test cases). This should probably be called "host".<dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 504 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 505 | <dt><b>prcontext</b> (%prcontext)</dt> |
| 506 | <dd>Path to the prcontext tcl script that prints some context around a |
| 507 | line that matches a pattern. This isn't strictly necessary as the test suite |
| 508 | is run with its PATH altered to include the test/Scripts directory where |
| 509 | the prcontext script is located. Note that this script is similar to |
| 510 | <tt>grep -C</tt> but you should use the <tt>prcontext</tt> script because |
| 511 | not all platforms support <tt>grep -C</tt>.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 512 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 513 | <dt><b>llvmgcc</b> (%llvmgcc)</dt> |
| 514 | <dd>The full path to the <tt>llvm-gcc</tt> executable as specified in the |
| 515 | configured LLVM environment</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 516 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 517 | <dt><b>llvmgxx</b> (%llvmgxx)</dt> |
| 518 | <dd>The full path to the <tt>llvm-gxx</tt> executable as specified in the |
| 519 | configured LLVM environment</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 520 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 521 | <dt><b>llvmgcc_version</b> (%llvmgcc_version)</dt> |
| 522 | <dd>The full version number of the <tt>llvm-gcc</tt> executable.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 523 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 524 | <dt><b>llvmgccmajvers</b> (%llvmgccmajvers)</dt> |
| 525 | <dd>The major version number of the <tt>llvm-gcc</tt> executable.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 526 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 527 | <dt><b>gccpath</b></dt> |
| 528 | <dd>The full path to the C compiler used to <i>build </i> LLVM. Note that |
| 529 | this might not be gcc.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 530 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 531 | <dt><b>gxxpath</b></dt> |
| 532 | <dd>The full path to the C++ compiler used to <i>build </i> LLVM. Note that |
| 533 | this might not be g++.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 534 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 535 | <dt><b>compile_c</b> (%compile_c)</dt> |
| 536 | <dd>The full command line used to compile LLVM C source code. This has all |
| 537 | the configured -I, -D and optimization options.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 538 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 539 | <dt><b>compile_cxx</b> (%compile_cxx)</dt> |
| 540 | <dd>The full command used to compile LLVM C++ source code. This has |
| 541 | all the configured -I, -D and optimization options.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 542 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 543 | <dt><b>link</b> (%link)</dt> |
| 544 | <dd>This full link command used to link LLVM executables. This has all the |
| 545 | configured -I, -L and -l options.</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 546 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 547 | <dt><b>shlibext</b> (%shlibext)</dt> |
| 548 | <dd>The suffix for the host platforms share library (dll) files. This |
| 549 | includes the period as the first character.</dd> |
| 550 | </dl> |
| 551 | <p>To add more variables, two things need to be changed. First, add a line in |
| 552 | the <tt>test/Makefile</tt> that creates the <tt>site.exp</tt> file. This will |
| 553 | "set" the variable as a global in the site.exp file. Second, in the |
| 554 | <tt>test/lib/llvm.exp</tt> file, in the substitute proc, add the variable name |
| 555 | to the list of "global" declarations at the beginning of the proc. That's it, |
| 556 | the variable can then be used in test scripts.</p> |
| 557 | </div> |
| 558 | |
| 559 | <!-- _______________________________________________________________________ --> |
| 560 | <div class="doc_subsection"><a name="dgfeatures">Other Features</a></div> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 561 | <!-- _______________________________________________________________________ --> |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 562 | <div class="doc_text"> |
| 563 | <p>To make RUN line writing easier, there are several shell scripts located |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 564 | in the <tt>llvm/test/Scripts</tt> directory. This directory is in the PATH |
| 565 | when running tests, so you can just call these scripts using their name. For |
| 566 | example:</p> |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 567 | <dl> |
| 568 | <dt><b>ignore</b></dt> |
| 569 | <dd>This script runs its arguments and then always returns 0. This is useful |
| 570 | in cases where the test needs to cause a tool to generate an error (e.g. to |
| 571 | check the error output). However, any program in a pipeline that returns a |
| 572 | non-zero result will cause the test to fail. This script overcomes that |
| 573 | issue and nicely documents that the test case is purposefully ignoring the |
| 574 | result code of the tool</dd> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 575 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 576 | <dt><b>not</b></dt> |
| 577 | <dd>This script runs its arguments and then inverts the result code from |
| 578 | it. Zero result codes become 1. Non-zero result codes become 0. This is |
| 579 | useful to invert the result of a grep. For example "not grep X" means |
| 580 | succeed only if you don't find X in the input.</dd> |
| 581 | </dl> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 582 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 583 | <p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL. |
| 584 | You can easily mark a test as XFAIL just by including <tt>XFAIL: </tt> on a |
| 585 | line near the top of the file. This signals that the test case should succeed |
| 586 | if the test fails. Such test cases are counted separately by DejaGnu. To |
| 587 | specify an expected fail, use the XFAIL keyword in the comments of the test |
| 588 | program followed by a colon and one or more regular expressions (separated by |
| 589 | a comma). The regular expressions allow you to XFAIL the test conditionally |
| 590 | by host platform. The regular expressions following the : are matched against |
| 591 | the target triplet or llvmgcc version number for the host machine. If there is |
| 592 | a match, the test is expected to fail. If not, the test is expected to |
| 593 | succeed. To XFAIL everywhere just specify <tt>XFAIL: *</tt>. When matching |
| 594 | the llvm-gcc version, you can specify the major (e.g. 3) or full version |
| 595 | (i.e. 3.4) number. Here is an example of an <tt>XFAIL</tt> line:</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 596 | |
| 597 | <div class="doc_code"> |
| 598 | <pre> |
| 599 | ; XFAIL: darwin,sun,llvmgcc4 |
| 600 | </pre> |
| 601 | </div> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 602 | |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 603 | <p>To make the output more useful, the <tt>llvm_runtest</tt> function wil |
| 604 | scan the lines of the test case for ones that contain a pattern that matches |
| 605 | PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 606 | is related to the test case. The number after "PR" specifies the LLVM bugzilla |
Reid Spencer | bbb2a7a | 2007-04-14 21:46:15 +0000 | [diff] [blame] | 607 | number. When a PR number is specified, it will be used in the pass/fail |
| 608 | reporting. This is useful to quickly get some context when a test fails.</p> |
| 609 | |
| 610 | <p>Finally, any line that contains "END." will cause the special |
| 611 | interpretation of lines to terminate. This is generally done right after the |
| 612 | last RUN: line. This has two side effects: (a) it prevents special |
| 613 | interpretation of lines that are part of the test program, not the |
| 614 | instructions to the test case, and (b) it speeds things up for really big test |
| 615 | cases by avoiding interpretation of the remainder of the file.</p> |
Tanya Lattner | 5026c7f | 2004-12-06 02:11:52 +0000 | [diff] [blame] | 616 | |
| 617 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 618 | |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 619 | <!--=========================================================================--> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 620 | <div class="doc_section"><a name="testsuitestructure">Test suite |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 621 | Structure</a></div> |
| 622 | <!--=========================================================================--> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 623 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 624 | <div class="doc_text"> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 625 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 626 | <p>The <tt>test-suite</tt> module contains a number of programs that can be compiled |
| 627 | with LLVM and executed. These programs are compiled using the native compiler |
| 628 | and various LLVM backends. The output from the program compiled with the |
| 629 | native compiler is assumed correct; the results from the other programs are |
| 630 | compared to the native program output and pass if they match.</p> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 631 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 632 | <p>When executing tests, it is usually a good idea to start out with a subset of |
| 633 | the available tests or programs. This makes test run times smaller at first and |
| 634 | later on this is useful to investigate individual test failures. To run some |
| 635 | test only on a subset of programs, simply change directory to the programs you |
| 636 | want tested and run <tt>gmake</tt> there. Alternatively, you can run a different |
| 637 | test using the <tt>TEST</tt> variable to change what tests or run on the |
| 638 | selected programs (see below for more info).</p> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 639 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 640 | <p>In addition for testing correctness, the <tt>llvm-test</tt> directory also |
| 641 | performs timing tests of various LLVM optimizations. It also records |
| 642 | compilation times for the compilers and the JIT. This information can be |
| 643 | used to compare the effectiveness of LLVM's optimizations and code |
| 644 | generation.</p> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 645 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 646 | <p><tt>llvm-test</tt> tests are divided into three types of tests: MultiSource, |
| 647 | SingleSource, and External.</p> |
Reid Spencer | 3281ead | 2004-12-08 16:52:51 +0000 | [diff] [blame] | 648 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 649 | <ul> |
| 650 | <li><tt>llvm-test/SingleSource</tt> |
| 651 | <p>The SingleSource directory contains test programs that are only a single |
| 652 | source file in size. These are usually small benchmark programs or small |
| 653 | programs that calculate a particular value. Several such programs are grouped |
| 654 | together in each directory.</p></li> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 655 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 656 | <li><tt>llvm-test/MultiSource</tt> |
| 657 | <p>The MultiSource directory contains subdirectories which contain entire |
| 658 | programs with multiple source files. Large benchmarks and whole applications |
| 659 | go here.</p></li> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 660 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 661 | <li><tt>llvm-test/External</tt> |
| 662 | <p>The External directory contains Makefiles for building code that is external |
| 663 | to (i.e., not distributed with) LLVM. The most prominent members of this |
| 664 | directory are the SPEC 95 and SPEC 2000 benchmark suites. The <tt>External</tt> |
Stuart Hastings | c4c268b | 2009-05-21 20:23:59 +0000 | [diff] [blame] | 665 | directory does not contain these actual tests, but only the Makefiles that know |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 666 | how to properly compile these programs from somewhere else. The presence and |
| 667 | location of these external programs is configured by the llvm-test |
| 668 | <tt>configure</tt> script.</p></li> |
| 669 | </ul> |
| 670 | |
| 671 | <p>Each tree is then subdivided into several categories, including applications, |
| 672 | benchmarks, regression tests, code that is strange grammatically, etc. These |
| 673 | organizations should be relatively self explanatory.</p> |
| 674 | |
| 675 | <p>Some tests are known to fail. Some are bugs that we have not fixed yet; |
| 676 | others are features that we haven't added yet (or may never add). In DejaGNU, |
| 677 | the result for such tests will be XFAIL (eXpected FAILure). In this way, you |
| 678 | can tell the difference between an expected and unexpected failure.</p> |
| 679 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 680 | <p>The tests in the test suite have no such feature at this time. If the |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 681 | test passes, only warnings and other miscellaneous output will be generated. If |
| 682 | a test fails, a large <program> FAILED message will be displayed. This |
| 683 | will help you separate benign warnings from actual test failures.</p> |
| 684 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 685 | </div> |
| 686 | |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 687 | <!--=========================================================================--> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 688 | <div class="doc_section"><a name="testsuiterun">Running the test suite</a></div> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 689 | <!--=========================================================================--> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 690 | |
| 691 | <div class="doc_text"> |
| 692 | |
| 693 | <p>First, all tests are executed within the LLVM object directory tree. They |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 694 | <i>are not</i> executed inside of the LLVM source tree. This is because the |
John Mosby | 3228abe | 2009-03-30 18:56:53 +0000 | [diff] [blame] | 695 | test suite creates temporary files during execution.</p> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 696 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 697 | <p>To run the test suite, you need to use the following steps:</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 698 | |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 699 | <ol> |
John Mosby | 24446d6 | 2009-03-30 04:37:51 +0000 | [diff] [blame] | 700 | <li><tt>cd</tt> into the <tt>llvm/projects</tt> directory in your source tree. |
| 701 | </li> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 702 | |
| 703 | <li><p>Check out the <tt>test-suite</tt> module with:</p> |
| 704 | |
| 705 | <div class="doc_code"> |
| 706 | <pre> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 707 | % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 708 | </pre> |
| 709 | </div> |
Stuart Hastings | c4c268b | 2009-05-21 20:23:59 +0000 | [diff] [blame] | 710 | <p>This will get the test suite into <tt>llvm/projects/test-suite</tt>.</p> |
John Mosby | 24446d6 | 2009-03-30 04:37:51 +0000 | [diff] [blame] | 711 | </li> |
Stuart Hastings | c4c268b | 2009-05-21 20:23:59 +0000 | [diff] [blame] | 712 | <li><p>Configure and build <tt>llvm</tt>.</p></li> |
| 713 | <li><p>Configure and build <tt>llvm-gcc</tt>.</p></li> |
| 714 | <li><p>Install <tt>llvm-gcc</tt> somewhere.</p></li> |
| 715 | <li><p><em>Re-configure</em> <tt>llvm</tt> from the top level of |
| 716 | each build tree (LLVM object directory tree) in which you want |
| 717 | to run the test suite, just as you do before building LLVM.</p> |
| 718 | <p>During the <em>re-configuration</em>, you must either: (1) |
| 719 | have <tt>llvm-gcc</tt> you just built in your path, or (2) |
| 720 | specify the directory where your just-built <tt>llvm-gcc</tt> is |
| 721 | installed using <tt>--with-llvmgccdir=$LLVM_GCC_DIR</tt>.</p> |
| 722 | <p>You must also tell the configure machinery that the test suite |
| 723 | is available so it can be configured for your build tree:</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 724 | <div class="doc_code"> |
| 725 | <pre> |
John Mosby | 24446d6 | 2009-03-30 04:37:51 +0000 | [diff] [blame] | 726 | % cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure [--with-llvmgccdir=$LLVM_GCC_DIR] |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 727 | </pre> |
| 728 | </div> |
John Mosby | 24446d6 | 2009-03-30 04:37:51 +0000 | [diff] [blame] | 729 | <p>[Remember that <tt>$LLVM_GCC_DIR</tt> is the directory where you |
| 730 | <em>installed</em> llvm-gcc, not its src or obj directory.]</p> |
Matthijs Kooijman | d30020a | 2008-05-20 10:28:55 +0000 | [diff] [blame] | 731 | </li> |
| 732 | |
John Mosby | 24446d6 | 2009-03-30 04:37:51 +0000 | [diff] [blame] | 733 | <li><p>You can now run the test suite from your build tree as follows:</p> |
| 734 | <div class="doc_code"> |
| 735 | <pre> |
| 736 | % cd $LLVM_OBJ_ROOT/projects/test-suite |
| 737 | % make |
| 738 | </pre> |
| 739 | </div> |
| 740 | </li> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 741 | </ol> |
| 742 | <p>Note that the second and third steps only need to be done once. After you |
| 743 | have the suite checked out and configured, you don't need to do it again (unless |
Matthijs Kooijman | d30020a | 2008-05-20 10:28:55 +0000 | [diff] [blame] | 744 | the test code or configure script changes).</p> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 745 | |
Shantonu Sen | 1b6d3da | 2009-06-26 05:44:53 +0000 | [diff] [blame^] | 746 | </div> |
| 747 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 748 | <!-- _______________________________________________________________________ --> |
| 749 | <div class="doc_subsection"> |
Stuart Hastings | c4c268b | 2009-05-21 20:23:59 +0000 | [diff] [blame] | 750 | <a name="testsuiteexternal">Configuring External Tests</a></div> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 751 | <!-- _______________________________________________________________________ --> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 752 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 753 | <div class="doc_text"> |
Stuart Hastings | c4c268b | 2009-05-21 20:23:59 +0000 | [diff] [blame] | 754 | <p>In order to run the External tests in the <tt>test-suite</tt> |
| 755 | module, you must specify <i>--with-externals</i>. This |
| 756 | must be done during the <em>re-configuration</em> step (see above), |
| 757 | and the <tt>llvm</tt> re-configuration must recognize the |
| 758 | previously-built <tt>llvm-gcc</tt>. If any of these is missing or |
| 759 | neglected, the External tests won't work.</p> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 760 | <dl> |
Dale Johannesen | 80b9902 | 2008-12-10 01:58:32 +0000 | [diff] [blame] | 761 | <dt><i>--with-externals</i></dt> |
| 762 | <dt><i>--with-externals=<<tt>directory</tt>></i></dt> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 763 | </dl> |
Dale Johannesen | 80b9902 | 2008-12-10 01:58:32 +0000 | [diff] [blame] | 764 | This tells LLVM where to find any external tests. They are expected to be |
| 765 | in specifically named subdirectories of <<tt>directory</tt>>. |
| 766 | If <tt>directory</tt> is left unspecified, |
| 767 | <tt>configure</tt> uses the default value |
| 768 | <tt>/home/vadve/shared/benchmarks/speccpu2000/benchspec</tt>. |
| 769 | Subdirectory names known to LLVM include: |
| 770 | <dl> |
| 771 | <dt>spec95</dt> |
| 772 | <dt>speccpu2000</dt> |
| 773 | <dt>speccpu2006</dt> |
| 774 | <dt>povray31</dt> |
| 775 | </dl> |
| 776 | Others are added from time to time, and can be determined from |
| 777 | <tt>configure</tt>. |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 778 | </div> |
| 779 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 780 | <!-- _______________________________________________________________________ --> |
| 781 | <div class="doc_subsection"> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 782 | <a name="testsuitetests">Running different tests</a></div> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 783 | <!-- _______________________________________________________________________ --> |
| 784 | <div class="doc_text"> |
Stuart Hastings | c4c268b | 2009-05-21 20:23:59 +0000 | [diff] [blame] | 785 | <p>In addition to the regular "whole program" tests, the <tt>test-suite</tt> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 786 | module also provides a mechanism for compiling the programs in different ways. |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 787 | If the variable TEST is defined on the <tt>gmake</tt> command line, the test system will |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 788 | include a Makefile named <tt>TEST.<value of TEST variable>.Makefile</tt>. |
| 789 | This Makefile can modify build rules to yield different results.</p> |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 790 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 791 | <p>For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt> to |
| 792 | create the nightly test reports. To run the nightly tests, run <tt>gmake |
| 793 | TEST=nightly</tt>.</p> |
| 794 | |
| 795 | <p>There are several TEST Makefiles available in the tree. Some of them are |
| 796 | designed for internal LLVM research and will not work outside of the LLVM |
| 797 | research group. They may still be valuable, however, as a guide to writing your |
| 798 | own TEST Makefile for any optimization or analysis passes that you develop with |
| 799 | LLVM.</p> |
| 800 | |
Bill Wendling | da51c4c | 2007-09-22 09:16:44 +0000 | [diff] [blame] | 801 | </div> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 802 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 803 | <!-- _______________________________________________________________________ --> |
| 804 | <div class="doc_subsection"> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 805 | <a name="testsuiteoutput">Generating test output</a></div> |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 806 | <!-- _______________________________________________________________________ --> |
| 807 | <div class="doc_text"> |
| 808 | <p>There are a number of ways to run the tests and generate output. The most |
| 809 | simple one is simply running <tt>gmake</tt> with no arguments. This will |
| 810 | compile and run all programs in the tree using a number of different methods |
| 811 | and compare results. Any failures are reported in the output, but are likely |
| 812 | drowned in the other output. Passes are not reported explicitely.</p> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 813 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 814 | <p>Somewhat better is running <tt>gmake TEST=sometest test</tt>, which runs |
| 815 | the specified test and usually adds per-program summaries to the output |
| 816 | (depending on which sometest you use). For example, the <tt>nightly</tt> test |
| 817 | explicitely outputs TEST-PASS or TEST-FAIL for every test after each program. |
| 818 | Though these lines are still drowned in the output, it's easy to grep the |
| 819 | output logs in the Output directories.</p> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 820 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 821 | <p>Even better are the <tt>report</tt> and <tt>report.format</tt> targets |
| 822 | (where <tt>format</tt> is one of <tt>html</tt>, <tt>csv</tt>, <tt>text</tt> or |
| 823 | <tt>graphs</tt>). The exact contents of the report are dependent on which |
| 824 | <tt>TEST</tt> you are running, but the text results are always shown at the |
| 825 | end of the run and the results are always stored in the |
| 826 | <tt>report.<type>.format</tt> file (when running with |
| 827 | <tt>TEST=<type></tt>). |
Chris Lattner | fd9d1b3 | 2004-06-24 20:53:09 +0000 | [diff] [blame] | 828 | |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 829 | The <tt>report</tt> also generate a file called |
| 830 | <tt>report.<type>.raw.out</tt> containing the output of the entire test |
| 831 | run. |
Chris Lattner | fd9d1b3 | 2004-06-24 20:53:09 +0000 | [diff] [blame] | 832 | </div> |
| 833 | |
Chris Lattner | eb82da8 | 2006-05-23 01:40:20 +0000 | [diff] [blame] | 834 | <!-- _______________________________________________________________________ --> |
| 835 | <div class="doc_subsection"> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 836 | <a name="testsuitecustom">Writing custom tests for the test suite</a></div> |
Chris Lattner | eb82da8 | 2006-05-23 01:40:20 +0000 | [diff] [blame] | 837 | <!-- _______________________________________________________________________ --> |
| 838 | |
| 839 | <div class="doc_text"> |
| 840 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 841 | <p>Assuming you can run the test suite, (e.g. "<tt>gmake TEST=nightly report</tt>" |
Chris Lattner | eb82da8 | 2006-05-23 01:40:20 +0000 | [diff] [blame] | 842 | should work), it is really easy to run optimizations or code generator |
| 843 | components against every program in the tree, collecting statistics or running |
| 844 | custom checks for correctness. At base, this is how the nightly tester works, |
| 845 | it's just one example of a general framework.</p> |
| 846 | |
| 847 | <p>Lets say that you have an LLVM optimization pass, and you want to see how |
| 848 | many times it triggers. First thing you should do is add an LLVM |
| 849 | <a href="ProgrammersManual.html#Statistic">statistic</a> to your pass, which |
| 850 | will tally counts of things you care about.</p> |
| 851 | |
| 852 | <p>Following this, you can set up a test and a report that collects these and |
| 853 | formats them for easy viewing. This consists of two files, an |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 854 | "<tt>test-suite/TEST.XXX.Makefile</tt>" fragment (where XXX is the name of your |
Chris Lattner | eb82da8 | 2006-05-23 01:40:20 +0000 | [diff] [blame] | 855 | test) and an "<tt>llvm-test/TEST.XXX.report</tt>" file that indicates how to |
| 856 | format the output into a table. There are many example reports of various |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 857 | levels of sophistication included with the test suite, and the framework is very |
Chris Lattner | eb82da8 | 2006-05-23 01:40:20 +0000 | [diff] [blame] | 858 | general.</p> |
| 859 | |
| 860 | <p>If you are interested in testing an optimization pass, check out the |
| 861 | "libcalls" test as an example. It can be run like this:<p> |
| 862 | |
| 863 | <div class="doc_code"> |
| 864 | <pre> |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 865 | % cd llvm/projects/test-suite/MultiSource/Benchmarks # or some other level |
Chris Lattner | eb82da8 | 2006-05-23 01:40:20 +0000 | [diff] [blame] | 866 | % make TEST=libcalls report |
| 867 | </pre> |
| 868 | </div> |
| 869 | |
| 870 | <p>This will do a bunch of stuff, then eventually print a table like this:</p> |
| 871 | |
| 872 | <div class="doc_code"> |
| 873 | <pre> |
| 874 | Name | total | #exit | |
| 875 | ... |
| 876 | FreeBench/analyzer/analyzer | 51 | 6 | |
| 877 | FreeBench/fourinarow/fourinarow | 1 | 1 | |
| 878 | FreeBench/neural/neural | 19 | 9 | |
| 879 | FreeBench/pifft/pifft | 5 | 3 | |
| 880 | MallocBench/cfrac/cfrac | 1 | * | |
| 881 | MallocBench/espresso/espresso | 52 | 12 | |
| 882 | MallocBench/gs/gs | 4 | * | |
| 883 | Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * | |
| 884 | Prolangs-C/agrep/agrep | 33 | 12 | |
| 885 | Prolangs-C/allroots/allroots | * | * | |
| 886 | Prolangs-C/assembler/assembler | 47 | * | |
| 887 | Prolangs-C/bison/mybison | 74 | * | |
| 888 | ... |
| 889 | </pre> |
| 890 | </div> |
| 891 | |
| 892 | <p>This basically is grepping the -stats output and displaying it in a table. |
| 893 | You can also use the "TEST=libcalls report.html" target to get the table in HTML |
| 894 | form, similarly for report.csv and report.tex.</p> |
| 895 | |
Matthijs Kooijman | 31ce08f | 2008-06-24 12:58:31 +0000 | [diff] [blame] | 896 | <p>The source for this is in test-suite/TEST.libcalls.*. The format is pretty |
Chris Lattner | eb82da8 | 2006-05-23 01:40:20 +0000 | [diff] [blame] | 897 | simple: the Makefile indicates how to run the test (in this case, |
| 898 | "<tt>opt -simplify-libcalls -stats</tt>"), and the report contains one line for |
| 899 | each column of the output. The first value is the header for the column and the |
| 900 | second is the regex to grep the output of the command for. There are lots of |
| 901 | example reports that can do fancy stuff.</p> |
| 902 | |
| 903 | </div> |
| 904 | |
| 905 | |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 906 | <!--=========================================================================--> |
Chris Lattner | fd9d1b3 | 2004-06-24 20:53:09 +0000 | [diff] [blame] | 907 | <div class="doc_section"><a name="nightly">Running the nightly tester</a></div> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 908 | <!--=========================================================================--> |
Chris Lattner | fd9d1b3 | 2004-06-24 20:53:09 +0000 | [diff] [blame] | 909 | |
| 910 | <div class="doc_text"> |
| 911 | |
| 912 | <p> |
Patrick Jenkins | 825c2a7 | 2006-08-11 23:27:02 +0000 | [diff] [blame] | 913 | The <a href="http://llvm.org/nightlytest/">LLVM Nightly Testers</a> |
Chris Lattner | fd9d1b3 | 2004-06-24 20:53:09 +0000 | [diff] [blame] | 914 | automatically check out an LLVM tree, build it, run the "nightly" |
Matthijs Kooijman | 6fce844 | 2008-05-23 11:45:18 +0000 | [diff] [blame] | 915 | program test (described above), run all of the DejaGNU tests, |
Patrick Jenkins | 825c2a7 | 2006-08-11 23:27:02 +0000 | [diff] [blame] | 916 | delete the checked out tree, and then submit the results to |
| 917 | <a href="http://llvm.org/nightlytest/">http://llvm.org/nightlytest/</a>. |
| 918 | After test results are submitted to |
| 919 | <a href="http://llvm.org/nightlytest/">http://llvm.org/nightlytest/</a>, |
| 920 | they are processed and displayed on the tests page. An email to |
| 921 | <a href="http://lists.cs.uiuc.edu/pipermail/llvm-testresults/"> |
| 922 | llvm-testresults@cs.uiuc.edu</a> summarizing the results is also generated. |
| 923 | This testing scheme is designed to ensure that programs don't break as well |
| 924 | as keep track of LLVM's progress over time.</p> |
Chris Lattner | fd9d1b3 | 2004-06-24 20:53:09 +0000 | [diff] [blame] | 925 | |
Patrick Jenkins | 825c2a7 | 2006-08-11 23:27:02 +0000 | [diff] [blame] | 926 | <p>If you'd like to set up an instance of the nightly tester to run on your |
| 927 | machine, take a look at the comments at the top of the |
| 928 | <tt>utils/NewNightlyTest.pl</tt> file. If you decide to set up a nightly tester |
| 929 | please choose a unique nickname and invoke <tt>utils/NewNightlyTest.pl</tt> |
Reid Spencer | 669ed45 | 2007-07-09 08:04:31 +0000 | [diff] [blame] | 930 | with the "-nickname [yournickname]" command line option. |
Chris Lattner | fd9d1b3 | 2004-06-24 20:53:09 +0000 | [diff] [blame] | 931 | |
Reid Spencer | 669ed45 | 2007-07-09 08:04:31 +0000 | [diff] [blame] | 932 | <p>You can create a shell script to encapsulate the running of the script. |
Misha Brukman | fc1a27b | 2005-03-10 22:51:59 +0000 | [diff] [blame] | 933 | The optimized x86 Linux nightly test is run from just such a script:</p> |
| 934 | |
| 935 | <div class="doc_code"> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 936 | <pre> |
| 937 | #!/bin/bash |
| 938 | BASE=/proj/work/llvm/nightlytest |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 939 | export BUILDDIR=$BASE/build |
| 940 | export WEBDIR=$BASE/testresults |
| 941 | export LLVMGCCDIR=/proj/work/llvm/cfrontend/install |
| 942 | export PATH=/proj/install/bin:$LLVMGCCDIR/bin:$PATH |
| 943 | export LD_LIBRARY_PATH=/proj/install/lib |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 944 | cd $BASE |
Patrick Jenkins | 825c2a7 | 2006-08-11 23:27:02 +0000 | [diff] [blame] | 945 | cp /proj/work/llvm/llvm/utils/NewNightlyTest.pl . |
| 946 | nice ./NewNightlyTest.pl -nice -release -verbose -parallel -enable-linscan \ |
Reid Spencer | 669ed45 | 2007-07-09 08:04:31 +0000 | [diff] [blame] | 947 | -nickname NightlyTester -noexternals > output.log 2>&1 |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 948 | </pre> |
Misha Brukman | fc1a27b | 2005-03-10 22:51:59 +0000 | [diff] [blame] | 949 | </div> |
Reid Spencer | 8284f1f | 2004-09-05 20:07:26 +0000 | [diff] [blame] | 950 | |
Patrick Jenkins | 825c2a7 | 2006-08-11 23:27:02 +0000 | [diff] [blame] | 951 | <p>It is also possible to specify the the location your nightly test results |
| 952 | are submitted. You can do this by passing the command line option |
| 953 | "-submit-server [server_address]" and "-submit-script [script_on_server]" to |
| 954 | <tt>utils/NewNightlyTest.pl</tt>. For example, to submit to the llvm.org |
| 955 | nightly test results page, you would invoke the nightly test script with |
| 956 | "-submit-server llvm.org -submit-script /nightlytest/NightlyTestAccept.cgi". |
| 957 | If these options are not specified, the nightly test script sends the results |
| 958 | to the llvm.org nightly test results page.</p> |
| 959 | |
| 960 | <p>Take a look at the <tt>NewNightlyTest.pl</tt> file to see what all of the |
| 961 | flags and strings do. If you start running the nightly tests, please let us |
| 962 | know. Thanks!</p> |
Misha Brukman | fc1a27b | 2005-03-10 22:51:59 +0000 | [diff] [blame] | 963 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 964 | </div> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 965 | |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 966 | <!-- *********************************************************************** --> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 967 | |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 968 | <hr> |
| 969 | <address> |
| 970 | <a href="http://jigsaw.w3.org/css-validator/check/referer"><img |
Misha Brukman | 4440870 | 2008-12-11 17:34:48 +0000 | [diff] [blame] | 971 | src="http://jigsaw.w3.org/css-validator/images/vcss-blue" alt="Valid CSS"></a> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 972 | <a href="http://validator.w3.org/check/referer"><img |
Misha Brukman | 4440870 | 2008-12-11 17:34:48 +0000 | [diff] [blame] | 973 | src="http://www.w3.org/Icons/valid-html401-blue" alt="Valid HTML 4.01"></a> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 974 | |
John Criswell | c6e32eb | 2005-05-13 19:48:07 +0000 | [diff] [blame] | 975 | John T. Criswell, Reid Spencer, and Tanya Lattner<br> |
Matthijs Kooijman | d30020a | 2008-05-20 10:28:55 +0000 | [diff] [blame] | 976 | <a href="http://llvm.org">The LLVM Compiler Infrastructure</a><br> |
Misha Brukman | 1d83e11 | 2004-03-01 18:21:04 +0000 | [diff] [blame] | 977 | Last modified: $Date$ |
| 978 | </address> |
Brian Gaeke | af19f2e | 2003-10-23 18:10:28 +0000 | [diff] [blame] | 979 | </body> |
John Criswell | 020cbd8 | 2003-10-10 18:42:49 +0000 | [diff] [blame] | 980 | </html> |