njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 1 | INTRO |
| 2 | ----- |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 3 | This directory (nightly/) contains a simple, automatic build-and-test |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 4 | system for Valgrind, intended to be run nightly by cron or a similar |
| 5 | program. |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 6 | |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 7 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 8 | BASIC OPERATIONS |
| 9 | ---------------- |
| 10 | When run, the system checks out two trees: the SVN trunk from 24 hours ago |
| 11 | and the SVN trunk from now. ("24 hours ago" and "now" are determined when |
| 12 | the script starts running, so if any commits happen while the tests are |
| 13 | running they will not be tested.) |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 14 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 15 | If the two trees are different (i.e. there have been commits in the past 24 |
| 16 | hours) it builds ("make"), installs ("make install") and runs the regression |
| 17 | tests ("make regtest") in both, and compares the results. Note that the |
| 18 | "make install" isn't necessary in order to run the tests because the |
| 19 | regression tests use the code built (with "make") within the tree, but it's |
| 20 | worth doing because it tests that "make install" isn't totally broken. |
| 21 | After checking both trees, it emails a summary of the results to a |
| 22 | recipient. All this typically takes something like 30 minutes. |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 23 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 24 | If the two trees are identical, the tests are not run and no results are |
| 25 | emailed. This avoids spamming people with uninteresting results emails when |
| 26 | no commits have happened recently. |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 27 | |
njn | 5a1ba02 | 2009-01-07 04:14:42 +0000 | [diff] [blame] | 28 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 29 | SETTING UP |
| 30 | ---------- |
| 31 | To set up nightly testing for a machine, do the following. |
njn | 5a1ba02 | 2009-01-07 04:14:42 +0000 | [diff] [blame] | 32 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 33 | (1) Check out just this directory from the repository, eg: |
| 34 | |
| 35 | svn co svn://svn.valgrind.org/valgrind/trunk/nightly $DIR |
| 36 | |
| 37 | where $DIR is the name of the directory you want it to be in. |
| 38 | |
| 39 | Note that this doesn't check out the whole Valgrind tree, just the |
| 40 | directory containing the nightly testing stuff. This is possible |
| 41 | because the testing script doesn't check the code in the tree it belongs |
| 42 | to; rather it checks out new trees (within $DIR) and tests them |
| 43 | independently. |
| 44 | |
| 45 | (2) Choose a tag that identifies the test results. This is usually the |
| 46 | machine name. We'll call it $TAG in what follows. |
| 47 | |
| 48 | (3) Create a configuration file $DIR/conf/$TAG.conf. It is sourced by the |
| 49 | 'nightly' script, and can define any or all of the following environment |
| 50 | variables. (In most cases, only ABT_DETAILS is needed.) |
| 51 | |
| 52 | - ABT_DETAILS: describes the machine in more detail, eg. the OS. The |
| 53 | default is empty, but you should define it. An example: |
| 54 | |
| 55 | export ABT_DETAILS="Ubuntu 9.04, Intel x86-64" |
| 56 | |
| 57 | You could also use some invocation of 'uname' or something similar |
| 58 | to generate this string. Eg. on Ubuntu Linux this works nicely: |
| 59 | |
| 60 | export ABT_DETAILS="`cat /etc/issue.net`, `uname -m`" |
| 61 | |
| 62 | And on Mac OS X this works nicely: |
| 63 | |
| 64 | export ABT_DETAILS=`uname -mrs` |
| 65 | |
| 66 | The advantage of doing it like this is that if you update the OS on |
| 67 | the test machine you won't have to update ABT_DETAILS manually. |
| 68 | |
| 69 | - ABT_CONFIGURE_OPTIONS: gives extra configure options. The default is |
| 70 | empty. |
| 71 | |
| 72 | - ABT_EVAL: if provided, it must be the name of a shell script that |
| 73 | executes the shell command $1 with arguments $2 .. ${$#}. Allows to |
| 74 | compile and run the Valgrind regression tests on another system than |
| 75 | the system the 'nightly' script runs on. It is assumed that the remote |
| 76 | system shares the local filesystem tree through e.g. NFS. It is the |
| 77 | responsibility of the shell script to set the remote working directory |
| 78 | such that it matches the local current directory ($PWD). |
| 79 | |
| 80 | - ABT_RUN_REGTEST: if provided, it must be the name of an argumentless |
| 81 | shell function (also specified in the $TAG.conf file) that will be used |
| 82 | to run the tests. If not specified, the usual "make regtest" will be |
| 83 | used. |
| 84 | |
| 85 | - ABT_JOBS: allows parallel builds -- it's passed as the argument to |
| 86 | "make -j" when building Valgrind and the tests. The default is 1. |
| 87 | [XXX: the .NOTPARALLEL that currently resides in Makefile.all.am foils |
| 88 | this!] |
| 89 | |
| 90 | Note that the appropriate syntax to use in this file will depend on the |
| 91 | shell from which the $DIR/bin/nightly script is run (which in turn may |
| 92 | depend on what shell is used by cron or any similar program). |
| 93 | |
| 94 | (4) Create a mailer script $DIR/conf/$TAG.sendmail. It must be executable. |
| 95 | It's used to send email results to the desired recipient (e.g. |
| 96 | valgrind-developers@lists.sourceforge.net) It must handle three command |
| 97 | line arguments. |
| 98 | |
| 99 | - The first argument is the email subject line. It contains |
| 100 | $ABT_DETAILS plus some other stuff. |
| 101 | |
| 102 | - The second argument is the name of the file containing the email's |
| 103 | body (which shows the tests that failed, and the differences between now |
| 104 | and 24 hours ago). |
| 105 | |
| 106 | - The third is the name of the file containing all the diffs from |
| 107 | failing tests. Depending on the test results you get, you could |
| 108 | inline this file into the email body, or attach it, or compress and |
| 109 | attach it, or even omit it. The right choice depends on how many |
| 110 | failures you typically get -- if you get few failures, inlining the |
| 111 | results make them easier to read; if you get many failures, |
| 112 | compressing might be a good idea to minimise the size of the emails. |
| 113 | |
| 114 | The best way to do this depends on how mail is set up on your machine. |
| 115 | You might be able to use /usr/bin/mail, or you might need something more |
| 116 | elaborate like using Mutt to send mail via an external account. |
| 117 | |
njn | 1d96fc5 | 2009-06-04 01:52:14 +0000 | [diff] [blame] | 118 | At first, you should probably just send emails to yourself for testing |
| 119 | purposes. After it's working, then sending it to others might be |
| 120 | appropriate. |
| 121 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 122 | (5) To run the tests, execute: |
| 123 | |
| 124 | $DIR/bin/nightly $DIR $TAG |
| 125 | |
| 126 | You probably want to put this command into a cron file or equivalent |
njn | 1d96fc5 | 2009-06-04 01:52:14 +0000 | [diff] [blame] | 127 | so it is run regularly (preferably every night). Actually, it's |
| 128 | probably better to put that command inside a script, and run the script |
| 129 | from cron, rather than running $DIR/bin/nightly directly. That way you |
| 130 | can put any other configuration stuff that's necessary inside the |
| 131 | script (e.g. make sure that programs used by the mailer script are in |
| 132 | your PATH). |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 133 | |
| 134 | |
| 135 | OUTPUT FILES |
| 136 | ------------ |
| 137 | If the tests are run, the following files are produced: |
| 138 | |
| 139 | - $DIR/old.verbose and $DIR/new.verbose contain full output of the whole |
| 140 | process for each of the two trees. |
| 141 | |
| 142 | - $DIR/old.short and $DIR/new.short contain summary output of the process |
| 143 | for each of the two trees. The diff between these two files goes in |
| 144 | $DIR/diff.short. |
| 145 | |
| 146 | - $DIR/final contains the overall summary, constructed from $DIR/old.short, |
| 147 | $DIR/new.short, $DIR/diff.short and some other bits and pieces. (The name |
| 148 | of this file is what's passed as the second argument to |
| 149 | $DIR/conf/$TAG.sendmail.) |
| 150 | |
| 151 | - $DIR/diffs holds the diffs from all the failing tests in the newer tree, |
| 152 | concatenated together; the diff from each failure is truncated at 100 |
| 153 | lines to minimise possible size blow-outs. (The name of this file is |
| 154 | what's passed as the third argument to $DIR/conf/$TAG.sendmail.) |
| 155 | |
| 156 | - $DIR/sendmail.log contains the output (stdout and stderr) from |
| 157 | $DIR/conf/$TAG.sendmail goes in $DIR/sendmail.log. |
| 158 | |
njn | e43358f | 2009-06-04 23:17:12 +0000 | [diff] [blame^] | 159 | - $DIR/valgrind-old/ and $DIR/valgrind-new/ contain the tested trees (and |
| 160 | $DIR/valgrind-old/Inst/ and $DIR/valgrind-new/Inst/ contain the installed |
| 161 | code). |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 162 | |
| 163 | If the tests aren't run, the following file is produced: |
| 164 | |
| 165 | - $DIR/unchanged.log is created only if no tests were run because the two |
| 166 | trees were identical. It will contain a short explanatory message. |
| 167 | |
| 168 | Each time the tests are run, all files from previous runs are deleted. |
| 169 | |
| 170 | |
| 171 | TROUBLESHOOTING |
| 172 | --------------- |
| 173 | If something goes wrong, looking at the output files can be useful. For |
| 174 | example, if no email was sent but you expected one, check sendmail.log to |
| 175 | see if the mailer script had a problem. Or check if unchanged.log exists. |
| 176 | |
| 177 | Occasionally the SVN server isn't available when the tests runs, for either |
| 178 | or both trees. When this happens the email will be sent but it won't be |
| 179 | very informative. Usually it's just a temporary server problem and it'll |
| 180 | run fine the next time without you having to do anything. |
| 181 | |
| 182 | Note that the test suite is imperfect: |
| 183 | - There are very few machines where all tests pass; that's why the old/new |
| 184 | diff is produced. Some of the tests may not be as portable as intended. |
| 185 | - Some tests are non-deterministic, and so may pass one day and fail the |
| 186 | next. |
| 187 | |
| 188 | Improving the test suite to avoid these problems is a long-term goal but it |
| 189 | isn't easy. |
| 190 | |
| 191 | |
| 192 | MAINTENANCE |
| 193 | ----------- |
| 194 | The scripts in the nightly/ directory occasionally get updated. If that |
| 195 | happens, you can just "svn update" within $DIR to get the updated versions, |
| 196 | which will then be used the next time the tests run. (It's possible that |
| 197 | the scripts will be changed in a way that requires changes to the files in |
| 198 | $DIR/conf/, but we try to avoid this.) |
| 199 | |
| 200 | If you want such updates to happen automatically, you could write a script |
| 201 | that does all the steps in SETTING UP above, and instead run that script |
| 202 | from cron. |
| 203 | |
| 204 | |