njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 1 | INTRO |
| 2 | ----- |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 3 | This directory (nightly/) contains a simple, automatic build-and-test |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 4 | system for Valgrind, intended to be run nightly by cron or a similar |
| 5 | program. |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 6 | |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 7 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 8 | BASIC OPERATIONS |
| 9 | ---------------- |
| 10 | When run, the system checks out two trees: the SVN trunk from 24 hours ago |
| 11 | and the SVN trunk from now. ("24 hours ago" and "now" are determined when |
| 12 | the script starts running, so if any commits happen while the tests are |
| 13 | running they will not be tested.) |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 14 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 15 | If the two trees are different (i.e. there have been commits in the past 24 |
njn | 8b68b64 | 2009-06-24 00:37:09 +0000 | [diff] [blame] | 16 | hours, either to the trunk or a branch) it builds ("make"), installs ("make |
| 17 | install") and runs the regression tests ("make regtest") in both, and |
| 18 | compares the results. Note that the "make install" isn't necessary in order |
| 19 | to run the tests because the regression tests use the code built (with |
| 20 | "make") within the tree, but it's worth doing because it tests that "make |
| 21 | install" isn't totally broken. After checking both trees, it emails a |
| 22 | summary of the results to a recipient. All this typically takes something |
| 23 | like 30 minutes. |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 24 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 25 | If the two trees are identical, the tests are not run and no results are |
| 26 | emailed. This avoids spamming people with uninteresting results emails when |
| 27 | no commits have happened recently. |
jseward | e2cac93 | 2004-02-25 13:14:39 +0000 | [diff] [blame] | 28 | |
njn | 5a1ba02 | 2009-01-07 04:14:42 +0000 | [diff] [blame] | 29 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 30 | SETTING UP |
| 31 | ---------- |
| 32 | To set up nightly testing for a machine, do the following. |
njn | 5a1ba02 | 2009-01-07 04:14:42 +0000 | [diff] [blame] | 33 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 34 | (1) Check out just this directory from the repository, eg: |
| 35 | |
| 36 | svn co svn://svn.valgrind.org/valgrind/trunk/nightly $DIR |
| 37 | |
| 38 | where $DIR is the name of the directory you want it to be in. |
| 39 | |
| 40 | Note that this doesn't check out the whole Valgrind tree, just the |
| 41 | directory containing the nightly testing stuff. This is possible |
| 42 | because the testing script doesn't check the code in the tree it belongs |
| 43 | to; rather it checks out new trees (within $DIR) and tests them |
| 44 | independently. |
| 45 | |
| 46 | (2) Choose a tag that identifies the test results. This is usually the |
| 47 | machine name. We'll call it $TAG in what follows. |
| 48 | |
| 49 | (3) Create a configuration file $DIR/conf/$TAG.conf. It is sourced by the |
| 50 | 'nightly' script, and can define any or all of the following environment |
| 51 | variables. (In most cases, only ABT_DETAILS is needed.) |
| 52 | |
| 53 | - ABT_DETAILS: describes the machine in more detail, eg. the OS. The |
| 54 | default is empty, but you should define it. An example: |
| 55 | |
| 56 | export ABT_DETAILS="Ubuntu 9.04, Intel x86-64" |
| 57 | |
| 58 | You could also use some invocation of 'uname' or something similar |
| 59 | to generate this string. Eg. on Ubuntu Linux this works nicely: |
| 60 | |
| 61 | export ABT_DETAILS="`cat /etc/issue.net`, `uname -m`" |
| 62 | |
| 63 | And on Mac OS X this works nicely: |
| 64 | |
| 65 | export ABT_DETAILS=`uname -mrs` |
| 66 | |
| 67 | The advantage of doing it like this is that if you update the OS on |
| 68 | the test machine you won't have to update ABT_DETAILS manually. |
| 69 | |
| 70 | - ABT_CONFIGURE_OPTIONS: gives extra configure options. The default is |
| 71 | empty. |
| 72 | |
| 73 | - ABT_EVAL: if provided, it must be the name of a shell script that |
njn | 8b68b64 | 2009-06-24 00:37:09 +0000 | [diff] [blame] | 74 | executes the shell command $1 with arguments $2 .. ${$#}. Allows to |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 75 | compile and run the Valgrind regression tests on another system than |
njn | 8b68b64 | 2009-06-24 00:37:09 +0000 | [diff] [blame] | 76 | the system the 'nightly' script runs on. It is assumed that the remote |
| 77 | system shares the local filesystem tree through e.g. NFS. It is the |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 78 | responsibility of the shell script to set the remote working directory |
| 79 | such that it matches the local current directory ($PWD). |
| 80 | |
| 81 | - ABT_RUN_REGTEST: if provided, it must be the name of an argumentless |
| 82 | shell function (also specified in the $TAG.conf file) that will be used |
| 83 | to run the tests. If not specified, the usual "make regtest" will be |
| 84 | used. |
| 85 | |
| 86 | - ABT_JOBS: allows parallel builds -- it's passed as the argument to |
| 87 | "make -j" when building Valgrind and the tests. The default is 1. |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 88 | |
philippe | f31fe6f | 2013-09-27 18:10:29 +0000 | [diff] [blame] | 89 | - ABT_PERF: unset or set to "" mean 'do not run perf tests' (default value) |
| 90 | set to "--vg=../valgrind-new" (run perf tests for new tree) |
| 91 | set to "--vg=../valgrind-new --vg=../valgrind-old" |
| 92 | (run perf tests for "new" and for "24 hours ago", |
| 93 | to compare the performances between the 2 trees). |
| 94 | |
| 95 | - ABT_PERF_TOOLS: --tools=.... option of perf/vg_perf. |
| 96 | (default value: all non experimental tools) |
| 97 | |
| 98 | - ABT_PERF_REPS: --reps=... option of perf/vg_perf |
| 99 | (default value: --reps=3) |
| 100 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 101 | Note that the appropriate syntax to use in this file will depend on the |
| 102 | shell from which the $DIR/bin/nightly script is run (which in turn may |
| 103 | depend on what shell is used by cron or any similar program). |
| 104 | |
| 105 | (4) Create a mailer script $DIR/conf/$TAG.sendmail. It must be executable. |
| 106 | It's used to send email results to the desired recipient (e.g. |
| 107 | valgrind-developers@lists.sourceforge.net) It must handle three command |
| 108 | line arguments. |
| 109 | |
| 110 | - The first argument is the email subject line. It contains |
| 111 | $ABT_DETAILS plus some other stuff. |
| 112 | |
| 113 | - The second argument is the name of the file containing the email's |
| 114 | body (which shows the tests that failed, and the differences between now |
| 115 | and 24 hours ago). |
| 116 | |
| 117 | - The third is the name of the file containing all the diffs from |
| 118 | failing tests. Depending on the test results you get, you could |
| 119 | inline this file into the email body, or attach it, or compress and |
| 120 | attach it, or even omit it. The right choice depends on how many |
| 121 | failures you typically get -- if you get few failures, inlining the |
| 122 | results make them easier to read; if you get many failures, |
| 123 | compressing might be a good idea to minimise the size of the emails. |
| 124 | |
| 125 | The best way to do this depends on how mail is set up on your machine. |
| 126 | You might be able to use /usr/bin/mail, or you might need something more |
| 127 | elaborate like using Mutt to send mail via an external account. |
| 128 | |
njn | 1d96fc5 | 2009-06-04 01:52:14 +0000 | [diff] [blame] | 129 | At first, you should probably just send emails to yourself for testing |
| 130 | purposes. After it's working, then sending it to others might be |
| 131 | appropriate. |
| 132 | |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 133 | (5) To run the tests, execute: |
| 134 | |
| 135 | $DIR/bin/nightly $DIR $TAG |
| 136 | |
| 137 | You probably want to put this command into a cron file or equivalent |
njn | 1d96fc5 | 2009-06-04 01:52:14 +0000 | [diff] [blame] | 138 | so it is run regularly (preferably every night). Actually, it's |
| 139 | probably better to put that command inside a script, and run the script |
| 140 | from cron, rather than running $DIR/bin/nightly directly. That way you |
| 141 | can put any other configuration stuff that's necessary inside the |
| 142 | script (e.g. make sure that programs used by the mailer script are in |
| 143 | your PATH). |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 144 | |
| 145 | |
| 146 | OUTPUT FILES |
| 147 | ------------ |
| 148 | If the tests are run, the following files are produced: |
| 149 | |
| 150 | - $DIR/old.verbose and $DIR/new.verbose contain full output of the whole |
| 151 | process for each of the two trees. |
| 152 | |
| 153 | - $DIR/old.short and $DIR/new.short contain summary output of the process |
| 154 | for each of the two trees. The diff between these two files goes in |
| 155 | $DIR/diff.short. |
| 156 | |
| 157 | - $DIR/final contains the overall summary, constructed from $DIR/old.short, |
| 158 | $DIR/new.short, $DIR/diff.short and some other bits and pieces. (The name |
| 159 | of this file is what's passed as the second argument to |
| 160 | $DIR/conf/$TAG.sendmail.) |
| 161 | |
| 162 | - $DIR/diffs holds the diffs from all the failing tests in the newer tree, |
| 163 | concatenated together; the diff from each failure is truncated at 100 |
| 164 | lines to minimise possible size blow-outs. (The name of this file is |
| 165 | what's passed as the third argument to $DIR/conf/$TAG.sendmail.) |
| 166 | |
| 167 | - $DIR/sendmail.log contains the output (stdout and stderr) from |
| 168 | $DIR/conf/$TAG.sendmail goes in $DIR/sendmail.log. |
| 169 | |
njn | e43358f | 2009-06-04 23:17:12 +0000 | [diff] [blame] | 170 | - $DIR/valgrind-old/ and $DIR/valgrind-new/ contain the tested trees (and |
| 171 | $DIR/valgrind-old/Inst/ and $DIR/valgrind-new/Inst/ contain the installed |
| 172 | code). |
njn | 21c307c | 2009-06-04 01:30:14 +0000 | [diff] [blame] | 173 | |
| 174 | If the tests aren't run, the following file is produced: |
| 175 | |
| 176 | - $DIR/unchanged.log is created only if no tests were run because the two |
| 177 | trees were identical. It will contain a short explanatory message. |
| 178 | |
| 179 | Each time the tests are run, all files from previous runs are deleted. |
| 180 | |
| 181 | |
| 182 | TROUBLESHOOTING |
| 183 | --------------- |
| 184 | If something goes wrong, looking at the output files can be useful. For |
| 185 | example, if no email was sent but you expected one, check sendmail.log to |
| 186 | see if the mailer script had a problem. Or check if unchanged.log exists. |
| 187 | |
| 188 | Occasionally the SVN server isn't available when the tests runs, for either |
| 189 | or both trees. When this happens the email will be sent but it won't be |
| 190 | very informative. Usually it's just a temporary server problem and it'll |
| 191 | run fine the next time without you having to do anything. |
| 192 | |
| 193 | Note that the test suite is imperfect: |
| 194 | - There are very few machines where all tests pass; that's why the old/new |
| 195 | diff is produced. Some of the tests may not be as portable as intended. |
| 196 | - Some tests are non-deterministic, and so may pass one day and fail the |
| 197 | next. |
| 198 | |
| 199 | Improving the test suite to avoid these problems is a long-term goal but it |
| 200 | isn't easy. |
| 201 | |
| 202 | |
| 203 | MAINTENANCE |
| 204 | ----------- |
| 205 | The scripts in the nightly/ directory occasionally get updated. If that |
| 206 | happens, you can just "svn update" within $DIR to get the updated versions, |
| 207 | which will then be used the next time the tests run. (It's possible that |
| 208 | the scripts will be changed in a way that requires changes to the files in |
| 209 | $DIR/conf/, but we try to avoid this.) |
| 210 | |
| 211 | If you want such updates to happen automatically, you could write a script |
| 212 | that does all the steps in SETTING UP above, and instead run that script |
| 213 | from cron. |
| 214 | |
| 215 | |