Ben Murdoch | b8a8cc1 | 2014-11-26 15:28:44 +0000 | [diff] [blame] | 1 | Test suite runner for V8, including support for distributed running. |
| 2 | ==================================================================== |
| 3 | |
| 4 | |
| 5 | Local usage instructions: |
| 6 | ========================= |
| 7 | |
| 8 | Run the main script with --help to get detailed usage instructions: |
| 9 | |
| 10 | $ tools/run-tests.py --help |
| 11 | |
| 12 | The interface is mostly the same as it was for the old test runner. |
| 13 | You'll likely want something like this: |
| 14 | |
| 15 | $ tools/run-tests.py --nonetwork --arch ia32 --mode release |
| 16 | |
| 17 | --nonetwork is the default on Mac and Windows. If you don't specify --arch |
| 18 | and/or --mode, all available values will be used and run in turn (e.g., |
| 19 | omitting --mode from the above example will run ia32 in both Release and Debug |
| 20 | modes). |
| 21 | |
| 22 | |
| 23 | Networked usage instructions: |
| 24 | ============================= |
| 25 | |
| 26 | Networked running is only supported on Linux currently. Make sure that all |
| 27 | machines participating in the cluster are binary-compatible (e.g. mixing |
| 28 | Ubuntu Lucid and Precise doesn't work). |
| 29 | |
| 30 | Setup: |
| 31 | ------ |
| 32 | |
| 33 | 1.) Copy tools/test-server.py to a new empty directory anywhere on your hard |
| 34 | drive (preferably not inside your V8 checkout just to keep things clean). |
| 35 | Please do create a copy, not just a symlink. |
| 36 | |
| 37 | 2.) Navigate to the new directory and let the server setup itself: |
| 38 | |
| 39 | $ ./test-server.py setup |
| 40 | |
| 41 | This will install PIP and UltraJSON, create a V8 working directory, and |
| 42 | generate a keypair. |
| 43 | |
| 44 | 3.) Swap public keys with someone who's already part of the networked cluster. |
| 45 | |
| 46 | $ cp trusted/`cat data/mypubkey`.pem /where/peers/can/see/it/myname.pem |
| 47 | $ ./test-server.py approve /wherever/they/put/it/yourname.pem |
| 48 | |
| 49 | |
| 50 | Usage: |
| 51 | ------ |
| 52 | |
| 53 | 1.) Start your server: |
| 54 | |
| 55 | $ ./test-server.py start |
| 56 | |
| 57 | 2.) (Optionally) inspect the server's status: |
| 58 | |
| 59 | $ ./test-server.py status |
| 60 | |
| 61 | 3.) From your regular V8 working directory, run tests: |
| 62 | |
| 63 | $ tool/run-tests.py --arch ia32 --mode debug |
| 64 | |
| 65 | 4.) (Optionally) enjoy the speeeeeeeeeeeeeeeed |
| 66 | |
| 67 | |
| 68 | Architecture overview: |
| 69 | ====================== |
| 70 | |
| 71 | Code organization: |
| 72 | ------------------ |
| 73 | |
| 74 | This section is written from the point of view of the tools/ directory. |
| 75 | |
| 76 | ./run-tests.py: |
| 77 | Main script. Parses command-line options and drives the test execution |
| 78 | procedure from a high level. Imports the actual implementation of all |
| 79 | steps from the testrunner/ directory. |
| 80 | |
| 81 | ./test-server.py: |
| 82 | Interface to interact with the server. Contains code to setup the server's |
| 83 | working environment and can start and stop server daemon processes. |
| 84 | Imports some stuff from the testrunner/server/ directory. |
| 85 | |
| 86 | ./testrunner/local/*: |
| 87 | Implementation needed to run tests locally. Used by run-tests.py. Inspired by |
| 88 | (and partly copied verbatim from) the original test.py script. |
| 89 | |
| 90 | ./testrunner/objects/*: |
| 91 | A bunch of data container classes, used by the scripts in the various other |
| 92 | directories; serializable for transmission over the network. |
| 93 | |
| 94 | ./testrunner/network/*: |
| 95 | Equivalents and extensions of some of the functionality in ./testrunner/local/ |
| 96 | as required when dispatching tests to peers on the network. |
| 97 | |
| 98 | ./testrunner/network/network_execution.py: |
| 99 | Drop-in replacement for ./testrunner/local/execution that distributes |
| 100 | test jobs to network peers instead of running them locally. |
| 101 | |
| 102 | ./testrunner/network/endpoint.py: |
| 103 | Receiving end of a network distributed job, uses the implementation |
| 104 | in ./testrunner/local/execution.py for actually running the tests. |
| 105 | |
| 106 | ./testrunner/server/*: |
| 107 | Implementation of the daemon that accepts and runs test execution jobs from |
| 108 | peers on the network. Should ideally have no dependencies on any of the other |
| 109 | directories, but that turned out to be impractical, so there are a few |
| 110 | exceptions. |
| 111 | |
| 112 | ./testrunner/server/compression.py: |
| 113 | Defines a wrapper around Python TCP sockets that provides JSON based |
| 114 | serialization, gzip based compression, and ensures message completeness. |
| 115 | |
| 116 | |
| 117 | Networking architecture: |
| 118 | ------------------------ |
| 119 | |
| 120 | The distribution stuff is designed to be a layer between deciding which tests |
| 121 | to run on the one side, and actually running them on the other. The frontend |
| 122 | that the user interacts with is the same for local and networked execution, |
| 123 | and the actual test execution and result gathering code is the same too. |
| 124 | |
| 125 | The server daemon starts four separate servers, each listening on another port: |
| 126 | - "Local": Communication with a run-tests.py script running on the same host. |
| 127 | The test driving script e.g. needs to ask for available peers. It then talks |
| 128 | to those peers directly (one of them will be the locally running server). |
| 129 | - "Work": Listens for test job requests from run-tests.py scripts on the network |
| 130 | (including localhost). Accepts an arbitrary number of connections at the |
| 131 | same time, but only works on them in a serialized fashion. |
| 132 | - "Status": Used for communication with other servers on the network, e.g. for |
| 133 | exchanging trusted public keys to create the transitive trust closure. |
| 134 | - "Discovery": Used to detect presence of other peers on the network. |
| 135 | In contrast to the other three, this uses UDP (as opposed to TCP). |
| 136 | |
| 137 | |
| 138 | Give us a diagram! We love diagrams! |
| 139 | ------------------------------------ |
| 140 | . |
| 141 | Machine A . Machine B |
| 142 | . |
| 143 | +------------------------------+ . |
| 144 | | run-tests.py | . |
| 145 | | with flag: | . |
| 146 | |--nonetwork --network | . |
| 147 | | | / | | . |
| 148 | | | / | | . |
| 149 | | v / v | . |
| 150 | |BACKEND / distribution | . |
| 151 | +--------- / --------| \ ------+ . |
| 152 | / | \_____________________ |
| 153 | / | . \ |
| 154 | / | . \ |
| 155 | +----- v ----------- v --------+ . +---- v -----------------------+ |
| 156 | | LocalHandler | WorkHandler | . | WorkHandler | LocalHandler | |
| 157 | | | | | . | | | | |
| 158 | | | v | . | v | | |
| 159 | | | BACKEND | . | BACKEND | | |
| 160 | |------------- +---------------| . |---------------+--------------| |
| 161 | | Discovery | StatusHandler <----------> StatusHandler | Discovery | |
| 162 | +---- ^ -----------------------+ . +-------------------- ^ -------+ |
| 163 | | . | |
| 164 | +---------------------------------------------------------+ |
| 165 | |
| 166 | Note that the three occurrences of "BACKEND" are the same code |
| 167 | (testrunner/local/execution.py and its imports), but running from three |
| 168 | distinct directories (and on two different machines). |