sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 1 | Automatic regression tests: |
| 2 | --------------------------- |
| 3 | The Valgrind gdbserver automatic tests are by default |
| 4 | executed as part of the Valgrind test suite: |
| 5 | make regtest |
| 6 | |
| 7 | By default, these tests are running with the gdb found at |
| 8 | Valgrind configure time. |
| 9 | |
| 10 | If you want to test with another gdb version, you can do: |
| 11 | make regtest GDB=/path/to/another/gdb |
| 12 | |
| 13 | or (to just run the gdbserver tests with another gdb): |
| 14 | cd gdbserver_tests |
| 15 | make check |
| 16 | cd .. |
| 17 | gdbserver_tests/make_local_links /path/to/another/gdb |
| 18 | perl tests/vg_regtest gdbserver_tests |
| 19 | |
sewardj | 47ffedb | 2011-10-24 07:36:57 +0000 | [diff] [blame] | 20 | The minimum version to run the tests is gdb >= 6.5. |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 21 | Previous versions do not have the 'target remote |' command. It |
sewardj | 47ffedb | 2011-10-24 07:36:57 +0000 | [diff] [blame] | 22 | would be possible to use an older version by using the option |
| 23 | --port of vgdb, and using the tcp/ip variant of the 'target remote' |
| 24 | GDB command. |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 25 | |
sewardj | 47ffedb | 2011-10-24 07:36:57 +0000 | [diff] [blame] | 26 | The tests have been run on various platforms using gdb versions >= 7.2 |
sewardj | d142f99 | 2011-05-17 17:15:07 +0000 | [diff] [blame] | 27 | and on some platforms gdb 7.0 and 7.1. |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 28 | Some gdb tests implies a gdb >= 7.2. (these are automatically disabled |
| 29 | if testing with a lower version). |
sewardj | d142f99 | 2011-05-17 17:15:07 +0000 | [diff] [blame] | 30 | Test behaviour with gdb < 7.0 is unknown: some might fail, |
| 31 | some might block or loop for a very long time. |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 32 | |
| 33 | Some tests implies to have a vgdb "ptrace invoker" capable. |
| 34 | |
| 35 | The prerequisite are established during make regtest (using marker files). |
| 36 | Each test verifies the prerequisite using the prereq: line. |
| 37 | |
sewardj | d8c12f1 | 2011-05-11 22:54:57 +0000 | [diff] [blame] | 38 | In case of failing tests |
| 39 | ------------------------ |
sewardj | d142f99 | 2011-05-17 17:15:07 +0000 | [diff] [blame] | 40 | When executed with a new gdb version and/or depending on the OS version, |
| 41 | gdbserver tests might often fail due to (irrelevant) differences. |
sewardj | d8c12f1 | 2011-05-11 22:54:57 +0000 | [diff] [blame] | 42 | Such irrelevant differences have to be filtered by gdbserver_tests/filter_gdb. |
| 43 | |
sewardj | d142f99 | 2011-05-17 17:15:07 +0000 | [diff] [blame] | 44 | You are welcome to fix such bugs by enhancing filter_gdb. |
| 45 | |
| 46 | Alternatively, to report such problems, the best is to re-run |
| 47 | the gdbserver tests the following way: |
sewardj | d8c12f1 | 2011-05-11 22:54:57 +0000 | [diff] [blame] | 48 | perl tests/vg_regtest --keep-unfiltered gdbserver_tests |
| 49 | |
| 50 | Then file a bug in bugzilla, giving the following information: |
| 51 | output of |
| 52 | gdbserver_tests/gdb --version |
| 53 | uname -a |
| 54 | cat /etc/issue |
| 55 | valgrind --version (and/or svn version) |
sewardj | d142f99 | 2011-05-17 17:15:07 +0000 | [diff] [blame] | 56 | and attach a tar file containing all the *.out and *.diff |
| 57 | files in gdbserver_tests directory |
| 58 | |
| 59 | If a gdbserver test fails for other reasons, you can run the test |
| 60 | manually in two windows: |
| 61 | In one window, the valgrind |
| 62 | In another window, you launch gdb yourself, and you copy paste |
| 63 | the command from xxxx.stdinB.gdb. This might help to see what is |
| 64 | wrong. |
| 65 | |
| 66 | Another good trick is also to execute the same kind of actions |
| 67 | using a gdb connected to the gdbserver part of gdb. |
| 68 | You can examine what is happening by enabling the trace |
| 69 | of the packets being sent using the gdb command: |
| 70 | set debug remote 1 |
| 71 | Note however that the packets might be different |
| 72 | (e.g. the Valgrind gdbserver understands the 'P' packet, |
| 73 | which might not be understood by the gdbserver of gdb). |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 74 | |
| 75 | Naming conventions: |
| 76 | ------------------- |
| 77 | |
| 78 | The gdbserver tests are done with various Valgrind tools. A gdbserver |
| 79 | test using a tool xxxxxx should have its name starting with the 'short |
| 80 | two letters code' of this tool. For example, the test mcvabits.vgtest |
| 81 | is using Memcheck tool, while the test mssnapshot.vgtest is using |
| 82 | massif tool. |
| 83 | |
| 84 | Typically, a gdbserver test implies to launch two programs: |
| 85 | prog: a program running under valgrind |
| 86 | progB: another program (typically, either gdb or vgdb standalone) |
| 87 | The conventions about how to specify the 2nd program in a .vgtest |
| 88 | are explained in the file ../tests/vg_regtest.in |
| 89 | Many tests are using gdb as progB. The input for gdb is named |
| 90 | xxxxxx.stdinB.gdb. |
| 91 | |
| 92 | One day, we might imagine to run tests in parallel. |
| 93 | So, each test gets its own '--vgdb-prefix' argument. |
| 94 | This also help to avoid interactions between two successive tests: |
| 95 | if a previous test stayed blocked, the vgdb of the next test |
| 96 | will not report an error due to multiple possible FIFOs. |
| 97 | |
| 98 | |
| 99 | Rational for test approach |
| 100 | -------------------------- |
| 101 | Two approaches have been looked at: |
| 102 | V: use the 'vg_regtest' approach used by the rest of Valgrind tests |
| 103 | G: use the gdb Dejagnu test framework. |
| 104 | |
bart | 1e45164 | 2012-07-28 13:06:45 +0000 | [diff] [blame] | 105 | Advantages of V: much simpler that G, known by Valgrind developers, |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 106 | no additional dependency for the Valgrind build and test. |
| 107 | |
| 108 | Disadvantages of V: not well suited to testing of interactive tools, |
| 109 | very unflexible way to test the results (everything is in "template" |
| 110 | files), templates contains often irrelevant data for the test but it |
| 111 | can make the test fail. After writing 13 tests, it looks like the |
| 112 | template diff approach is quite fragile (e.g. changing the gdb version |
| 113 | and/or the OS version influences the output of irrelevant things which |
| 114 | are part of the template). |
| 115 | |
| 116 | A more flexible template filtering is needed. |
| 117 | Maybe something like: |
| 118 | The program under test is outputting its instructions to be filtered in |
| 119 | special markers e.g. |
| 120 | #pushf filter_addresses | filter_messages |
| 121 | ... some output |
| 122 | #pushf an_additional_filter |
| 123 | ... some other output, filtered by both the first and second push |
| 124 | #popf |
| 125 | ... here output filtered only by the first pushf |
| 126 | #popf |
| 127 | |
| 128 | Advantages of G: much more powerful, well suited to test a gdb with a |
| 129 | gdbserver, tests can verify specifically some output without being |
| 130 | impacted by other output, allow to test Valgrind gdbserver with the |
| 131 | all of the gdb test suite (but a lot of tests will rather test gdb |
| 132 | than Valgrind gdbserver). |
| 133 | |
| 134 | Disadvantages: not an easy beast to understand and master, running the |
| 135 | whole gdb testsuite with Valgrind gdbserver looks to be a challenge. |
| 136 | |
| 137 | Currently, tests are written with vg_regtest. Approach G will be looked at it |
| 138 | again (e.g. to complement V) once a basic set of tests are available. |
| 139 | |
| 140 | |
| 141 | Manual tests still to automate: |
| 142 | ------------------------------- |
| 143 | |
| 144 | Validate monitor commands abbreviation recognition |
| 145 | *************************************************** |
sewardj | 30b3eca | 2011-06-28 08:20:39 +0000 | [diff] [blame] | 146 | mo v.info all_errors # to show all errors recorded so far |
| 147 | mo v.i a # the same |
| 148 | mo v # must give an error: v can match v.set v.info |
| 149 | mo v # the same v |
| 150 | mo v. # the same v. |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 151 | |
| 152 | test of gdb detaching or dying |
| 153 | ****************************** |
| 154 | valgrind --vgdb=yes --vgdb-error=0 --vgdb-poll=500 ./t |
| 155 | |
| 156 | in another window |
| 157 | |
| 158 | gdb ./t |
| 159 | set remotetimeout 100 |
| 160 | target remote|vgdb |
sewardj | 30b3eca | 2011-06-28 08:20:39 +0000 | [diff] [blame] | 161 | detach valgrind continues |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 162 | target remote|vgdb reattach |
sewardj | 30b3eca | 2011-06-28 08:20:39 +0000 | [diff] [blame] | 163 | detach valgrind continues |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 164 | target remote|vgdb reattach |
sewardj | 30b3eca | 2011-06-28 08:20:39 +0000 | [diff] [blame] | 165 | monitor v.wait no effect |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 166 | |
| 167 | |
| 168 | |
| 169 | test of valgrind/gdb output redirection |
| 170 | *************************************** |
| 171 | valgrind --vgdb=yes --vgdb-error=1 --vgdb-poll=500 ./t |
| 172 | |
| 173 | in another window |
| 174 | |
| 175 | **** command to type*** ****** expected behaviour |
| 176 | gdb ./t |
| 177 | set remotetimeout 1000 |
| 178 | target remote | vgdb |
sewardj | 30b3eca | 2011-06-28 08:20:39 +0000 | [diff] [blame] | 179 | mo v.set vgdb-error 1000 # so that valgrind does stop only at error 1000 and after |
| 180 | mo v.set gdb_output # to have further valgrind errors output in gdb |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 181 | c # continue, some errors will appear |
| 182 | C-c # interrupt program |
sewardj | 30b3eca | 2011-06-28 08:20:39 +0000 | [diff] [blame] | 183 | mo v.set log_output # to set back the valgrind output to normal process log output |
| 184 | mo l # leak output to appear in log of process |
| 185 | mo v.set mixed_output |
| 186 | mo l # leak output to appear in gdb |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 187 | |
| 188 | |
| 189 | |
| 190 | test with a big executable: firefox |
| 191 | *********************************** |
| 192 | valgrind --vgdb=yes --vgdb-error=1000 --vgdb-poll=50000 --trace-children=yes firefox 2>&1 | tee f.out |
| 193 | |
| 194 | wait for some messages from the "big" firefox executable to appear. |
| 195 | Then: |
| 196 | |
| 197 | gdb /usr/lib/firefox-3.5/firefox |
| 198 | target remote | vgdb |
| 199 | ... then you can do various next/print/bt/bt full/break/... to see it is working |
| 200 | |
| 201 | bulk test with the above |
| 202 | ************************ |
| 203 | to verify there is no race condition/no reentrance problem |
| 204 | between gdbserver code and valgrind: |
| 205 | start firefox like in the previous test. |
| 206 | In another window, do: |
| 207 | while true |
| 208 | do |
sewardj | 30b3eca | 2011-06-28 08:20:39 +0000 | [diff] [blame] | 209 | vgdb leak |
sewardj | 3b29048 | 2011-05-06 21:02:55 +0000 | [diff] [blame] | 210 | sleep 1 |
| 211 | done |
| 212 | |
| 213 | NB: this will make firefox run extremely slow, as it will do a leak |
| 214 | search every second. |
| 215 | |
| 216 | |
| 217 | Test of "jump" functionality |
| 218 | ---------------------------- |
| 219 | ... to be done : put two breaks, jump over one. |
| 220 | ... same but when error is encountered |
| 221 | |
| 222 | |
| 223 | * test with --max-invoke-ms=0 |
| 224 | ----------------------------- |
| 225 | valgrind --vgdb=yes ./t |
| 226 | ... wait till you see "petachounok sleeping 4 of 15 |
| 227 | then try to gdb it |
| 228 | |
| 229 | !!!! this often causes gdb to report a protocol timeout. |
| 230 | use gdb set remotetimeout <a big time> to avoid that. |
| 231 | The symptoms of a timeout are: |
| 232 | (gdb) tar rem|vgdb --max-invoke-ms=0 |
| 233 | Remote debugging using |vgdb --max-invoke-ms=0 |
| 234 | relaying data between gdb and process 2930 |
| 235 | Ignoring packet error, continuing... |
| 236 | warning: unrecognized item "timeout" in "qSupported" response |
| 237 | |
| 238 | * tests of shadow registers |
| 239 | ---------------------------- |
| 240 | Show/modify shadow registers |