README_DEVELOPERS
1 Automatic regression tests: 2 --------------------------- 3 The Valgrind gdbserver automatic tests are by default 4 executed as part of the Valgrind test suite: 5 make regtest 6 7 By default, these tests are running with the gdb found at 8 Valgrind configure time. 9 10 If you want to test with another gdb version, you can do: 11 make regtest GDB=/path/to/another/gdb 12 13 or (to just run the gdbserver tests with another gdb): 14 cd gdbserver_tests 15 make check 16 cd .. 17 gdbserver_tests/make_local_links /path/to/another/gdb 18 perl tests/vg_regtest gdbserver_tests 19 20 The minimum version to run the tests is gdb >= 6.5. 21 Previous versions do not have the 'target remote |' command. It 22 would be possible to use an older version by using the option 23 --port of vgdb, and using the tcp/ip variant of the 'target remote' 24 GDB command. 25 26 The tests have been run on various platforms using gdb versions >= 7.2 27 and on some platforms gdb 7.0 and 7.1. 28 Some gdb tests implies a gdb >= 7.2. (these are automatically disabled 29 if testing with a lower version). 30 Test behaviour with gdb < 7.0 is unknown: some might fail, 31 some might block or loop for a very long time. 32 33 Some tests implies to have a vgdb "ptrace invoker" capable. 34 35 The prerequisite are established during make regtest (using marker files). 36 Each test verifies the prerequisite using the prereq: line. 37 38 In case of failing tests 39 ------------------------ 40 When executed with a new gdb version and/or depending on the OS version, 41 gdbserver tests might often fail due to (irrelevant) differences. 42 Such irrelevant differences have to be filtered by gdbserver_tests/filter_gdb. 43 44 You are welcome to fix such bugs by enhancing filter_gdb. 45 46 Alternatively, to report such problems, the best is to re-run 47 the gdbserver tests the following way: 48 perl tests/vg_regtest --keep-unfiltered gdbserver_tests 49 50 Then file a bug in bugzilla, giving the following information: 51 output of 52 gdbserver_tests/gdb --version 53 uname -a 54 cat /etc/issue 55 valgrind --version (and/or svn version) 56 and attach a tar file containing all the *.out and *.diff 57 files in gdbserver_tests directory 58 59 If a gdbserver test fails for other reasons, you can run the test 60 manually in two windows: 61 In one window, the valgrind 62 In another window, you launch gdb yourself, and you copy paste 63 the command from xxxx.stdinB.gdb. This might help to see what is 64 wrong. 65 66 Another good trick is also to execute the same kind of actions 67 using a gdb connected to the gdbserver part of gdb. 68 You can examine what is happening by enabling the trace 69 of the packets being sent using the gdb command: 70 set debug remote 1 71 Note however that the packets might be different 72 (e.g. the Valgrind gdbserver understands the 'P' packet, 73 which might not be understood by the gdbserver of gdb). 74 75 Naming conventions: 76 ------------------- 77 78 The gdbserver tests are done with various Valgrind tools. A gdbserver 79 test using a tool xxxxxx should have its name starting with the 'short 80 two letters code' of this tool. For example, the test mcvabits.vgtest 81 is using Memcheck tool, while the test mssnapshot.vgtest is using 82 massif tool. 83 84 Typically, a gdbserver test implies to launch two programs: 85 prog: a program running under valgrind 86 progB: another program (typically, either gdb or vgdb standalone) 87 The conventions about how to specify the 2nd program in a .vgtest 88 are explained in the file ../tests/vg_regtest.in 89 Many tests are using gdb as progB. The input for gdb is named 90 xxxxxx.stdinB.gdb. 91 92 One day, we might imagine to run tests in parallel. 93 So, each test gets its own '--vgdb-prefix' argument. 94 This also help to avoid interactions between two successive tests: 95 if a previous test stayed blocked, the vgdb of the next test 96 will not report an error due to multiple possible FIFOs. 97 98 99 Rational for test approach 100 -------------------------- 101 Two approaches have been looked at: 102 V: use the 'vg_regtest' approach used by the rest of Valgrind tests 103 G: use the gdb Dejagnu test framework. 104 105 Advantages of V: much simpler that G, known by Valgrind developers, 106 no additional dependency for the Valgrind build and test. 107 108 Disadvantages of V: not well suited to testing of interactive tools, 109 very unflexible way to test the results (everything is in "template" 110 files), templates contains often irrelevant data for the test but it 111 can make the test fail. After writing 13 tests, it looks like the 112 template diff approach is quite fragile (e.g. changing the gdb version 113 and/or the OS version influences the output of irrelevant things which 114 are part of the template). 115 116 A more flexible template filtering is needed. 117 Maybe something like: 118 The program under test is outputting its instructions to be filtered in 119 special markers e.g. 120 #pushf filter_addresses | filter_messages 121 ... some output 122 #pushf an_additional_filter 123 ... some other output, filtered by both the first and second push 124 #popf 125 ... here output filtered only by the first pushf 126 #popf 127 128 Advantages of G: much more powerful, well suited to test a gdb with a 129 gdbserver, tests can verify specifically some output without being 130 impacted by other output, allow to test Valgrind gdbserver with the 131 all of the gdb test suite (but a lot of tests will rather test gdb 132 than Valgrind gdbserver). 133 134 Disadvantages: not an easy beast to understand and master, running the 135 whole gdb testsuite with Valgrind gdbserver looks to be a challenge. 136 137 Currently, tests are written with vg_regtest. Approach G will be looked at it 138 again (e.g. to complement V) once a basic set of tests are available. 139 140 141 Manual tests still to automate: 142 ------------------------------- 143 144 Validate monitor commands abbreviation recognition 145 *************************************************** 146 mo v.info all_errors # to show all errors recorded so far 147 mo v.i a # the same 148 mo v # must give an error: v can match v.set v.info 149 mo v # the same v 150 mo v. # the same v. 151 152 test of gdb detaching or dying 153 ****************************** 154 valgrind --vgdb=yes --vgdb-error=0 --vgdb-poll=500 ./t 155 156 in another window 157 158 gdb ./t 159 set remotetimeout 100 160 target remote|vgdb 161 detach valgrind continues 162 target remote|vgdb reattach 163 detach valgrind continues 164 target remote|vgdb reattach 165 monitor v.wait no effect 166 167 168 169 test of valgrind/gdb output redirection 170 *************************************** 171 valgrind --vgdb=yes --vgdb-error=1 --vgdb-poll=500 ./t 172 173 in another window 174 175 **** command to type*** ****** expected behaviour 176 gdb ./t 177 set remotetimeout 1000 178 target remote | vgdb 179 mo v.set vgdb-error 1000 # so that valgrind does stop only at error 1000 and after 180 mo v.set gdb_output # to have further valgrind errors output in gdb 181 c # continue, some errors will appear 182 C-c # interrupt program 183 mo v.set log_output # to set back the valgrind output to normal process log output 184 mo l # leak output to appear in log of process 185 mo v.set mixed_output 186 mo l # leak output to appear in gdb 187 188 189 190 test with a big executable: firefox 191 *********************************** 192 valgrind --vgdb=yes --vgdb-error=1000 --vgdb-poll=50000 --trace-children=yes firefox 2>&1 | tee f.out 193 194 wait for some messages from the "big" firefox executable to appear. 195 Then: 196 197 gdb /usr/lib/firefox-3.5/firefox 198 target remote | vgdb 199 ... then you can do various next/print/bt/bt full/break/... to see it is working 200 201 bulk test with the above 202 ************************ 203 to verify there is no race condition/no reentrance problem 204 between gdbserver code and valgrind: 205 start firefox like in the previous test. 206 In another window, do: 207 while true 208 do 209 vgdb leak 210 sleep 1 211 done 212 213 NB: this will make firefox run extremely slow, as it will do a leak 214 search every second. 215 216 217 Test of "jump" functionality 218 ---------------------------- 219 ... to be done : put two breaks, jump over one. 220 ... same but when error is encountered 221 222 223 * test with --max-invoke-ms=0 224 ----------------------------- 225 valgrind --vgdb=yes ./t 226 ... wait till you see "petachounok sleeping 4 of 15 227 then try to gdb it 228 229 !!!! this often causes gdb to report a protocol timeout. 230 use gdb set remotetimeout <a big time> to avoid that. 231 The symptoms of a timeout are: 232 (gdb) tar rem|vgdb --max-invoke-ms=0 233 Remote debugging using |vgdb --max-invoke-ms=0 234 relaying data between gdb and process 2930 235 Ignoring packet error, continuing... 236 warning: unrecognized item "timeout" in "qSupported" response 237 238 * tests of shadow registers 239 ---------------------------- 240 Show/modify shadow registers 241
