Home | History | Annotate | Download | only in testrunner 
      1 Test suite runner for V8, including support for distributed running.
      2 ====================================================================
      3 
      4 
      5 Local usage instructions:
      6 =========================
      7 
      8 Run the main script with --help to get detailed usage instructions:
      9 
     10 $ tools/run-tests.py --help
     11 
     12 The interface is mostly the same as it was for the old test runner.
     13 You'll likely want something like this:
     14 
     15 $ tools/run-tests.py --nonetwork --arch ia32 --mode release
     16 
     17 --nonetwork is the default on Mac and Windows. If you don't specify --arch
     18 and/or --mode, all available values will be used and run in turn (e.g.,
     19 omitting --mode from the above example will run ia32 in both Release and Debug
     20 modes).
     21 
     22 
     23 Networked usage instructions:
     24 =============================
     25 
     26 Networked running is only supported on Linux currently. Make sure that all
     27 machines participating in the cluster are binary-compatible (e.g. mixing
     28 Ubuntu Lucid and Precise doesn't work).
     29 
     30 Setup:
     31 ------
     32 
     33 1.) Copy tools/test-server.py to a new empty directory anywhere on your hard
     34     drive (preferably not inside your V8 checkout just to keep things clean).
     35     Please do create a copy, not just a symlink.
     36 
     37 2.) Navigate to the new directory and let the server setup itself:
     38 
     39 $ ./test-server.py setup
     40 
     41     This will install PIP and UltraJSON, create a V8 working directory, and
     42     generate a keypair.
     43 
     44 3.) Swap public keys with someone who's already part of the networked cluster.
     45 
     46 $ cp trusted/`cat data/mypubkey`.pem /where/peers/can/see/it/myname.pem
     47 $ ./test-server.py approve /wherever/they/put/it/yourname.pem
     48 
     49 
     50 Usage:
     51 ------
     52 
     53 1.) Start your server:
     54 
     55 $ ./test-server.py start
     56 
     57 2.) (Optionally) inspect the server's status:
     58 
     59 $ ./test-server.py status
     60 
     61 3.) From your regular V8 working directory, run tests:
     62 
     63 $ tool/run-tests.py --arch ia32 --mode debug
     64 
     65 4.) (Optionally) enjoy the speeeeeeeeeeeeeeeed
     66 
     67 
     68 Architecture overview:
     69 ======================
     70 
     71 Code organization:
     72 ------------------
     73 
     74 This section is written from the point of view of the tools/ directory.
     75 
     76 ./run-tests.py:
     77   Main script. Parses command-line options and drives the test execution
     78   procedure from a high level. Imports the actual implementation of all
     79   steps from the testrunner/ directory.
     80 
     81 ./test-server.py:
     82   Interface to interact with the server. Contains code to setup the server's
     83   working environment and can start and stop server daemon processes.
     84   Imports some stuff from the testrunner/server/ directory.
     85 
     86 ./testrunner/local/*:
     87   Implementation needed to run tests locally. Used by run-tests.py. Inspired by
     88   (and partly copied verbatim from) the original test.py script.
     89 
     90 ./testrunner/local/old_statusfile.py:
     91   Provides functionality to read an old-style <testsuite>.status file and
     92   convert it to new-style syntax. This can be removed once the new-style
     93   syntax becomes authoritative (and old-style syntax is no longer supported).
     94   ./status-file-converter.py provides a stand-alone interface to this.
     95 
     96 ./testrunner/objects/*:
     97   A bunch of data container classes, used by the scripts in the various other
     98   directories; serializable for transmission over the network.
     99 
    100 ./testrunner/network/*:
    101   Equivalents and extensions of some of the functionality in ./testrunner/local/
    102   as required when dispatching tests to peers on the network.
    103 
    104 ./testrunner/network/network_execution.py:
    105   Drop-in replacement for ./testrunner/local/execution that distributes
    106   test jobs to network peers instead of running them locally.
    107 
    108 ./testrunner/network/endpoint.py:
    109   Receiving end of a network distributed job, uses the implementation
    110   in ./testrunner/local/execution.py for actually running the tests.
    111 
    112 ./testrunner/server/*:
    113   Implementation of the daemon that accepts and runs test execution jobs from
    114   peers on the network. Should ideally have no dependencies on any of the other
    115   directories, but that turned out to be impractical, so there are a few
    116   exceptions.
    117 
    118 ./testrunner/server/compression.py:
    119   Defines a wrapper around Python TCP sockets that provides JSON based
    120   serialization, gzip based compression, and ensures message completeness.
    121 
    122 
    123 Networking architecture:
    124 ------------------------
    125 
    126 The distribution stuff is designed to be a layer between deciding which tests
    127 to run on the one side, and actually running them on the other. The frontend
    128 that the user interacts with is the same for local and networked execution,
    129 and the actual test execution and result gathering code is the same too.
    130 
    131 The server daemon starts four separate servers, each listening on another port:
    132 - "Local": Communication with a run-tests.py script running on the same host.
    133   The test driving script e.g. needs to ask for available peers. It then talks
    134   to those peers directly (one of them will be the locally running server).
    135 - "Work": Listens for test job requests from run-tests.py scripts on the network
    136   (including localhost). Accepts an arbitrary number of connections at the
    137   same time, but only works on them in a serialized fashion.
    138 - "Status": Used for communication with other servers on the network, e.g. for
    139   exchanging trusted public keys to create the transitive trust closure.
    140 - "Discovery": Used to detect presence of other peers on the network.
    141   In contrast to the other three, this uses UDP (as opposed to TCP).
    142 
    143 
    144 Give us a diagram! We love diagrams!
    145 ------------------------------------
    146                                      .
    147                          Machine A   .  Machine B
    148                                      .
    149 +------------------------------+     .
    150 |        run-tests.py          |     .
    151 |         with flag:           |     .
    152 |--nonetwork   --network       |     .
    153 |   |          /    |          |     .
    154 |   |         /     |          |     .
    155 |   v        /      v          |     .
    156 |BACKEND    /   distribution   |     .
    157 +--------- / --------| \ ------+     .
    158           /          |  \_____________________
    159          /           |               .        \
    160         /            |               .         \
    161 +----- v ----------- v --------+     .    +---- v -----------------------+
    162 | LocalHandler | WorkHandler   |     .    | WorkHandler   | LocalHandler |
    163 |              |     |         |     .    |     |         |              |
    164 |              |     v         |     .    |     v         |              |
    165 |              |  BACKEND      |     .    |  BACKEND      |              |
    166 |------------- +---------------|     .    |---------------+--------------|
    167 | Discovery    | StatusHandler <----------> StatusHandler | Discovery    |
    168 +---- ^ -----------------------+     .    +-------------------- ^ -------+
    169       |                              .                          |
    170       +---------------------------------------------------------+
    171 
    172 Note that the three occurrences of "BACKEND" are the same code
    173 (testrunner/local/execution.py and its imports), but running from three
    174 distinct directories (and on two different machines).
    175