Home | History | Annotate | Download | only in testing 
      1 # Src-side test specifications
      2 
      3 Src-side test specifications enable developers to quickly add tests running on
      4 specific bots on V8's continuous infrastructure (CI) or tryserver. Features to
      5 be tested must live behind runtime flags, which are mapped to named testing
      6 variants specified [here](https://chromium.googlesource.com/v8/v8/+/master/tools/testrunner/local/variants.py).
      7 Changes to src-side test specifications go through CQ like any other CL and
      8 require tests added for specific trybots to pass.
      9 
     10 The test specifications are defined in a V8-side python-literal file
     11 `infra/testing/builders.pyl`.
     12 
     13 The structure of the file is:
     14 ```
     15 {
     16   <buildername>: {
     17     'tests': [
     18       {
     19         'name': <test-spec name>,
     20         'suffix': <step suffix>,
     21         'variant': <variant name>,
     22         'shards': <number of shards>,
     23         'test_args': <list of flags>,
     24         'swarming_task_attrs': {...},
     25         'swarming_dimensions': {...},
     26       },
     27       ...
     28     ],
     29     'swarming_task_attrs': {...},
     30     'swarming_dimensions': {...},
     31   },
     32   ...
     33 }
     34 ```
     35 The `<buildername>` is a string name of the builder to execute the tests.
     36 `<test-spec name>` is a label defining a test specification matching the
     37 [infra-side](https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave/recipe_modules/v8/testing.py#58).
     38 The optional `suffix` will be appended to test-step names for disambiguation.
     39 The optional `variant` is a testing variant specified
     40 [here](https://chromium.googlesource.com/v8/v8/+/master/tools/testrunner/local/variants.py).
     41 The optional `shards` (default 1) can be provided to increase the swarming
     42 shards for long-running tests.
     43 The optional `test_args` is a list of string flags that will be passed to the
     44 V8 test driver.
     45 The optional `swarming_task_attrs` is a dict allowing to override the defaults
     46 for `priority`, `expiration` and `hard_timeout`.
     47 The optional `swarming_dimensions` is a dict allowing to override the defaults
     48 for `cpu`, `cores` and `os`.
     49 Both `swarming_task_attrs` and `swarming_dimensions` can be defined per builder
     50 and per test, whereas the latter takes precedence.
     51 
     52 Example:
     53 ```
     54 {
     55   'v8_linux64_rel_ng_triggered': {
     56     'tests': [
     57       {
     58         'name': 'v8testing',
     59         'suffix': 'stress',
     60         'variant': 'nooptimization',
     61         'shards': 2,
     62         'test_args': ['--gc-stress'],
     63         'swarming_dimensions': {'os': 'Ubuntu-14.4'},
     64       },
     65     ],
     66     'swarming_properties': {'priority': 35},
     67     'swarming_dimensions': {'os': 'Ubuntu'},
     68   },
     69 }
     70 ```
     71 
     72 ## Guidelines
     73 
     74 Please keep trybots and continuous bots in sync. E.g. add the same configuration
     75 for the release and debug CI bots and the corresponding trybot (where
     76 applicable). E.g.
     77 
     78 ```
     79 tryserver.v8:
     80   v8_linux64_rel_ng_triggered
     81 client.v8:
     82   V8 Linux64
     83   V8 Linux64 - debug
     84 ```
     85 
     86 Please only add tests that are expected to pass, or skip failing tests via
     87 status file for the selected testing variants only. If you want to add FYI tests
     88 (i.e. not closing the tree and not blocking CQ) you can do so for the following
     89 set of bots:
     90 
     91 ```
     92 tryserver.v8:
     93   v8_linux64_fyi_rel_ng_triggered
     94 client.v8:
     95   V8 Linux64 - fyi
     96   V8 Linux64 - debug - fyi
     97 ```
     98