Home | History | Annotate | Download | only in docs 
      1 ## Introduction
      2 
      3 `test_that` is the supported mechanism to run autotests against Chrome OS
      4 devices at your desk.  `test_that` replaces an older script, `run_remote_tests`.
      5 
      6 Features for testing a local device:
      7   - CTRL+C kills `test_that` and all its autoserv children. Orphaned processes
      8     are no longer left behind.
      9   - Tests that require binary autotest dependencies will just work, because
     10     test_that always runs from the sysroot location.
     11   - Running emerge after python-only test changes is no longer necessary.
     12     test_that uses autotest_quickmerge to copy your python changes to the
     13     sysroot.
     14   - Tests are generally specified to `test_that` by the NAME field of their
     15     control file. Matching tests by filename is supported using f:[file
     16     pattern]
     17 
     18 In addition to running tests against local device, `test_that` can be used to
     19 launch jobs in the ChromeOS Hardware Lab (or against a local Autotest instance
     20 or a Moblab). This feature is only supported for infrastructure-produced builds
     21 that were uploaded to google storage.
     22 
     23 ### Example uses (inside the chroot)
     24 
     25 Run the test(s) named dummy\_Pass:
     26 
     27 ```
     28 $ test_that -b ${board} ${host} dummy_Pass
     29 ```
     30 
     31 Run the test(s) named dummy\_Pass.suspend:
     32 
     33 ```
     34 $ test_that -b ${board} ${host} dummy_Pass.suspend
     35 ```
     36 
     37 Run the smoke suite against dut:
     38 
     39 ```
     40 $ test_that -b ${board} ${host} suite:smoke
     41 ```
     42 
     43 Run all tests whose names match the regular expression `^login_.*$`. Note that
     44 even though these tests have binary dependencies, there is no longer a need to
     45 specify extra flags.
     46 
     47 ```
     48 $ test_that -b ${board} ${host} e:login_.*
     49 ```
     50 
     51 Run all tests whose control file filename matches the regular expression
     52 `^.*control.dummy$`:
     53 
     54 ```
     55 $ test_that -b ${board} ${host} f:.*control.dummy
     56 ```
     57 
     58 ### Running jobs in the lab
     59 
     60 `test_that` now allows you to run jobs in the test lab. The usage is similar to
     61 running tests against a specified host. The keyword :lab: is used as
     62 test\_that's REMOTE argument, and the -i/--build argument is required, and takes
     63 a trybot, paladin, or canary build number. To learn how to build a trybot image
     64 with a new test that you're iterating on, see "dynamic suite" codelab or the
     65 [Chromium OS Remote
     66 Trybots](https://sites.google.com/a/chromium.org/dev/chromium-os/build/using-remote-trybots)
     67 guide.
     68 
     69 For instance:
     70 
     71 ```
     72 $ test_that -b lumpy -i lumpy-paladin/R38-6009.0.0-rc4 :lab: dummy_Pass
     73 ```
     74 
     75 This will kick off a suite in the lab that consists of just 1 job, dummy\_Pass,
     76 to run in this case on board lumpy using the image
     77 lumpy-paladin/R38-6009.0.0-rc4. The lab's scheduler will take responsibility
     78 for finding a suitable set of hosts, provisioning them to the correct image,
     79 and running the tests. `test_that` will return after the suite finishes running,
     80 with a suite run report.
     81 
     82 You can specify multiple tests or test-matching expressions in the same way as
     83 before:
     84 
     85 ```
     86 $ test_that -b lumpy -i ${latest_image} :lab: dummy_Pass dummy_Fail
     87 $ test_that -b lumpy -i ${latest_image} :lab: e:login_.*
     88 ```
     89 
     90 Kicking off a run in the lab should be useful whenever you need to run a
     91 particular test on a board or image that you do not have readily available
     92 locally.For occasional runs of ad-hoc suites in the lab, this will also avoid
     93 the need to create a suite control file and wait for it to end up in an image.
     94 
     95 You can also kick off a suite, for example with:
     96 
     97 ```
     98 test_that -b peach_pit :lab: suite:pyauto_perf -i 'peach_pit-release/R32-4763.0.0'
     99 ```
    100 
    101 That told me that my job ID was 5196037. I could follow along by going to
    102 http://cautotest/afe/#tab_id=view_job&object_id=5195962.
    103 
    104 Things to note about running in the lab:
    105 
    106  - This feature will only work on builds that produced autotest test artifacts.
    107    If your build doesn't have such artifacts, you will see a
    108    [confusing error](https://crbug.com/354556). The easiest way today to
    109    guarantee that hwtest artifacts are produced is to make sure that your tryjob
    110    is launched with the --hwtest flag. Once [this bug](https://crbug.com/299838)
    111    is fixed that will no longer be the case.
    112  - By default, jobs will be scheduled in the `suites` machine pool. That can be
    113    overridden with the `-p` flag.
    114  - This will only work with images newer than Sept 20, 2013 (specifically, builds
    115    that contain Ifa73d7de7aac9c6efebd5f559708623804ad3691).
    116 
    117 
    118 ### Running jobs against a local Autotest setup or MobLab
    119 
    120 `test_that` allows you to run jobs against a local Autotest setup or a
    121 MobLab instance. This usage is similar to running tests in the lab. The argument
    122 --web allows you to specify the web address of the Autotest instance you want to
    123 run tests within.
    124 
    125 For instance:
    126 ```
    127 $ test_that -b lumpy -i lumpy-paladin/R38-6009.0.0-rc4 --web 100.96.51.136 :lab:
    128 dummy_Pass
    129 ```
    130 
    131 This will kick off the dummy_Pass test on a lumpy device on the Autotest
    132 instance located at 100.96.51.136
    133