1 ## Introduction 2 3 `test_that` is the supported mechanism to run autotests against Chrome OS 4 devices at your desk. `test_that` replaces an older script, `run_remote_tests`. 5 6 Features: 7 - CTRL+C kills `test_that` and all its autoserv children. Orphaned processes 8 are no longer left behind. 9 - Tests that require binary autotest dependencies will just work, because 10 test_that always runs from the sysroot location. 11 - Running emerge after python-only test changes is no longer necessary. 12 test_that uses autotest_quickmerge to copy your python changes to the 13 sysroot. 14 - Tests are generally specified to `test_that` by the NAME field of their 15 control file. Matching tests by filename is supported using f:[file 16 pattern] 17 18 ### Example uses (inside the chroot) 19 20 Run the test(s) named dummy\_Pass: 21 22 ``` 23 $ test_that -b ${board} ${host} dummy_Pass 24 ``` 25 26 Run the test(s) named dummy\_Pass.suspend: 27 28 ``` 29 $ test_that -b ${board} ${host} dummy_Pass.suspend 30 ``` 31 32 Run the smoke suite against dut: 33 34 ``` 35 $ test_that -b ${board} ${host} suite:smoke 36 ``` 37 38 Run all tests whose names match the regular expression `^login_.*$`. Note that 39 even though these tests have binary dependencies, there is no longer a need to 40 specify extra flags. 41 42 ``` 43 $ test_that -b ${board} ${host} e:login_.* 44 ``` 45 46 Run all tests whose control file filename matches the regular expression 47 `^.*control.dummy$`: 48 49 ``` 50 $ test_that -b ${board} ${host} f:.*control.dummy 51 ``` 52 53 ### Running jobs in the lab 54 55 `test_that` now allows you to run jobs in the test lab. The usage is similar to 56 running tests against a specified host. The keyword :lab: is used as 57 test\_that's REMOTE argument, and the -i/--build argument is required, and takes 58 a trybot, paladin, or canary build number. To learn how to build a trybot image 59 with a new test that you're iterating on, see "dynamic suite" codelab. 60 61 For instance: 62 63 ``` 64 $ test_that -b lumpy -i lumpy-paladin/R38-6009.0.0-rc4 :lab: dummy_Pass 65 ``` 66 67 This will kick off a suite in the lab that consists of just 1 job, dummy\_Pass, 68 to run in this case on board lumpy using the image 69 lumpy-paladin/R38-6009.0.0-rc4. The lab's scheduler will take responsibility 70 for finding a suitable set of hosts, provisioning them to the correct image, 71 and running the tests. `test_that` will return after the suite finishes running, 72 with a suite run report. 73 74 You can specify multiple tests or test-matching expressions in the same way as 75 before: 76 77 ``` 78 $ test_that -b lumpy -i ${latest_image} :lab: dummy_Pass dummy_Fail 79 $ test_that -b lumpy -i ${latest_image} :lab: e:login_.* 80 ``` 81 82 Kicking off a run in the lab should be useful whenever you need to run a 83 particular test on a board or image that you do not have readily available 84 locally.For occasional runs of ad-hoc suites in the lab, this will also avoid 85 the need to create a suite control file and wait for it to end up in an image. 86 87 You can also kick off a suite, for example with: 88 89 ``` 90 test_that -b peach_pit :lab: suite:pyauto_perf -i 'peach_pit-release/R32-4763.0.0' 91 ``` 92 93 That told me that my job ID was 5196037. I could follow along by going to 94 http://cautotest/afe/#tab_id=view_job&object_id=5195962. 95 96 ### Things to note: 97 98 This will only work with images newer than Sept 20, 2013 (specifically, builds 99 that contain Ifa73d7de7aac9c6efebd5f559708623804ad3691). Jobs will be scheduled 100 in the pool:try-bot machine pool. 101
