1 # Autotest Best Practices 2 When the Chrome OS team started using autotest, we tried our best to figure out 3 how to fit our code and our tests into the upstream style with little guidance 4 and poor documentation. This went poorly. With the benefit of hindsight, 5 were going to lay out some best-practices that wed like to enforce going 6 forward. In many cases, there is legacy code that contradicts this style; we 7 should go through and refactor that code to fit these guidelines as time 8 allows. 9 10 ## Upstream Documentation 11 12 There is a sizeable volume of general Autotest documentation available on 13 github: 14 https://github.com/autotest/autotest/wiki 15 16 ## Coding style 17 18 Basically PEP-8. See [docs/coding-style.md](docs/coding-style.md) 19 20 ## Where should my code live? 21 22 | Type of Code | Relative Path | 23 |---------------------------|-------------------------| 24 | client-side tests | client/site_tests/ | 25 | server-side tests | server/site_tests | 26 | common library code | client/common_lib/cros/ | 27 | server-only library code | server/cros | 28 29 30 ## Writing tests 31 32 An autotest is really defined by its control file. A control file contains 33 important metadata about the test (name, author, description, duration, what 34 suite its in, etc) and then pulls in and executes the actual test code. This 35 test code can be shared among multiple distinct test cases by parameterizing it 36 and passing those parameters in from separate control files. 37 38 Autotests *must*: 39 40 * Be self-contained: assume nothing about the condition of the device 41 * Be hermetic: requiring the Internet to be reachable in order for your test 42 to succeed is unacceptable. 43 * Be automatic: avoid user interaction and run-time specification of input 44 values. 45 * Be integration tests: if you can test the feature in a unit test (or a 46 chrome browser test), do so. 47 * Prefer object composition to inheritance: avoid subclassing test.test to 48 implement common functionality for multiple tests. Instead, create a class 49 that your tests can instantiate to perform common operations. This enables 50 us to write tests that use both PyAuto and Servo without dealing with 51 multiple inheritance, for example. 52 * Be deterministic: a test should not validate the timing of some operation. 53 Instead, write a test that records the timing in performance keyvals so that 54 we can track the numbers over time. 55 56 Autotests *must not*: 57 58 * Put significant logic in the control file: control files are really just 59 python, so one can put arbitrary logic in there. Dont. Run your test 60 code, perhaps with some parameters. 61 62 Autotests *may*: 63 64 * Share parameterized fixtures: a test is defined by a control file. Control 65 files import and run test code, and can pass simple parameters to the code 66 they run through a well-specified interface. 67 68 Autotest has a notion of both client-side tests and server-side tests. Code in 69 a client-side test runs only on the device under test (DUT), and as such isnt 70 capable of maintaining state across reboots or handling a failed suspend/resume 71 and the like. If possible, an autotest should be written as a client-side 72 test. A server test runs on the autotest server, but gets assigned a DUT 73 just like a client-side test. It can use various autotest primitives (and 74 library code written by the CrOS team) to manipulate that device. Most, if not 75 all, tests that use Servo or remote power management should be server-side 76 tests, as an example. 77 78 Adding a test involves putting a control file and a properly-written test 79 wrapper in the right place in the source tree. There are conventions that must 80 be followed, and a variety of primitives available for use. When writing any 81 code, whether client-side test, server-side test, or library, have a strong 82 bias towards using autotest utility code. This keeps the codebase consistent. 83 84 85 ## Writing a test 86 87 This section explains considerations and requirements for any autotest, whether 88 client or server. 89 90 ### Control files 91 92 Upstream documentation 93 Our local conventions for autotest control files deviate from the above a bit, 94 but the indication about which fields are mandatory still holds. 95 96 | Variable | Required | Value | 97 |--------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 98 | AUTHOR | Yes | A comma-delimited string of at least one responsible engineer and a backup engineer -- or at worst a backup mailing list. i.e. AUTHOR = msb, snanda | 99 | DEPENDENCIES | No | list of tags known to the HW test lab. | 100 | DOC | Yes | Long description of the test, pass/fail criteria | 101 | NAME | Yes | Display name of the test. Generally this is the directory where your test lives e.g. hardware_TPMCheck. If you are using multiple run_test calls in the same control file or multiple control files with one test wrapper in the same suite, problems arise with the displaying of your test name. crosbug.com/35795. When in doubt ask. | 102 | SYNC\_COUNT | No | Integer >= 1. Number of simultaneous devices needed for a test run. | 103 | TIME | Yes | Test duration: 'FAST' (<1m), 'MEDIUM' (<10m), 'LONG' (<20m), 'LENGTHY' (>30m) | 104 | TEST\_TYPE | Yes | Client or Server | 105 | SUITE | No | A comma-delimited string of suite names that this test should be a part of. | 106 107 ### Choosing a Suite 108 109 Currently existing suites are defined in the test\_suites/ subdirectory at the 110 top level of the autotest repo. Read the docstrings there to see if your new 111 test fits into one thats already defined. 112 113 When first adding a test, it should not go into the BVT suite. A test should 114 only be added to the BVT after it has been running in some non-BVT suite long 115 enough to establish a track record showing that the test does not fail when run 116 against working software. A suite named experimental exists for tests intended 117 for the BVT, and for which there is no more convenient home. 118 119 ### Pure python 120 121 Lie, cheat and steal to keep your tests in pure python. It will be easier to 122 debug failures, it will be easier to generate meaningful error output, it will 123 be simpler to get your tests installed and run, and it will be simpler for the 124 lab team to build tools that allow you to quickly iterate. 125 126 Shelling out to existing command-line tools is done fairly often, and isnt a 127 terrible thing. The test author can wind up having to do a lot of output 128 parsing, which is often brittle, but this can be a decent tradeoff in lieu of 129 having to reimplement large pieces of functionality in python. 130 131 Note that you will need to be sure that any commands you use are installed on 132 the host. For a client-side test, the host means the DUT. For a 133 server-side test, the host typically means the system running autoserv; 134 however, if you use SiteHost.run(), the command will run on the DUT. On the 135 server, your tests will have access to all tools common to both a typical CrOS 136 chroot environment and standard Goobuntu. 137 138 If you want to use a tool on the DUT, it may be appropriate to include it as a 139 dependency of the chromeos-base/chromeos-test package. This ensures that the 140 tool is pre-installed on every test image for every device, and will always be 141 available for use. Otherwise, the tool must be installed as an autotest dep. 142 143 _Never install your own shell scripts and call them._ Anything you can do in 144 shell, you can do in python. 145 146 ### Reporting failures 147 148 Autotest supports several kinds of failure statuses: 149 150 | Status | Exception | Reason | 151 |----------|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 152 | WARN | error.TestWarn | error.TestWarn should be used when side effects to the test running are encountered but are not directly related to the test running. For example, if you are testing Wifi and powerd crashes. *Currently* there are not any clear usecases for this and error.TestWarn should be generally avoided until further notice. | 153 | TEST\_NA | error.TestNAError | This test does not apply in the current environment. | 154 | ERROR | error.TestError | The test was unable to validate the desired behavior. | 155 | FAIL | error.TestFail | The test determined the desired behavior failed to occur. | 156 157 158 ### Considerations when writing client-side tests 159 160 All client-side tests authored at Google must live in the client/site\_tests sub-directory of the autotest source tree. 161 162 ###Compiling and executing binaries 163 164 It is possible to compile source thats included with your test and use the 165 products at test runtime. The build infrastructure will compile this code for 166 the appropriate target architecture and package it up along with the rest of 167 your tests resources, but this increases your development iteration time as 168 you need to actually re-build and re-package your test to deploy it to the 169 device. While we hope to improve tooling support for this use case in the 170 future, avoiding this issue is the ideal. 171 172 If you cant avoid this, heres how to get your code compiled and installed as 173 a part of your test: 174 1. Create a src/ directory next to your control file. 175 2. Put your source, including its Makefile, in src/ 176 3. define a method in your test class called setup(self) that takes no arguments. 177 4. setup(self) should perform all tasks necessary to build your tool. There are some helpful utility functions in client/common_lib/base_utils.py. Trivial example: 178 179 ``` 180 def setup(self): 181 os.chdir(self.srcdir) 182 utils.make('OUT_DIR=.') 183 ``` 184 185 ### Reusing code (fixtures) 186 187 Any autotest is, essentially, a single usage of a re-usable test fixture. This 188 is because run\_once() in your test wrapper can take any arguments you want. As 189 such, multiple control files can re-use the same wrapper -- and should, where 190 it makes sense. 191 192 ### Considerations when writing server-side tests 193 194 All server-side tests authored at Google must live in the server/site\_tests 195 sub-directory of the autotest source tree. 196 197 It should be even easier to keep the server-side of a test in pure python, as 198 you should simply be driving the DUT and verifying state. 199 200 ### When/why to write a server-side test 201 202 Server-side tests are appropriate when some operation in the test can't be 203 executed on the DUT. The prototypical example is rebooting the DUT. Other 204 examples include tests that manipulate the network around the DUT (e.g. WiFi 205 tests), tests that power off the DUT, and tests that rely on a Servo attached 206 to the DUT. 207 208 One simple criterion for whether to write a server-side test is this: Is the 209 DUT an object that the test must manipulate? If the answer is yes, then a 210 server-side test makes sense. 211 212 ### Control files for server-side tests 213 214 Server-side tests commonly operate on the DUT as an object. Autotest 215 represents the DUT with an instance of class Host; the instance is constructed 216 and passed to the test from the control file. Creating the host object in the 217 control file can be done using certain definitions present in the global 218 environment of every control file: 219 220 * Function hosts.create\_host() will create a host object from a string with 221 the name of the host (an IP address as a string is also acceptable). 222 * Variable machines is a list of the host names available to the test. 223 224 Below is a sample fragment for a control file that runs a simple server side test in parallel on all the hosts specified for the test. The fragment is a complete control file, except for the missing boilerplate comments and documentation definitions required in all control files. 225 226 ``` 227 def run(machine): 228 host = hosts.create_host(machine) 229 job.run_test("platform_ServerTest", host=host) 230 231 parallel_simple(run, machines) 232 ``` 233 234 Note: The sample above relies on a common convention that the run\_once() 235 method of a server-side test defines an argument named host with a default 236 value, e.g. 237 238 ``` 239 def run_once(self, host=None): 240 # test code goes here. 241 ``` 242 243 ### Operations on Host objects 244 245 A Host object supports various methods to operate on a DUT. Below is a short list of important methods supported by instances of Host: 246 247 * run(command) - run a shell command on the host 248 * reboot() - reboot the host, and wait for it to be back on the network 249 * wait_up() - wait for the host to be active on the network 250 * wait_down() - wait until the host is no longer on the network, or until it is known to have rebooted. 251 252 More details, including a longer list of available methods, and more about how 253 they work can be found in the Autotest documentation for autoserv and Autotest 254 documentation for Host. 255 256 ### Servo-based tests 257 258 For server-side tests that use a servo-attached DUT, the host object has a 259 servo attribute. If Autotest determines that the DUT has a Servo attached, the 260 servo attribute will be a valid instance of a Servo client object; otherwise 261 the attribute will be None. 262 263 For a DUT in the lab, Autotest will automatically determine whether there is a 264 servo available; however, if a test requires Servo, its control file must have 265 additional code to guarantee a properly initialized servo object on the host. 266 267 Below is a code snippet outlining the requirements; portions of the control file have been omitted for brevity: 268 269 ``` 270 # ... Standard boilerplate variable assignments... 271 DEPENDENCIES = "servo" 272 # ... more standard boilerplate... 273 274 args_dict = utils.args_to_dict(args) 275 servo_args = hosts.SiteHost.get_servo_arguments(args_dict) 276 277 def run(machine): 278 host = hosts.create_host(machine, servo_args=servo_args) 279 job.run_test("platform_SampleServoTest", host=host) 280 281 parallel_simple(run, machines) 282 ``` 283 284 The `DEPENDENCIES` setting guarantees that if the test is scheduled in the lab, 285 it will be assigned to a DUT that has a servo. 286 287 The setting of `servo_args` guarantees two distinct things: First, it forces 288 checks that will make sure that the Servo is functioning properly; this 289 guarantees that the host's `servo` attribute will not be None. Second, the code 290 allows you to pass necessary servo specific command-line arguments to 291 `test_that`. 292 293 If the test control file follows the formula above, the test can be reliably called in a variety of ways: 294 * When used for hosts in the lab, the hosts servo object will use the servo attached to the host, and the test can assume that the servo object is not None. 295 * If you start servod manually on your desktop using the default port, you can use test_that without any special options. 296 * If you need to specify a non-default host or port number (e.g. because servod is remote, or because you have more than one servo board), you can specify them with commands like these: 297 298 ``` 299 test_that --args=servo_host=... 300 test_that --args=servo_port=... 301 test_that --args=servo_host=... servo_port=... ... 302 ``` 303 304 ### Calling client-side tests from a server-side test 305 306 Commonly, server-side tests need to do more on the DUT than simply run short 307 shell commands. In those cases, a client-side test should be written and 308 invoked from the server-side test. In particular, a client side test allows 309 the client side code to be written in Python that uses standard Autotest 310 infrastructure, such as various utility modules or the logging infrastructure. 311 312 Below is a short snippet showing the standard form for calling a client-side 313 test from server-side code: 314 315 ``` 316 from autotest_lib.server import autotest 317 318 # ... inside some function, e.g. in run_once() 319 client_at = autotest.Autotest(host) 320 client_at.run_test("platform_ClientTest") 321 ``` 322 323 ### Writing library code 324 325 There is a large quantity of Chromium OS specific code in the autotest 326 codebase. Much of this exists to provide re-usable modules that enable tests 327 to talk to system services. The guidelines from above apply here as well. 328 This code should be as pure python as possible, though it is reasonable to 329 shell out to command line tools from time to time. In some cases weve done 330 this where we could (now) use the services DBus APIs directly. If youre 331 adding code to allow tests to communicate with your service, it is strongly 332 recommended that you use DBus where possible, instead of munging config files 333 directly or using command-line tools. 334 335 Currently, our library code lives in a concerning variety of places in the 336 autotest tree. This is due to a poor initial understanding of how to do 337 things, and new code should follow the following conventions instead: 338 339 * Used only in server-side tests: server/cros 340 * Used in both server- and client-side tests, or only client: 341 client/common\_lib/cros 342 343 ### Adding test deps 344 345 This does not refer to the optional `DEPENDENCIES` field in test control files. 346 Rather, this section discusses how and when to use code/data/tools that are not 347 pre-installed on test images, and should (or can) not be included right in with 348 the test source. 349 350 Unfortunately, there is no hard-and-fast rule here. Generally, if this is some 351 small tool or blob of data you need for a single test, you should include it as 352 discussed above in Writing client-side tests. If youre writing the tool, and 353 it has use for developers as well as in one or more tests that youre writing, 354 then make it a first-class CrOS project. Write an ebuild, write unit tests, 355 and then add it to the test image by default. This can be done by RDEPENDing 356 on your new test package from the chromeos-test ebuild. 357 358 If your code/data falls in the middle (useful to several tests, not to devs), 359 and/or is large (hundreds of megabytes as opposed to tens) then using an 360 autotest dep may be the right choice. Conceptually, an autotest test dep is 361 simply another kind of archive that the autotest infrastructure knows how to 362 fetch and unpack. There are two components to including a dependency from an 363 autotest test -- setup during build time and installing it on your DUT when 364 running a test. The setup phase must be run from your tests setup() method 365 like so: 366 367 ``` 368 def setup(self): 369 self.job.setup_dep([mydep]) 370 logging.debug(mydep is at %s % (os.path.join(self.autodir, 371 deps/mydep)) 372 ``` 373 374 The above gets run when you build the test. 375 376 The other half of this equation is actually installing the dependency so you 377 can use it while running a test. To do this, add the following to either your 378 run\_once or initialize methods: 379 380 ``` 381 dep = dep_name 382 dep_dir = os.path.join(self.autodir, 'deps', dep=dep) 383 self.job.install_pkg(dep, 'dep', dep_dir) 384 ``` 385 386 387 You can now reference the content of your dep using dep_dir. 388 389 Now that you know how to include a dep, the next question is how to write one. 390 Before you read further, you should check out client/deps/\* for many examples 391 of deps in our autotest tree. 392 393 ### Create a dep from a third-party package 394 395 There are many examples of how to do this in the client/deps directory already. 396 The key component is to check in a tarball of the version of the dependency 397 youd like to include under client/deps/your\_dep. 398 399 All deps require a control file and an actual python module by the same name. 400 They will also need a copy of common.py to import utils.update\_version. Both 401 the control and common are straightforward, the python module does all the 402 magic. 403 404 The deps python module follows a standard convention: a setup function and a 405 call to utils.update\_version. update\_version is used instead of directly 406 calling setup as it maintains additional versioning logic ensuring setup is 407 only done 1x per dep. The following is its method signature: 408 409 ``` 410 def update_version(srcdir, preserve_srcdir, new_version, install, 411 *args, **dargs) 412 ``` 413 414 415 Notably, install should be a pointer to your setup function and `*args` should 416 be filled in with params to said setup function. 417 418 If you are using a tarball, your setup function should look something like: 419 420 ``` 421 def setup(tarball, my_dir) 422 utils.extract_tarball_to_dir(tarball, my_dir) 423 os.chdir(my_dir) 424 utils.make() # this assumes your tarball has a Makefile. 425 ``` 426 427 And you would invoke this with: 428 429 ``` 430 utils.update_version(os.getcwd(), True, version, setup, tarball_path, 431 os.getcwd()) 432 ``` 433 434 435 Note: The developer needs to call this because def setup is a function they are 436 defining that can take any number of arguments or install the dep in any way 437 they see fit. The above example uses tarballs but some are distributed as 438 straight source under the src dir so their setup function only takes a top 439 level path. We could avoid this by forcing a convention but that would be 440 artificially constraining the deps mechanism. 441 442 Once youve created the dep, you will also have to add the dep to the 443 autotest-deps package in chromiumos-overlay/chromeos-base/autotest-deps, 444 cros\_workon start it, and re-emerge it. 445 446 ### Create a dep from other chrome-os packages 447 448 One can also create autotest deps from code that lives in other CrOS packages, 449 or from build products generated by other packages. This is similar as above 450 but you can reference code using the `CHROMEOS_ROOT` env var that points to the 451 root of the CrOS source checkout, or the SYSROOT env var (which points to 452 /build/<board>) to refer to build products. Again, read the above. Heres an 453 example of the former with the files I want in 454 chromeos\_tree/chromite/my\_dep/\* where this will be the python code in 455 autotest/files/client/deps/my\_dep/my\_dep.py module. 456 457 ``` 458 import common, os, shutil 459 from autotest_lib.client.bin import utils 460 461 version = 1 462 463 def setup(setup_dir): 464 my_dep_dir = os.path.join(os.environ['CHROMEOS_ROOT'], 'chromite', 465 'buildbot') 466 shutil.copytree(my_dep_dir, setup_dir) 467 468 469 work_dir = os.path.join(os.getcwd(), 'src') 470 utils.update_version(os.getcwd(), True, version, setup, work_dir) 471 ``` 472
