1 <html devsite> 2 <head> 3 <title>End-to-End Test Example</title> 4 <meta name="project_path" value="/_project.yaml" /> 5 <meta name="book_path" value="/_book.yaml" /> 6 </head> 7 <body> 8 <!-- 9 Copyright 2017 The Android Open Source Project 10 11 Licensed under the Apache License, Version 2.0 (the "License"); 12 you may not use this file except in compliance with the License. 13 You may obtain a copy of the License at 14 15 http://www.apache.org/licenses/LICENSE-2.0 16 17 Unless required by applicable law or agreed to in writing, software 18 distributed under the License is distributed on an "AS IS" BASIS, 19 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 20 See the License for the specific language governing permissions and 21 limitations under the License. 22 --> 23 24 25 26 <p>This tutorial guides you through creating a "hello world" Trade Federation 27 (TF) test configuration and gives you a hands-on introduction to the TF 28 framework. Starting from a development environment, you will create a simple 29 configuration and add features.</p> 30 31 <p>The tutorial presents the test development process as a set of exercises, 32 each consisting of several steps, that demonstrate how to build and gradually 33 refine your configuration. All sample code you need to complete the test 34 configuration is provided, and the title of each exercise is annotated with a 35 letter describing the roles involved in that step:</p> 36 <ul> 37 <li><strong>D</strong> for Developer</li> 38 <li><strong>I</strong> for Integrator</li> 39 <li><strong>R</strong> for Test Runner</li> 40 </ul> 41 42 <p>After completing the tutorial, you will have a functioning TF configuration 43 and understand many important concepts in the TF framework.</p> 44 45 <h2 id="setup">Setting up the Trade Federation development environment</h2> 46 <p>For details on seting up the TF development environment, see 47 <a href="/devices/tech/test_infra/tradefed/fundamentals/machine_setup.html">Machine 48 Setup</a>. The rest of this tutorial assumes you have a shell open that has been 49 initialized to the TF environment.</p> 50 51 <p>For simplicity, this tutorial illustrates adding a configuration and its 52 classes to the TF framework core library. This can be extended to developing 53 modules outside the source tree by compiling the tradefed JAR, then compiling 54 your modules against that JAR.</p> 55 56 <h2 id="testclass">Creating a test class (D)</h2> 57 <p>Lets create a hello world test that just dumps a message to stdout. A 58 tradefed test generally implements the 59 <a href="/reference/com/android/tradefed/testtype/IRemoteTest.html">IRemoteTest</a> 60 interface. Here's an implementation for the HelloWorldTest:</p> 61 <pre class="prettyprint"> 62 package com.android.tradefed.example; 63 64 import com.android.tradefed.device.DeviceNotAvailableException; 65 import com.android.tradefed.result.ITestInvocationListener; 66 import com.android.tradefed.testtype.IRemoteTest; 67 68 public class HelloWorldTest implements IRemoteTest { 69 @Override 70 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 71 System.out.println("Hello, TF World!"); 72 } 73 } 74 </pre> 75 76 <p>Save this sample code to 77 <code><tree>/tools/tradefederation/core/prod-tests/src/com/android/tradefed/example/HelloWorldTest.java</code> 78 and rebuild tradefed from your shell:</p> 79 <pre class="devsite-terminal devsite-click-to-copy"> 80 m -jN 81 </pre> 82 83 <p>Note that <code>System.out</code> in the example above may not actually 84 direct output to the console. While this is acceptable for this test example, 85 you should establish logging in Trade Federation as described in <a 86 href="#logging">Logging (D, I, R)</a>.</p> 87 88 <p>If the build does not succeed, consult 89 <a href="/devices/tech/test_infra/tradefed/fundamentals/machine_setup.html">Machine 90 Setup</a> to ensure you didn't miss a step.</p> 91 92 <h2 id="createconfig">Creating a Configuration (I)</h2> 93 <p>Trade Federation tests are made executable by creating a 94 <strong>Configuration</strong>, an XML file that instructs tradefed on which 95 test (or tests) to run, as well as which other modules to execute and in what 96 order.</p> 97 98 <p>Lets create a new Configuration for our HelloWorldTest (note the full class 99 name of the HelloWorldTest):</p> 100 <pre class="prettyprint"> 101 <configuration description="Runs the hello world test"> 102 <test class="com.android.tradefed.example.HelloWorldTest" /> 103 </configuration></pre> 104 105 <p>Save this data to a <code>helloworld.xml</code> file anywhere on your local 106 filesystem (e.g. <code>/tmp/helloworld.xml</code>). TF will parse the 107 Configuration XML file (aka <b>config</b>), load the specified class using 108 reflection, instantiate it, cast it to a <code>IRemoteTest</code>, and call its 109 <code>run</code> method.</p> 110 111 <h2 id="runconfig">Running the config (R)</h2> 112 <p>From your shell, launch the tradefed console:</p> 113 <pre class="devsite-terminal devsite-click-to-copy"> 114 tradefed.sh 115 </pre> 116 117 <p>Ensure a device is connected to the host machine and is visible to tradefed:</p> 118 <pre class="devsite-click-to-copy"> 119 tf> list devices 120 Serial State Product Variant Build Battery 121 004ad9880810a548 Available mako mako JDQ39 100 122 </pre> 123 124 <p>Configurations can be executed using the <code>run <config></code> 125 console command. Try:</p> 126 <pre class="devsite-click-to-copy"> 127 tf> run /tmp/helloworld.xml 128 05-12 13:19:36 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 129 Hello, TF World! 130 </pre> 131 <p>You should see "Hello, TF World!" output on the terminal.</p> 132 133 <h2 id="addconfig">Adding the config to the Classpath (D, I, R)</h2> 134 <p>For convenience of deployment, you can also bundle configs into the tradefed 135 JARs themselves. Tradefed automatically recognizes all configurations placed in 136 <em>config</em> folders on the classpath.</p> 137 138 <p>To illustrate, move the <code>helloworld.xml</code> file into the tradefed 139 core library 140 (<code><tree>/tools/tradefederation/core/prod-tests/res/config/example/helloworld.xml</code>). 141 Rebuild tradefed, restart the tradefed console, then ask tradefed to display the 142 list of configurations from the classpath:</p> 143 <pre class="devsite-click-to-copy"> 144 tf> list configs 145 [] 146 example/helloworld: Runs the hello world test 147 </pre> 148 149 <p>You can now run the helloworld config using:</p> 150 <pre class="devsite-click-to-copy"> 151 tf> run example/helloworld 152 05-12 13:21:21 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 153 Hello, TF World! 154 </pre> 155 156 <h2 id="deviceinteract">Interacting with a device (D, R)</h2> 157 <p>So far, our HelloWorldTest isn't doing anything interesting. Tradefed's 158 specialty is running tests using Android devices, so lets add an Android device 159 to the test.</p> 160 161 <p>Tests can get a reference to an Android device by implementing the 162 <a href="/reference/com/android/tradefed/testtype/IDeviceTest.html">IDeviceTest</a> 163 interface. Here's a sample implementation of what this looks like:</p> 164 <pre class="prettyprint"> 165 public class HelloWorldTest implements IRemoteTest, IDeviceTest { 166 private ITestDevice mDevice; 167 @Override 168 public void setDevice(ITestDevice device) { 169 mDevice = device; 170 } 171 172 @Override 173 public ITestDevice getDevice() { 174 return mDevice; 175 } 176 177 } 178 </pre> 179 180 <p>The Trade Federation framework will inject the <code>ITestDevice</code> 181 reference into your test via the <code>IDeviceTest#setDevice</code> method, 182 before the <code>IRemoteTest#run</code> method is called.</p> 183 184 <p>Let's modify the HelloWorldTest print message to display the serial number of 185 the device:</p> 186 <pre class="prettyprint"> 187 @Override 188 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 189 System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber()); 190 } 191 </pre> 192 193 <p>Now rebuild tradefed and check the list of devices:</p> 194 <pre class="devsite-terminal devsite-click-to-copy"> 195 tradefed.sh 196 </pre> 197 <pre class="devsite-click-to-copy"> 198 tf> list devices 199 Serial State Product Variant Build Battery 200 004ad9880810a548 Available mako mako JDQ39 100 201 </pre> 202 203 <p>Take note of the serial number listed as <strong>Available</strong>; that is 204 the device that should be allocated to HelloWorld:</p> 205 <pre class="devsite-click-to-copy"> 206 tf> run example/helloworld 207 05-12 13:26:18 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 208 Hello, TF World! I have device 004ad9880810a548 209 </pre> 210 211 <p>You should see the new print message displaying the serial number of the 212 device.</p> 213 214 <h2 id="sendresults">Sending test results (D)</h2> 215 <p><code>IRemoteTest</code> reports results by calling methods on the 216 <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html">ITestInvocationListener</a> 217 instance provided to the <code>#run</code> method. The TF framework itself is 218 responsible for reporting the start (via 219 <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html#invocationStarted(com.android.tradefed.build.IBuildInfo)">ITestInvocationListener#invocationStarted</a>) 220 and end (via 221 <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html#invocationEnded(long)" 222 >ITestInvocationListener#invocationEnded</a>) 223 of each Invocation.</p> 224 225 <p>A <b>test run</b> is a logical collection of tests. To report test results, 226 <code>IRemoteTest</code> is responsible for reporting the start of a test run, 227 the start and end of each test, and the end of the test run.</p> 228 229 <p>Here's what the HelloWorldTest implementation might look like with a single 230 failed test result.</p> 231 <pre class="prettyprint"> 232 @Override 233 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 234 System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber()); 235 236 TestIdentifier testId = new TestIdentifier("com.example.TestClassName", "sampleTest"); 237 listener.testRunStarted("helloworldrun", 1); 238 listener.testStarted(testId); 239 listener.testFailed(testId, "oh noes, test failed"); 240 listener.testEnded(testId, Collections.emptyMap()); 241 listener.testRunEnded(0, Collections.emptyMap()); 242 } 243 </pre> 244 245 <p>TF includes several <code>IRemoteTest</code> implementations you can reuse 246 instead of writing your own from scratch. For example, 247 <a href="/reference/com/android/tradefed/testtype/InstrumentationTest.html">InstrumentationTest</a> 248 can run an Android application's tests remotely on an Android device, parse the 249 results, and forward those results to the <code>ITestInvocationListener</code>). 250 For details, see 251 <a href="/reference/com/android/tradefed/testtype/package-summary.html">Test 252 Types</a>.</p> 253 254 <h2 id="storeresults">Storing test results (I)</h2> 255 <p>The default test listener implementation for a TF config is 256 <a href="/reference/com/android/tradefed/result/TextResultReporter.html">TextResultReporter</a>, 257 which dumps the results of an invocation to stdout. To illustrate, run the 258 HelloWorldTest config from the previous section:</p> 259 <pre class="devsite-terminal devsite-click-to-copy"> 260 ./tradefed.sh 261 </pre> 262 <pre class="devsite-click-to-copy"> 263 tf> run example/helloworld 264 05-16 20:03:15 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 265 Hello, TF World! I have device 004ad9880810a548 266 05-16 20:03:15 I/InvocationToJUnitResultForwarder: run helloworldrun started: 1 tests 267 Test FAILURE: com.example.TestClassName#sampleTest 268 stack: oh noes, test failed 269 05-16 20:03:15 I/InvocationToJUnitResultForwarder: run ended 0 ms 270 </pre> 271 272 <p>To store the results of an invocation elsewhere, such as in a file, specify a 273 custom <code>ITestInvocationListener</code> implementation using the 274 <code>result_reporter</code> tag in your configuration.</p> 275 276 <p>TF also includes the 277 <a href="/reference/com/android/tradefed/result/XmlResultReporter.html">XmlResultReporter</a> 278 listener, which writes test results to an XML file in a format similar to that 279 used by the <em>ant</em> JUnit XML writer. To specify the result_reporter in the 280 configuration, edit the <code>/res/config/example/helloworld.xml</code> 281 config:</p> 282 <pre class="prettyprint"> 283 <configuration description="Runs the hello world test"> 284 <test class="com.android.tradefed.example.HelloWorldTest" /> 285 <result_reporter class="com.android.tradefed.result.XmlResultReporter" /> 286 </configuration> 287 </pre> 288 289 <p>Now rebuild tradefed and re-run the hello world sample:</p> 290 <pre class="devsite-click-to-copy"> 291 tf> run example/helloworld 292 05-16 21:07:07 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 293 Hello, TF World! I have device 004ad9880810a548 294 05-16 21:07:07 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_2991649128735283633/device_logcat_6999997036887173857.txt 295 05-16 21:07:07 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_2991649128735283633/host_log_6307746032218561704.txt 296 05-16 21:07:07 I/XmlResultReporter: XML test result file generated at /tmp/0/inv_2991649128735283633/test_result_536358148261684076.xml. Total tests 1, Failed 1, Error 0 297 </pre> 298 299 <p>Notice the log message stating that an XML file has been generated; the 300 generated file should look like this:</p> 301 <pre class="prettyprint"> 302 <?xml version='1.0' encoding='UTF-8' ?> 303 <testsuite name="stub" tests="1" failures="1" errors="0" time="9" timestamp="2011-05-17T04:07:07" hostname="localhost"> 304 <properties /> 305 <testcase name="sampleTest" classname="com.example.TestClassName" time="0"> 306 <failure>oh noes, test failed 307 </failure> 308 </testcase> 309 </testsuite> 310 </pre> 311 312 <p>You can also write your own custom invocation listeners—they simply 313 need to implement the 314 <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html">ITestInvocationListener</a> 315 interface.</p> 316 317 <p>Tradefed supports multiple invocation listeners, so you can send test results 318 to multiple independent destinations. To do this, just specify multiple 319 <code><result_reporter></code> tags in your config.</p> 320 321 <h2 id="logging">Logging (D, I, R)</h2> 322 <p>TF's logging facilities include the ability to:</p> 323 <ol> 324 <li>Capture logs from the device (aka device logcat)</li> 325 <li>Record logs from the TradeFederation framework running on the host machine 326 (aka host log)</li> 327 </ol> 328 329 <p>The TF framework automatically captures the logcat from the allocated device 330 and sends it to the invocation listener for processing. 331 <code>XmlResultReporter</code> then saves the captured device logcat as a file. 332 </p> 333 334 <p>TF host logs are reported using the 335 <a href="/reference/com/android/tradefed/log/LogUtil.CLog.html">CLog wrapper</a> 336 for the ddmlib Log class. Let's convert the 337 previous <code>System.out.println</code> call in HelloWorldTest to a 338 <code>CLog</code> call:</p> 339 <pre class="prettyprint"> 340 @Override 341 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 342 CLog.i("Hello, TF World! I have device %s", getDevice().getSerialNumber()); 343 </pre> 344 345 <p><code>CLog</code> handles string interpolation directly, similar to 346 <code>String.format</code>. When you rebuild and rerun TF, you should see the 347 log message on stdout:</p> 348 <pre class="devsite-click-to-copy"> 349 tf> run example/helloworld 350 351 05-16 21:30:46 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548 352 353 </pre> 354 355 <p>By default, tradefed 356 <a href"/reference/com/android/tradefed/log/StdoutLogger.html">outputs host log 357 messages to stdout</a>. TF also includes a log implementation that writes 358 messages to a file: 359 <a href="/reference/com/android/tradefed/log/FileLogger.html">FileLogger</a>. 360 To add file logging, add a <code>logger</code> tag to the config, specifying the 361 full class name of <code>FileLogger</code>:</p> 362 <pre class="prettyprint"> 363 <configuration description="Runs the hello world test"> 364 <test class="com.android.tradefed.example.HelloWorldTest" /> 365 <result_reporter class="com.android.tradefed.result.XmlResultReporter" /> 366 <logger class="com.android.tradefed.log.FileLogger" /> 367 </configuration> 368 </pre> 369 370 <p>Now, rebuild and run the helloworld example again:</p> 371 <pre class="devsite-click-to-copy"> 372 tf >run example/helloworld 373 374 05-16 21:38:21 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_6390011618174565918/device_logcat_1302097394309452308.txt 375 05-16 21:38:21 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt 376 377 </pre> 378 <p>The log message indicates the path of the host log, which, when viewed, 379 should contain your HelloWorldTest log message:</p> 380 <pre class="devsite-terminal devsite-click-to-copy"> 381 more /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt</pre> 382 <p>Example output:</p> 383 <pre class="devsite-click-to-copy"> 384 385 05-16 21:38:21 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548 386 </pre> 387 388 <h2 id="optionhandling">Handling options (D, I, R)</h2> 389 <p>Objects loaded from a TF Configuration (aka <b>Configuration objects</b>) 390 can also receive data from command line arguments through the use of the 391 <code>@Option</code> annotation.<p> 392 393 <p>To participate, a Configuration object class applies the <code>@Option</code> 394 annotation to a member field and provides it a unique name. This enables that 395 member field value to be populated via a command line option (and also 396 automatically adds that option to the configuration help system).</p> 397 398 <p class="note"><strong>Note:</strong> Not all field types are supported. For a 399 description of supported types, see 400 <a href="/reference/com/android/tradefed/config/OptionSetter.html">OptionSetter</a>. 401 </p> 402 403 <p>Let's add an <code>@Option</code> to HelloWorldTest:</p> 404 <pre class="prettyprint"> 405 @Option(name="my_option", 406 shortName='m', 407 description="this is the option's help text", 408 // always display this option in the default help text 409 importance=Importance.ALWAYS) 410 private String mMyOption = "thisisthedefault"; 411 </pre> 412 413 <p>Next, let's add a log message to display the value of the option in 414 HelloWorldTest so we can demonstrate it was received correctly:</p> 415 <pre class="prettyprint"> 416 @Override 417 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 418 419 CLog.logAndDisplay(LogLevel.INFO, "I received option '%s'", mMyOption); 420 </pre> 421 422 <p>Finally, rebuild TF and run helloworld; you should see a log message with the 423 <code>my_option</code> default value:</p> 424 <pre class="devsite-click-to-copy"> 425 tf> run example/helloworld 426 427 05-24 18:30:05 I/HelloWorldTest: I received option 'thisisthedefault' 428 </pre> 429 430 <h3 id="passclivalues">Passing values from the command line</h3> 431 <p>Pass in a value for <code>my_option</code>; you should see 432 <code>my_option</code> populated with that value:</p> 433 <pre class="devsite-click-to-copy"> 434 tf> run example/helloworld --my_option foo 435 436 05-24 18:33:44 I/HelloWorldTest: I received option 'foo' 437 </pre> 438 439 <p>TF configurations also include a help system, which automatically displays 440 help text for <code>@Option</code> fields. Try it now, and you should see the 441 help text for <code>my_option</code>:</p> 442 <pre class="devsite-click-to-copy"> 443 tf> run example/helloworld --help 444 Printing help for only the important options. To see help for all options, use the --help-all flag 445 446 cmd_options options: 447 --[no-]help display the help text for the most important/critical options. Default: false. 448 --[no-]help-all display the full help text for all options. Default: false. 449 --[no-]loop keep running continuously. Default: false. 450 451 test options: 452 -m, --my_option this is the option's help text Default: thisisthedefault. 453 454 'file' logger options: 455 --log-level-display the minimum log level to display on stdout. Must be one of verbose, debug, info, warn, error, assert. Default: error. 456 </pre> 457 458 <p>Note the message about "printing only the important options." To reduce 459 option help clutter, TF uses the <code>Option#importance</code> attribute to 460 determine whether to show a particular <code>@Option</code> field help text when 461 <code>--help</code> is specified. <code>--help-all</code> always shows help for 462 all <code>@Option</code> fields, regardless of importance. For details, see 463 <a href="/reference/com/android/tradefed/config/Option.Importance.html">Option.Importance</a>. 464 </p> 465 466 <h3 id="passconfvalues">Passing values from a configuration</h3> 467 <p>You can also specify an Option value within the config by adding a 468 <code><option name="" value=""></code> element. Test it using 469 <code>helloworld.xml</code>:</p> 470 <pre class="prettyprint"> 471 <test class="com.android.tradefed.example.HelloWorldTest" > 472 <option name="my_option" value="fromxml" /> 473 </test> 474 </pre> 475 476 <p>Re-building and running helloworld should now produce this output:</p> 477 <pre class="devsite-click-to-copy"> 478 05-24 20:38:25 I/HelloWorldTest: I received option 'fromxml' 479 </pre> 480 481 <p>The configuration help should also update to indicate the default value of 482 <code>my_option</code>:</p> 483 <pre class="devsite-click-to-copy"> 484 tf> run example/helloworld --help 485 test options: 486 -m, --my_option this is the option's help text Default: fromxml. 487 </pre> 488 489 <p>Other configuration objects included in the helloworld config, such as 490 <code>FileLogger</code>, also accept options. The option 491 <code>--log-level-display</code> is interesting because it filters the logs that 492 show up on stdout. Earlier in the tutorial, you may have noticed the "Hello, TF 493 World! I have device ' log message stopped being displayed on stdout after we 494 switched to using <code>FileLogger</code>. You can increase the verbosity of 495 logging to stdout by passing in the <code>--log-level-display</code> arg.</p> 496 497 <p>Try this now, and you should see the 'I have device' log message reappear on 498 stdout, in addition to being logged to a file:</p> 499 <pre class="devsite-click-to-copy"> 500 tf> run example/helloworld --log-level-display info 501 502 05-24 18:53:50 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548 503 </pre> 504 505 <h2 id="conclusion">That's all, folks!</h2> 506 <p>As a reminder, if you're stuck on something, the 507 <a href="https://android.googlesource.com/platform/tools/tradefederation/+/master">Trade 508 Federation source code</a> has a lot of useful information that isn't exposed in 509 the documentation. If all else fails, try asking on the 510 <a href="/source/community.html">android-platform</a> Google Group, 511 with "Trade Federation" in the message subject.</p> 512 513 </body> 514 </html> 515
