1 page.title=End-to-End Test Example 2 @jd:body 3 4 <!-- 5 Copyright 2015 The Android Open Source Project 6 7 Licensed under the Apache License, Version 2.0 (the "License"); 8 you may not use this file except in compliance with the License. 9 You may obtain a copy of the License at 10 11 http://www.apache.org/licenses/LICENSE-2.0 12 13 Unless required by applicable law or agreed to in writing, software 14 distributed under the License is distributed on an "AS IS" BASIS, 15 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 See the License for the specific language governing permissions and 17 limitations under the License. 18 --> 19 <div id="qv-wrapper"> 20 <div id="qv"> 21 <h2>In this document</h2> 22 <ol id="auto-toc"> 23 </ol> 24 </div> 25 </div> 26 27 <p>This tutorial guides you through the construction of a "hello world" Trade Federation test 28 configuration, and gives you a hands-on introduction to the Trade Federation framework. Starting 29 from the TF development environment, it guides you through the process of creating a simple Trade 30 Federation config and gradually adding more features to it.</p> 31 32 <p>The tutorial presents the TF test development process as a set of exercises, each consisting of 33 several steps. The exercises demonstrate how to gradually build and refine your configuration, and 34 provide all the sample code you need to complete the test configuration. The title of each 35 exercise is annotated with a letter describing which roles are involved in that step: <b>D</b> for 36 Developer, <b>I</b> for Integrator, and/or <b>R</b> for Test Runner.</p> 37 38 <p>When you are finished with the tutorial, you will have created a functioning TF configuration and 39 will have learned many of the most important concepts in the TF framework.</p> 40 41 <h2 id="setup">Set up Trade Federation development environment</h2> 42 <p>See the <a href="/devices/tech/test_infra/tradefed/fundamentals/machine_setup.html" 43 >Machine Setup</a> page for how to setup the development environment. The rest of this tutorial 44 assumes you have a shell open that has been initialized to the Trade Federation environment.</p> 45 46 <p>For simplicity, this tutorial will illustrate adding a configuration and its classes to the 47 Trade Federation framework core library. This can be extended to developing modules outside the 48 source tree by simply compiling the tradefed JAR, and compiling your modules against that JAR.</p> 49 50 <h2 id="testclass">Creating a test class (D)</h2> 51 <p>Lets create a hello world test that just dumps a message to stdout. A tradefed test will 52 generally implement the <a href="/reference/com/android/tradefed/testtype/IRemoteTest.html" 53 >IRemoteTest</a> interface.</p> 54 55 <p>Here's an implementation for the HelloWorldTest:</p> 56 <pre><code>package com.android.tradefed.example; 57 58 import com.android.tradefed.device.DeviceNotAvailableException; 59 import com.android.tradefed.result.ITestInvocationListener; 60 import com.android.tradefed.testtype.IRemoteTest; 61 62 public class HelloWorldTest implements IRemoteTest { 63 @Override 64 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 65 System.out.println("Hello, TF World!"); 66 } 67 } 68 </code></pre> 69 70 <p>Save this sample code to 71 <code><tree>/tools/tradefederation/prod-tests/src/com/android/tradefed/example/HelloWorldTest.java</code> 72 and rebuild tradefed from your shell:</p> 73 <pre><code>m -jN</code></pre> 74 75 <p>If the build does not succeed, consult the 76 <a href="/devices/tech/test_infra/tradefed/fundamentals/machine_setup.html">Machine Setup</a> page 77 to ensure that you didn't miss any steps.</p> 78 79 <h2 id="createconfig">Creating a Configuration (I)</h2> 80 <p>Trade Federation tests are made executable by creating a <b>Configuration</b>, which is an XML file 81 that instructs tradefed on which test (or tests) to run, as well as which other modules to 82 execute, and in what order.</p> 83 84 <p>Lets create a new Configuration for our HelloWorldTest:</p> 85 <pre><code><configuration description="Runs the hello world test"> 86 <test class="com.android.tradefed.example.HelloWorldTest" /> 87 </configuration></code></pre> 88 89 <p>TF will parse the Configuration XML file (aka <b>config</b>), load the specified class using 90 reflection, instantiate it, cast it to a <code>IRemoteTest</code>, and call its <code>run</code> 91 method.</p> 92 93 <p>Note that we've specified the full class name of the HelloWorldTest. Save this data to a 94 <code>helloworld.xml</code> file anywhere on your local filesystem (eg <code>/tmp/helloworld.xml</code>).</p> 95 96 <h2 id="runconfig">Running the config (R)</h2> 97 <p>From your shell, launch the tradefed console</p> 98 <pre><code>$ tradefed.sh 99 </code></pre> 100 101 <p>Ensure that a device is connected to the host machine and is visible to tradefed</p> 102 <pre><code>tf >list devices 103 Serial State Product Variant Build Battery 104 004ad9880810a548 Available mako mako JDQ39 100 105 </code></pre> 106 107 <p>Configurations can be executed using the <code>run <config></code> console command. Try this:</p> 108 <pre><code>tf> run /tmp/helloworld.xml 109 05-12 13:19:36 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 110 Hello, TF World! 111 </code></pre> 112 <p>You should see "Hello, TF World!" outputted on the terminal.</p> 113 114 <h2 id="addconfig">Adding the config to the Classpath (D, I, R)</h2> 115 <p>For convenience of deployment, you can also bundle configs into the tradefed jars 116 themselves. Tradefed will automatically recognize all configurations placed in 'config' folders on 117 the classpath.</p> 118 119 <p>Lets illustrate this now by moving the helloworld.xml into the tradefed core library.</p> 120 <p>Move the <code>helloworld.xml</code> file into 121 <code><tree>/tools/tradefederation/prod-tests/res/config/example/helloworld.xml</code>.</p> 122 <p>Rebuild tradefed, and restart the tradefed console. </p> 123 <p>Ask tradefed to display the list of configurations from the classpath:</p> 124 <pre><code>tf> list configs 125 [] 126 example/helloworld: Runs the hello world test 127 </code></pre> 128 129 <p>You can now run the helloworld config via the following command</p> 130 <pre><code>tf >run example/helloworld 131 05-12 13:21:21 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 132 Hello, TF World! 133 </code></pre> 134 135 <h2 id="deviceinteract">Interacting with a Device (D, R)</h2> 136 <p>So far our hello world test isn't doing anything interesting. Tradefed's specialty is running 137 tests using Android devices, so lets add an Android device to the test.</p> 138 139 <p>Tests can get a reference to an Android device by implementing the 140 <a href="/reference/com/android/tradefed/testtype/IDeviceTest.html">IDeviceTest</a> interface.</p> 141 142 <p>Here's a sample implementation of what this looks like:</p> 143 <pre><code>public class HelloWorldTest implements IRemoteTest, IDeviceTest { 144 private ITestDevice mDevice; 145 @Override 146 public void setDevice(ITestDevice device) { 147 mDevice = device; 148 } 149 150 @Override 151 public ITestDevice getDevice() { 152 return mDevice; 153 } 154 155 } 156 </code></pre> 157 158 <p>The Trade Federation framework will inject the <code>ITestDevice</code> reference into your 159 test via the <code>IDeviceTest#setDevice</code> method, before the <code>IRemoteTest#run</code> 160 method is called.</p> 161 162 <p>Let's modify the HelloWorldTest print message to display the serial number of the device.</p> 163 <pre><code>@Override 164 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 165 System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber()); 166 } 167 </code></pre> 168 169 <p>Now rebuild tradefed, and check the list of devices:</p> 170 <pre><code>$ tradefed.sh 171 tf >list devices 172 Serial State Product Variant Build Battery 173 004ad9880810a548 Available mako mako JDQ39 100 174 </code></pre> 175 176 <p>Take note of the serial number listed as Available above. That is the device that should be 177 allocated to HelloWorld.</p> 178 <pre><code>tf >run example/helloworld 179 05-12 13:26:18 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 180 Hello, TF World! I have device 004ad9880810a548 181 </code></pre> 182 183 <p>You should see the new print message displaying the serial number of the device.</p> 184 185 <h2 id="sendresults">Sending Test Results (D)</h2> 186 <p><code>IRemoteTest</code>s report results by calling methods on the 187 <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html" 188 >ITestInvocationListener</a> instance provided to their <code>#run</code> method. Note that the 189 TF framework itself is responsible for reporting the start and end of each Invocation, (via 190 the <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html#invocationStarted(com.android.tradefed.build.IBuildInfo)" 191 >ITestInvocationListener#invocationStarted</a> and 192 <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html#invocationEnded(long)" 193 >ITestInvocationListener#invocationEnded</a> methods, respectively).</p> 194 195 <p>A <b>test run</b> is a logical collection of tests. To report test results, 196 <code>IRemoteTest</code>s are responsible 197 for reporting the start of a test run, the start and end of each test, and the end of the test run.</p> 198 199 <p>Here's what the HelloWorldTest implementation might look like with a single failed test result.</p> 200 <pre><code>@Override 201 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 202 System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber()); 203 204 TestIdentifier testId = new TestIdentifier("com.example.TestClassName", "sampleTest"); 205 listener.testRunStarted("helloworldrun", 1); 206 listener.testStarted(testId); 207 listener.testFailed(TestFailure.FAILURE, testId, "oh noes, test failed"); 208 listener.testEnded(testId, Collections.emptyMap()); 209 listener.testRunEnded(0, Collections.emptyMap()); 210 }</code></pre> 211 212 <p>Note that Trade Federation also includes several <code>IRemoteTest</code> implementations that 213 you can reuse instead of writing your own from scratch. These include, for instance, 214 <a href="/reference/com/android/tradefed/testtype/InstrumentationTest.html" 215 >InstrumentationTest</a>, which can run an Android application's tests remotely on an Android 216 device, parse the results, and forward them to the <code>ITestInvocationListener</code>). See the 217 <a href="/reference/com/android/tradefed/testtype/package-summary.html">Test Types 218 documentation</a> for more details.</p> 219 220 <h2 id="storeresults">Storing Test Results (I)</h2> 221 <p>By default, a TF config will use the 222 <a href="/reference/com/android/tradefed/result/TextResultReporter.html">TextResultReporter</a> as 223 the test listener implementation. <code>TextResultReporter</code> will dump the results of an 224 invocation to stdout. To illustrate, try running the hello-world config from the previous 225 section:</p> 226 <pre><code>$ ./tradefed.sh 227 tf >run example/helloworld 228 05-16 20:03:15 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 229 Hello, TF World! I have device 004ad9880810a548 230 05-16 20:03:15 I/InvocationToJUnitResultForwarder: run helloworldrun started: 1 tests 231 Test FAILURE: com.example.TestClassName#sampleTest 232 stack: oh noes, test failed 233 05-16 20:03:15 I/InvocationToJUnitResultForwarder: run ended 0 ms 234 </code></pre> 235 236 <p>If you want to store the results of an invocation elsewhere, such as in a file, you need to 237 specify a custom <code>ITestInvocationListener</code> implementation by using the 238 <code>result_reporter</code> tag in your configuration.</p> 239 240 <p>Trade Federation includes the 241 <a href="/reference/com/android/tradefed/result/XmlResultReporter.html">XmlResultReporter</a> 242 listener, which will write test results to an XML file, in a format similar to that used by the 243 <em>ant</em> JUnit XML writer.</p> 244 245 <p>Let's specify the result_reporter in the configuration now. Edit the 246 <code>/res/config/example/helloworld.xml</code> config like this:</p> 247 <pre><code><configuration description="Runs the hello world test"> 248 <test class="com.android.tradefed.example.HelloWorldTest" /> 249 <result_reporter class="com.android.tradefed.result.XmlResultReporter" /> 250 </configuration> 251 </code></pre> 252 253 <p>Now rebuild tradefed and re-run the hello world sample:</p> 254 <pre><code>tf >run example/helloworld 255 05-16 21:07:07 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 256 Hello, TF World! I have device 004ad9880810a548 257 05-16 21:07:07 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_2991649128735283633/device_logcat_6999997036887173857.txt 258 05-16 21:07:07 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_2991649128735283633/host_log_6307746032218561704.txt 259 05-16 21:07:07 I/XmlResultReporter: XML test result file generated at /tmp/0/inv_2991649128735283633/test_result_536358148261684076.xml. Total tests 1, Failed 1, Error 0 260 </code></pre> 261 262 <p>Notice the log message stating that an XML file has been generated. The generated file should look like this:</p> 263 <pre><code><?xml version='1.0' encoding='UTF-8' ?> 264 <testsuite name="stub" tests="1" failures="1" errors="0" time="9" timestamp="2011-05-17T04:07:07" hostname="localhost"> 265 <properties /> 266 <testcase name="sampleTest" classname="com.example.TestClassName" time="0"> 267 <failure>oh noes, test failed 268 </failure> 269 </testcase> 270 </testsuite> 271 </code></pre> 272 273 <p>You can also write your own custom invocation listeners. It just needs to implement the 274 <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html" 275 >ITestInvocationListener</a> interface.</p> 276 277 <p>Also note that tradefed supports multiple invocation listeners, meaning that you can send test 278 results to multiple independent destinations. Just specify multiple 279 <code><result_reporter></code> tags in your config to do this.</p> 280 281 <h2 id="logging">Logging (D, I, R)</h2> 282 <p>TradeFederation includes two logging facilities:</p> 283 <ol> 284 <li>ability to capture logs from the device (aka device logcat)</li> 285 <li>ability to record logs from the TradeFederation framework running on the host machine (aka the 286 host log)</li> 287 </ol> 288 <p>Lets focus on #2 for now. Trade Federation's host logs are reported using the 289 <a href="/reference/com/android/tradefed/log/LogUtil.CLog.html">CLog wrapper</a> for the 290 ddmlib Log class.</p> 291 292 <p>Let's convert the previous <code>System.out.println</code> call in HelloWorldTest to a 293 <code>CLog</code> call:</p> 294 <pre><code>@Override 295 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 296 CLog.i("Hello, TF World! I have device %s", getDevice().getSerialNumber()); 297 </code></pre> 298 299 <p>Note that <code>CLog</code> handles string interpolation directly, akin to 300 <code>String.format</code>. At this point, when you rebuild and rerun TF, you should see the 301 log message on stdout.</p> 302 <pre><code>tf> run example/helloworld 303 304 05-16 21:30:46 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548 305 306 </code></pre> 307 308 <p>By default, tradefed will 309 <a href"/reference/com/android/tradefed/log/StdoutLogger.html">output host log messages to 310 stdout</a>. TF also includes a log implementation that will write messages to a file: 311 <a href="/reference/com/android/tradefed/log/FileLogger.html">FileLogger</a>. To add file logging, 312 add a <code>logger</code> tag to the config, specifying the full class name of 313 <code>FileLogger</code>.</p> 314 <pre><code><configuration description="Runs the hello world test"> 315 <test class="com.android.tradefed.example.HelloWorldTest" /> 316 <result_reporter class="com.android.tradefed.result.XmlResultReporter" /> 317 <logger class="com.android.tradefed.log.FileLogger" /> 318 </configuration> 319 </code></pre> 320 321 <p>Now rebuild and run the helloworld example again.</p> 322 <pre><code>tf >run example/helloworld 323 324 05-16 21:38:21 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_6390011618174565918/device_logcat_1302097394309452308.txt 325 05-16 21:38:21 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt 326 327 </code></pre> 328 <p>Note the log message indicating the path of the host log. View the contents of that file, and you 329 should see your HelloWorldTest log message</p> 330 <pre><code>$ more /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt 331 332 05-16 21:38:21 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548 333 </code></pre> 334 335 <p>The TradeFederation framework will also automatically capture the logcat from the allocated device, 336 and send it the invocation listener for processing. <code>XmlResultReporter</code> will save the 337 captured device logcat as a file.</p> 338 339 <h2 id="optionhandling">Option Handling (D, I, R)</h2> 340 <p>Objects loaded from a Trade Federation Configuration (aka <b>Configuration objects</b>) also have the 341 ability to receive data from command line arguments.</p> 342 <p>This is accomplished via the <code>@Option</code> annotation. To participate, a Configuration object class 343 would apply the <code>@Option</code> annotation to a member field, and provide it a unique name. This would 344 allow that member field's value to be populated via a command line option, and would also 345 automatically add that option to the configuration help system.</p> 346 <p class="note"><strong>Note:</strong> Not all field types are supported. See the 347 <a href="/reference/com/android/tradefed/config/OptionSetter.html">OptionSetter javadoc</a> for a 348 description of supported types.</p> 349 350 <p>Let's add an <code>@Option</code> to the HelloWorldTest:</p> 351 <pre><code>@Option(name="my_option", 352 shortName='m', 353 description="this is the option's help text", 354 // always display this option in the default help text 355 importance=Importance.ALWAYS) 356 private String mMyOption = "thisisthedefault"; 357 </code></pre> 358 359 <p>And let's add a log message to display the value of the option in HelloWorldTest, so we can 360 demonstrate that it was received correctly.</p> 361 <pre><code>@Override 362 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 363 364 CLog.logAndDisplay(LogLevel.INFO, "I received option '%s'", mMyOption); 365 </code></pre> 366 367 <p>Rebuild TF and run helloworld; you should see a log message with <code>my_option</code>'s 368 default value.</p> 369 <pre><code>tf> run example/helloworld 370 371 05-24 18:30:05 I/HelloWorldTest: I received option 'thisisthedefault' 372 </code></pre> 373 374 <h3 id="passclivalues">Passing Values from the Command Line</h3> 375 <p>Now pass in a value for my_option: you should see my_option getting populated with that value</p> 376 <pre><code>tf> run example/helloworld --my_option foo 377 378 05-24 18:33:44 I/HelloWorldTest: I received option 'foo' 379 </code></pre> 380 381 <p>TF configurations also include a help system, which automatically displays help text for 382 <code>@Option</code> fields. Try it now, and you should see the help text for 383 <code>my_option</code>:</p> 384 <pre><code>tf> run --help example/helloworld 385 Printing help for only the important options. To see help for all options, use the --help-all flag 386 387 cmd_options options: 388 --[no-]help display the help text for the most important/critical options. Default: false. 389 --[no-]help-all display the full help text for all options. Default: false. 390 --[no-]loop keep running continuously. Default: false. 391 392 test options: 393 -m, --my_option this is the option's help text Default: thisisthedefault. 394 395 'file' logger options: 396 --log-level-display the minimum log level to display on stdout. Must be one of verbose, debug, info, warn, error, assert. Default: error. 397 </code></pre> 398 399 <p>Note the message at the top about "printing only the important options." To reduce option help 400 clutter, TF uses the <code>Option#importance</code> attribute to determine whether to show a 401 particular <code>@Option</code> field's help text 402 when <code>--help</code> is specified. <code>--help-all</code> will always show help for all 403 <code>@Option</code> fields, regardless of importance. See the 404 <a href="/reference/com/android/tradefed/config/Option.Importance.html" 405 >Option.Importance javadoc</a> for details.</p> 406 407 <h3 id="passconfvalues">Passing Values from a Configuration</h3> 408 <p>You can also specify an Option's value within the config by adding a 409 <code><option name="" value=""></code> element. Let's see how this looks in 410 <code>helloworld.xml</code>:</p> 411 <pre><code><test class="com.android.tradefed.example.HelloWorldTest" > 412 <option name="my_option" value="fromxml" /> 413 </test> 414 </code></pre> 415 416 <p>Re-building and running helloworld should now produce this output:</p> 417 <pre><code>05-24 20:38:25 I/HelloWorldTest: I received option 'fromxml' 418 </code></pre> 419 420 <p>The configuration help should also be updated to indicate my_option's new default value:</p> 421 <pre><code>tf> run --help example/helloworld 422 test options: 423 -m, --my_option this is the option's help text Default: fromxml. 424 </code></pre> 425 <p>Also note that other configuration objects included in the helloworld config, such as 426 <code>FileLogger</code>, also accept options. The option <code>--log-level-display</code> is 427 interesting because it filters the logs that show up on stdout. You may have noticed from earlier 428 in the tutorial that the "Hello, TF World! I have device ' log message stopped being displayed 429 on stdout once we switched to using <code>FileLogger</code>. You can increase the verbosity of 430 logging to stdout by passing in the <code>--log-level-display</code> arg.</p> 431 432 <p>Try this now, and you should see the 'I have device' log message reappear on stdout, in 433 addition to being logged to a file.</p> 434 <pre><code>tf >run --log-level-display info example/helloworld 435 436 05-24 18:53:50 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548 437 </code></pre> 438 439 <h2 id="conclusion">That's All, Folks!</h2> 440 <p>As a reminder, if you're stuck on something, the 441 <a href="https://android.googlesource.com/platform/tools/tradefederation/+/master" 442 >Trade Federation source code</a> has a lot of useful information that isn't 443 exposed in the documentation. And if all else fails, try asking on the 444 <a href="{@docRoot}source/community.html">android-platform</a> Google Group, with "Trade Federation" 445 in the message subject.</p> 446
