1 page.title=End-to-End Test Example 2 @jd:body 3 4 <!-- 5 Copyright 2013 The Android Open Source Project 6 7 Licensed under the Apache License, Version 2.0 (the "License"); 8 you may not use this file except in compliance with the License. 9 You may obtain a copy of the License at 10 11 http://www.apache.org/licenses/LICENSE-2.0 12 13 Unless required by applicable law or agreed to in writing, software 14 distributed under the License is distributed on an "AS IS" BASIS, 15 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 See the License for the specific language governing permissions and 17 limitations under the License. 18 --> 19 20 <p>This tutorial guides you through the construction of a "hello world" Trade Federation test 21 configuration, and gives you a hands-on introduction to the Trade Federation framework. Starting 22 from the TF development environment, it guides you through the process of creating a simple Trade 23 Federation config and gradually adding more features to it.</p> 24 25 <p>The tutorial presents the TF test development process as a set of exercises, each consisting of 26 several steps. The exercises demonstrate how to gradually build and refine your configuration, and 27 provide all the sample code you need to complete the test configuration. The title of each 28 exercise is annotated with a letter describing which roles are involved in that step: <b>D</b> for 29 Developer, <b>I</b> for Integrator, and/or <b>R</b> for Test Runner.</p> 30 31 <p>When you are finished with the tutorial, you will have created a functioning TF configuration and 32 will have learned many of the most important concepts in the TF framework.</p> 33 34 <h2>Set up TradeFederation development environment</h2> 35 <p>See the <a href="/devices/tech/test_infra/tradefed/fundamentals/machine_setup.html" 36 >Machine Setup</a> page for how to setup the development environment. The rest of this tutorial 37 assumes you have a shell open that has been initialized to the Trade Federation environment.</p> 38 39 <p>For simplicity, this tutorial will illustrate adding a configuration and its classes to the 40 Trade Federation framework core library. This can be extended to developing modules outside the 41 source tree by simply compiling the tradefed JAR, and compiling your modules against that JAR.</p> 42 43 <h2>Creating a test class (D)</h2> 44 <p>Lets create a hello world test that just dumps a message to stdout. A tradefed test will 45 generally implement the <a href="/reference/com/android/tradefed/testtype/IRemoteTest.html" 46 >IRemoteTest</a> interface.</p> 47 48 <p>Here's an implementation for the HelloWorldTest:</p> 49 <pre><code>package com.android.tradefed.example; 50 51 import com.android.tradefed.device.DeviceNotAvailableException; 52 import com.android.tradefed.result.ITestInvocationListener; 53 import com.android.tradefed.testtype.IRemoteTest; 54 55 public class HelloWorldTest implements IRemoteTest { 56 @Override 57 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 58 System.out.println("Hello, TF World!"); 59 } 60 } 61 </code></pre> 62 63 <p>Save this sample code to 64 <code><tree>/tools/tradefederation/prod-tests/src/com/android/tradefed/example/HelloWorldTest.java</code> 65 and rebuild tradefed from your shell:</p> 66 <pre><code>m -jN</code></pre> 67 68 <p>If the build does not succeed, consult the 69 <a href="/devices/tech/test_infra/tradefed/fundamentals/machine_setup.html">Machine Setup</a> page 70 to ensure that you didn't miss any steps.</p> 71 72 <h2>Creating a Configuration (I)</h2> 73 <p>Trade Federation tests are made executable by creating a <b>Configuration</b>, which is an XML file 74 that instructs tradefed on which test (or tests) to run, as well as which other modules to 75 execute, and in what order.</p> 76 77 <p>Lets create a new Configuration for our HelloWorldTest:</p> 78 <pre><code><configuration description="Runs the hello world test"> 79 <test class="com.android.tradefed.example.HelloWorldTest" /> 80 </configuration></code></pre> 81 82 <p>TF will parse the Configuration XML file (aka <b>config</b>), load the specified class using 83 reflection, instantiate it, cast it to a <code>IRemoteTest</code>, and call its <code>run</code> 84 method.</p> 85 86 <p>Note that we've specified the full class name of the HelloWorldTest. Save this data to a 87 <code>helloworld.xml</code> file anywhere on your local filesystem (eg <code>/tmp/helloworld.xml</code>).</p> 88 89 <h2>Running the config (R)</h2> 90 <p>From your shell, launch the tradefed console</p> 91 <pre><code>$ tradefed.sh 92 </code></pre> 93 94 <p>Ensure that a device is connected to the host machine and is visible to tradefed</p> 95 <pre><code>tf >list devices 96 Serial State Product Variant Build Battery 97 004ad9880810a548 Available mako mako JDQ39 100 98 </code></pre> 99 100 <p>Configurations can be executed using the <code>run <config></code> console command. Try this:</p> 101 <p>FIXME: redo this</p> 102 <pre><code>tf> run /tmp/helloworld.xml 103 05-12 13:19:36 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 104 Hello, TF World! 105 </code></pre> 106 <p>You should see "Hello, TF World!" outputted on the terminal.</p> 107 108 <h2>Adding the config to the Classpath (D, I, R)</h2> 109 <p>For convenience of deployment, you can also bundle configs into the tradefed jars 110 themselves. Tradefed will automatically recognize all configurations placed in 'config' folders on 111 the classpath.</p> 112 113 <p>Lets illustrate this now by moving the helloworld.xml into the tradefed core library.</p> 114 <p>Move the <code>helloworld.xml</code> file into 115 <code><tree>/tools/tradefederation/prod-tests/res/config/example/helloworld.xml</code>.</p> 116 <p>Rebuild tradefed, and restart the tradefed console. </p> 117 <p>Ask tradefed to display the list of configurations from the classpath:</p> 118 <pre><code>tf> list configs 119 [] 120 example/helloworld: Runs the hello world test 121 </code></pre> 122 123 <p>You can now run the helloworld config via the following command</p> 124 <pre><code>tf >run example/helloworld 125 05-12 13:21:21 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 126 Hello, TF World! 127 </code></pre> 128 129 <h2>Interacting with a Device (D, R)</h2> 130 <p>So far our hello world test isn't doing anything interesting. Tradefed's specialty is running 131 tests using Android devices, so lets add an Android device to the test.</p> 132 133 <p>Tests can get a reference to an Android device by implementing the 134 <a href="/reference/com/android/tradefed/testtype/IDeviceTest.html">IDeviceTest</a> interface.</p> 135 136 <p>Here's a sample implementation of what this looks like:</p> 137 <pre><code>public class HelloWorldTest implements IRemoteTest, IDeviceTest { 138 private ITestDevice mDevice; 139 @Override 140 public void setDevice(ITestDevice device) { 141 mDevice = device; 142 } 143 144 @Override 145 public ITestDevice getDevice() { 146 return mDevice; 147 } 148 149 } 150 </code></pre> 151 152 <p>The Trade Federation framework will inject the <code>ITestDevice</code> reference into your 153 test via the <code>IDeviceTest#setDevice</code> method, before the <code>IRemoteTest#run</code> 154 method is called.</p> 155 156 <p>Let's modify the HelloWorldTest print message to display the serial number of the device.</p> 157 <pre><code>@Override 158 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 159 System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber()); 160 } 161 </code></pre> 162 163 <p>Now rebuild tradefed, and check the list of devices:</p> 164 <pre><code>$ tradefed.sh 165 tf >list devices 166 Serial State Product Variant Build Battery 167 004ad9880810a548 Available mako mako JDQ39 100 168 </code></pre> 169 170 <p>Take note of the serial number listed as Available above. That is the device that should be 171 allocated to HelloWorld.</p> 172 <pre><code>tf >run example/helloworld 173 05-12 13:26:18 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 174 Hello, TF World! I have device 004ad9880810a548 175 </code></pre> 176 177 <p>You should see the new print message displaying the serial number of the device.</p> 178 179 <h2>Sending Test Results (D)</h2> 180 <p><code>IRemoteTest</code>s report results by calling methods on the 181 <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html" 182 >ITestInvocationListener</a> instance provided to their <code>#run</code> method. Note that the 183 TF framework itself is responsible for reporting the start and end of each Invocation, (via 184 the <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html#invocationStarted(com.android.tradefed.build.IBuildInfo)" 185 >ITestInvocationListener#invocationStarted</a> and 186 <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html#invocationEnded(long)" 187 >ITestInvocationListener#invocationEnded</a> methods, respectively).</p> 188 189 <p>A <b>test run</b> is a logical collection of tests. To report test results, 190 <code>IRemoteTest</code>s are responsible 191 for reporting the start of a test run, the start and end of each test, and the end of the test run.</p> 192 193 <p>Here's what the HelloWorldTest implementation might look like with a single failed test result.</p> 194 <pre><code>@Override 195 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 196 System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber()); 197 198 TestIdentifier testId = new TestIdentifier("com.example.TestClassName", "sampleTest"); 199 listener.testRunStarted("helloworldrun", 1); 200 listener.testStarted(testId); 201 listener.testFailed(TestFailure.FAILURE, testId, "oh noes, test failed"); 202 listener.testEnded(testId, Collections.emptyMap()); 203 listener.testRunEnded(0, Collections.emptyMap()); 204 }</code></pre> 205 206 <p>Note that Trade Federation also includes several <code>IRemoteTest</code> implementations that 207 you can reuse instead of writing your own from scratch. These include, for instance, 208 <a href="/reference/com/android/tradefed/testtype/InstrumentationTest.html" 209 >InstrumentationTest</a>, which can run an Android application's tests remotely on an Android 210 device, parse the results, and forward them to the <code>ITestInvocationListener</code>). See the 211 <a href="/reference/com/android/tradefed/testtype/package-summary.html">Test Types 212 documentation</a> for more details.</p> 213 214 <h2>Storing Test Results (I)</h2> 215 <p>By default, a TF config will use the 216 <a href="/reference/com/android/tradefed/result/TextResultReporter.html">TextResultReporter</a> as 217 the test listener implementation. <code>TextResultReporter</code> will dump the results of an 218 invocation to stdout. To illustrate, try running the hello-world config from the previous 219 section:</p> 220 <pre><code>$ ./tradefed.sh 221 tf >run example/helloworld 222 05-16 20:03:15 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 223 Hello, TF World! I have device 004ad9880810a548 224 05-16 20:03:15 I/InvocationToJUnitResultForwarder: run helloworldrun started: 1 tests 225 Test FAILURE: com.example.TestClassName#sampleTest 226 stack: oh noes, test failed 227 05-16 20:03:15 I/InvocationToJUnitResultForwarder: run ended 0 ms 228 </code></pre> 229 230 <p>If you want to store the results of an invocation elsewhere, such as in a file, you need to 231 specify a custom <code>ITestInvocationListener</code> implementation by using the 232 <code>result_reporter</code> tag in your configuration.</p> 233 234 <p>Trade Federation includes the 235 <a href="/reference/com/android/tradefed/result/XmlResultReporter.html">XmlResultReporter</a> 236 listener, which will write test results to an XML file, in a format similar to that used by the 237 <em>ant</em> JUnit XML writer.</p> 238 239 <p>Let's specify the result_reporter in the configuration now. Edit the 240 <code>/res/config/example/helloworld.xml</code> config like this:</p> 241 <pre><code><configuration description="Runs the hello world test"> 242 <test class="com.android.tradefed.example.HelloWorldTest" /> 243 <result_reporter class="com.android.tradefed.result.XmlResultReporter" /> 244 </configuration> 245 </code></pre> 246 247 <p>Now rebuild tradefed and re-run the hello world sample:</p> 248 <pre><code>tf >run example/helloworld 249 05-16 21:07:07 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548 250 Hello, TF World! I have device 004ad9880810a548 251 05-16 21:07:07 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_2991649128735283633/device_logcat_6999997036887173857.txt 252 05-16 21:07:07 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_2991649128735283633/host_log_6307746032218561704.txt 253 05-16 21:07:07 I/XmlResultReporter: XML test result file generated at /tmp/0/inv_2991649128735283633/test_result_536358148261684076.xml. Total tests 1, Failed 1, Error 0 254 </code></pre> 255 256 <p>Notice the log message stating that an XML file has been generated. The generated file should look like this:</p> 257 <pre><code><?xml version='1.0' encoding='UTF-8' ?> 258 <testsuite name="stub" tests="1" failures="1" errors="0" time="9" timestamp="2011-05-17T04:07:07" hostname="localhost"> 259 <properties /> 260 <testcase name="sampleTest" classname="com.example.TestClassName" time="0"> 261 <failure>oh noes, test failed 262 </failure> 263 </testcase> 264 </testsuite> 265 </code></pre> 266 267 <p>You can also write your own custom invocation listeners. It just needs to implement the 268 <a href="/reference/com/android/tradefed/result/ITestInvocationListener.html" 269 >ITestInvocationListener</a> interface.</p> 270 271 <p>Also note that tradefed supports multiple invocation listeners, meaning that you can send test 272 results to multiple independent destinations. Just specify multiple 273 <code><result_reporter></code> tags in your config to do this.</p> 274 275 <h2>Logging (D, I, R)</h2> 276 <p>TradeFederation includes two logging facilities:</p> 277 <ol> 278 <li>ability to capture logs from the device (aka device logcat)</li> 279 <li>ability to record logs from the TradeFederation framework running on the host machine (aka the 280 host log)</li> 281 </ol> 282 <p>Lets focus on #2 for now. Trade Federation's host logs are reported using the 283 <a href="/reference/com/android/tradefed/log/LogUtil.CLog.html">CLog wrapper</a> for the 284 ddmlib Log class.</p> 285 286 <p>Let's convert the previous <code>System.out.println</code> call in HelloWorldTest to a 287 <code>CLog</code> call:</p> 288 <pre><code>@Override 289 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 290 CLog.i("Hello, TF World! I have device %s", getDevice().getSerialNumber()); 291 </code></pre> 292 293 <p>Note that <code>CLog</code> handles string interpolation directly, akin to 294 <code>String.format</code>. At this point, when you rebuild and rerun TF, you should see the 295 log message on stdout.</p> 296 <pre><code>tf> run example/helloworld 297 298 05-16 21:30:46 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548 299 300 </code></pre> 301 302 <p>By default, tradefed will 303 <a href"/reference/com/android/tradefed/log/StdoutLogger.html">output host log messages to 304 stdout</a>. TF also includes a log implementation that will write messages to a file: 305 <a href="/reference/com/android/tradefed/log/FileLogger.html">FileLogger</a>. To add file logging, 306 add a <code>logger</code> tag to the config, specifying the full class name of 307 <code>FileLogger</code>.</p> 308 <pre><code><configuration description="Runs the hello world test"> 309 <test class="com.android.tradefed.example.HelloWorldTest" /> 310 <result_reporter class="com.android.tradefed.result.XmlResultReporter" /> 311 <logger class="com.android.tradefed.log.FileLogger" /> 312 </configuration> 313 </code></pre> 314 315 <p>Now rebuild and run the helloworld example again.</p> 316 <pre><code>tf >run example/helloworld 317 318 05-16 21:38:21 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_6390011618174565918/device_logcat_1302097394309452308.txt 319 05-16 21:38:21 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt 320 321 </code></pre> 322 <p>Note the log message indicating the path of the host log. View the contents of that file, and you 323 should see your HelloWorldTest log message</p> 324 <pre><code>$ more /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt 325 326 05-16 21:38:21 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548 327 </code></pre> 328 329 <p>The TradeFederation framework will also automatically capture the logcat from the allocated device, 330 and send it the invocation listener for processing. <code>XmlResultReporter</code> will save the 331 captured device logcat as a file.</p> 332 333 <h2>Option Handling (D, I, R)</h2> 334 <p>Objects loaded from a Trade Federation Configuration (aka <b>Configuration objects</b>) also have the 335 ability to receive data from command line arguments.</p> 336 <p>This is accomplished via the <code>@Option</code> annotation. To participate, a Configuration object class 337 would apply the <code>@Option</code> annotation to a member field, and provide it a unique name. This would 338 allow that member field's value to be populated via a command line option, and would also 339 automatically add that option to the configuration help system (Note: not all field types are 340 supported: see the 341 <a href="/reference/com/android/tradefed/config/OptionSetter.html">OptionSetter javadoc</a> for a 342 description of supported types).</p> 343 344 <p>Let's add an <code>@Option</code> to the HelloWorldTest:</p> 345 <pre><code>@Option(name="my_option", 346 shortName='m', 347 description="this is the option's help text", 348 // always display this option in the default help text 349 importance=Importance.ALWAYS) 350 private String mMyOption = "thisisthedefault"; 351 </code></pre> 352 353 <p>And let's add a log message to display the value of the option in HelloWorldTest, so we can 354 demonstrate that it was received correctly.</p> 355 <pre><code>@Override 356 public void run(ITestInvocationListener listener) throws DeviceNotAvailableException { 357 358 CLog.logAndDisplay(LogLevel.INFO, "I received option '%s'", mMyOption); 359 </code></pre> 360 361 <p>Rebuild TF and run helloworld; you should see a log message with <code>my_option</code>'s 362 default value.</p> 363 <pre><code>tf> run example/helloworld 364 365 05-24 18:30:05 I/HelloWorldTest: I received option 'thisisthedefault' 366 </code></pre> 367 368 <h3>Passing Values from the Command Line</h3> 369 <p>Now pass in a value for my_option: you should see my_option getting populated with that value</p> 370 <pre><code>tf> run example/helloworld --my_option foo 371 372 05-24 18:33:44 I/HelloWorldTest: I received option 'foo' 373 </code></pre> 374 375 <p>TF configurations also include a help system, which automatically displays help text for 376 <code>@Option</code> fields. Try it now, and you should see the help text for 377 <code>my_option</code>:</p> 378 <pre><code>tf> run --help example/helloworld 379 Printing help for only the important options. To see help for all options, use the --help-all flag 380 381 cmd_options options: 382 --[no-]help display the help text for the most important/critical options. Default: false. 383 --[no-]help-all display the full help text for all options. Default: false. 384 --[no-]loop keep running continuously. Default: false. 385 386 test options: 387 -m, --my_option this is the option's help text Default: thisisthedefault. 388 389 'file' logger options: 390 --log-level-display the minimum log level to display on stdout. Must be one of verbose, debug, info, warn, error, assert. Default: error. 391 </code></pre> 392 393 <p>Note the message at the top about "printing only the important options." To reduce option help 394 clutter, TF uses the <code>Option#importance</code> attribute to determine whether to show a 395 particular <code>@Option</code> field's help text 396 when <code>--help</code> is specified. <code>--help-all</code> will always show help for all 397 <code>@Option</code> fields, regardless of importance. See the 398 <a href="/reference/com/android/tradefed/config/Option.Importance.html" 399 >Option.Importance javadoc</a> for details.</p> 400 401 <h3>Passing Values from a Configuration</h3> 402 <p>You can also specify an Option's value within the config by adding a 403 <code><option name="" value=""></code> element. Let's see how this looks in 404 <code>helloworld.xml</code>:</p> 405 <pre><code><test class="com.android.tradefed.example.HelloWorldTest" > 406 <option name="my_option" value="fromxml" /> 407 </test> 408 </code></pre> 409 410 <p>Re-building and running helloworld should now produce this output:</p> 411 <pre><code>05-24 20:38:25 I/HelloWorldTest: I received option 'fromxml' 412 </code></pre> 413 414 <p>The configuration help should also be updated to indicate my_option's new default value:</p> 415 <pre><code>tf> run --help example/helloworld 416 test options: 417 -m, --my_option this is the option's help text Default: fromxml. 418 </code></pre> 419 <p>Also note that other configuration objects included in the helloworld config, such as 420 <code>FileLogger</code>, also accept options. The option <code>--log-level-display</code> is 421 interesting because it filters the logs that show up on stdout. You may have noticed from earlier 422 in the tutorial that the "Hello, TF World! I have device ' log message stopped being displayed 423 on stdout once we switched to using <code>FileLogger</code>. You can increase the verbosity of 424 logging to stdout by passing in the <code>--log-level-display</code> arg.</p> 425 426 <p>Try this now, and you should see the 'I have device' log message reappear on stdout, in 427 addition to being logged to a file.</p> 428 <pre><code>tf >run --log-level-display info example/helloworld 429 430 05-24 18:53:50 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548 431 </code></pre> 432 433 <h2>That's All, Folks!</h2> 434 <p>As a reminder, if you're stuck on something, the 435 <a href="https://android.googlesource.com/platform/tools/tradefederation/+/master" 436 >Trade Federation source code</a> has a lot of useful information that isn't 437 exposed in the documentation. And if all else fails, try asking on the 438 <a href="/source/community/index.html">android-platform</a> Google Group, with "Trade Federation" 439 in the message subject.</p> 440 441
