1 <html devsite> 2 <head> 3 <title>Testing Device Provisioning</title> 4 <meta name="project_path" value="/_project.yaml" /> 5 <meta name="book_path" value="/_book.yaml" /> 6 </head> 7 <body> 8 <!-- 9 Copyright 2017 The Android Open Source Project 10 11 Licensed under the Apache License, Version 2.0 (the "License"); 12 you may not use this file except in compliance with the License. 13 You may obtain a copy of the License at 14 15 http://www.apache.org/licenses/LICENSE-2.0 16 17 Unless required by applicable law or agreed to in writing, software 18 distributed under the License is distributed on an "AS IS" BASIS, 19 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 20 See the License for the specific language governing permissions and 21 limitations under the License. 22 --> 23 24 25 26 <p>The Android for Work (AfW) Test Harness is a test suite for validating the 27 AfW compatibility of Android devices. It includes support apps, test cases, 28 configuration files, and a test runner (<code>afw-test-tradefed</code>) built on 29 <code>cts-tradefed</code>. You should setup and run the AfW Test Harness after 30 completing <a href="/devices/tech/admin/provision.html">Provisioning 31 for Device Administration</a>.</p> 32 33 <p class=note><strong>Note:</strong> Building and running the AfW Test Harness 34 is similar to building and running the Android 35 <a href="http://source.android.com/compatibility/cts/index.html">Compatibility 36 Test Suite (CTS)</a>.</p> 37 38 <h2 id=setup_env>Setting up a development environment</h2> 39 <p>The development environment for the AfW Test Harness is similar to Android 40 OS. Follow the steps in 41 <a href="/source/requirements.html">Requirements</a> to set up a 42 development machine.</p> 43 44 <h2 id=download_source>Downloading source code</h2> 45 <p>Download the AfW Test Harness source code using the steps in 46 <a href="/source/downloading.html">Downloading the Source</a>. The AfW 47 Test Harness source code is in the <code>./test/AfwTestHarness</code> project. 48 The branch name determines the version of AfW Test Harness to download (each 49 Android platform has a separate version of AfW Test Harness). For Android 7.0, 50 the branch name is <code>afw-test-harness-nougat-dev</code>. To initialize 51 the repo and download source code for this branch, use:</p> 52 53 <pre class="devsite-click-to-copy"> 54 <code class="devsite-terminal">mkdir WORKING_DIRECTORY</code> 55 <code class="devsite-terminal">cd WORKING_DIRECTORY</code> 56 <code class="devsite-terminal">git config --global user.name "Your Name"</code> 57 <code class="devsite-terminal">git config --global user.email "you (a] example.com"</code> 58 <code class="devsite-terminal">repo init -u https://android.googlesource.com/platform/manifest -b afw-test-harness-nougat-dev</code> 59 <code class="devsite-terminal">repo sync -j24</code> 60 </pre> 61 62 <p>To check out the source code for a different version, specify the branch with 63 the corresponding tag. Available branches include:</p> 64 65 <table> 66 <tr> 67 <th>Branch Name</th> 68 <th>Supported Android Platform</th> 69 </tr> 70 <tr> 71 <td>afw-test-harness-nougat-dev</td> 72 <td>Android 7.0</td> 73 </tr> 74 <tr> 75 <td>afw-test-harness-2.1</td> 76 <td>Android 7.0</td> 77 </tr> 78 <tr> 79 <td>afw-test-harness-marshmallow-dev</td> 80 <td>Android 6.0</td> 81 </tr> 82 <tr> 83 <td>afw-test-harness-1.5</td> 84 <td>Android 6.0</td> 85 </tr> 86 </table> 87 88 <p>Other dependency projects required to build the harness are also downloaded 89 with the source code.</p> 90 91 <h3 id=view_studio>Viewing in Android Studio</h3> 92 <p>To view and edit AfW source code in Android Studio:</p> 93 <ol> 94 <li>Run the following commands 95 <pre class="devsite-click-to-copy"> 96 <code class="devsite-terminal">make idegen</code> 97 <code class="devsite-terminal">development/tools/idegen/idegen.sh</code> 98 </pre> 99 </li> 100 <li>In Android Studio, open <code>android.ipr</code>.</li> 101 </ol> 102 103 <p>The AfW Test Harness source code is in <code>test/AfwTestHarness</code>.</p> 104 105 <h2 id=config_harness>Configuring the AfW Test Harness</h2> 106 <p>You can customize the harness by configuring 107 <code>test/AfwTestHarness/afw-test.props</code>. To run the harness 108 successfully, complete the following steps:</p> 109 <ol> 110 <li>Configure the Wi-Fi network in <code>afw-test.props</code> with the 111 following properties: 112 <pre class="devsite-click-to-copy"> 113 wifi_ssid 114 wifi_password (optional) 115 wifi_security_type (optional, available options are: NONE, WEP or WPA) 116 </pre> 117 </li> 118 <li>Obtain at least one account from a domain that is bound to Test DPC as 119 its device policy controller. Specify the details in <code>afw-test.props</code> 120 with the following properties: 121 <pre class="devsite-click-to-copy"> 122 work_account_username 123 work_account_password 124 </pre> 125 <p>The AfW Test Harness uses Test DPC to test provisioning flows, so accounts 126 <strong>must</strong> bind to Test DPC to run the test harness.</p> 127 </li> 128 </ol> 129 130 <h2 id=build_harness>Building the AfW Test Harness</h2> 131 <p>Initialize the build configuration using:</p> 132 <pre class="devsite-click-to-copy"> 133 <code class="devsite-terminal">source build/envsetup.sh</code> 134 <code class="devsite-terminal">lunch</code> 135 </pre> 136 137 <p>Select a device type and press <strong>Enter</strong>.</p> 138 139 <p>Build the harness using:</p> 140 <pre class="devsite-terminal devsite-click-to-copy"> 141 make afw-test-harness -j32 142 </pre> 143 <p>This creates a directory (<code>out/host/linux-x86/afw-th/android-cts</code>) 144 with all necessary binaries, configuration files, and tools to run the test 145 harness. This directory is also zipped into a file 146 (<code>out/host/linux-x86/afw-th/android-afw-test-harness.zip</code>) 147 for distribution.</p> 148 149 <h2 id=run_harness>Running the AfW Test Harness</h2> 150 <p>Use the following steps to run the AfW Test Harness:</p> 151 <ol> 152 <li>In your build environment, launch the test runner using: 153 <pre class="devsite-terminal devsite-click-to-copy"> 154 afw-test-tradefed 155 </pre> 156 This starts the <code>cts-tf</code> console, loads test plans, test cases, 157 and <code>afw-test.props</code> from 158 <code>out/host/linux-x86/afw-th/android-cts</code>.</li> 159 <li>From the unzipped folder of <code>android-afw-test-harness.zip</code>, 160 launch the test runner using: 161 <pre class="devsite-click-to-copy"> 162 cts-tf> ./androidcts/tools/afw-testtradefed 163 </pre> 164 This loads test plans, test cases, and <code>afw-test.props</code> from 165 <code>android-cts</code> directory. Ensure 166 <code>./androidcts/repository/testcases/afw-test.props</code> has the work 167 account and Wi-Fi configuration.</li> 168 169 <li>Run a test plan. Each test plan is an XML file that contains a set of test 170 packages from the <code>AfwTestHarness/tests</code> test package directory. 171 Common plans include: 172 173 <ul> 174 <li><code><strong>afw-userdebug-build</strong></code>. Contains all test 175 packages that require a userdebug build.</li> 176 <li><code><strong>afw-user-build</strong></code>. Runs on a user build but 177 requires the test device to be set up properly, including completing the initial 178 setup and enabling USB debugging.</li> 179 </ul> 180 181 <br>To run the test plan <code>afw-userdebug-build</code>, use: 182 <pre class="devsite-click-to-copy"> 183 cts-tf> run cts --plan afw-userdebug-build 184 </pre> 185 To see all test plans, use the command <code>list plans</code>. To view plan 186 definitions, refer to 187 <code>out/host/linux-x86/afw-th/android-cts/repository/plans</code>. 188 <br> 189 </li> 190 <li>Run a test package. To run a single test package, use 191 <pre class="devsite-click-to-copy"> 192 cts-tf> run cts --package com.android.afwtest.NfcProvisioning 193 </pre> 194 To view all packages, use the command <code>list packages</code>. For more 195 options, use the command <code>run cts --help</code>.</li> 196 </ol> 197 198 <h2 id=debug_harness>Debugging the AfW Test Harness</h2> 199 <p>Run all commands in the afw-test-tradefed console (<code>cts-tf</code>), 200 which you can launch by running <code>afw-test-tradefed</code>.</p> 201 <ul> 202 203 <li>Display more information with the <code>-l INFO</code> or <code>-l 204 DEBUG</code> flags. Example: 205 <pre class="devsite-click-to-copy"> 206 cts-tf> run cts plan afw-userdebug-build -l DEBUG 207 </pre> 208 </li> 209 210 <li>Run the test harness on a specific device with the <code>-s</code> flag. 211 Example: 212 <pre class="devsite-click-to-copy"> 213 cts-tf> run cts plan afw-userdebug-build -l DEBUG -s device_sn 214 </pre> 215 </li> 216 217 <li>Run test harness on all connected devices with the 218 <code>--all-devices</code> flag. Example: 219 <pre class="devsite-click-to-copy"> 220 cts-tf> run cts plan afw-userdebug-build -l DEBUG --all-devices 221 </pre> 222 </li> 223 224 <li>View current running executions using <code>list invocations</code> or 225 <code>l i</code>.</li> 226 227 <li>View summary of past test executions using <code>list results</code> or 228 <code>l r</code>.</li> 229 230 <li>View other <code>list</code> commands using <code>help list</code>.</li> 231 232 <li>Monitor real-time logcat with filter using <code>afwtest</code>, then open 233 another terminal and start logcat using: <code>adb logcat | grep afwtest</code>. 234 After a test completes: 235 <ul> 236 <li>View logs in 237 <code>out/host/linux-x86/afw-th/android-cts/repository/logs/<em>start-time</em></code>. 238 The full device logcat and host log (<code>afw-test-tradefed</code> logs) are 239 saved in separate zip files.</li> 240 241 <li>Find relevant information by searching the device logcat for 242 <strong>afwtest</strong>. Example: <code>zless 243 out/host/linux-x86/afw-th/android-cts/repository/logs/<em>start-time</em>/device_logcat_<em>random-number</em>.zip 244 | grep afwtest</code></li> 245 246 <li>To view the full afw-test-tradefed log, use: <code>zless 247 out/host/linux-x86/afw-th/android-cts/repository/logs/<em>start-time</em>/host_log_<em>random-number</em>.zip</code> 248 </li> 249 </ul> 250 </li> 251 <li>A test package automates an AfW provisioning flow by going through UI pages 252 and recording a navigation log in the device logcat file for each page. 253 Example: <code>afwtest.AutomationDriver: 254 Navigating:com.android.afwtest.uiautomator.pages.gms.AddAccountPage</code> 255 <br>UI pages for test package 256 <code>com.android.afwtest.NfcProvisioning</code> include:<ul> 257 <li> 258 <code>com.android.afwtest.uiautomator.pages.managedprovisioning.NfcProvisioningPage</code> 259 </li> 260 <li><code>com.android.afwtest.uiautomator.pages.PageSkipper</code></li> 261 <li><code>com.android.afwtest.uiautomator.pages.LandingPage</code></li> 262 </ul> 263 </li> 264 <li>If a test failed during the provisioning process, logcat contains an error 265 similar to: 266 <pre class="devsite-click-to-copy"> 267 TestRunner: java.lang.RuntimeException: Failed to load page: com.android.afwtest.uiautomator.pages.packageinstaller.DeviceAccessPage 268 </pre> 269 This is typically caused by errors in a previous UI page or the page that 270 failed to load, so try to find other error messages in logcat before this error, 271 then try to reproduce it manually following the provisioning flow.</li> 272 <li>If a test package fails: 273 <ul> 274 <li>A screenshot is saved to 275 <code>out/host/linux-x86/afw-th/android-cts/repository/logs/<em>start-time</em></code> 276 using the following syntax: 277 <code>screenshot-test_<em>test_class_full_name</em>_<em>test_case_name</em>-<em>random_number</em>.png</code>. 278 This information is also logged in the host log.</li> 279 <li>A bug report is saved to 280 <code>out/host/linux-x86/afw-th/android-cts/repository/logs/<em>start-time</em></code> 281 as: 282 <code>bug-<em>test_class_full_name</em>_<em>test_case_name</em>-<em>random_number</em>.zip</code>. 283 </li> 284 </ul> 285 </li> 286 <li>After all test packages execute, a screenshot is taken and saved to 287 <code>out/host/linux-x86/afw-th/android-cts/repository/logs/<em>start-time</em></code> 288 as: <code>screenshot-<em>random_number</em>.png</code>. 289 This information is also logged in the host log.</li> 290 </ul> 291 292 <h2 id=faq>FAQ</h2> 293 <p>For help with questions not answered below, contact 294 <a href="mailto:afw-testharness-support (a] google.com">afw-testharness-support (a] google.com</a>. 295 </p> 296 297 <p><strong>Can I run test plan <code>afw-userdebug-build</code> on a device 298 flashed with user build?</strong></p> 299 <p><em>No. Test packages in the <code>afw-userdebug-build</code> plan factory 300 reset the testing device before running the actual test flow and require 301 <code>adb</code> debugging to be auto-enabled. With a user build, 302 <code>adb</code> debugging can be enabled only by manually changing the 303 setting in Developer options.</em></p> 304 305 <p><strong>Can I run test plan <code>afw-user-build</code> on a device flashed 306 with userdebug build?</strong></p> 307 <p><em>Yes, but we recommend that you run this test plan on a user build.</em></p> 308 309 <p><strong>Sometimes my test fails because UI loading takes too much time. How 310 can I fix this?</strong></p> 311 <p><em>Configure the <code>timeout_size</code> setting 312 in <code>./android-cts/repository/testcases/afw-test.props</code>. Valid 313 settings are: S, M, L, XL, XXL.</em></p> 314 315 <p><strong>The test package 316 <code>com.android.afwtest.NfcProvisioning</code> (or 317 <code>SuwDoProvisioning</code>) fails on my device because the installed initial 318 setup (i.e. Setup Wizard) shows customized UI (such as Term & Conditions) 319 after provisioning is complete. How can I skip this customized UI?</strong></p> 320 <p><em>There should be minimal UI after the provisioning process. The test 321 harness will automatically skip such UI if the UI has a button that has 322 meaningful text or content description that contains any of the following words: 323 Skip, Finish, Done, Accept, Agree, Next, Continue, or Proceed. Alternatively, 324 you can define a button in <code>afw-test.props</code> to configure the test 325 harness to skip your UI. Example:</em></p> 326 <pre class="devsite-click-to-copy"> 327 oem_widgets=your_btn 328 your_btn.text=your_customized_text 329 your_btn.package=your_package 330 your_btn.action=click 331 </pre> 332 <p><em>To define multiple widgets, separate using commas.</em></p> 333 334 <p><strong>The test package 335 <code>com.android.afwtest.NfcProvisioning</code> (or 336 <code>SuwDoProvisioning</code>) failed and the last UI screen is "Verify your 337 account." Why does this happen and how can I recover the testing device? 338 </strong></p> 339 <p><em>This failure occurs because the previous test package failed to clear 340 Factory Reset Protection at the end of the test. You must manually enter the 341 account to unlock the device.</em></p> 342 343 <p><strong>My device needs more time to factory reset. Can I extend the factory 344 reset timeout?</strong></p> 345 <p><em>Yes. Configure the <code>factory_reset_timeout_min</code> setting in 346 <code>afw-test.props</code>. Valid settings are in minutes; you can set to any 347 number of minutes that works with your device.</em></p> 348 349 </body> 350 </html> 351
