1 page.title=Camera HAL Testing Checklist 2 @jd:body 3 4 <!-- 5 Copyright 2016 The Android Open Source Project 6 7 Licensed under the Apache License, Version 2.0 (the "License"); 8 you may not use this file except in compliance with the License. 9 You may obtain a copy of the License at 10 11 http://www.apache.org/licenses/LICENSE-2.0 12 13 Unless required by applicable law or agreed to in writing, software 14 distributed under the License is distributed on an "AS IS" BASIS, 15 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 See the License for the specific language governing permissions and 17 limitations under the License. 18 --> 19 <div id="qv-wrapper"> 20 <div id="qv"> 21 <h2>In this document</h2> 22 <ol id="auto-toc"> 23 </ol> 24 </div> 25 </div> 26 27 <p>This document lists all tests available for evaluating the Android camera 28 hardware abstraction layer (HAL). It is intended for OEMs and application 29 processor (AP) vendors so they may ensure proper implementation of the camera 30 HAL with minimum defects. Although this is a voluntary addition to the Android 31 Compatibility Test Suite (CTS), it greatly increases camera test coverage and 32 will certainly identify potential bugs.</p> 33 34 <p>By passing these tests, original equipment manufacturers (OEM) validate whether 35 they have properly integrated the latest Android camera hardware abstraction 36 layer (HAL) 3.2 interfaces. When conforming with all items in the checklist, a 37 device implementation may be considered <em>full</em> with respect to the new Android 38 Camera HAL interfaces. This will in turn enable a device to properly support 39 the new <code>android.hardware.camera2</code> package that camera apps build upon.</p> 40 41 <h2 id=camera_hal_3_2_specification>[ ] 1. Camera HAL 3.2 specification</h2> 42 43 <p>The Android Camera HAL 3.2 specification is the authoritative source of 44 information on what devices must satisfy; the document here provides a summary 45 of all tests that can be used as a checklist. Camera HAL implementers (e.g. AP 46 vendors) should go through the HAL 3.2 specification line-by-line and ensure 47 their devices conform to it.</p> 48 49 <p>The current HAL 3.2 specification is defined in these files within the L 50 generic Android Platform Development Kit (PDK):</p> 51 52 <ul> 53 <li><em>Camera HAL 3.x interface and spec</em>: <code><a 54 href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera3.h">hardware/libhardware/include/hardware/camera3.h</a></code>, 55 <code><a 56 href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera_common.h">hardware/libhardware/include/hardware/camera_common.h</a></code> 57 <li><em>Camera HAL 3.x metadata spec</em>: <code><a 58 href="https://android.googlesource.com/platform/system/media/+/master/camera/docs/docs.html">system/media/camera/docs/docs.html</a></code> 59 <li><em>HAL pixel format interface and spec</em>: <code><a 60 href="https://android.googlesource.com/platform/system/core/+/master/include/system/graphics.h">system/core/include/system/graphics.h</a></code> 61 </ul> 62 63 <h2 id=camera_test_types>[ ] 2. Camera test types</h2> 64 65 <p>Here are the primary types of tests available for the latest Android camera 66 along with references to associated instructions below:</p> 67 68 <ul> 69 <li><em><a href="#native_tests">Native</a>:</em> Tests that directly test the camera HAL interface 70 <li><em><a href="#cts_tests">Compatibility Test Suite (CTS)</a></em>: Standard, automated Android 71 tests to ensure device compatibility - see the <a 72 href="{@docRoot}compatibility/cts/index.html">CTS introduction</a> and the <a 73 href="{@docRoot}devices/tech/test_infra/tradefed/index.html">Trade Federation 74 Overview</a> 75 <li><em><a href="#its_tests">Image Test Suite (ITS)</a>:</em> Manually run tests to ensure image 76 correctness - see the top-level and 77 test-specific <code>README</code> files and tutorial.py for setup instructions 78 <li><em><a href="#manual_tests_with_aosp_camera_app">Manual tests with the 79 Android Open Source Project (AOSP) Camera App</a>:</em> User-like testing of 80 common camera functions 81 <li><em><a href="#manual_testingcam_tests">Manual TestingCam tests</a>:</em> 82 Run from the source in <code>pdk/apps/TestingCamera/</code> 83 <li><em><a href="#manual_testingcam2_tests">Manual TestingCam2.1 84 tests</a>:</em> Run from the source in <code>pdk/apps/TestingCamera2/</code> 85 </ul> 86 87 <p>All of these test types are described in detail below. These tests are 88 presented in the chronological order in which OEMs are expected to execute 89 them.</p> 90 91 <p>For instance, if a device fails the native tests, it will assuredly fail the 92 subsequent Compatibility Test Suite (CTS) tests. And if a device fails CTS, 93 there is little use in proceeding to the Image Test Suite (ITS). We recommend 94 addressing failures in each test type before proceeding to the next set of 95 tests.</p> 96 97 <h2 id=native_tests>[ ] 3. Native tests</h2> 98 99 <p>These tests directly test the camera HAL interface.</p> 100 101 <p>The starting path for Camera native tests is: 102 <code>platform/hardware/libhardware</code></p> 103 104 <p>To set up these tests:</p> 105 106 <pre> 107 $ cd hardware/libhardware/tests/camera*/ 108 $ mm 109 $ adb remount; adb sync 110 </pre> 111 112 <h3 id=hal_3_x_tests>[ ] 3.1. HAL 3.x tests</h3> 113 114 <p>Find these camera tests under: 115 <code>hardware/libhardware/tests/camera3/*</code></p> 116 117 <p>To run all tests:</p> 118 119 <pre> 120 $ adb shell /data/nativetest/camera3_test/camera3_test 121 </pre> 122 123 <p>You receive an <strong>OK</strong> for each passed test. Errors are logged 124 at the end of output along with a summary of tests passed.</p> 125 126 <h3 id=hal_2_3_tests>[ ] 3.2. HAL 2/3 tests</h3> 127 128 <p>Find these camera tests under: 129 <code>hardware/libhardware/tests/camera2/*</code></p> 130 131 <p>To run all tests:</p> 132 133 <pre> 134 $ adb shell /data/nativetest/camera3_test/camera3_test 135 </pre> 136 137 <p>To run a single test, pass the <code>--gtest_filter</code> argument and the 138 test name, like so:</p> 139 140 <pre> 141 $ adb shell /data/nativetest/camera2_test/camera2_test \ 142 --gtest_filter=Camera2Test.OpenClose 143 </pre> 144 145 <p>To run a subset of tests, use a wildcard with the 146 <code>--gtest_filter</code> argument, like so:</p> 147 148 <pre> 149 $ adb shell /data/nativetest/camera2_test/camera2_test \ 150 --gtest_filter=Camera2Test.* 151 </pre> 152 153 <h3 id=3_tests_that_interact_with_the_camera_service>[ ] 3.3. Tests that 154 interact with the camera service</h3> 155 156 <p>Find these camera tests under: <code>frameworks/av/camera/tests/*</code></p> 157 158 <h3 id=camera_metadata_tests>[ ] 3.4. Camera metadata tests</h3> 159 160 <p>Find these camera tests under: <code>system/media/camera/tests/*</code></p> 161 162 <h2 id=cts_tests>[ ] 4. Compatibility Test Suite (CTS) tests</h2> 163 164 <p>Camera Android Compatibility Test Suite (CTS) tests focus upon device 165 compatibility. They do not require a specific test environment (the field of 166 view or FOV CTS Verifier test being the lone exception).</p> 167 168 <p>The starting path for Camera CTS tests is: <code>platform/cts</code></p> 169 170 <p>See the <a href="{@docRoot}compatibility/cts/index.html">CTS 171 introduction</a> and its subpages for general instructions on running CTS.</p> 172 173 <h3 id=cts_tests_for_the_android_hardware_camera_api>[ ] 4.1. CTS tests for 174 the <code>android.hardware.Camera</code> API</h3> 175 176 <p>Find these camera tests under <code>cts/tests/tests/</code>:</p> 177 178 <ul> 179 <li><code>hardware/src/android/hardware/cts/CameraTest.java</code> 180 <li><code>hardware/src/android/hardware/cts/CameraGLTest.java</code> 181 <li><code>hardware/src/android/hardware/cts/Camera_SizeTest.java</code> 182 <li><code>permission/src/android/permission/cts/CameraPermissionTest.java</code> 183 </ul> 184 185 <h3 id=cts_tests_for_the_android_hardware_camera2_api>[ ] 4.2. CTS tests for 186 the <code>android.hardware.camera2</code> API</h3> 187 188 <p>Find these camera tests under <code>cts/tests/tests/</code>:</p> 189 190 <ul> 191 <li><code>hardware/src/android/hardware/camera2/cts/*</code> 192 <li><code>permission/src/android/permission/cts/Camera2PermissionTest.java</code> 193 </ul> 194 195 <h3 id=cts_verifier_camera_tests>[ ] 4.3. CTS Verifier camera tests</h3> 196 197 <p>Find these camera tests under: 198 <code>cts/apps/CtsVerifier/src/com/android/cts/verifier/camera/*</code></p> 199 200 <h2 id=its_tests>[ ] 5. Image Test Suite (ITS) tests</h2> 201 202 <p>The CameraITS tests focus upon image correctness. These Python scripts are 203 manually run on a workstation with the Android device connected over USB. The 204 workstation can run any operating system as long as it has the requisite Python 205 2.7 environment.</p> 206 207 <p class="note">Since ITS is a CTS Verifier subtest, start CTS Verifier and the 208 ITS subtest before running the python scripts so they have processes with which 209 to communicate.</p> 210 211 <p>The CameraITS infrastructure and tests are located under: <code>cts/apps/CameraITS</code></p> 212 213 <p>See the latest <code>README</code> file in this top-level folder for 214 instructions on how to set up and run the tests. For setup: <code>make 215 cts</code></p> 216 217 <pre> 218 $ extract root/out/host/linux-x86/cts-verfier/android-cts-verifier.zip 219 $ cd android-cts-verifier 220 $ adb install -r CtsVerifier.apk 221 $ cd CameraITS 222 </pre> 223 224 <p>See <code>tutorial.py</code> in the <code>tests</code> subdirectory for a 225 walkthrough of the script's use. Each test resides in a corresponding 226 <code>tests/scene<#></code> subdirectory. See the <code>README</code> file in 227 each subdirectory for specific test instructions.</p> 228 229 <p>You will need a simple physical environment with a specific, reusable target 230 such as a white wall, grey card, and desk lamp. The Android device is mounted 231 on a tripod and its camera functions are exercised by the scripts. Most tests 232 are pass or fail but some offer metrics, as well.</p> 233 234 <p>These tests are works-in-progress and are not yet comprehensive enough for 235 full automated pass/fail validation of the camera HAL. However, these scripts 236 do test scenarios that are not tested in CTS and are an important component of 237 the overall HAL 3.2 test plan.</p> 238 239 <h3 id=its_tests_on_scene_0_plain>[ ] 5.1. ITS tests on scene 0 (plain)</h3> 240 241 <p>This test requires no specific setup. Pass all of the tests in the 242 <code>tests/scene0</code> folder, for all cameras (back + front + any 243 others).</p> 244 245 <h3 id=its_tests_on_scene_1_grey_card>[ ] 5.2. ITS tests on scene 1 (grey card)</h3> 246 247 <p>Pass all of the tests in the <code>tests/scene1</code> folder, for all 248 cameras (back + front + any others). The <code>tests/scene1/README</code> file 249 describes the scene setup.</p> 250 251 <h3 id=its_tests_on_scene_2_camera_lab>[ ] 5.3. ITS tests on scene 2 (camera lab)</h3> 252 253 <p>Pass all of the tests in the <code>tests/scene2</code> folder, for all 254 cameras (back + front + any others). The <code>tests/scene2/README</code> file 255 describes the scene setup.</p> 256 257 <h2 id=manual_tests_with_aosp_camera_app>[ ] 6. Manual tests with the AOSP App</h2> 258 259 <h3 id=camera_mode_aosp_camera_app>[ ] 6.1. Camera mode</h3> 260 261 <p>For all cameras on the device (front, back, and any others), verify:</p> 262 263 <ol> 264 <li>Images can be captured and reviewed on the device, and the images look good 265 with no obvious problems. 266 <li>Tap-to-focus, continuous autofocus, macro focus, infinity focus, AWB, and AEC 267 are all reliable. 268 <li>Tap-to-focus, continuous autofocus, AWB, and AEC are reliable when using 269 digital zoom (during capture). 270 <li>Flash settings (on/off/auto) are reliable and synchronize well with the 3As. 271 </ol> 272 273 <h3 id=video_mode_aosp_camera_app>[ ] 6.2. Video mode</h3> 274 275 <p>For all cameras on the device (front, back, and any others), verify:</p> 276 277 <ol> 278 <li>Videos can be captured and reviewed on the device, and the videos look good 279 with no obvious problems. 280 <li>Capturing a snapshot while in the middle of recording a video works. 281 <li>Tap-to-focus, continuous autofocus, macro focus, infinity focus, AWB, and AEC 282 are all reliable. 283 <li>Tap-to-focus, continuous autofocus, AWB, and AEC are reliable when using 284 digital zoom (during capture). 285 <li>Torch settings (on/off) are reliable and synchronize well with the 3As. 286 </ol> 287 288 <h3 id=camera_settings_resolution>[ ] 6.3. Camera settings: resolution</h3> 289 290 <p>For all cameras on the device (front, back, and any others), and for all 291 resolutions available in the menu, verify that correct resolution settings are 292 returned and applied for:</p> 293 294 <ul> 295 <li>Camera mode 296 <li>Video mode 297 <li>LensBlur 298 <li>PhotoSphere 299 <li>Panorama 300 </ul> 301 302 <h3 id=camera_settings_exposure_compensation>[ ] 6.4. Camera settings: 303 exposure compensation</h3> 304 305 <p>Verify that exposure compensation is applied (at +2 and -2).</p> 306 307 <h3 id=photosphere>[ ] 6.5. PhotoSphere</h3> 308 309 <p>Capture full 360-degree PhotoSphere images shot with each of the front and 310 rear cameras. Verify all of the individual frames are focused at infinity and the 311 exposure and white balance match between shots.</p> 312 313 <h3 id=panorama>[ ] 6.6. Panorama</h3> 314 315 <p>Capture vertical, horizontal, and wide-angle panoramas (with both front and 316 rear cameras), and verify all of the individual frames are focused at infinity 317 and the exposure and white balance matches between shots.</p> 318 319 <h3 id=lensblur>[ ] 6.7. LensBlur</h3> 320 321 <p>Capture a LensBlur image with both front and rear cameras, and verify 322 refocusing to different depths (while reviewing the captured shots) works.</p> 323 324 <p>Also verify tap-to-focus, continuous autofocus, AWB, and AEC are reliable in 325 this mode.</p> 326 327 <h2 id=media_framework_tests>[ ] 7. Media Framework tests</h2> 328 329 <p>Pass all of the camera-related media tests in MediaFrameworkTest. Please note, 330 these tests require the mediaframeworktest.apk be installed on the Android 331 device. You will need to <code>make mediaframeworktest</code> and then use adb 332 to install the resulting .apk. Example commands are included below.</p> 333 334 <p>The starting path for Camera-related media framework tests is: 335 <code>platform/frameworks/base</code></p> 336 337 <p>Find the source code for the tests here: 338 <code>frameworks/base/media/tests/MediaFrameworkTest</code></p> 339 340 <p>To set up these tests:</p> 341 342 <pre> 343 $ make mediaframeworktest 344 $ adb install out/target/product/<em><name></em>/data/app/mediaframeworktest.apk 345 </pre> 346 347 <p>Where the <em><code><name></code></em> variable represents the directory 348 containing the vendor's product.</p> 349 350 <p>Find all of the tests in the following directory or its subdirectories:</p> 351 352 <pre> 353 frameworks/base/media/tests/MediaFrameworkTest/src/com/android/mediaframeworktest 354 </pre> 355 356 <p>Each subdirectory represents a class of tests:</p> 357 358 <ul> 359 <li><code>functional/</code> 360 <li><code>integration/</code> 361 <li><code>performance/</code> 362 <li><code>power/</code> 363 <li><code>stress/</code> 364 <li><code>unit/</code> 365 </ul> 366 367 <h3 id=running_media_framework_tests>[ ] 7.1. Running Media Framework tests</h3> 368 369 <p>To see all of the available tests::</p> 370 371 <pre> 372 $ adb shell pm list instrumentation 373 </pre> 374 375 <p>This will yield results resembling:</p> 376 377 <pre> 378 instrumentation:com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner 379 (target=com.android.mediaframeworktest) 380 instrumentation:com.android.mediaframeworktest/.MediaRecorderStressTestRunner 381 (target=com.android.mediaframeworktest) 382 instrumentation:com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner 383 (target=com.android.mediaframeworktest) 384 instrumentation:com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner 385 (target=com.android.mediaframeworktest) 386 </pre> 387 388 <p>Identify and extract the component (between <code>instrumentation:</code> 389 and <code>(target=com.android.mediaframeworktest) </code>from each test line. 390 The component is composed of the target package name 391 (<code>com.android.mediaframeworktest</code>) and the test runner name 392 (<code>MediaFramework<type>TestRunner</code>).</p> 393 394 <p>For instance:</p> 395 396 <pre> 397 com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner 398 com.android.mediaframeworktest/.MediaRecorderStressTestRunner 399 com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner 400 com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner 401 </pre> 402 403 <p>You may then pass each component to <code>adb shell am instrument</code> like so:</p> 404 405 <pre> 406 $ adb shell am instrument -w <component.name> 407 </pre> 408 409 <p>Where the <component.name> equals the extracted value above. For example:</p> 410 411 <pre> 412 $ adb shell am instrument -w \ 413 com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner 414 </pre> 415 416 <p>Please note, while the class path is the Java package + class name, the 417 instrumentation package isn't necessarily the same as the Java package. Make 418 sure you use the AndroidManifest.xml package when concatenating the component 419 name, not the Java package in which the test runner class resides.</p> 420 421 <p>To run a single class of tests, pass the -e class <test-class> argument, like 422 so:</p> 423 424 <pre> 425 $ adb shell am instrument -e class \ 426 com.android.mediaframeworktest.integration.CameraBinderTest -w \ 427 com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner 428 </pre> 429 430 <p>To run only a single method in a test class, append a pound (#) sign and the 431 method name (in this case, <code>testConnectPro</code>) to the class name, like so:</p> 432 433 <pre> 434 $ adb shell am instrument -e class \ 435 'com.android.mediaframeworktest.integration.CameraBinderTest#testConnectPro' -w 436 \ com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner 437 </pre> 438 439 <h3 id=media_settings_functional_tests>[ ] 7.2. Media settings functional tests</h3> 440 441 <p>Here is an example run of a functional test. This test verifies the basic 442 functionality of different combinations of camera settings. (ie, Flash, 443 exposure, WB, scene, picture size and geoTag)</p> 444 445 <p>Run the test command:</p> 446 <pre> 447 $ adb shell am instrument -w -r -e delay_msec 15 -e log true -e class \ 448 com.android.mediaframeworktest.functional.camera.CameraPairwiseTest \ 449 com.android.mediaframeworktest/com.android.mediaframeworktest.CameraStressTestRunner 450 </pre> 451 452 <h3 id=media_integration_tests>[ ] 7.3. Media integration tests</h3> 453 454 <p>Here is an example run of an integration test, in this case 455 mediaframeworktest/integration/CameraBinderTest.java and 456 mediaframeworktest/CameraStressTestRunner.java:</p> 457 458 <pre> 459 $ adb shell am instrument -e class \ 'com.android.mediaframeworktest.<strong>integration</strong>.<strong>CameraBinderTest'</strong> -w \ 'com.android.mediaframeworktest/.<strong>CameraStressTestRunner'</strong> 460 </pre> 461 462 <p>If successful, this results in output resembling:</p> 463 464 <pre> 465 ----- 466 467 com.android.mediaframeworktest.integration.CameraBinderTest:........... 468 Test results for CameraStressTestRunner=........... 469 Time: 3.328 470 471 OK (11 tests) 472 473 ----- 474 </pre> 475 476 <h3 id=media_performance_tests>[ ] 7.4. Media performance tests</h3> 477 478 <p>This preview memory test will open and release the camera preview for 200 479 times. In each 20 iterations, the snapshot of ps mediaserver will be recorded 480 and it will compare the memory usage different after 200 iterations. Test will 481 fail If the difference is greater than 150kM.</p> 482 483 <p>Run the test command:</p> 484 <pre> 485 $ adb shell am instrument -w -r -e class \ 486 com.android.mediaframeworktest.performance.MediaPlayerPerformance#testCameraPreviewMemoryUsage 487 \ com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner 488 </pre> 489 490 <p>More detailed output can be found in: 491 <code>/sdcard/mediaMemOutput.txt</code></p> 492 493 <h3 id=media_unit_tests>[ ] 7.5. Media unit tests</h3> 494 495 <p>The commands to run unit tests are all similar. For example, for 496 CameraMetadataTest.java, the command would be:</p> 497 498 <pre> 499 $ adb shell am instrument -e class \ 500 'com.android.mediaframeworktest.unit.CameraMetadataTest' -w \ 501 'com.android.mediaframeworktest/.CameraStressTestRunner' 502 </pre> 503 504 <h3 id=media_stress_tests>[ ] 7.6. Media stress tests</h3> 505 506 <p>This test is to stress out the camera image capture and video recording.</p> 507 508 <p>Run the test command:</p> 509 510 <pre> 511 $ adb shell am instrument -w 512 com.google.android.camera.tests/com.android.camera.stress.CameraStressTestRunner 513 </pre> 514 515 <p>All tests should pass.</p> 516 517 <h2 id=manual_testingcam_tests>[ ] 8. Manual TestingCam tests</h2> 518 519 <p>The TestingCam app should be run manually with the following checks performed. 520 The source for TestingCam is here: <code>pdk/apps/TestingCamera/</code></p> 521 522 <h3 id=infinity_focus_with_camera_tilt>[ ] 8.1. Infinity focus with camera tilt</h3> 523 524 <p>Start TestingCam, turn on preview, and ensure that autofocus mode is set to 525 infinity. Using the <strong>Take picture</strong> button, capture shots of 526 distant subjects (at least 10m away) with the camera pointed horizontally, 527 upwards (close to vertical), and downwards (close to vertical); an example of 528 the upwards shot could be high leaves/branches of a tree from beneath and an 529 example of the downwards shot could be the street as seen from the roof of a 530 building. In all cases, the distant subject should be sharp and in focus. Save 531 and view the shots in the gallery view so that you can zoom in and inspect the 532 sharpness more easily.</p> 533 534 <p>Note that for a camera with a VCM actuator to pass this test, it will require 535 either a closed-loop AF control system, or it will need some sort of SW 536 correction based on using accelerometer data to determine camera orientation. 537 Reliable factory calibration of the lens infinity position will also be needed.</p> 538 539 <h2 id=manual_testingcam2_tests>[ ] 9. Manual TestingCam2 tests</h2> 540 541 <p>The TestingCam2 app should be run manually, with the following checks 542 performed. The source for TestingCam2 is here: <code>pdk/apps/TestingCamera2/</code></p> 543 544 <h3 id=9_1_jpeg_capture>[ ] 9.1. JPEG capture</h3> 545 546 <p>Start TestingCam2, and press the <strong>JPEG</strong> button. The image 547 that appears to the right of the viewfinder image should appear the same as the 548 viewfinder, including having the same orientation.</p> 549