This document lists all tests available for evaluating the Android camera hardware abstraction layer (HAL). It is intended for OEMs and application processor (AP) vendors so they may ensure proper implementation of the camera HAL with minimum defects. Although this is a voluntary addition to the Android Compatibility Test Suite (CTS), it greatly increases camera test coverage and will certainly identify potential bugs.
By passing these tests, original equipment manufacturers (OEM) validate whether
they have properly integrated the latest Android camera hardware abstraction
layer (HAL) 3.2 interfaces. When conforming with all items in the checklist, a
device implementation may be considered full with respect to the new Android
Camera HAL interfaces. This will in turn enable a device to properly support
the new android.hardware.camera2 package that camera apps build upon.
The Android Camera HAL 3.2 specification is the authoritative source of information on what devices must satisfy; the document here provides a summary of all tests that can be used as a checklist. Camera HAL implementers (e.g. AP vendors) should go through the HAL 3.2 specification line-by-line and ensure their devices conform to it.
The current HAL 3.2 specification is defined in these files within the L generic Android Platform Development Kit (PDK):
hardware/libhardware/include/hardware/camera3.h,
hardware/libhardware/include/hardware/camera_common.h
system/media/camera/docs/docs.html
system/core/include/system/graphics.h
Here are the primary types of tests available for the latest Android camera along with references to associated instructions below:
README files and tutorial.py for setup instructions
pdk/apps/TestingCamera/
pdk/apps/TestingCamera2/
All of these test types are described in detail below. These tests are presented in the chronological order in which OEMs are expected to execute them.
For instance, if a device fails the native tests, it will assuredly fail the subsequent Compatibility Test Suite (CTS) tests. And if a device fails CTS, there is little use in proceeding to the Image Test Suite (ITS). We recommend addressing failures in each test type before proceeding to the next set of tests.
These tests directly test the camera HAL interface.
The starting path for Camera native tests is:
platform/hardware/libhardware
To set up these tests:
cd hardware/libhardware/tests/camera*/mmadb remount; adb sync
Find these camera tests under:
hardware/libhardware/tests/camera3/*
To run all tests:
adb shell /data/nativetest/camera3_test/camera3_test
You receive an OK for each passed test. Errors are logged at the end of output along with a summary of tests passed.
Find these camera tests under:
hardware/libhardware/tests/camera2/*
To run all tests:
adb shell /data/nativetest/camera3_test/camera3_test
To run a single test, pass the --gtest_filter argument and the
test name, like so:
adb shell /data/nativetest/camera2_test/camera2_test --gtest_filter=Camera2Test.OpenClose
To run a subset of tests, use a wildcard with the
--gtest_filter argument, like so:
adb shell /data/nativetest/camera2_test/camera2_test --gtest_filter=Camera2Test.*
Find these camera tests under: frameworks/av/camera/tests/*
Find these camera tests under: system/media/camera/tests/*
Camera Android Compatibility Test Suite (CTS) tests focus upon device compatibility. They do not require a specific test environment (the field of view or FOV CTS Verifier test being the lone exception).
The starting path for Camera CTS tests is: platform/cts
See the CTS introduction and its subpages for general instructions on running CTS.
android.hardware.Camera APIFind these camera tests under cts/tests/tests/:
hardware/src/android/hardware/cts/CameraTest.java
hardware/src/android/hardware/cts/CameraGLTest.java
hardware/src/android/hardware/cts/Camera_SizeTest.java
permission/src/android/permission/cts/CameraPermissionTest.java
android.hardware.camera2 APIFind these camera tests under cts/tests/tests/:
hardware/src/android/hardware/camera2/cts/*
permission/src/android/permission/cts/Camera2PermissionTest.java
Find these camera tests under:
cts/apps/CtsVerifier/src/com/android/cts/verifier/camera/*
The CameraITS tests focus upon image correctness. These Python scripts are manually run on a workstation with the Android device connected over USB. The workstation can run any operating system as long as it has the requisite Python 2.7 environment.
Since ITS is a CTS Verifier subtest, start CTS Verifier and the ITS subtest before running the python scripts so they have processes with which to communicate.
The CameraITS infrastructure and tests are located under: cts/apps/CameraITS
See the latest README file in this top-level folder for
instructions on how to set up and run the tests. For setup: make
cts
extract root/out/host/linux-x86/cts-verfier/android-cts-verifier.zipcd android-cts-verifieradb install -r CtsVerifier.apkcd CameraITS
See tutorial.py in the tests subdirectory for a
walkthrough of the script's use. Each test resides in a corresponding
tests/scene<#> subdirectory. See the README file in
each subdirectory for specific test instructions.
You will need a simple physical environment with a specific, reusable target such as a white wall, grey card, and desk lamp. The Android device is mounted on a tripod and its camera functions are exercised by the scripts. Most tests are pass or fail but some offer metrics, as well.
These tests are works-in-progress and are not yet comprehensive enough for full automated pass/fail validation of the camera HAL. However, these scripts do test scenarios that are not tested in CTS and are an important component of the overall HAL 3.2 test plan.
This test requires no specific setup. Pass all of the tests in the
tests/scene0 folder, for all cameras (back + front + any
others).
Pass all of the tests in the tests/scene1 folder, for all
cameras (back + front + any others). The tests/scene1/README file
describes the scene setup.
Pass all of the tests in the tests/scene2 folder, for all
cameras (back + front + any others). The tests/scene2/README file
describes the scene setup.
For all cameras on the device (front, back, and any others), verify:
For all cameras on the device (front, back, and any others), verify:
For all cameras on the device (front, back, and any others), and for all resolutions available in the menu, verify that correct resolution settings are returned and applied for:
Verify that exposure compensation is applied (at +2 and -2).
Capture full 360-degree PhotoSphere images shot with each of the front and rear cameras. Verify all of the individual frames are focused at infinity and the exposure and white balance match between shots.
Capture vertical, horizontal, and wide-angle panoramas (with both front and rear cameras), and verify all of the individual frames are focused at infinity and the exposure and white balance matches between shots.
Capture a LensBlur image with both front and rear cameras, and verify refocusing to different depths (while reviewing the captured shots) works.
Also verify tap-to-focus, continuous autofocus, AWB, and AEC are reliable in this mode.
Pass all of the camera-related media tests in MediaFrameworkTest. Please note,
these tests require the mediaframeworktest.apk be installed on the Android
device. You will need to make mediaframeworktest and then use adb
to install the resulting .apk. Example commands are included below.
The starting path for Camera-related media framework tests is:
platform/frameworks/base
Find the source code for the tests here:
frameworks/base/media/tests/MediaFrameworkTest
To set up these tests:
make mediaframeworktestadb install out/target/product/<name>/data/app/mediaframeworktest.apk
Where the
Find all of the tests in the following directory or its subdirectories:
frameworks/base/media/tests/MediaFrameworkTest/src/com/android/mediaframeworktest
Each subdirectory represents a class of tests:
functional/
integration/
performance/
power/
stress/
unit/
To see all of the available tests::
adb shell pm list instrumentation
This will yield results resembling:
instrumentation:com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner (target=com.android.mediaframeworktest) instrumentation:com.android.mediaframeworktest/.MediaRecorderStressTestRunner (target=com.android.mediaframeworktest) instrumentation:com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner (target=com.android.mediaframeworktest) instrumentation:com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner (target=com.android.mediaframeworktest)
Identify and extract the component (between instrumentation:
and (target=com.android.mediaframeworktest) from each test line.
The component is composed of the target package name
(com.android.mediaframeworktest) and the test runner name
(MediaFramework).
For instance:
com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner com.android.mediaframeworktest/.MediaRecorderStressTestRunner com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner
You may then pass each component to adb shell am instrument like so:
adb shell am instrument -w <component.name>
Where the <component.name> equals the extracted value above. For example:
adb shell am instrument -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
Please note, while the class path is the Java package + class name, the instrumentation package isn't necessarily the same as the Java package. Make sure you use the AndroidManifest.xml package when concatenating the component name, not the Java package in which the test runner class resides.
To run a single class of tests, pass the -e class
adb shell am instrument -e class com.android.mediaframeworktest.integration.CameraBinderTest -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
To run only a single method in a test class, append a pound (#) sign and the
method name (in this case, testConnectPro) to the class name, like so:
adb shell am instrument -e class 'com.android.mediaframeworktest.integration.CameraBinderTest#testConnectPro' -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
Here is an example run of a functional test. This test verifies the basic functionality of different combinations of camera settings. (ie, Flash, exposure, WB, scene, picture size and geoTag)
Run the test command:
adb shell am instrument -w -r -e delay_msec 15 -e log true -e class com.android.mediaframeworktest.functional.camera.CameraPairwiseTest com.android.mediaframeworktest/com.android.mediaframeworktest.CameraStressTestRunner
Here is an example run of an integration test, in this case mediaframeworktest/integration/CameraBinderTest.java and mediaframeworktest/CameraStressTestRunner.java:
adb shell am instrument -e class \ 'com.android.mediaframeworktest.integration.CameraBinderTest' -w \ 'com.android.mediaframeworktest/.CameraStressTestRunner'
If successful, this results in output resembling:
----- com.android.mediaframeworktest.integration.CameraBinderTest:........... Test results for CameraStressTestRunner=........... Time: 3.328 OK (11 tests) -----
This preview memory test will open and release the camera preview for 200 times. In each 20 iterations, the snapshot of ps mediaserver will be recorded and it will compare the memory usage different after 200 iterations. Test will fail If the difference is greater than 150kM.
Run the test command:
adb shell am instrument -w -r -e class com.android.mediaframeworktest.performance.MediaPlayerPerformance#testCameraPreviewMemoryUsage com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner
More detailed output can be found in:
/sdcard/mediaMemOutput.txt
The commands to run unit tests are all similar. For example, for CameraMetadataTest.java, the command would be:
adb shell am instrument -e class 'com.android.mediaframeworktest.unit.CameraMetadataTest' -w 'com.android.mediaframeworktest/.CameraStressTestRunner'
This test is to stress out the camera image capture and video recording.
Run the test command:
adb shell am instrument -w com.google.android.camera.tests/com.android.camera.stress.CameraStressTestRunner
All tests should pass.
The TestingCam app should be run manually with the following checks performed.
The source for TestingCam is here: pdk/apps/TestingCamera/
Start TestingCam, turn on preview, and ensure that autofocus mode is set to infinity. Using the Take picture button, capture shots of distant subjects (at least 10m away) with the camera pointed horizontally, upwards (close to vertical), and downwards (close to vertical); an example of the upwards shot could be high leaves/branches of a tree from beneath and an example of the downwards shot could be the street as seen from the roof of a building. In all cases, the distant subject should be sharp and in focus. Save and view the shots in the gallery view so that you can zoom in and inspect the sharpness more easily.
Note that for a camera with a VCM actuator to pass this test, it will require either a closed-loop AF control system, or it will need some sort of SW correction based on using accelerometer data to determine camera orientation. Reliable factory calibration of the lens infinity position will also be needed.
The TestingCam2 app should be run manually, with the following checks
performed. The source for TestingCam2 is here: pdk/apps/TestingCamera2/
Start TestingCam2, and press the JPEG button. The image that appears to the right of the viewfinder image should appear the same as the viewfinder, including having the same orientation.