1 # Copyright 2013 The Android Open Source Project 2 # 3 # Licensed under the Apache License, Version 2.0 (the "License"); 4 # you may not use this file except in compliance with the License. 5 # You may obtain a copy of the License at 6 # 7 # http://www.apache.org/licenses/LICENSE-2.0 8 # 9 # Unless required by applicable law or agreed to in writing, software 10 # distributed under the License is distributed on an "AS IS" BASIS, 11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 # See the License for the specific language governing permissions and 13 # limitations under the License. 14 15 16 Android Camera Imaging Test Suite (ITS) 17 ======================================= 18 19 1. Introduction 20 --------------- 21 22 The ITS is a framework for running tests on the images produced by an Android 23 camera. The general goal of each test is to configure the camera in a desired 24 manner and capture one or more shots, and then examine the shots to see if 25 they contain the expected image data. Many of the tests will require that the 26 camera is pointed at a specific target chart or be illuminated at a specific 27 intensity. 28 29 2. Setup 30 -------- 31 32 There are two components to the ITS: 33 1. The Android device running ItsService.apk. 34 2. A host machine connected to the Android device that runs Python tests. 35 36 2.1. Device setup 37 ----------------- 38 39 Build and install ItsService.apk for your device. After setting up your 40 shell for Android builds, from the pdk/apps/CameraITS directory run the 41 following commands: 42 43 cd service 44 mma -j32 45 adb install -r <YOUR_OUTPUT_PATH>/ItsService.apk 46 47 using whatever path is appropriate to your output ItsService.apk file. 48 49 2.2. Host PC setup 50 ------------------ 51 52 The first pre-requisite is the Android SDK, as adb is used to communicate with 53 the device. 54 55 The test framework is based on Python on the host machine. It requires 56 Python 2.7 and the scipy/numpy stack, including the Python Imaging Library. 57 58 (For Ubuntu users) 59 60 sudo apt-get install python-numpy python-scipy python-matplotlib 61 62 (For other users) 63 64 All of these pieces can be installed on your host machine separately, 65 however it is highly recommended to install a bundled distribution of 66 Python that comes with these modules. Some different bundles are listed 67 here: 68 69 http://www.scipy.org/install.html 70 71 Of these, Anaconda has been verified to work with these scripts, and it is 72 available on Mac, Linux, and Windows from here: 73 74 http://continuum.io/downloads 75 76 Note that the Anaconda python executable's directory must be at the front of 77 your PATH environment variable, assuming that you are using this Python 78 distribution. The Anaconda installer may set this up for you automatically. 79 80 Once your Python installation is ready, set up the test environment. 81 82 2.2.1. Linux + Mac OS X 83 ----------------------- 84 85 On Linux or Mac OS X, run the following command (in a terminal) from the 86 pdk/apps/CameraITS directory, from a bash shell: 87 88 source build/envsetup.sh 89 90 This will do some basic sanity checks on your Python installation, and set up 91 the PYTHONPATH environment variable. 92 93 2.2.2. Windows 94 -------------- 95 96 On Windows, the bash script won't run (unless you have cygwin (which has not 97 been tested)), but all you need to do is set your PYTHONPATH environment 98 variable in your shell to point to the pdk/apps/CameraITS/pymodules directory, 99 giving an absolute path. Without this, you'll get "import" errors when running 100 the test scripts. 101 102 3. Python framework overview 103 ---------------------------- 104 105 The Python modules are under the pymodules directory, in the "its" package. 106 107 * its.device: encapsulates communication with ItsService.apk service running 108 on the device 109 * its.objects: contains a collection of functions for creating Python objects 110 corresponding to the Java objects which ItsService.apk uses 111 * its.image: contains a collection of functions (built on numpy arrays) for 112 processing captured images 113 * its.error: the exception/error class used in this framework 114 * its.target: functions to set and measure the exposure level to use for 115 manual shots in tests, to ensure that the images are exposed well for the 116 target scene 117 * its.dng: functions to work with DNG metadata 118 119 All of these module have associated unit tests; to run the unit tests, execute 120 the modules (rather than importing them). 121 122 3.1. Device control 123 ------------------- 124 125 The its.device.ItsSession class encapsulates a session with a connected device 126 under test (which is running ItsService.apk). The session is over TCP, which is 127 forwarded over adb. 128 129 As an overview, the ItsSession.do_capture() function takes a Python dictionary 130 object as an argument, converts that object to JSON, and sends it to the 131 device over tcp which then deserializes from the JSON object representation to 132 Camera2 Java objects (CaptureRequests) which are used to specify one or more 133 captures. Once the captures are complete, the resultant images are copied back 134 to the host machine (over tcp again), along with JSON representations of the 135 CaptureResult and other objects that describe the shot that was actually taken. 136 137 The Python capture request object(s) can contain key/value entries corresponding 138 to any of the Java CaptureRequest object fields. 139 140 The output surface's width, height, and format can also be specified. Currently 141 supported formats are "jpg", "raw", "raw10", "dng", and "yuv", where "yuv" is 142 YUV420 fully planar. The default output surface is a full sensor YUV420 frame. 143 144 The metadata that is returned along with the captured images is also in JSON 145 format, serialized from the CaptureRequest and CaptureResult objects that were 146 passed to the capture listener, as well as the CameraProperties object. 147 148 3.2. Image processing and analysis 149 ---------------------------------- 150 151 The its.image module is a collection of Python functions, built on top of numpy 152 arrays, for manipulating captured images. Some functions of note include: 153 154 load_yuv420_to_rgb_image 155 apply_lut_to_image 156 apply_matrix_to_image 157 write_image 158 159 The scripts in the tests directory make use of these modules. 160 161 Note that it's important to do heavy image processing using the efficient numpy 162 ndarray operations, rather than writing complex loops in standard Python to 163 process pixels. Refer to online docs and examples of numpy for information on 164 this. 165 166 3.3. Tests 167 ---------- 168 169 The tests directory contains a number of self-contained test scripts. All 170 tests should pass if the tree is in a good state. 171 172 Most of the tests save various files in the current directory. To have all the 173 output files put into a separate directory, run the script from that directory, 174 for example: 175 176 mkdir out 177 cd out 178 python ../tests/scene1/test_linearity.py 179 180 Any test can be specified to reboot the camera prior to capturing any shots, by 181 adding a "reboot" or "reboot=N" command line argument, where N is the number of 182 seconds to wait after rebooting the device before sending any commands; the 183 default is 30 seconds. 184 185 python tests/scene1/test_linearity.py reboot 186 python tests/scene1/test_linearity.py reboot=20 187 188 It's possible that a test could leave the camera in a bad state, in particular 189 if there are any bugs in the HAL or the camera framework. Rebooting the device 190 can be used to get it into a known clean state again. 191 192 Each test assumes some sort of target or scene. There are multiple scene<N> 193 folders under the tests directory, and each contains a README file which 194 describes the scene for the scripts in that folder. 195 196 By default, camera device id=0 is opened when the script connects to the unit, 197 however this can be specified by adding a "camera=1" or similar argument to 198 the script command line. On a typical device, camera=0 is the main (rear) 199 camera, and camera=1 is the front-facing camera. 200 201 python tests/scene1/test_linearity.py camera=1 202 203 The tools/run_all_tests.py script should be executed from the top-level 204 CameraITS directory, and it will run all of the tests in an automated fashion, 205 saving the generated output files along with the stdout and stderr dumps to 206 a temporary directory. 207 208 python tools/run_all_tests.py 209 210 This can be run with the "noinit" argument, and in general any args provided 211 to this command line will be passed to each script as it is executed. 212 213 The tests/inprog directory contains a mix of unfinished, in-progress, and 214 incomplete tests. These may or may not be useful in testing a HAL impl., 215 and as these tests are copmleted they will be moved into the scene<N> folders. 216 217 When running individual tests from the command line (as in the examples here), 218 each test run will ensure that the ItsService is running on the device and is 219 ready to accept TCP connections. When using a separate test harness to control 220 this infrastructure, the "noinit" command line argument can be provided to 221 skip this step; in this case, the test will just try to open a socket to the 222 service on the device, and will fail if it's not running and ready. 223 224 python tests/scene1/test_linearity.py noinit 225 226 3.4. Target exposure 227 -------------------- 228 229 The tools/config.py script is a wrapper for the its.target module, which is 230 used to set an exposure level based on the scene that the camera is imaging. 231 The purpose of this is to be able to have tests which use hard-coded manual 232 exposure controls, while at the same time ensuring that the captured images 233 are properly exposed for the test (and aren't clamped to white or black). 234 235 If no argument is provided, the script will use the camera to measure the 236 scene to determine the exposure level. An argument can be provided to hard- 237 code the exposure level. 238 239 python tools/config.py 240 python tools/config.py 16531519962 241 242 This creates a file named its.target.cfg in the current directory, storing the 243 target exposure level. Tests that use the its.target module will be reusing 244 this value, if they are run from the same directory and if they contain the 245 "target" command line argument: 246 247 python tests/scene1/test_linearity.py target 248 249 If the "target" argument isn't present, then the script won't use any cached 250 its.target.cfg values that may be present in the current directory. 251 252 3.5. Docs 253 --------- 254 255 The pydoc tool can generate HTML docs for the ITS Python modules, using the 256 following command (run after PYTHONPATH has been set up as described above): 257 258 pydoc -w its its.device its.image its.error its.objects its.dng its.target 259 260 There is a tutorial script in the tests folder (named tutorial.py). It 261 illustrates a number of the its.image and its.device primitives, and shows 262 how to work with image data in general using this infrastructure. (Its code 263 is commented with explanatory remarks.) 264 265 python tests/tutorial.py 266 267 3.6. List of command line args 268 --------------------------------- 269 270 The above doc sections describe the following command line arguments that may 271 be provided when running a test: 272 273 reboot 274 reboot=N 275 target 276 noinit 277 camera=N 278 279 4. Known issues 280 --------------- 281 282 The Python test scripts don't work if multiple devices are connected to the 283 host machine; currently, the its.device module uses a simplistic "adb -d" 284 approach to communicating with the device, assuming that there is only one 285 device connected. Fixing this is a TODO. 286 287
