1 <html devsite> 2 <head> 3 <title>HAL subsystem</title> 4 <meta name="project_path" value="/_project.yaml" /> 5 <meta name="book_path" value="/_book.yaml" /> 6 </head> 7 <body> 8 <!-- 9 Copyright 2017 The Android Open Source Project 10 11 Licensed under the Apache License, Version 2.0 (the "License"); 12 you may not use this file except in compliance with the License. 13 You may obtain a copy of the License at 14 15 http://www.apache.org/licenses/LICENSE-2.0 16 17 Unless required by applicable law or agreed to in writing, software 18 distributed under the License is distributed on an "AS IS" BASIS, 19 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 20 See the License for the specific language governing permissions and 21 limitations under the License. 22 --> 23 24 25 26 <h2 id="requests">Requests</h2> 27 <p> The app framework issues requests for captured results to the camera subsystem. 28 One request corresponds to one set of results. A request encapsulates all 29 configuration information about the capturing and processing of those results. 30 This includes things such as resolution and pixel format; manual sensor, lens, 31 and flash control; 3A operating modes; RAW to YUV processing control; and 32 statistics generation. This allows for much more control over the results' 33 output and processing. Multiple requests can be in flight at once, and 34 submitting requests is non-blocking. And the requests are always processed in 35 the order they are received.<br/> 36 <img src="images/camera_model.png" alt="Camera request model" id="figure1" /> 37 <p class="img-caption"> 38 <strong>Figure 1.</strong> Camera model 39 </p> 40 <h2 id="hal-subsystem">The HAL and camera subsystem</h2> 41 <p> The camera subsystem includes the implementations for components in the camera 42 pipeline such as the 3A algorithm and processing controls. The camera HAL 43 provides interfaces for you to implement your versions of these components. To 44 maintain cross-platform compatibility between multiple device manufacturers and 45 Image Signal Processor (ISP, or camera sensor) vendors, the camera pipeline 46 model is virtual and does not directly correspond to any real ISP. However, it 47 is similar enough to real processing pipelines so that you can map it to your 48 hardware efficiently. In addition, it is abstract enough to allow for multiple 49 different algorithms and orders of operation without compromising either 50 quality, efficiency, or cross-device compatibility.<br/> 51 The camera pipeline also supports triggers that the app framework can initiate 52 to turn on things such as auto-focus. It also sends notifications back to the 53 app framework, notifying apps of events such as an auto-focus lock or errors.<br/> 54 <img src="images/camera_hal.png" alt="Camera hardware abstraction layer" id="figure2" /> 55 <p class="img-caption"> 56 <strong>Figure 2.</strong> Camera pipeline 57 </p> 58 Please note, some image processing blocks shown in the diagram above are not 59 well-defined in the initial release.<br/> 60 The camera pipeline makes the following assumptions:</p> 61 <ul> 62 <li>RAW Bayer output undergoes no processing inside the ISP.</li> 63 <li>Statistics are generated based off the raw sensor data.</li> 64 <li>The various processing blocks that convert raw sensor data to YUV are in an 65 arbitrary order.</li> 66 <li>While multiple scale and crop units are shown, all scaler units share the 67 output region controls (digital zoom). However, each unit may have a different 68 output resolution and pixel format.</li> 69 </ul> 70 <p><strong>Summary of API use</strong><br/> 71 This is a brief summary of the steps for using the Android camera API. See the 72 Startup and expected operation sequence section for a detailed breakdown of 73 these steps, including API calls.</p> 74 <ol> 75 <li>Listen for and enumerate camera devices.</li> 76 <li>Open device and connect listeners.</li> 77 <li>Configure outputs for target use case (such as still capture, recording, 78 etc.).</li> 79 <li>Create request(s) for target use case.</li> 80 <li>Capture/repeat requests and bursts.</li> 81 <li>Receive result metadata and image data.</li> 82 <li>When switching use cases, return to step 3.</li> 83 </ol> 84 <p><strong>HAL operation summary</strong></p> 85 <ul> 86 <li>Asynchronous requests for captures come from the framework.</li> 87 <li>HAL device must process requests in order. And for each request, produce 88 output result metadata, and one or more output image buffers.</li> 89 <li>First-in, first-out for requests and results, and for streams referenced by 90 subsequent requests. </li> 91 <li>Timestamps must be identical for all outputs from a given request, so that the 92 framework can match them together if needed. </li> 93 <li>All capture configuration and state (except for the 3A routines) is 94 encapsulated in the requests and results.</li> 95 </ul> 96 <img src="images/camera-hal-overview.png" alt="Camera HAL overview" id="figure3" /> 97 <p class="img-caption"> 98 <strong>Figure 3.</strong> Camera HAL overview 99 </p> 100 <h2 id="startup">Startup and expected operation sequence</h2> 101 <p>This section contains a detailed explanation of the steps expected when using 102 the camera API. Please see <a href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera3.h">platform/hardware/libhardware/include/hardware/camera3.h</a> for definitions of these structures and methods.</p> 103 <ol> 104 <li>Framework calls camera_module_t->common.open(), which returns a 105 hardware_device_t structure.</li> 106 <li>Framework inspects the hardware_device_t->version field, and instantiates the 107 appropriate handler for that version of the camera hardware device. In case 108 the version is CAMERA_DEVICE_API_VERSION_3_0, the device is cast to a 109 camera3_device_t.</li> 110 <li>Framework calls camera3_device_t->ops->initialize() with the framework 111 callback function pointers. This will only be called this one time after 112 open(), before any other functions in the ops structure are called.</li> 113 <li>The framework calls camera3_device_t->ops->configure_streams() with a list of 114 input/output streams to the HAL device.</li> 115 <li>The framework allocates gralloc buffers and calls 116 camera3_device_t->ops->register_stream_buffers() for at least one of the 117 output streams listed in configure_streams. The same stream is registered 118 only once.</li> 119 <li>The framework requests default settings for some number of use cases with 120 calls to camera3_device_t->ops->construct_default_request_settings(). This 121 may occur any time after step 3.</li> 122 <li>The framework constructs and sends the first capture request to the HAL with 123 settings based on one of the sets of default settings, and with at least one 124 output stream that has been registered earlier by the framework. This is sent 125 to the HAL with camera3_device_t->ops->process_capture_request(). The HAL 126 must block the return of this call until it is ready for the next request to 127 be sent.</li> 128 <li>The framework continues to submit requests, and possibly call 129 register_stream_buffers() for not-yet-registered streams, and call 130 construct_default_request_settings to get default settings buffers for other 131 use cases.</li> 132 <li>When the capture of a request begins (sensor starts exposing for the 133 capture), the HAL calls camera3_callback_ops_t->notify() with the SHUTTER 134 event, including the frame number and the timestamp for start of exposure. 135 This notify call must be made before the first call to 136 process_capture_result() for that frame number.</li> 137 <li>After some pipeline delay, the HAL begins to return completed captures to 138 the framework with camera3_callback_ops_t->process_capture_result(). These 139 are returned in the same order as the requests were submitted. Multiple 140 requests can be in flight at once, depending on the pipeline depth of the 141 camera HAL device.</li> 142 <li>After some time, the framework may stop submitting new requests, wait for 143 the existing captures to complete (all buffers filled, all results 144 returned), and then call configure_streams() again. This resets the camera 145 hardware and pipeline for a new set of input/output streams. Some streams 146 may be reused from the previous configuration; if these streams' buffers had 147 already been registered with the HAL, they will not be registered again. The 148 framework then continues from step 7, if at least one registered output 149 stream remains. (Otherwise, step 5 is required first.)</li> 150 <li>Alternatively, the framework may call camera3_device_t->common->close() to 151 end the camera session. This may be called at any time when no other calls 152 from the framework are active, although the call may block until all 153 in-flight captures have completed (all results returned, all buffers 154 filled). After the close call returns, no more calls to the 155 camera3_callback_ops_t functions are allowed from the HAL. Once the close() 156 call is underway, the framework may not call any other HAL device functions.</li> 157 <li>In case of an error or other asynchronous event, the HAL must call 158 camera3_callback_ops_t->notify() with the appropriate error/event message. 159 After returning from a fatal device-wide error notification, the HAL should 160 act as if close() had been called on it. However, the HAL must either cancel 161 or complete all outstanding captures before calling notify(), so that once 162 notify() is called with a fatal error, the framework will not receive 163 further callbacks from the device. Methods besides close() should return 164 -ENODEV or NULL after the notify() method returns from a fatal error 165 message.</li> 166 </ol> 167 <img src="images/camera-ops-flow.png" width="600" height="434" alt="Camera operations flow" id="figure4" /> 168 <p class="img-caption"> 169 <strong>Figure 4.</strong> Camera operational flow 170 </p> 171 <h2 id="ops-modes">Operational modes</h2> 172 <p>The camera 3 HAL device can implement one of two possible operational modes: 173 limited and full. Full support is expected from new higher-end devices. Limited 174 mode has hardware requirements roughly in line with those for a camera HAL 175 device v1 implementation, and is expected from older or inexpensive devices. 176 Full is a strict superset of limited, and they share the same essential 177 operational flow, as documented above.</p> 178 <p>The HAL must indicate its level of support with the 179 android.info.supportedHardwareLevel static metadata entry, with 0 indicating 180 limited mode, and 1 indicating full mode support.</p> 181 <p>Roughly speaking, limited-mode devices do not allow for application control of 182 capture settings (3A control only), high-rate capture of high-resolution images, 183 raw sensor readout, or support for YUV output streams above maximum recording 184 resolution (JPEG only for large images).<br/> 185 Here are the details of limited-mode behavior:</p> 186 <ul> 187 <li>Limited-mode devices do not need to implement accurate synchronization between 188 capture request settings and the actual image data captured. Instead, changes 189 to settings may take effect some time in the future, and possibly not for the 190 same output frame for each settings entry. Rapid changes in settings may 191 result in some settings never being used for a capture. However, captures that 192 include high-resolution output buffers ( > 1080p ) have to use the settings as 193 specified (but see below for processing rate).</li> 194 <li>Captures in limited mode that include high-resolution (> 1080p) output buffers 195 may block in process_capture_request() until all the output buffers have been 196 filled. A full-mode HAL device must process sequences of high-resolution 197 requests at the rate indicated in the static metadata for that pixel format. 198 The HAL must still call process_capture_result() to provide the output; the 199 framework must simply be prepared for process_capture_request() to block until 200 after process_capture_result() for that request completes for high-resolution 201 captures for limited-mode devices.</li> 202 <li>Limited-mode devices do not need to support most of the settings/result/static 203 info metadata. Only the following settings are expected to be consumed or 204 produced by a limited-mode HAL device: 205 <ul> 206 <li>android.control.aeAntibandingMode (controls)</li> 207 <li>android.control.aeExposureCompensation (controls)</li> 208 <li>android.control.aeLock (controls)</li> 209 <li>android.control.aeMode (controls)</li> 210 <li>[OFF means ON_FLASH_TORCH]</li> 211 <li>android.control.aeRegions (controls)</li> 212 <li>android.control.aeTargetFpsRange (controls)</li> 213 <li>android.control.afMode (controls)</li> 214 <li>[OFF means infinity focus]</li> 215 <li>android.control.afRegions (controls)</li> 216 <li>android.control.awbLock (controls)</li> 217 <li>android.control.awbMode (controls)</li> 218 <li>[OFF not supported]</li> 219 <li>android.control.awbRegions (controls)</li> 220 <li>android.control.captureIntent (controls)</li> 221 <li>android.control.effectMode (controls)</li> 222 <li>android.control.mode (controls)</li> 223 <li>[OFF not supported]</li> 224 <li>android.control.sceneMode (controls)</li> 225 <li>android.control.videoStabilizationMode (controls)</li> 226 <li>android.control.aeAvailableAntibandingModes (static)</li> 227 <li>android.control.aeAvailableModes (static)</li> 228 <li>android.control.aeAvailableTargetFpsRanges (static)</li> 229 <li>android.control.aeCompensationRange (static)</li> 230 <li>android.control.aeCompensationStep (static)</li> 231 <li>android.control.afAvailableModes (static)</li> 232 <li>android.control.availableEffects (static)</li> 233 <li>android.control.availableSceneModes (static)</li> 234 <li>android.control.availableVideoStabilizationModes (static)</li> 235 <li>android.control.awbAvailableModes (static)</li> 236 <li>android.control.maxRegions (static)</li> 237 <li>android.control.sceneModeOverrides (static)</li> 238 <li>android.control.aeRegions (dynamic)</li> 239 <li>android.control.aeState (dynamic)</li> 240 <li>android.control.afMode (dynamic)</li> 241 <li>android.control.afRegions (dynamic)</li> 242 <li>android.control.afState (dynamic)</li> 243 <li>android.control.awbMode (dynamic)</li> 244 <li>android.control.awbRegions (dynamic)</li> 245 <li>android.control.awbState (dynamic)</li> 246 <li>android.control.mode (dynamic)</li> 247 <li>android.flash.info.available (static)</li> 248 <li>android.info.supportedHardwareLevel (static)</li> 249 <li>android.jpeg.gpsCoordinates (controls)</li> 250 <li>android.jpeg.gpsProcessingMethod (controls)</li> 251 <li>android.jpeg.gpsTimestamp (controls)</li> 252 <li>android.jpeg.orientation (controls)</li> 253 <li>android.jpeg.quality (controls)</li> 254 <li>android.jpeg.thumbnailQuality (controls)</li> 255 <li>android.jpeg.thumbnailSize (controls)</li> 256 <li>android.jpeg.availableThumbnailSizes (static)</li> 257 <li>android.jpeg.maxSize (static)</li> 258 <li>android.jpeg.gpsCoordinates (dynamic)</li> 259 <li>android.jpeg.gpsProcessingMethod (dynamic)</li> 260 <li>android.jpeg.gpsTimestamp (dynamic)</li> 261 <li>android.jpeg.orientation (dynamic)</li> 262 <li>android.jpeg.quality (dynamic)</li> 263 <li>android.jpeg.size (dynamic)</li> 264 <li>android.jpeg.thumbnailQuality (dynamic)</li> 265 <li>android.jpeg.thumbnailSize (dynamic)</li> 266 <li>android.lens.info.minimumFocusDistance (static)</li> 267 <li>android.request.id (controls)</li> 268 <li>android.request.id (dynamic)</li> 269 <li>android.scaler.cropRegion (controls)</li> 270 <li>[ignores (x,y), assumes center-zoom]</li> 271 <li>android.scaler.availableFormats (static)</li> 272 <li>[RAW not supported]</li> 273 <li>android.scaler.availableJpegMinDurations (static)</li> 274 <li>android.scaler.availableJpegSizes (static)</li> 275 <li>android.scaler.availableMaxDigitalZoom (static)</li> 276 <li>android.scaler.availableProcessedMinDurations (static)</li> 277 <li>android.scaler.availableProcessedSizes (static)</li> 278 <li>[full resolution not supported]</li> 279 <li>android.scaler.maxDigitalZoom (static)</li> 280 <li>android.scaler.cropRegion (dynamic)</li> 281 <li>android.sensor.orientation (static)</li> 282 <li>android.sensor.timestamp (dynamic)</li> 283 <li>android.statistics.faceDetectMode (controls)</li> 284 <li>android.statistics.info.availableFaceDetectModes (static)</li> 285 <li>android.statistics.faceDetectMode (dynamic)</li> 286 <li>android.statistics.faceIds (dynamic)</li> 287 <li>android.statistics.faceLandmarks (dynamic)</li> 288 <li>android.statistics.faceRectangles (dynamic)</li> 289 <li>android.statistics.faceScores (dynamic)</li> 290 </ul> 291 </li> 292 </ul> 293 <h2 id="interaction">Interaction between the application capture request, 3A 294 control, and the processing pipeline</h2> 295 <p>Depending on the settings in the 3A control block, the camera pipeline ignores 296 some of the parameters in the application's capture request and uses the values 297 provided by the 3A control routines instead. For example, when auto-exposure is 298 active, the exposure time, frame duration, and sensitivity parameters of the 299 sensor are controlled by the platform 3A algorithm, and any app-specified values 300 are ignored. The values chosen for the frame by the 3A routines must be reported 301 in the output metadata. The following table describes the different modes of the 302 3A control block and the properties that are controlled by these modes. See 303 the <a href="https://android.googlesource.com/platform/system/media/+/master/camera/docs/docs.html">platform/system/media/camera/docs/docs.html</a> file for definitions of these properties.</p> 304 <table> 305 <tr> 306 <th>Parameter</th> 307 <th>State</th> 308 <th>Properties controlled</th> 309 </tr> 310 <tr> 311 <td>android.control.aeMode</td> 312 <td>OFF</td> 313 <td>None</td> 314 </tr> 315 <tr> 316 <td></td> 317 <td>ON</td> 318 <td>android.sensor.exposureTime 319 android.sensor.frameDuration 320 android.sensor.sensitivity 321 android.lens.aperture (if supported) 322 android.lens.filterDensity (if supported)</td> 323 </tr> 324 <tr> 325 <td></td> 326 <td>ON_AUTO_FLASH</td> 327 <td>Everything is ON, plus android.flash.firingPower, android.flash.firingTime, and android.flash.mode</td> 328 </tr> 329 <tr> 330 <td></td> 331 <td>ON_ALWAYS_FLASH</td> 332 <td>Same as ON_AUTO_FLASH</td> 333 </tr> 334 <tr> 335 <td></td> 336 <td>ON_AUTO_FLASH_RED_EYE</td> 337 <td>Same as ON_AUTO_FLASH</td> 338 </tr> 339 <tr> 340 <td>android.control.awbMode</td> 341 <td>OFF</td> 342 <td>None</td> 343 </tr> 344 <tr> 345 <td></td> 346 <td>WHITE_BALANCE_*</td> 347 <td>android.colorCorrection.transform. Platform-specific adjustments if android.colorCorrection.mode is FAST or HIGH_QUALITY.</td> 348 </tr> 349 <tr> 350 <td>android.control.afMode</td> 351 <td>OFF</td> 352 <td>None</td> 353 </tr> 354 <tr> 355 <td></td> 356 <td>FOCUS_MODE_*</td> 357 <td>android.lens.focusDistance</td> 358 </tr> 359 <tr> 360 <td>android.control.videoStabilization</td> 361 <td>OFF</td> 362 <td>None</td> 363 </tr> 364 <tr> 365 <td></td> 366 <td>ON</td> 367 <td>Can adjust android.scaler.cropRegion to implement video stabilization</td> 368 </tr> 369 <tr> 370 <td>android.control.mode</td> 371 <td>OFF</td> 372 <td>AE, AWB, and AF are disabled</td> 373 </tr> 374 <tr> 375 <td></td> 376 <td>AUTO</td> 377 <td>Individual AE, AWB, and AF settings are used</td> 378 </tr> 379 <tr> 380 <td></td> 381 <td>SCENE_MODE_*</td> 382 <td>Can override all parameters listed above. Individual 3A controls are disabled.</td> 383 </tr> 384 </table> 385 <p>The controls exposed for the 3A algorithm mostly map 1:1 to the old API's 386 parameters (such as exposure compensation, scene mode, or white balance mode).<br/> 387 The controls in the Image Processing block in Figure 2</a> all 388 operate on a similar principle, and generally each block has three modes:</p> 389 <ul> 390 <li>OFF: This processing block is disabled. The demosaic, color correction, and 391 tone curve adjustment blocks cannot be disabled.</li> 392 <li>FAST: In this mode, the processing block may not slow down the output frame 393 rate compared to OFF mode, but should otherwise produce the best-quality 394 output it can given that restriction. Typically, this would be used for 395 preview or video recording modes, or burst capture for still images. On some 396 devices, this may be equivalent to OFF mode (no processing can be done without 397 slowing down the frame rate), and on some devices, this may be equivalent to 398 HIGH_QUALITY mode (best quality still does not slow down frame rate).</li> 399 <li>HIGHQUALITY: In this mode, the processing block should produce the best 400 quality result possible, slowing down the output frame rate as needed. 401 Typically, this would be used for high-quality still capture. Some blocks 402 include a manual control which can be optionally selected instead of FAST or 403 HIGHQUALITY. For example, the color correction block supports a color 404 transform matrix, while the tone curve adjustment supports an arbitrary global 405 tone mapping curve.</li> 406 </ul> 407 <p>The maximum frame rate that can be supported by a camera subsystem is a function 408 of many factors:</p> 409 <ul> 410 <li>Requested resolutions of output image streams</li> 411 <li>Availability of binning / skipping modes on the imager</li> 412 <li>The bandwidth of the imager interface</li> 413 <li>The bandwidth of the various ISP processing blocks</li> 414 </ul> 415 <p>Since these factors can vary greatly between different ISPs and sensors, the 416 camera HAL interface tries to abstract the bandwidth restrictions into as simple 417 model as possible. The model presented has the following characteristics:</p> 418 <ul> 419 <li>The image sensor is always configured to output the smallest resolution 420 possible given the application's requested output stream sizes. The smallest 421 resolution is defined as being at least as large as the largest requested 422 output stream size.</li> 423 <li>Since any request may use any or all the currently configured output streams, 424 the sensor and ISP must be configured to support scaling a single capture to 425 all the streams at the same time. </li> 426 <li>JPEG streams act like processed YUV streams for requests for which they are 427 not included; in requests in which they are directly referenced, they act as 428 JPEG streams.</li> 429 <li>The JPEG processor can run concurrently to the rest of the camera pipeline but 430 cannot process more than one capture at a time.</li> 431 </ul> 432 433 </body> 434 </html> 435
