1 <html devsite> 2 <head> 3 <title>Implementing the Hardware Composer HAL</title> 4 <meta name="project_path" value="/_project.yaml" /> 5 <meta name="book_path" value="/_book.yaml" /> 6 </head> 7 <body> 8 <!-- 9 Copyright 2017 The Android Open Source Project 10 11 Licensed under the Apache License, Version 2.0 (the "License"); 12 you may not use this file except in compliance with the License. 13 You may obtain a copy of the License at 14 15 http://www.apache.org/licenses/LICENSE-2.0 16 17 Unless required by applicable law or agreed to in writing, software 18 distributed under the License is distributed on an "AS IS" BASIS, 19 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 20 See the License for the specific language governing permissions and 21 limitations under the License. 22 --> 23 24 25 26 27 <p>The Hardware Composer HAL (HWC) is used by SurfaceFlinger to composite 28 surfaces to the screen. The HWC abstracts objects such as overlays and 2D 29 blitters and helps offload some work that would normally be done with OpenGL.</p> 30 31 <p>Android 7.0 includes a new version of HWC (HWC2) used by SurfaceFlinger to 32 talk to specialized window composition hardware. SurfaceFlinger contains a 33 fallback path that uses the 3D graphics processor (GPU) to perform the task of 34 window composition, but this path is not ideal for a couple of reasons:</p> 35 36 <ul> 37 <li>Typically, GPUs are not optimized for this use case and may use more power 38 than necessary to perform composition.</li> 39 <li>Any time SurfaceFlinger is using the GPU for composition is time that 40 applications cannot use the processor for their own rendering, so it is 41 preferable to use specialized hardware for composition instead of the GPU 42 whenever possible.</li> 43 </ul> 44 45 <h2 id="guidance">General guidance</h2> 46 47 <p>As the physical display hardware behind the Hardware Composer abstraction 48 layer can vary from device to device, it's difficult to give recommendations on 49 specific features. In general, use the following guidance:</p> 50 51 <ul> 52 <li>The HWC should support at least four overlays (status bar, system bar, 53 application, and wallpaper/background).</li> 54 <li>Layers can be bigger than the screen, so the HWC should be able to handle 55 layers that are larger than the display (for example, a wallpaper).</li> 56 <li>Pre-multiplied per-pixel alpha blending and per-plane alpha blending 57 should be supported at the same time.</li> 58 <li>The HWC should be able to consume the same buffers the GPU, camera, and 59 video decoder are producing, so supporting some of the following 60 properties is helpful: 61 <ul> 62 <li>RGBA packing order</li> 63 <li>YUV formats</li> 64 <li>Tiling, swizzling, and stride properties</li> 65 </ul> 66 <li>To support protected content, a hardware path for protected video playback 67 must be present.</li> 68 </ul> 69 70 <p>The general recommendation is to implement a non-operational HWC first; after 71 the structure is complete, implement a simple algorithm to delegate composition 72 to the HWC (for example, delegate only the first three or four surfaces to the 73 overlay hardware of the HWC).</p> 74 75 <p>Focus on optimization, such as intelligently selecting the surfaces to send 76 to the overlay hardware that maximizes the load taken off of the GPU. Another 77 optimization is to detect whether the screen is updating; if it isn't, delegate 78 composition to OpenGL instead of the HWC to save power. When the screen updates 79 again, continue to offload composition to the HWC.</p> 80 81 <p>Prepare for common use cases, such as:</p> 82 83 <ul> 84 <li>Full-screen games in portrait and landscape mode</li> 85 <li>Full-screen video with closed captioning and playback control</li> 86 <li>The home screen (compositing the status bar, system bar, application 87 window, and live wallpapers)</li> 88 <li>Protected video playback</li> 89 <li>Multiple display support</li> 90 </ul> 91 92 <p>These use cases should address regular, predictable uses rather than edge 93 cases that are rarely encountered (otherwise, optimizations will have little 94 benefit). Implementations must balance two competing goals: animation smoothness 95 and interaction latency.</p> 96 97 98 <h2 id="interface_activities">HWC2 interface activities</h2> 99 100 <p>HWC2 provides a few primitives (layer, display) to represent composition work 101 and its interaction with the display hardware.</p> 102 <p>A <em>layer</em> is the most important unit of composition; every layer has a 103 set of properties that define how it interacts with other layers. Property 104 categories include the following:</p> 105 106 <ul> 107 <li><strong>Positional</strong>. Defines where the layer appears on its display. 108 Includes information such as the positions of a layer's edges and its <em>Z 109 order</em> relative to other layers (whether it should be in front of or behind 110 other layers).</li> 111 <li><strong>Content</strong>. Defines how content displayed on the layer should 112 be presented within the bounds defined by the positional properties. Includes 113 information such as crop (to expand a portion of the content to fill the bounds 114 of the layer) and transform (to show rotated or flipped content).</li> 115 <li><strong>Composition</strong>. Defines how the layer should be composited 116 with other layers. Includes information such as blending mode and a layer-wide 117 alpha value for 118 <a href="https://en.wikipedia.org/wiki/Alpha_compositing#Alpha_blending">alpha 119 compositing</a>.</li> 120 <li><strong>Optimization</strong>. Provides information not strictly necessary 121 to correctly composite the layer, but which can be used by the HWC device to 122 optimize how it performs composition. Includes information such as the visible 123 region of the layer and which portion of the layer has been updated since the 124 previous frame.</li> 125 </ul> 126 127 <p>A <em>display</em> is another important unit of composition. Every layer can 128 be present only on one display. A system can have multiple displays, and 129 displays can be added or removed during normal system operations. This 130 addition/removal can come at the request of the HWC device (typically in 131 response to an external display being plugged into or removed from the device, 132 called <em>hotplugging</em>), or at the request of the client, which permits the 133 creation of <em>virtual displays</em> whose contents are rendered into an 134 off-screen buffer instead of to a physical display.</p> 135 <p>HWC2 provides functions to determine the properties of a given display, to 136 switch between different configurations (e.g., 4k or 1080p resolution) and color 137 modes (e.g., native color or true sRGB), and to turn the display on, off, or 138 into a low-power mode if supported.</p> 139 <p>In addition to layers and displays, HWC2 also provides control over the 140 hardware vertical sync (VSYNC) signal along with a callback into the client to 141 notify it of when a vsync event has occurred.</p> 142 143 <h3 id="func_pointers">Function pointers</h3> 144 <p>In this section and in HWC2 header comments, HWC interface functions are 145 referred to by lowerCamelCase names that do not actually exist in the interface 146 as named fields. Instead, almost every function is loaded by requesting a 147 function pointer using <code>getFunction</code> provided by 148 <code>hwc2_device_t</code>. For example, the function <code>createLayer</code> 149 is a function pointer of type <code>HWC2_PFN_CREATE_LAYER</code>, which is 150 returned when the enumerated value <code>HWC2_FUNCTION_CREATE_LAYER</code> is 151 passed into <code>getFunction</code>.</p> 152 <p>For detailed documentation on functions (including functions required for 153 every HWC2 implementation), refer to the 154 <a href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/hwcomposer2.h">HWC2 header</a>.</p> 155 156 <h3 id="layer_display_handles">Layer and display handles</h3> 157 <p>Layers and displays are manipulated by opaque handles.</p> 158 <p>When SurfaceFlinger wants to create a new layer, it calls the 159 <code>createLayer</code> function, which then returns an opaque handle of type 160 <code>hwc2_layer_t</code>. From that point on, any time SurfaceFlinger wants to 161 modify a property of that layer, it passes that <code>hwc2_layer_t</code> value 162 into the appropriate modification function, along with any other information 163 needed to make the modification. The <code>hwc2_layer_t</code> type was made 164 large enough to be able to hold either a pointer or an index, and it will be 165 treated as opaque by SurfaceFlinger to provide HWC implementers maximum 166 flexibility.</p> 167 <p>Most of the above also applies to display handles, though handles are created 168 differently depending on whether they are hotplugged (where the handle is passed 169 through the hotplug callback) or requested by the client as a virtual display 170 (where the handle is returned from <code>createVirtualDisplay</code>).</p> 171 172 <h2 id="display_comp_ops">Display composition operations</h2> 173 <p>Once per hardware vsync, SurfaceFlinger wakes if it has new content to 174 composite. This new content could be new image buffers from applications or just 175 a change in the properties of one or more layers. When it wakes, it performs the 176 following steps:</p> 177 178 <ol> 179 <li>Apply transactions, if present. Includes changes in the properties of layers 180 specified by the window manager but not changes in the contents of layers (i.e., 181 graphic buffers from applications).</li> 182 <li>Latch new graphic buffers (acquire their handles from their respective 183 applications), if present.</li> 184 <li>If step 1 or 2 resulted in a change to the display contents, perform a new 185 composition (described below).</li> 186 </ol> 187 188 <p>Steps 1 and 2 have some nuances (such as deferred transactions and 189 presentation timestamps) that are outside the scope of this section. However, 190 step 3 involves the HWC interface and is detailed below.</p> 191 <p>At the beginning of the composition process, SurfaceFlinger will create and 192 destroy layers or modify layer state as applicable. It will also update the 193 layers with their current contents, using calls such as 194 <code>setLayerBuffer</code> or <code>setLayerColor</code>. After all layers have 195 been updated, it will call <code>validateDisplay</code>, which tells the device 196 to examine the state of the various layers and determine how composition will 197 proceed. By default, SurfaceFlinger usually attempts to configure every layer 198 such that it will be composited by the device, though there may be some 199 circumstances where it will mandate that it be composited by the client.</p> 200 <p>After the call to <code>validateDisplay</code>, SurfaceFlinger will follow up 201 with a call to <code>getChangedCompositionTypes</code> to see if the device 202 wants any of the layers' composition types changed before performing the actual 203 composition. SurfaceFlinger may choose to:</p> 204 205 <ul> 206 <li>Change some of the layer composition types and re-validate the display.</li> 207 </ul> 208 209 <em><strong>OR</strong></em> 210 211 <ul> 212 <li>Call <code>acceptDisplayChanges</code>, which has the same effect as 213 changing the composition types as requested by the device and re-validating 214 without actually having to call <code>validateDisplay</code> again.</li> 215 </ul> 216 217 <p>In practice, SurfaceFlinger always takes the latter path (calling 218 <code>acceptDisplayChanges</code>) though this behavior may change in the 219 future.</p> 220 <p>At this point, the behavior differs depending on whether any of the layers 221 have been marked for client composition. If any (or all) layers have been marked 222 for client composition, SurfaceFlinger will now composite all of those layers 223 into the client target buffer. This buffer will be provided to the device using 224 the <code>setClientTarget</code> call so that it may be either displayed 225 directly on the screen or further composited with layers that have not been 226 marked for client composition. If no layers have been marked for client 227 composition, then the client composition step is bypassed.</p> 228 <p>Finally, after all of the state has been validated and client composition has 229 been performed if needed, SurfaceFlinger will call <code>presentDisplay</code>. 230 This is the HWC device's cue to complete the composition process and display the 231 final result.</p> 232 233 <h2 id="multiple_displays">Multiple displays in Android 7.0</h2> 234 <p>While the HWC2 interface is quite flexible when it comes to the number of 235 displays in the system, the rest of the Android framework is not yet as 236 flexible. When designing a HWC2 implementation intended for use on Android 7.0, 237 there are some additional restrictions not present in the HWC definition itself: 238 </p> 239 240 <ul> 241 <li>It is assumed that there is exactly one <em>primary</em> display; that is, 242 that there is one physical display that will be hotplugged immediately during 243 the initialization of the device (specifically after the hotplug callback is 244 registered).</li> 245 <li>In addition to the primary display, exactly one <em>external</em> display 246 may be hotplugged during normal operation of the device.</li> 247 </ul> 248 249 <p>While the SurfaceFlinger operations described above are performed per-display 250 (eventual goal is to be able to composite displays independently of each other), 251 they are currently performed sequentially for all active displays, even if only 252 the contents of one display are updated.</p> 253 <p>For example, if only the external display is updated, the sequence is:</p> 254 255 <pre class="devsite-click-to-copy"> 256 // Update state for internal display 257 // Update state for external display 258 validateDisplay(<internal display>) 259 validateDisplay(<external display>) 260 presentDisplay(<internal display>) 261 presentDisplay(<external display>) 262 </pre> 263 264 265 <h2 id="sync_fences">Synchronization fences</h2> 266 <p>Synchronization (sync) fences are a crucial aspect of the Android graphics 267 system. Fences allow CPU work to proceed independently from concurrent GPU work, 268 blocking only when there is a true dependency.</p> 269 <p>For example, when an application submits a buffer that is being produced on 270 the GPU, it will also submit a fence object; this fence signals only when the 271 GPU has finished writing into the buffer. Since the only part of the system that 272 truly needs the GPU write to have finished is the display hardware (the hardware 273 abstracted by the HWC HAL), the graphics pipeline is able to pass this fence 274 along with the buffer through SurfaceFlinger to the HWC device. Only immediately 275 before that buffer would be displayed does the device need to actually check 276 that the fence has signaled.</p> 277 <p>Sync fences are integrated tightly into HWC2 and organized in the following 278 categories:</p> 279 280 <ol> 281 <li>Acquire fences are passed along with input buffers to the 282 <code>setLayerBuffer</code> and <code>setClientTarget</code> calls. These 283 represent a pending write into the buffer and must signal before the HWC client 284 or device attempts to read from the associated buffer to perform composition. 285 </li> 286 <li>Release fences are retrieved after the call to <code>presentDisplay</code> 287 using the <code>getReleaseFences</code> call and are passed back to the 288 application along with buffers that will be replaced during the next 289 composition. These represent a pending read from the buffer, and must signal 290 before the application attempts to write new contents into the buffer.</li> 291 <li>Retire fences are returned, one per frame, as part of the call to 292 <code>presentDisplay</code> and represent when the composition of this frame 293 has completed, or alternately, when the composition result of the prior frame is 294 no longer needed. For physical displays, this is when the current frame appears 295 on the screen and can also be interpreted as the time after which it is safe to 296 write to the client target buffer again (if applicable). For virtual displays, 297 this is the time when it is safe to read from the output buffer.</li> 298 </ol> 299 300 <h3 id="hwc2_changes">Changes in HWC2</h3> 301 <p>The meaning of sync fences in HWC 2.0 has changed significantly relative to 302 previous versions of the HAL.</p> 303 <p>In HWC v1.x, the release and retire fences were speculative. A release fence 304 for a buffer or a retire fence for the display retrieved in frame N would not 305 signal any sooner than frame N + 1. In other words, the meaning of the fence 306 was "the content of the buffer you provided for frame N is no longer needed." 307 This is speculative because in theory SurfaceFlinger may not run again after 308 frame N for an indeterminate period of time, which would leave those fences 309 unsignaled for the same period.</p> 310 <p>In HWC 2.0, release and retire fences are non-speculative. A release or 311 retire fence retrieved in frame N will signal as soon as the content of the 312 associated buffers replaces the contents of the buffers from frame N - 1, or in 313 other words, the meaning of the fence is "the content of the buffer you provided 314 for frame N has now replaced the previous content." This is non-speculative, 315 since this fence should signal shortly after <code>presentDisplay</code> is 316 called as soon as the hardware presents this frame's content.</p> 317 <p>For implementation details, refer to the 318 <a href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/hwcomposer2.h">HWC2 header</a>.</p> 319 320 </body> 321 </html> 322
