1 page.title=TextureView 2 @jd:body 3 4 <!-- 5 Copyright 2014 The Android Open Source Project 6 7 Licensed under the Apache License, Version 2.0 (the "License"); 8 you may not use this file except in compliance with the License. 9 You may obtain a copy of the License at 10 11 http://www.apache.org/licenses/LICENSE-2.0 12 13 Unless required by applicable law or agreed to in writing, software 14 distributed under the License is distributed on an "AS IS" BASIS, 15 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 See the License for the specific language governing permissions and 17 limitations under the License. 18 --> 19 <div id="qv-wrapper"> 20 <div id="qv"> 21 <h2>In this document</h2> 22 <ol id="auto-toc"> 23 </ol> 24 </div> 25 </div> 26 27 28 <p>The TextureView class introduced in Android 4.0 and is the most complex of 29 the View objects discussed here, combining a View with a SurfaceTexture.</p> 30 31 <h2 id=render_gles>Rendering with GLES</h2> 32 <p>Recall that the SurfaceTexture is a "GL consumer", consuming buffers of graphics 33 data and making them available as textures. TextureView wraps a SurfaceTexture, 34 taking over the responsibility of responding to the callbacks and acquiring new 35 buffers. The arrival of new buffers causes TextureView to issue a View 36 invalidate request. When asked to draw, the TextureView uses the contents of 37 the most recently received buffer as its data source, rendering wherever and 38 however the View state indicates it should.</p> 39 40 <p>You can render on a TextureView with GLES just as you would SurfaceView. Just 41 pass the SurfaceTexture to the EGL window creation call. However, doing so 42 exposes a potential problem.</p> 43 44 <p>In most of what we've looked at, the BufferQueues have passed buffers between 45 different processes. When rendering to a TextureView with GLES, both producer 46 and consumer are in the same process, and they might even be handled on a single 47 thread. Suppose we submit several buffers in quick succession from the UI 48 thread. The EGL buffer swap call will need to dequeue a buffer from the 49 BufferQueue, and it will stall until one is available. There won't be any 50 available until the consumer acquires one for rendering, but that also happens 51 on the UI thread so we're stuck.</p> 52 53 <p>The solution is to have BufferQueue ensure there is always a buffer 54 available to be dequeued, so the buffer swap never stalls. One way to guarantee 55 this is to have BufferQueue discard the contents of the previously-queued buffer 56 when a new buffer is queued, and to place restrictions on minimum buffer counts 57 and maximum acquired buffer counts. (If your queue has three buffers, and all 58 three buffers are acquired by the consumer, then there's nothing to dequeue and 59 the buffer swap call must hang or fail. So we need to prevent the consumer from 60 acquiring more than two buffers at once.) Dropping buffers is usually 61 undesirable, so it's only enabled in specific situations, such as when the 62 producer and consumer are in the same process.</p> 63 64 <h2 id=surface_or_texture>SurfaceView or TextureView?</h2> 65 SurfaceView and TextureView fill similar roles, but have very different 66 implementations. To decide which is best requires an understanding of the 67 trade-offs.</p> 68 69 <p>Because TextureView is a proper citizen of the View hierarchy, it behaves like 70 any other View, and can overlap or be overlapped by other elements. You can 71 perform arbitrary transformations and retrieve the contents as a bitmap with 72 simple API calls.</p> 73 74 <p>The main strike against TextureView is the performance of the composition step. 75 With SurfaceView, the content is written to a separate layer that SurfaceFlinger 76 composites, ideally with an overlay. With TextureView, the View composition is 77 always performed with GLES, and updates to its contents may cause other View 78 elements to redraw as well (e.g. if they're positioned on top of the 79 TextureView). After the View rendering completes, the app UI layer must then be 80 composited with other layers by SurfaceFlinger, so you're effectively 81 compositing every visible pixel twice. For a full-screen video player, or any 82 other application that is effectively just UI elements layered on top of video, 83 SurfaceView offers much better performance.</p> 84 85 <p>As noted earlier, DRM-protected video can be presented only on an overlay plane. 86 Video players that support protected content must be implemented with 87 SurfaceView.</p> 88 89 <h2 id=grafika>Case Study: Grafika's Play Video (TextureView)</h2> 90 91 <p>Grafika includes a pair of video players, one implemented with TextureView, the 92 other with SurfaceView. The video decoding portion, which just sends frames 93 from MediaCodec to a Surface, is the same for both. The most interesting 94 differences between the implementations are the steps required to present the 95 correct aspect ratio.</p> 96 97 <p>While SurfaceView requires a custom implementation of FrameLayout, resizing 98 SurfaceTexture is a simple matter of configuring a transformation matrix with 99 <code>TextureView#setTransform()</code>. For the former, you're sending new 100 window position and size values to SurfaceFlinger through WindowManager; for 101 the latter, you're just rendering it differently.</p> 102 103 <p>Otherwise, both implementations follow the same pattern. Once the Surface has 104 been created, playback is enabled. When "play" is hit, a video decoding thread 105 is started, with the Surface as the output target. After that, the app code 106 doesn't have to do anything -- composition and display will either be handled by 107 SurfaceFlinger (for the SurfaceView) or by TextureView.</p> 108 109 <h2 id=decode>Case Study: Grafika's Double Decode</h2> 110 111 <p>This activity demonstrates manipulation of the SurfaceTexture inside a 112 TextureView.</p> 113 114 <p>The basic structure of this activity is a pair of TextureViews that show two 115 different videos playing side-by-side. To simulate the needs of a 116 videoconferencing app, we want to keep the MediaCodec decoders alive when the 117 activity is paused and resumed for an orientation change. The trick is that you 118 can't change the Surface that a MediaCodec decoder uses without fully 119 reconfiguring it, which is a fairly expensive operation; so we want to keep the 120 Surface alive. The Surface is just a handle to the producer interface in the 121 SurfaceTexture's BufferQueue, and the SurfaceTexture is managed by the 122 TextureView;, so we also need to keep the SurfaceTexture alive. So how do we deal 123 with the TextureView getting torn down?</p> 124 125 <p>It just so happens TextureView provides a <code>setSurfaceTexture()</code> call 126 that does exactly what we want. We obtain references to the SurfaceTextures 127 from the TextureViews and save them in a static field. When the activity is 128 shut down, we return "false" from the <code>onSurfaceTextureDestroyed()</code> 129 callback to prevent destruction of the SurfaceTexture. When the activity is 130 restarted, we stuff the old SurfaceTexture into the new TextureView. The 131 TextureView class takes care of creating and destroying the EGL contexts.</p> 132 133 <p>Each video decoder is driven from a separate thread. At first glance it might 134 seem like we need EGL contexts local to each thread; but remember the buffers 135 with decoded output are actually being sent from mediaserver to our 136 BufferQueue consumers (the SurfaceTextures). The TextureViews take care of the 137 rendering for us, and they execute on the UI thread.</p> 138 139 <p>Implementing this activity with SurfaceView would be a bit harder. We can't 140 just create a pair of SurfaceViews and direct the output to them, because the 141 Surfaces would be destroyed during an orientation change. Besides, that would 142 add two layers, and limitations on the number of available overlays strongly 143 motivate us to keep the number of layers to a minimum. Instead, we'd want to 144 create a pair of SurfaceTextures to receive the output from the video decoders, 145 and then perform the rendering in the app, using GLES to render two textured 146 quads onto the SurfaceView's Surface.</p> 147
