1 Android Hardware OpenGLES emulation design overview 2 =================================================== 3 4 Introduction: 5 ------------- 6 7 Hardware GLES emulation in the Android platform is implemented with a mix 8 of components, which are: 9 10 - Several host "translator" libraries. They implement the EGL, GLES 1.1 and 11 GLES 2.0 ABIs defined by Khronos, and translate the corresponding function 12 calls into calls to the appropriate desktop APIs, i.e.: 13 14 - Xgl (Linux), AGL (OS X) or WGL (Windows) for EGL 15 - desktop GL 2.0 for GLES 1.1 and GLES 2.0 16 17 _________ __________ __________ 18 | | | | | | HOST 19 |TRANSLATOR |TRANSLATOR| |TRANSLATOR| HOST 20 | EGL | | GLES 1.1 | | GLES 2.0 | TRANSLATOR 21 |_________| |__________| |__________| LIBRARIES 22 | | | 23 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 24 | | | 25 ____v____ ____v_____ _____v____ HOST 26 | | | | | | SYSTEM 27 | Xgl | | GL 2.0 | | GL 2.0 | LIBRARIES 28 |_________| |__________| |__________| 29 30 31 32 - Several system libraries inside the emulated guest system that implement 33 the same EGL / GLES 1.1 and GLES 2.0 ABIs. 34 35 They collect the sequence of EGL/GLES function calls and translate then 36 into a custom wire protocol stream that is sent to the emulator program 37 through a high-speed communication channel called a "QEMU Pipe". 38 39 For now, all you need to know is that the pipe is implemented with a 40 custom kernel driver, and provides for _very_ fast bandwidth. All read() 41 and writes() from/to the pipes are essentially instantaneous from the 42 guest's point of view. 43 44 45 _________ __________ __________ 46 | | | | | | 47 |EMULATION| |EMULATION | |EMULATION | GUEST 48 | EGL | | GLES 1.1 | | GLES 2.0 | SYSTEM 49 |_________| |__________| |__________| LIBRARIES 50 | | | 51 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 52 | | | 53 ____v____________________v____________________v____ GUEST 54 | | KERNEL 55 | QEMU PIPE | 56 |___________________________________________________| 57 | 58 - - - - - - - - - - - - - -|- - - - - - - - - - - - - - - - 59 | 60 v 61 EMULATOR 62 63 - Specific code inside the emulator program that is capable of transmitting 64 the wire protocol stream to a special rendering library or process (called 65 the "renderer" here), which understands the format. 66 67 | 68 | PROTOCOL BYTE STREAM 69 _____v_____ 70 | | 71 | EMULATOR | 72 |___________| 73 | 74 | UNMODIFIED PROTOCOL BYTE STREAM 75 _____v_____ 76 | | 77 | RENDERER | 78 |___________| 79 80 81 - The renderer decodes the EGL/GLES commands from the wire 82 protocol stream, and dispatches them to the translator libraries 83 appropriately. 84 85 | 86 | PROTOCOL BYTE STREAM 87 _____v_____ 88 | | 89 | RENDERER | 90 |___________| 91 | | | 92 +-----------------+ | +-----------------+ 93 | | | 94 ____v____ ___v______ ____v_____ 95 | | | | | | HOST 96 |TRANSLATOR |TRANSLATOR| |TRANSLATOR| HOST 97 | EGL | | GLES 1.1 | | GLES 2.0 | TRANSLATOR 98 |_________| |__________| |__________| LIBRARIES 99 100 101 102 - In reality, the protocol stream flows in both directions, even though most 103 of the commands result in data going from the guest to the host. A complete 104 picture of the emulation would thus be: 105 106 107 108 109 110 _________ __________ __________ 111 | | | | | | 112 |EMULATION| |EMULATION | |EMULATION | GUEST 113 | EGL | | GLES 1.1 | | GLES 2.0 | SYSTEM 114 |_________| |__________| |__________| LIBRARIES 115 ^ ^ ^ 116 | | | 117 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 118 | | | 119 ____v____________________v____________________v____ GUEST 120 | | KERNEL 121 | QEMU PIPE | 122 |___________________________________________________| 123 ^ 124 | 125 - - - - - - - - - - - - - -|- - - - - - - - - - - - - - - - 126 | 127 | PROTOCOL BYTE STREAM 128 _____v_____ 129 | | 130 | EMULATOR | 131 |___________| 132 ^ 133 | UNMODIFIED PROTOCOL BYTE STREAM 134 _____v_____ 135 | | 136 | RENDERER | 137 |___________| 138 ^ ^ ^ 139 | | | 140 +-----------------+ | +-----------------+ 141 | | | 142 ____v____ ___v______ ____v_____ 143 | | | | | | 144 |TRANSLATOR |TRANSLATOR| |TRANSLATOR| HOST 145 | EGL | | GLES 1.1 | | GLES 2.0 | TRANSLATOR 146 |_________| |__________| |__________| LIBRARIES 147 ^ ^ ^ 148 | | | 149 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 150 | | | 151 ____v____ ____v_____ _____v____ HOST 152 | | | | | | SYSTEM 153 | Xgl | | GL 2.0 | | GL 2.0 | LIBRARIES 154 |_________| |__________| |__________| 155 156 (NOTE: 'Xgl' is for Linux only, replace 'AGL' on OS X, and 'WGL' on Windows). 157 158 159 Note that, in the above graphics, only the host system libraries at the bottom 160 are _not_ provided by Android. 161 162 163 Design Requirements: 164 -------------------- 165 166 The above design comes from several important requirements that were decided 167 early in the project: 168 169 1 - The ability to run the renderer in a separate process from the emulator 170 itself is important. 171 172 For various practical reasons, we plan to completely separate the core QEMU 173 emulation from the UI window by using two distinct processes. As such, the 174 renderer will be implemented as a library inside the UI program, but will 175 need to receive protocol bytes from the QEMU process. 176 177 The communication channel will be either a fast Unix socket or a Win32 178 named pipe between these two. A shared memory segment with appropriate 179 synchronization primitives might also be used if performance becomes 180 an issue. 181 182 This explains why the emulator doesn't alter or even try to parse the 183 protocol byte stream. It only acts as a dumb proxy between the guest 184 system and the renderer. This also avoids adding lots of GLES-specific 185 code inside the QEMU code base which is terribly complex. 186 187 2 - The ability to use vendor-specific desktop EGL/GLES libraries is 188 important. 189 190 GPU vendors like NVidia, AMD or ARM all provide host versions of the 191 EGL/GLES libraries that emulate their respectivie embedded graphics 192 chipset. 193 194 The renderer library can be configured to use these instead of the 195 translator libraries provided with this project. This can be useful to 196 more accurately emulate the behaviour of specific devices. 197 198 Moreover, these vendor libraries typically expose vendor-specific 199 extensions that are not provided by the translator libraries. We cannot 200 expose them without modifying our code, but it's important to be able 201 to do so without too much pain. 202 203 204 Code organization: 205 ------------------ 206 207 All source code for the components above is spread over multiple directories 208 in the Android source trees: 209 210 - The emulator sources are under $ANDROID/external/qemu, which we'll 211 call $QEMU in the rest of this document. 212 213 - The guest and system libraries are under 214 $ANDROID/development/tools/emulator/opengl, which we'll call $EMUGL 215 216 - The QEMU Pipe kernel driver is under $KERNEL/drivers/misc/qemupipe 217 218 Where $ANDROID is the top of the open-source Android source tree, and 219 $KERNEL is the top of the qemu-specific kernel source tree (using one 220 of the android-goldfish-xxxx branches here). 221 222 The emulator sources related to this projects are: 223 224 $QEMU/hw/goldfish_pipe.c -> implement QEMU pipe virtual hardware 225 $QEMU/hw/opengles.c -> implement GLES initialization 226 $QEMU/hw/hw-pipe-net.c -> implements the communication channel 227 between the QEMU Pipe and the renderer library 228 229 The other sources are: 230 231 $EMUGL/system -> system libraries 232 $EMUGL/host -> host libraries (translator + renderer) 233 $EMUGL/shared -> shared libraries, used both in the guest and the host 234 $EMUGL/tests -> various test programs 235 236 237 Translator libraries: 238 --------------------- 239 240 There are three translator host libraries provided by this project: 241 242 libEGL_translator -> EGL 1.2 translation 243 libGLES_CM_translator -> GLES 1.1 translation 244 libGLES_V2_translator -> GLES 2.0 translation 245 246 The full name of the library will depend on the host system. 247 For simplicity, only the library name suffix will change (i.e. the 248 'lib' prefix is not dropped on Windows), i.e.: 249 250 libEGL_translator.so -> for Linux 251 libEGL_translator.dylib -> for OS X 252 libEGL_translator.dll -> for Windows 253 254 The source code for these libraries is located under the following 255 path in the Android source tree: 256 257 $EMUGL/host/libs/Translator/EGL 258 $EMUGL/host/libs/Translator/GLES_CM 259 $EMUGL/host/libs/Translator/GLES_V2 260 261 The translator libraries also use a common routines defined under: 262 263 $EMUGL/host/libs/Translator/GLcommon 264 265 266 Wire Protocol Overiew: 267 ---------------------- 268 269 The "wire protocol" is implemented as follows: 270 271 - EGL/GLES function calls are described through several "specification" 272 files, which describes the types, function signatures and various 273 attributes for each one of them. 274 275 - These files are read by a tool called "emugen" which generates C 276 source files and headers based on the specification. These correspond 277 to both encoding, decoding and "wrappers" (more on this later). 278 279 - System "encoder" static libraries are built using some of these generated 280 files. They contain code that can serialize EGL/GLES calls into simple 281 byte messages and send it through a generic "IOStream" object. 282 283 - Host "decoder" static libraries are also built using some of these 284 generated files. Their code retrieves byte messages from an "IOStream" 285 object, and translates them into function callbacks. 286 287 IOStream abstraction: 288 - - - - - - - - - - - 289 290 The "IOStream" is a very simple abstract class used to send byte messages 291 both in the guest and host. It is defined through a shared header under 292 $EMUGL/host/include/libOpenglRender/IOStream.h 293 294 Note that despite the path, this header is included by *both* host and guest 295 source code. The main idea around IOStream's design is that to send a message, 296 one does the following: 297 298 1/ call stream->allocBuffer(size), which returns the address of a 299 memory buffer of at least 'size' bytes. 300 301 2/ write the content of the serialized command (usually a header + some 302 payload) directly into the buffer 303 304 3/ call stream->commitBuffer() to send it. 305 306 Alternatively, one can also pack several commands into a single buffer with 307 stream->alloc() and stream->flush(), as in: 308 309 1/ buf1 = stream->alloc(size1) 310 2/ write first command bytes into buf1 311 3/ buf2 = stream->alloc(size2) 312 4/ write second command bytes into buf2 313 5/ stream->flush() 314 315 Finally, there are also explict read/write methods like stream->readFully() 316 or stream->writeFully() which can be used when you don't want an intermediate 317 buffer. This is used in certain cases by the implementation, e.g. to avoid 318 an intermediate memory copy when sending texture data from the guest to the 319 host. 320 321 The host IOStream implementations are under $EMUGL/shared/OpenglCodecCommon/, 322 see in particular: 323 324 $EMUGL/shared/OpenglCodecCommon/TcpStream.cpp -> using local TCP sockets 325 $EMUGL/shared/OpenglCodecCommon/UnixStream.cpp -> using Unix sockets 326 $EMUGL/shared/OpenglCodecCommon/Win32PipeStream.cpp -> using Win32 named pipes 327 328 The guest IOStream implementation uses the TcpStream.cpp above, as well as 329 an alternative QEMU-specific source: 330 331 $EMUGL/system/OpenglSystemCommon/QemuPipeStream.cpp -> uses QEMU pipe from the guest 332 333 The QEMU Pipe implementation is _significantly_ faster (about 20x) due to 334 several reasons: 335 336 - all succesful read() and write() operations through it are instantaneous 337 from the guest's point of view. 338 339 - all buffer/memory copies are performed directly by the emulator, and thus 340 much faster than performing the same thing inside the kernel with emulated 341 ARM instructions. 342 343 - it doesn't need to go through a kernel TCP/IP stack that will wrap the 344 data into TCP/IP/MAC packets, send them to an emulated ethernet device, 345 which is itself connected to an internal firewall implementation that 346 will unwrap the packets, re-assemble them, then send them through BSD 347 sockets to the host kernel. 348 349 However, would it be necessary, you could write a guest IOStream implementation 350 that uses a different transport. If you do, please look at 351 $EMUGL/system/OpenglCodecCommon/HostConnection.cpp which contains the code 352 used to connect the guest to the host, on a per-thread basis. 353 354 355 Source code auto-generation: 356 - - - - - - - - - - - - - - 357 358 The 'emugen' tool is located under $EMUGL/host/tools/emugen. There is a README 359 file that explains how it works. 360 361 You can also look at the following specifications files: 362 363 For GLES 1.1: 364 $EMUGL/system/GLESv1_enc/gl.types 365 $EMUGL/system/GLESv1_enc/gl.in 366 $EMUGL/system/GLESv1_enc/gl.attrib 367 $EMUGL/system/GLESv1_enc/gl.addon 368 369 For GLES 2.0: 370 $EMUGL/system/GLESv2_enc/gl2.types 371 $EMUGL/system/GLESv2_enc/gl2.in 372 $EMUGL/system/GLESv2_enc/gl2.attrib 373 $EMUGL/system/GLESv2_enc/gl2.addon 374 375 For EGL: 376 $EMUGL/system/renderControl_enc/renderControl.types 377 $EMUGL/system/renderControl_enc/renderControl.in 378 $EMUGL/system/renderControl_enc/renderControl.attrib 379 $EMUGL/system/renderControl_enc/renderControl.addon 380 381 Note that the EGL specification files are under a directory named 382 "renderControl_enc" and have filenames that begin with "renderControl" 383 384 This is mainly for historic reasons now, but is also related to the fact that 385 this part of the wire protocol contains support functions/calls/specifications 386 that are not part of the EGL specification itself, but add a few features 387 required to make everything works. For example, they have calls related to 388 the "gralloc" system library module used to manage graphics surfaces at a 389 lower level than EGL. 390 391 Generally speaking, guest encoder sources are located under directories 392 named $EMUGL/system/<name>_enc/, while the corresponding host decoder 393 sources will be under $EMUGL/host/libs/<name>_dec/ 394 395 However, all these sources use the same spec files located under the 396 encoding directories. The decoders may even need to include a few 397 non-auto-generated header files from the encoder directories. 398 399 400 401 System libraries: 402 ----------------- 403 404 Meta EGL/GLES system libraries, and egl.cfg: 405 - - - - - - - - - - - - - - - - - - - - - - 406 407 It is important to understand that the emulation-specific EGL/GLES libraries 408 are not directly linked by applications at runtime. Instead, the system 409 provides a set of "meta" EGL/GLES libraries that will load the appropriate 410 hardware-specific libraries on first use. 411 412 More specifically, the system libEGL.so contains a "loader" which will try 413 to load: 414 415 - hardware-specific EGL/GLES libraries 416 - the software-based rendering libraries (called "libagl") 417 418 The system libEGL.so is also capable of merging the EGL configs of both the 419 hardware and software libraries transparently to the application. The system 420 libGLESv1_CM.so and libGLESv2.so, work with it to ensure that the thread's 421 current context will be linked to either the hardware or software libraries 422 depending on the config selected. 423 424 For the record, the loader's source code in under 425 frameworks/base/opengl/libs/EGL/Loader.cpp. It depends on a file named 426 /system/lib/egl/egl.cfg which must contain two lines that look like: 427 428 0 1 <name> 429 0 0 android 430 431 The first number in each line is a display number, and must be 0 since the 432 system's EGL/GLES libraries don't support anything else. 433 434 The second number must be 1 to indicate hardware libraries, and 0 to indicate 435 a software one. The line corresponding to the hardware library, if any, must 436 always appear before the one for the software library. 437 438 The third field is a name corresponding to a shared library suffix. It really 439 means that the corresponding libraries will be named libEGL_<name>.so, 440 libGLESv1_CM_<name>.so and libGLESv2_<name>.so. Moreover these libraries must 441 be placed under /system/lib/egl/ 442 443 The name "android" is reserved for the system software renderer. 444 445 The egl.cfg that comes with this project uses the name "emulation" for the 446 hardware libraries. This means that it provides an egl.cfg file that contains 447 the following lines: 448 449 0 1 emulation 450 0 0 android 451 452 See $EMUGL/system/egl/egl.cfg and more generally the following build files: 453 454 $EMUGL/system/egl/Android.mk 455 $EMUGL/system/GLESv1/Android.mk 456 $EMUGL/system/GLESv2/Android.mk 457 458 to see how the libraries are named and placed under /system/lib/egl/ by the 459 build system. 460 461 462 Emulation libraries: 463 - - - - - - - - - - - 464 465 The emulator-specific libraries are under the following: 466 467 $EMUGL/system/egl/ 468 $EMUGL/system/GLESv1/ 469 $EMUGL/system/GLESv2/ 470 471 The code for GLESv1 and GLESv2 is pretty small, since it mostly link against 472 the static encoding libraries. 473 474 The code for EGL is a bit more complex, because it needs to deal with 475 extensions dynamically. I.e. if an extension is not available on the host 476 it shouldn't be exposed by the library at runtime. So the EGL code queries 477 the host for the list of available extensions in order to return them to 478 clients. Similarly, it must query the list of valid EGLConfigs for the 479 current host system. 480 481 482 "gralloc" module implementation: 483 - - - - - - - - - - - - - - - - - 484 485 In addition to EGL/GLES libraries, the Android system requires a 486 hardware-specific library to manage graphics surfaces at a level lower than 487 EGL. This library must be what is called in Android land as a "HAL module". 488 489 A "HAL module" must provide interfaces defined by Android's HAL 490 (Hardware Abstraction Library). These interface definitions can be found 491 under $ANDROID/hardware/libhardware/include/ 492 493 Of all possible HAL modules, the "gralloc" one is used by the system's 494 SurfaceFlinger to allocate framebuffers and other graphics memory regions, 495 as well as eventually lock/unlock/swap them when needed. 496 497 The code under $EMUGL/system/gralloc/ implements the module required by the 498 GLES emulation project. It's not very long, but there are a few things to 499 notice here: 500 501 - first, it will probe the guest system to determine if the emulator that 502 is running the virtual device really supports GPU emulation. In certain 503 circumstances this may not be possible. 504 505 If this is the case, then the module will redirect all calls to the 506 "default" gralloc module that is normally used by the system when 507 software-only rendering is enabled. 508 509 The probing happens in the function "fallback_init" which gets called 510 when the module is first opened. This initializes the 'sFallback' variable 511 to a pointer to the default gralloc module when required. 512 513 - second, this module is used by SurfaceFlinger to display "software surfaces", 514 i.e. those that are backed by system memory pixel buffers, and written to 515 directly through the Skia graphics library (i.e. the non-accelerated ones). 516 517 the default module simply copies the pixel data from the surface to the 518 virtual framebuffer i/o memory, but this project's gralloc module sends it 519 to the renderer through the QEMU Pipe instead. 520 521 It turns out that this results in _faster_ rendering/frame-rates overall, 522 because memory copies inside the guest are slow, while QEMU pipe transfers 523 are done directly in the emulator. 524 525 526 Host Renderer: 527 -------------- 528 529 The host renderer library is located under $EMUGL/host/libs/libOpenglRender, 530 and it provides an interface described by the headers under 531 $EMUGL/host/include/libOpenglRender/render_api.h (e.g. for use by the emulator). 532 533 In a nutshell, the rendering library is responsible for the following: 534 535 - Providing a virtual off-screen video surface where everything will get 536 rendered at runtime. Its dimensions are fixed by the call to 537 initOpenglRender() that must happen just after the library is 538 initialized. 539 540 - Provide a way to display the virtual video surface on a host application's 541 UI. This is done by calling createOpenGLSubWindow() which takes as argument 542 the window ID or handle of a parent window, some display dimensions and 543 a rotation angle. This allows the surface to be scaled/rotated when it is 544 displayed, even if the dimensions of the video surface do not change. 545 546 - Provide a way to listen to incoming EGL/GLES commands from the guest. 547 This is done by providing a so-called "port number" to initOpenglRender(). 548 549 By default, the port number corresponds to a local TCP port number that the 550 renderer will bind to and listen. Every new connection to this port will 551 correspond to the creation of a new guest host connection, each such 552 connection corresponding to a distinct thread in the guest system. 553 554 For performance reasons, it is possible to listen to either Unix sockets 555 (on Linux and OS X), or to a Win32 named pipe (on Windows). To do so, one 556 had to call setStreamType() between library initialization 557 (i.e. initLibrary()) and construction (i.e. initOpenglRender()). 558 559 Note that in these modes, the port number is still used to differentiate 560 between several emulator instances. These details are normally handled by 561 the emulator code so you shouldn't care too much. 562 563 Note that an earlier version of the interface allowed a client of the renderer 564 library to provide its own IOStream implementation. However, this wasn't very 565 convenient for a number of reasons. This maybe something that could be done 566 again if it makes sense, but for now the performance numbers are pretty good. 567 568 569 Host emulator: 570 -------------- 571 572 The code under $QEMU/android/opengles.c is in charge of dynamically loading 573 the rendering library and initializing / constructing it properly. 574 575 QEMU pipe connections to the 'opengles' service are piped through the code 576 in $QEMU/android/hw-pipe-net.c. Look for the openglesPipe_init() function, 577 which is in charge of creating a connection to the renderer library 578 (either through a TCP socket, or a Unix pipe depending on configuration. 579 support for Win32 named pipes hasn't been implemented yet in the emulator) 580 whenever a guest process opens the "opengles" service through /dev/qemu_pipe. 581 582 There is also some support code for the display of the GLES framebuffer 583 (through the renderer library's subwindow) under $QEMU/skin/window. 584 585 Note that at the moment, scaling and rotation are supported. However, 586 brightness emulation (which used to modify the pixel values from the 587 hardware framebuffer before displaying them) doesn't work. 588 589 Another issue is that it is not possible to display anything on top of the 590 GL subwindow at the moment. E.g. this will obscure the emulated trackball 591 image (that is normally toggled with Ctrl-T during emulation, or enabled 592 by pressing the Delete key). 593 594
