1 Android Hardware OpenGLES emulation design overview 2 =================================================== 3 4 Introduction: 5 ------------- 6 7 Hardware GLES emulation in the Android platform is implemented with a mix 8 of components, which are: 9 10 - Several host "translator" libraries. They implement the EGL, GLES 1.1 and 11 GLES 2.0 ABIs defined by Khronos, and translate the corresponding function 12 calls into calls to the appropriate desktop APIs, i.e.: 13 14 - Xgl (Linux), AGL (OS X) or WGL (Windows) for EGL 15 - desktop GL 2.0 for GLES 1.1 and GLES 2.0 16 17 _________ __________ __________ 18 | | | | | | HOST 19 |TRANSLATOR |TRANSLATOR| |TRANSLATOR| HOST 20 | EGL | | GLES 1.1 | | GLES 2.0 | TRANSLATOR 21 |_________| |__________| |__________| LIBRARIES 22 | | | 23 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 24 | | | 25 ____v____ ____v_____ _____v____ HOST 26 | | | | | | SYSTEM 27 | Xgl | | GL 2.0 | | GL 2.0 | LIBRARIES 28 |_________| |__________| |__________| 29 30 31 32 - Several system libraries inside the emulated guest system that implement 33 the same EGL / GLES 1.1 and GLES 2.0 ABIs. 34 35 They collect the sequence of EGL/GLES function calls and translate then 36 into a custom wire protocol stream that is sent to the emulator program 37 through a high-speed communication channel called a "QEMU Pipe". 38 39 For now, all you need to know is that the pipe is implemented with a 40 custom kernel driver, and provides for _very_ fast bandwidth. All read() 41 and writes() from/to the pipes are essentially instantaneous from the 42 guest's point of view. 43 44 45 _________ __________ __________ 46 | | | | | | 47 |EMULATION| |EMULATION | |EMULATION | GUEST 48 | EGL | | GLES 1.1 | | GLES 2.0 | SYSTEM 49 |_________| |__________| |__________| LIBRARIES 50 | | | 51 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 52 | | | 53 ____v____________________v____________________v____ GUEST 54 | | KERNEL 55 | QEMU PIPE | 56 |___________________________________________________| 57 | 58 - - - - - - - - - - - - - -|- - - - - - - - - - - - - - - - 59 | 60 v 61 EMULATOR 62 63 - Specific code inside the emulator program that is capable of transmitting 64 the wire protocol stream to a special rendering library or process (called 65 the "renderer" here), which understands the format. 66 67 | 68 | PROTOCOL BYTE STREAM 69 _____v_____ 70 | | 71 | EMULATOR | 72 |___________| 73 | 74 | UNMODIFIED PROTOCOL BYTE STREAM 75 _____v_____ 76 | | 77 | RENDERER | 78 |___________| 79 80 81 - The renderer decodes the EGL/GLES commands from the wire 82 protocol stream, and dispatches them to the translator libraries 83 appropriately. 84 85 | 86 | PROTOCOL BYTE STREAM 87 _____v_____ 88 | | 89 | RENDERER | 90 |___________| 91 | | | 92 +-----------------+ | +-----------------+ 93 | | | 94 ____v____ ___v______ ____v_____ 95 | | | | | | HOST 96 |TRANSLATOR |TRANSLATOR| |TRANSLATOR| HOST 97 | EGL | | GLES 1.1 | | GLES 2.0 | TRANSLATOR 98 |_________| |__________| |__________| LIBRARIES 99 100 101 102 - In reality, the protocol stream flows in both directions, even though most 103 of the commands result in data going from the guest to the host. A complete 104 picture of the emulation would thus be: 105 106 107 108 109 110 _________ __________ __________ 111 | | | | | | 112 |EMULATION| |EMULATION | |EMULATION | GUEST 113 | EGL | | GLES 1.1 | | GLES 2.0 | SYSTEM 114 |_________| |__________| |__________| LIBRARIES 115 ^ ^ ^ 116 | | | 117 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 118 | | | 119 ____v____________________v____________________v____ GUEST 120 | | KERNEL 121 | QEMU PIPE | 122 |___________________________________________________| 123 ^ 124 | 125 - - - - - - - - - - - - - -|- - - - - - - - - - - - - - - - 126 | 127 | PROTOCOL BYTE STREAM 128 _____v_____ 129 | | 130 | EMULATOR | 131 |___________| 132 ^ 133 | UNMODIFIED PROTOCOL BYTE STREAM 134 _____v_____ 135 | | 136 | RENDERER | 137 |___________| 138 ^ ^ ^ 139 | | | 140 +-----------------+ | +-----------------+ 141 | | | 142 ____v____ ___v______ ____v_____ 143 | | | | | | 144 |TRANSLATOR |TRANSLATOR| |TRANSLATOR| HOST 145 | EGL | | GLES 1.1 | | GLES 2.0 | TRANSLATOR 146 |_________| |__________| |__________| LIBRARIES 147 ^ ^ ^ 148 | | | 149 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 150 | | | 151 ____v____ ____v_____ _____v____ HOST 152 | | | | | | SYSTEM 153 | Xgl | | GL 2.0 | | GL 2.0 | LIBRARIES 154 |_________| |__________| |__________| 155 156 (NOTE: 'Xgl' is for Linux only, replace 'AGL' on OS X, and 'WGL' on Windows). 157 158 159 Note that, in the above graphics, only the host system libraries at the bottom 160 are _not_ provided by Android. 161 162 163 Design Requirements: 164 -------------------- 165 166 The above design comes from several important requirements that were decided 167 early in the project: 168 169 1 - The ability to run the renderer in a separate process from the emulator 170 itself is important. 171 172 For various practical reasons, we plan to completely separate the core QEMU 173 emulation from the UI window by using two distinct processes. As such, the 174 renderer will be implemented as a library inside the UI program, but will 175 need to receive protocol bytes from the QEMU process. 176 177 The communication channel will be either a fast Unix socket or a Win32 178 named pipe between these two. A shared memory segment with appropriate 179 synchronization primitives might also be used if performance becomes 180 an issue. 181 182 This explains why the emulator doesn't alter or even try to parse the 183 protocol byte stream. It only acts as a dumb proxy between the guest 184 system and the renderer. This also avoids adding lots of GLES-specific 185 code inside the QEMU code base which is terribly complex. 186 187 2 - The ability to use vendor-specific desktop EGL/GLES libraries is 188 important. 189 190 GPU vendors like NVidia, AMD or ARM all provide host versions of the 191 EGL/GLES libraries that emulate their respectivie embedded graphics 192 chipset. 193 194 The renderer library can be configured to use these instead of the 195 translator libraries provided with this project. This can be useful to 196 more accurately emulate the behaviour of specific devices. 197 198 Moreover, these vendor libraries typically expose vendor-specific 199 extensions that are not provided by the translator libraries. We cannot 200 expose them without modifying our code, but it's important to be able 201 to do so without too much pain. 202 203 204 Code organization: 205 ------------------ 206 207 All source code for the components above is spread over multiple directories 208 in the Android source trees: 209 210 - The emulator sources are under $ANDROID/external/qemu, which we'll 211 call $QEMU in the rest of this document. 212 213 - The guest libraries are under 214 $ANDROID/development/tools/emulator/opengl, which we'll call $EMUGL_GUEST 215 216 - The host renderer and translator libraries are under 217 $ANDROID/sdk/emulator/opengl, which we'll call $EMUGL_HOST 218 219 - The QEMU Pipe kernel driver is under $KERNEL/drivers/misc/qemupipe 220 221 Where $ANDROID is the top of the open-source Android source tree, and 222 $KERNEL is the top of the qemu-specific kernel source tree (using one 223 of the android-goldfish-xxxx branches here). 224 225 The emulator sources related to this projects are: 226 227 $QEMU/hw/goldfish_pipe.c -> implement QEMU pipe virtual hardware 228 $QEMU/hw/opengles.c -> implement GLES initialization 229 $QEMU/hw/hw-pipe-net.c -> implements the communication channel 230 between the QEMU Pipe and the renderer library 231 232 The other sources are: 233 234 $EMUGL_GUEST/system -> system libraries 235 $EMUGL_GUEST/shared -> guest copy of shared libraries 236 $EMUGL_GUEST/tests -> various test programs 237 $EMUGL_HOST/host -> host libraries (translator + renderer) 238 $EMUGL_HOST/shared -> host copy of shared libraries 239 $EMUGL_HOST/tests -> various test programs 240 241 The reason the shared libraries aren't actually shared is historical: at one 242 point both guest and host code lived in the same place. That turned out to be 243 impractical with the way the Android SDK is branched, and didn't support the 244 requirement that a single emulator binary be able to run several generations 245 of Android. 246 247 248 Translator libraries: 249 --------------------- 250 251 There are three translator host libraries provided by this project: 252 253 libEGL_translator -> EGL 1.2 translation 254 libGLES_CM_translator -> GLES 1.1 translation 255 libGLES_V2_translator -> GLES 2.0 translation 256 257 The full name of the library will depend on the host system. 258 For simplicity, only the library name suffix will change (i.e. the 259 'lib' prefix is not dropped on Windows), i.e.: 260 261 libEGL_translator.so -> for Linux 262 libEGL_translator.dylib -> for OS X 263 libEGL_translator.dll -> for Windows 264 265 The source code for these libraries is located under the following 266 path in the Android source tree: 267 268 $EMUGL_HOST/host/libs/Translator/EGL 269 $EMUGL_HOST/host/libs/Translator/GLES_CM 270 $EMUGL_HOST/host/libs/Translator/GLES_V2 271 272 The translator libraries also use a common routines defined under: 273 274 $EMUGL_HOST/host/libs/Translator/GLcommon 275 276 277 Wire Protocol Overiew: 278 ---------------------- 279 280 The "wire protocol" is implemented as follows: 281 282 - EGL/GLES function calls are described through several "specification" 283 files, which describes the types, function signatures and various 284 attributes for each one of them. 285 286 - These files are read by a tool called "emugen" which generates C 287 source files and headers based on the specification. These correspond 288 to both encoding, decoding and "wrappers" (more on this later). 289 290 - System "encoder" static libraries are built using some of these generated 291 files. They contain code that can serialize EGL/GLES calls into simple 292 byte messages and send it through a generic "IOStream" object. 293 294 - Host "decoder" static libraries are also built using some of these 295 generated files. Their code retrieves byte messages from an "IOStream" 296 object, and translates them into function callbacks. 297 298 IOStream abstraction: 299 - - - - - - - - - - - 300 301 The "IOStream" is a very simple abstract class used to send byte messages 302 both in the guest and host. It is defined through a shared header under 303 $EMUGL/host/include/libOpenglRender/IOStream.h 304 305 Note that despite the path, this header is included by *both* host and guest 306 source code. The main idea around IOStream's design is that to send a message, 307 one does the following: 308 309 1/ call stream->allocBuffer(size), which returns the address of a 310 memory buffer of at least 'size' bytes. 311 312 2/ write the content of the serialized command (usually a header + some 313 payload) directly into the buffer 314 315 3/ call stream->commitBuffer() to send it. 316 317 Alternatively, one can also pack several commands into a single buffer with 318 stream->alloc() and stream->flush(), as in: 319 320 1/ buf1 = stream->alloc(size1) 321 2/ write first command bytes into buf1 322 3/ buf2 = stream->alloc(size2) 323 4/ write second command bytes into buf2 324 5/ stream->flush() 325 326 Finally, there are also explict read/write methods like stream->readFully() 327 or stream->writeFully() which can be used when you don't want an intermediate 328 buffer. This is used in certain cases by the implementation, e.g. to avoid 329 an intermediate memory copy when sending texture data from the guest to the 330 host. 331 332 The host IOStream implementations are under $EMUGL/shared/OpenglCodecCommon/, 333 see in particular: 334 335 $EMUGL_HOST/shared/OpenglCodecCommon/TcpStream.cpp 336 -> using local TCP sockets 337 $EMUGL_HOST/shared/OpenglCodecCommon/UnixStream.cpp 338 -> using Unix sockets 339 $EMUGL_HOST/shared/OpenglCodecCommon/Win32PipeStream.cpp 340 -> using Win32 named pipes 341 342 The guest IOStream implementation uses the TcpStream.cpp above, as well as 343 an alternative QEMU-specific source: 344 345 $EMUGL_GUEST/system/OpenglSystemCommon/QemuPipeStream.cpp 346 -> uses QEMU pipe from the guest 347 348 The QEMU Pipe implementation is _significantly_ faster (about 20x) due to 349 several reasons: 350 351 - all succesful read() and write() operations through it are instantaneous 352 from the guest's point of view. 353 354 - all buffer/memory copies are performed directly by the emulator, and thus 355 much faster than performing the same thing inside the kernel with emulated 356 ARM instructions. 357 358 - it doesn't need to go through a kernel TCP/IP stack that will wrap the 359 data into TCP/IP/MAC packets, send them to an emulated ethernet device, 360 which is itself connected to an internal firewall implementation that 361 will unwrap the packets, re-assemble them, then send them through BSD 362 sockets to the host kernel. 363 364 However, would it be necessary, you could write a guest IOStream implementation 365 that uses a different transport. If you do, please look at 366 $EMUGL_GUEST/system/OpenglCodecCommon/HostConnection.cpp which contains the 367 code used to connect the guest to the host, on a per-thread basis. 368 369 370 Source code auto-generation: 371 - - - - - - - - - - - - - - 372 373 The 'emugen' tool is located under $EMUGL_HOST/host/tools/emugen. There is a 374 README file that explains how it works. 375 376 You can also look at the following specifications files: 377 378 For GLES 1.1: 379 $EMUGL_HOST/host/GLESv1_dec/gl.types 380 $EMUGL_HOST/host/GLESv1_dec/gl.in 381 $EMUGL_HOST/host/GLESv1_dec/gl.attrib 382 383 For GLES 2.0: 384 $EMUGL_HOST/host/GLESv2_dec/gl2.types 385 $EMUGL_HOST/host/GLESv2_dec/gl2.in 386 $EMUGL_HOST/host/GLESv2_dec/gl2.attrib 387 388 For EGL: 389 $EMUGL_HOST/host/renderControl_dec/renderControl.types 390 $EMUGL_HOST/host/renderControl_dec/renderControl.in 391 $EMUGL_HOST/host/renderControl_dec/renderControl.attrib 392 393 Note that the EGL specification files are under a directory named 394 "renderControl_dec" and have filenames that begin with "renderControl" 395 396 This is mainly for historic reasons now, but is also related to the fact that 397 this part of the wire protocol contains support functions/calls/specifications 398 that are not part of the EGL specification itself, but add a few features 399 required to make everything works. For example, they have calls related to 400 the "gralloc" system library module used to manage graphics surfaces at a 401 lower level than EGL. 402 403 Generally speaking, guest encoder sources are located under directories 404 named $EMUGL_GUEST/system/<name>_enc/, while the corresponding host decoder 405 sources will be under $EMUGL_HOST/host/libs/<name>_dec/ 406 407 However, all these sources use the same spec files located under the 408 decoding directories. 409 410 The encoder files are built from emugen and spec files located in $EMUGL_HOST 411 and copied to the encoder directories in $EMUGL_GUEST by the gen-encoder.sh 412 script. They are checked in, so that a given version of Android supports a 413 specific version of the protocol, even if newer versions of the renderer (and 414 future Android versions) support a newer protocol version. This step needs to 415 be done manually when the protocol changes; these changes also need to be 416 accompanied by changes in the renderer to handle the old version of the 417 protocol. 418 419 420 System libraries: 421 ----------------- 422 423 Meta EGL/GLES system libraries, and egl.cfg: 424 - - - - - - - - - - - - - - - - - - - - - - 425 426 It is important to understand that the emulation-specific EGL/GLES libraries 427 are not directly linked by applications at runtime. Instead, the system 428 provides a set of "meta" EGL/GLES libraries that will load the appropriate 429 hardware-specific libraries on first use. 430 431 More specifically, the system libEGL.so contains a "loader" which will try 432 to load: 433 434 - hardware-specific EGL/GLES libraries 435 - the software-based rendering libraries (called "libagl") 436 437 The system libEGL.so is also capable of merging the EGL configs of both the 438 hardware and software libraries transparently to the application. The system 439 libGLESv1_CM.so and libGLESv2.so, work with it to ensure that the thread's 440 current context will be linked to either the hardware or software libraries 441 depending on the config selected. 442 443 For the record, the loader's source code in under 444 frameworks/base/opengl/libs/EGL/Loader.cpp. It depends on a file named 445 /system/lib/egl/egl.cfg which must contain two lines that look like: 446 447 0 1 <name> 448 0 0 android 449 450 The first number in each line is a display number, and must be 0 since the 451 system's EGL/GLES libraries don't support anything else. 452 453 The second number must be 1 to indicate hardware libraries, and 0 to indicate 454 a software one. The line corresponding to the hardware library, if any, must 455 always appear before the one for the software library. 456 457 The third field is a name corresponding to a shared library suffix. It really 458 means that the corresponding libraries will be named libEGL_<name>.so, 459 libGLESv1_CM_<name>.so and libGLESv2_<name>.so. Moreover these libraries must 460 be placed under /system/lib/egl/ 461 462 The name "android" is reserved for the system software renderer. 463 464 The egl.cfg that comes with this project uses the name "emulation" for the 465 hardware libraries. This means that it provides an egl.cfg file that contains 466 the following lines: 467 468 0 1 emulation 469 0 0 android 470 471 See $EMUGL_GUEST/system/egl/egl.cfg and more generally the following build 472 files: 473 474 $EMUGL_GUEST/system/egl/Android.mk 475 $EMUGL_GUEST/system/GLESv1/Android.mk 476 $EMUGL_GUEST/system/GLESv2/Android.mk 477 478 to see how the libraries are named and placed under /system/lib/egl/ by the 479 build system. 480 481 482 Emulation libraries: 483 - - - - - - - - - - - 484 485 The emulator-specific libraries are under the following: 486 487 $EMUGL_GUEST/system/egl/ 488 $EMUGL_GUEST/system/GLESv1/ 489 $EMUGL_GUEST/system/GLESv2/ 490 491 The code for GLESv1 and GLESv2 is pretty small, since it mostly link against 492 the static encoding libraries. 493 494 The code for EGL is a bit more complex, because it needs to deal with 495 extensions dynamically. I.e. if an extension is not available on the host 496 it shouldn't be exposed by the library at runtime. So the EGL code queries 497 the host for the list of available extensions in order to return them to 498 clients. Similarly, it must query the list of valid EGLConfigs for the 499 current host system. 500 501 502 "gralloc" module implementation: 503 - - - - - - - - - - - - - - - - - 504 505 In addition to EGL/GLES libraries, the Android system requires a 506 hardware-specific library to manage graphics surfaces at a level lower than 507 EGL. This library must be what is called in Android land as a "HAL module". 508 509 A "HAL module" must provide interfaces defined by Android's HAL 510 (Hardware Abstraction Library). These interface definitions can be found 511 under $ANDROID/hardware/libhardware/include/ 512 513 Of all possible HAL modules, the "gralloc" one is used by the system's 514 SurfaceFlinger to allocate framebuffers and other graphics memory regions, 515 as well as eventually lock/unlock/swap them when needed. 516 517 The code under $EMUGL/system/gralloc/ implements the module required by the 518 GLES emulation project. It's not very long, but there are a few things to 519 notice here: 520 521 - first, it will probe the guest system to determine if the emulator that 522 is running the virtual device really supports GPU emulation. In certain 523 circumstances this may not be possible. 524 525 If this is the case, then the module will redirect all calls to the 526 "default" gralloc module that is normally used by the system when 527 software-only rendering is enabled. 528 529 The probing happens in the function "fallback_init" which gets called 530 when the module is first opened. This initializes the 'sFallback' variable 531 to a pointer to the default gralloc module when required. 532 533 - second, this module is used by SurfaceFlinger to display "software surfaces", 534 i.e. those that are backed by system memory pixel buffers, and written to 535 directly through the Skia graphics library (i.e. the non-accelerated ones). 536 537 the default module simply copies the pixel data from the surface to the 538 virtual framebuffer i/o memory, but this project's gralloc module sends it 539 to the renderer through the QEMU Pipe instead. 540 541 It turns out that this results in _faster_ rendering/frame-rates overall, 542 because memory copies inside the guest are slow, while QEMU pipe transfers 543 are done directly in the emulator. 544 545 546 Host Renderer: 547 -------------- 548 549 The host renderer library is located under 550 $EMUGL_HOST/host/libs/libOpenglRender, and it provides an interface described 551 by the headers under $EMUGL_HOST/host/include/libOpenglRender/render_api.h 552 (e.g. for use by the emulator). 553 554 In a nutshell, the rendering library is responsible for the following: 555 556 - Providing a virtual off-screen video surface where everything will get 557 rendered at runtime. Its dimensions are fixed by the call to 558 initOpenglRender() that must happen just after the library is 559 initialized. 560 561 - Provide a way to display the virtual video surface on a host application's 562 UI. This is done by calling createOpenGLSubWindow() which takes as argument 563 the window ID or handle of a parent window, some display dimensions and 564 a rotation angle. This allows the surface to be scaled/rotated when it is 565 displayed, even if the dimensions of the video surface do not change. 566 567 - Provide a way to listen to incoming EGL/GLES commands from the guest. 568 This is done by providing a so-called "port number" to initOpenglRender(). 569 570 By default, the port number corresponds to a local TCP port number that the 571 renderer will bind to and listen. Every new connection to this port will 572 correspond to the creation of a new guest host connection, each such 573 connection corresponding to a distinct thread in the guest system. 574 575 For performance reasons, it is possible to listen to either Unix sockets 576 (on Linux and OS X), or to a Win32 named pipe (on Windows). To do so, one 577 had to call setStreamType() between library initialization 578 (i.e. initLibrary()) and construction (i.e. initOpenglRender()). 579 580 Note that in these modes, the port number is still used to differentiate 581 between several emulator instances. These details are normally handled by 582 the emulator code so you shouldn't care too much. 583 584 Note that an earlier version of the interface allowed a client of the renderer 585 library to provide its own IOStream implementation. However, this wasn't very 586 convenient for a number of reasons. This maybe something that could be done 587 again if it makes sense, but for now the performance numbers are pretty good. 588 589 590 Host emulator: 591 -------------- 592 593 The code under $QEMU/android/opengles.c is in charge of dynamically loading 594 the rendering library and initializing / constructing it properly. 595 596 QEMU pipe connections to the 'opengles' service are piped through the code 597 in $QEMU/android/hw-pipe-net.c. Look for the openglesPipe_init() function, 598 which is in charge of creating a connection to the renderer library 599 (either through a TCP socket, or a Unix pipe depending on configuration. 600 support for Win32 named pipes hasn't been implemented yet in the emulator) 601 whenever a guest process opens the "opengles" service through /dev/qemu_pipe. 602 603 There is also some support code for the display of the GLES framebuffer 604 (through the renderer library's subwindow) under $QEMU/skin/window. 605 606 Note that at the moment, scaling and rotation are supported. However, 607 brightness emulation (which used to modify the pixel values from the 608 hardware framebuffer before displaying them) doesn't work. 609 610 Another issue is that it is not possible to display anything on top of the 611 GL subwindow at the moment. E.g. this will obscure the emulated trackball 612 image (that is normally toggled with Ctrl-T during emulation, or enabled 613 by pressing the Delete key). 614 615
