1 .. _devguide-coding-3D-graphics: 2 3 ########### 4 3D Graphics 5 ########### 6 7 Native Client applications use the `OpenGL ES 2.0 8 <http://en.wikipedia.org/wiki/OpenGL_ES>`_ API for 3D rendering. This document 9 describes how to call the OpenGL ES 2.0 interface in a Native Client module and 10 how to build an efficient rendering loop. It also explains how to validate GPU 11 drivers and test for specific GPU capabilities, and provides tips to help ensure 12 your rendering code runs efficiently. 13 14 .. Note:: 15 :class: note 16 17 **Note**: 3D drawing and OpenGL are complex topics. This document deals only 18 with issues directly related to programming in the Native Client 19 environment. To learn more about OpenGL ES 2.0 itself, see the `OpenGL ES 2.0 20 Programming Guide <http://opengles-book.com/>`_. 21 22 Validating the client graphics platform 23 ======================================= 24 25 Native Client is a software technology that lets you code an application once 26 and run it on multiple platforms without worrying about the implementation 27 details on every possible target platform. It's difficult to provide the same 28 support at the hardware level. Graphics hardware comes from many different 29 manufacturers and is controlled by drivers of varying quality. A particular GPU 30 driver may not support every OpenGL ES 2.0 feature, and some drivers are known 31 to have vulnerabilities that can be exploited. 32 33 Even if the GPU driver is safe to use, your program should perform a validation 34 check before you launch your application to ensure that the driver supports all 35 the features you need. 36 37 Vetting the driver in JavaScript 38 -------------------------------- 39 40 At startup, the application should perform a few additional tests that can be 41 implemented in JavaScript on its hosting web page. The script that performs 42 these tests should be included before the module's ``embed`` tag, and ideally 43 the ``embed`` tag should appear on the hosting page only if these tests succeed. 44 45 The first thing to check is whether you can create a graphics context. If you 46 can, use the context to confirm the existence of any required OpenGL ES 2.0 47 extensions. You may want to refer to the `extension registry 48 <http://www.khronos.org/registry/webgl/extensions/>`_ and include `vendor 49 prefixes <https://developer.mozilla.org/en-US/docs/WebGL/Using_Extensions>`_ 50 when checking for extensions. 51 52 Vetting the driver in Native Client 53 ----------------------------------- 54 55 Create a context 56 ^^^^^^^^^^^^^^^^ 57 58 Once you've passed the JavaScript validation tests, it's safe to add a Native 59 Client embed tag to the hosting web page and load the module. As part of the 60 module initialization code, you must create a graphics context for the app by 61 either creating a C++ ``Graphics3D`` object or calling ``PPB_Graphics3D`` API 62 function ``Create``. Don't assume this will always succeed; you still might have 63 problems creating the context. If you are in development mode and can't create 64 the context, try creating a simpler version to see if you're asking for an 65 unsupported feature or exceeding a driver resource limit. Your production code 66 should always check that the context was created and fail gracefully if that's 67 not the case. 68 69 Check for extensions and capabilities 70 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 71 72 Not every GPU supports every extension or has the same amount of texture units, 73 vertex attributes, etc. On startup, call ``glGetString(GL_EXTENSIONS)`` and 74 check for the extensions and the features you need. For example: 75 76 * If you are using non power-of-2 texture with mipmaps, make sure 77 ``GL_OES_texture_npot`` exists. 78 79 * If you are using floating point textures, make sure ``GL_OES_texture_float`` 80 exists. 81 82 * If you are using DXT1, DXT3, or DXT5 textures, make sure the corresponding 83 extensions ``EXT_texture_compression_dxt1``, 84 ``GL_CHROMIUM_texture_compression_dxt3``, and 85 ``GL_CHROMIUM_texture_compression_dxt5`` exist. 86 87 * If you are using the functions ``glDrawArraysInstancedANGLE``, 88 ``glDrawElementsInstancedANGLE``, ``glVertexAttribDivisorANGLE``, or the PPAPI 89 interface ``PPB_OpenGLES2InstancedArrays``, make sure the corresponding 90 extension ``GL_ANGLE_instanced_arrays`` exists. 91 92 * If you are using the function ``glRenderbufferStorageMultisampleEXT``, or the 93 PPAPI interface ``PPB_OpenGLES2FramebufferMultisample``, make sure the 94 corresponding extension ``GL_CHROMIUM_framebuffer_multisample`` exists. 95 96 * If you are using the functions ``glGenQueriesEXT``, ``glDeleteQueriesEXT``, 97 ``glIsQueryEXT``, ``glBeginQueryEXT``, ``glEndQueryEXT``, ``glGetQueryivEXT``, 98 ``glGetQueryObjectuivEXT``, or the PPAPI interface ``PPB_OpenGLES2Query``, 99 make sure the corresponding extension ``GL_EXT_occlusion_query_boolean`` 100 exists. 101 102 * If you are using the functions ``glMapBufferSubDataCHROMIUM``, 103 ``glUnmapBufferSubDataCHROMIUM``, ``glMapTexSubImage2DCHROMIUM``, 104 ``glUnmapTexSubImage2DCHROMIUM``, or the PPAPI interface 105 ``PPB_OpenGLES2ChromiumMapSub``, make sure the corresponding extension 106 ``GL_CHROMIUM_map_sub`` exists. 107 108 Check for system capabilites with ``glGetIntegerv`` and adjust shader programs 109 as well as texture and vertex data accordingly: 110 111 * If you are using textures in vertex shaders, make sure 112 ``glGetIntegerv(GL_MAX_VERTEX_TEXTURE_IMAGE_UNITS, ...)`` and 113 ``glGetIntegerv(GL_MAX_TEXTURE_SIZE, ...)`` return values greater than 0. 114 115 * If you are using more than 8 textures in a single shader, make sure 116 ``glGetIntegerv(GL_MAX_TEXTURE_IMAGE_UNITS, ...)`` returns a value greater 117 than or equal to the number of simultaneous textures you need. 118 119 Vetting the driver in the Chrome Web Store 120 ------------------------------------------ 121 122 If you choose to place your application in the `Chrome Web Store </webstore>`_, 123 its Web Store `manifest file </extensions/manifest>`_ can include the ``webgl`` 124 feature in the requirements parameter. It looks like this: 125 126 .. naclcode:: 127 128 "requirements": { 129 "3D": { 130 "features": ["webgl"] 131 } 132 } 133 134 While WebGL is technically a JavaScript API, specifying the ``webgl`` feature 135 also works for OpenGL ES 2.0 because both interfaces use the same driver. 136 137 This manifest item is not required, but if you include it, the Chrome Web Store 138 will prevent a user from installing the application if the browser is running on 139 a machine that does not support OpenGL ES 2.0 or that is using a known 140 blacklisted GPU driver that could invite an attack. 141 142 If the Web Store determines that the user's driver is deficient, the app won't 143 appear on the store's tile display. However, it will appear in store search 144 results or if the user links to it directly, in which case the user could still 145 download it. But the manifest requirements will be checked when the user reaches 146 the install page, and if there is a problem, the browser will display the 147 message "This application is not supported on this computer. Installation has 148 been disabled." 149 150 The manifest-based check applies only to downloads directly from the Chrome Web 151 Store. It is not performed when an application is loaded via `inline 152 installation </webstore/inline_installation>`_. 153 154 What to do when there are problems 155 ---------------------------------- 156 157 Using the vetting procedure described above, you should be able to detect the 158 most common problems before your application runs. If there are problems, your 159 code should describe the issue as clearly as possible. That's easy if there is a 160 missing feature. Failure to create a graphics context is tougher to diagnose. At 161 the very least, you can suggest that the user try to update the driver. You 162 might want to linke to the Chrome page that describes `how to do updates 163 <http://support.google.com/chrome/bin/answer.py?hl=en&answer=1202946>`_. 164 165 If a user can't update the driver, or their problem persists, be sure to gather 166 information about their graphics environment. Ask for the contents of the Chrome 167 ``about:gpu`` page. 168 169 Document unreliable drivers 170 --------------------------- 171 172 It can be helpful to include information about known dubious drivers in your 173 user documentation. This might help identify if a rogue driver is the cause of a 174 problem. There are many sources of GPU driver blacklists. Two such lists can be 175 found at the `Chromium project 176 <http://src.chromium.org/viewvc/chrome/trunk/deps/gpu/software_rendering_list/software_rendering_list.json>`_ 177 and `Khronos <http://www.khronos.org/webgl/wiki/BlacklistsAndWhitelists>`_. You 178 can use these lists to include information in your documentation that warns 179 users about dangerous drivers. 180 181 Test your defenses 182 ------------------ 183 184 You can test your driver validation code by running Chrome with the following 185 flags (all at once) and watching how your application responds: 186 187 * ``--disable-webgl`` 188 * ``--disable-pepper-3d`` 189 * ``--disable_multisampling`` 190 * ``--disable-accelerated-compositing`` 191 * ``--disable-accelerated-2d-canvas`` 192 193 Calling OpenGL ES 2.0 commands 194 ============================== 195 196 There are three ways to write OpenGL ES 2.0 calls in Native Client. 197 198 Use "pure" OpenGL ES 2.0 function calls 199 --------------------------------------- 200 201 You can make OpenGL ES 2.0 calls through a Pepper extension library. The SDK 202 example ``examples/api/graphics_3d`` works this way. In the file 203 ``graphics_3d.cc``, the key initialization steps are as follows: 204 205 * Add these includes at the top of the file: 206 207 .. naclcode:: 208 209 #include <GLES2/gl2.h> 210 #include "ppapi/lib/gl/gles2/gl2ext_ppapi.h" 211 212 * Define the function ``InitGL``. The exact specification of ``attrib_list`` 213 will be application specific. 214 215 .. naclcode:: 216 217 bool InitGL(int32_t new_width, int32_t new_height) { 218 if (!glInitializePPAPI(pp::Module::Get()->get_browser_interface())) { 219 fprintf(stderr, "Unable to initialize GL PPAPI!\n"); 220 return false; 221 } 222 223 const int32_t attrib_list[] = { 224 PP_GRAPHICS3DATTRIB_ALPHA_SIZE, 8, 225 PP_GRAPHICS3DATTRIB_DEPTH_SIZE, 24, 226 PP_GRAPHICS3DATTRIB_WIDTH, new_width, 227 PP_GRAPHICS3DATTRIB_HEIGHT, new_height, 228 PP_GRAPHICS3DATTRIB_NONE 229 }; 230 231 context_ = pp::Graphics3D(this, attrib_list); 232 if (!BindGraphics(context_)) { 233 fprintf(stderr, "Unable to bind 3d context!\n"); 234 context_ = pp::Graphics3D(); 235 glSetCurrentContextPPAPI(0); 236 return false; 237 } 238 239 glSetCurrentContextPPAPI(context_.pp_resource()); 240 return true; 241 } 242 243 * Include logic in ``Instance::DidChangeView`` to call ``InitGL`` whenever 244 necessary: upon application launch (when the graphics context is NULL) and 245 whenever the module's View changes size. 246 247 Use Regal 248 --------- 249 250 If you are porting an OpenGL ES 2.0 application, or are comfortable writing in 251 OpenGL ES 2.0, you should stick with the Pepper APIs or pure OpenGL ES 2.0 calls 252 described above. If you are porting an application that uses features not in 253 OpenGL ES 2.0, consider using Regal. Regal is an open source library that 254 supports many versions of OpenGL. Regal recently added support for Native 255 Client. Regal forwards most OpenGL calls directly to the underlying graphics 256 library, but it can also emulate other calls that are not included (when 257 hardware support exists). See `libregal 258 <http://www.altdevblogaday.com/2012/09/04/bringing-regal-opengl-to-native-client/>`_ 259 for more info. 260 261 Use the Pepper API 262 ------------------ 263 264 Your code can call the Pepper PPB_OpenGLES2 API directly, as with any Pepper 265 interface. When you write in this way, each invocation of an OpenGL ES 2.0 266 function must begin with a reference to the Pepper interface, and the first 267 argument is the graphics context. To invoke the function ``glCompileShader``, 268 your code might look like: 269 270 .. naclcode:: 271 272 ppb_g3d_interface->CompileShader(graphicsContext, shader); 273 274 This approach specifically targets the Pepper APIs. Each call corresponds to a 275 OpenGL ES 2.0 function, but the syntax is unique to Native Client, so the source 276 file is not portable. 277 278 Implementing a rendering loop 279 ============================= 280 281 Graphics applications require a continuous frame render-and-redraw cycle that 282 runs at a high frequency. To achieve the best frame rate, is important to 283 understand how the OpenGL ES 2.0 code in a Native Client module interacts with 284 Chrome. 285 286 The Chrome and Native Client processes 287 -------------------------------------- 288 289 Chrome is a multi-process browser. Each Chrome tab is a separate process that is 290 running an application with its own main thread (we'll call it the Chrome main 291 thread). When an application launches a Native Client module, the module runs in 292 a new, separate sandboxed process. The module's process has its own main thread 293 (the Native Client thread). The Chrome and Native Client processes communicate 294 with each other using Pepper API calls on their main threads. 295 296 When the Chrome main thread calls the Native Client thread (keyboard and mouse 297 callbacks, for example), the Chrome main thread will block. This means that 298 lengthy operations on the Native Client thread can steal cycles from Chrome, and 299 performing blocking operations on the Native Client thread can bring your app to 300 a standstill. 301 302 Native Client uses callback functions to synchronize the main threads of the 303 two processes. Only certain Pepper functions use callbacks; `SwapBuffers 304 </native-client/pepper_stable/c/struct_p_p_b___graphics3_d__1__0#a293c6941c0da084267ffba3954793497>`_ 305 is one. 306 307 ``SwapBuffers`` and its callback function 308 ----------------------------------------- 309 310 ``SwapBuffers`` is non-blocking; it is called from the Native Client thread and 311 returns immediately. When ``SwapBuffers`` is called, it runs asynchronously on 312 the Chrome main thread. It switches the graphics data buffers, handles any 313 needed compositing operations, and redraws the screen. When the screen update is 314 complete, the callback function that was included as one of ``SwapBuffer``'s 315 arguments will be called from the Chrome thread and executed on the Native 316 Client thread. 317 318 To create a rendering loop, your Native Client module should include a function 319 that does the rendering work and then executes ``SwapBuffers``, passing itself 320 as the ``SwapBuffer`` callback. If your rendering code is efficient and runs 321 quickly, this scheme will achieve the highest frame rate possible. The 322 documentation for ``SwapBuffers`` explains why this is optimal: because the 323 callback is executed only when the plugin's current state is actually on the 324 screen, this function provides a way to rate-limit animations. By waiting until 325 the image is on the screen before painting the next frame, you can ensure you're 326 not generating updates faster than the screen can be updated. 327 328 The following diagram illustrates the interaction between the Chrome and Native 329 Client processes. The application-specific rendering code runs in the function 330 called ``Draw`` on the Native Client thread. Blue down-arrows are blocking calls 331 from the main thread to Native Client, green up-arrows are non-blocking 332 ``SwapBuffers`` calls from Native Client to the main thread. All OpenGL ES 2.0 333 calls are made from ``Draw`` in the Native Client thread. 334 335 .. image:: /images/3d-graphics-render-loop.png 336 337 SDK example ``graphics_3d`` 338 --------------------------- 339 340 The SDK example ``graphics_3d`` uses the function ``MainLoop`` (in 341 ``hello_world.cc``) to create a rendering loop as described above. ``MainLoop`` 342 calls ``Render`` to do the rendering work, and then invokes ``SwapBuffers``, 343 passing itself as the callback. 344 345 .. naclcode:: 346 347 void MainLoop(void* foo, int bar) { 348 if (g_LoadCnt == 3) { 349 InitProgram(); 350 g_LoadCnt++; 351 } 352 if (g_LoadCnt > 3) { 353 Render(); 354 PP_CompletionCallback cc = PP_MakeCompletionCallback(MainLoop, 0); 355 ppb_g3d_interface->SwapBuffers(g_context, cc); 356 } else { 357 PP_CompletionCallback cc = PP_MakeCompletionCallback(MainLoop, 0); 358 ppb_core_interface->CallOnMainThread(0, cc, 0); 359 } 360 } 361 362 Managing the OpenGL ES 2.0 pipeline 363 =================================== 364 365 OpenGL ES 2.0 commands do not run in the Chrome or Native Client processes. They 366 are passed into a FIFO queue in shared memory which is best understood as a `GPU 367 command buffer 368 <http://www.chromium.org/developers/design-documents/gpu-command-buffer>`_. The 369 command buffer is shared by a dedicated GPU process. By using a separate GPU 370 process, Chrome implements another layer of runtime security, vetting all OpenGL 371 ES 2.0 commands and their arguments before they are sent on to the 372 GPU. Buffering commands through the FIFO also speeds up your code, since each 373 OpenGL ES 2.0 call in your Native Client thread returns immediately, while the 374 processing may be delayed as the GPU works down the commands queued up in the 375 FIFO. 376 377 Before the screen is updated, all the intervening OpenGL ES 2.0 commands must be 378 processed by the GPU. Programmers often try to ensure this by using the 379 ``glFlush`` and ``glFinish`` commands in their rendering code. In the case of 380 Native Client this is usually unnecessary. The ``SwapBuffers`` command does an 381 implicit flush, and the Chrome team is continually tweaking the GPU code to 382 consume the OpenGL ES 2.0 FIFO as fast as possible. 383 384 Sometimes a 3D application can write to the FIFO in a way that's difficult to 385 handle. The command pipeline may fill up and your code will have to wait for the 386 GPU to flush the FIFO. If this is the case, you may be able to add ``glFlush`` 387 calls to speed up the flow of the OpenGL ES 2.0 command FIFO. Before you start 388 to add your own flushes, first try to determine if pipeline saturation is really 389 the problem by monitoring the rendering time per frame and looking for irregular 390 spikes that do not consistently fall on the same OpenGL ES 2.0 call. If you're 391 convinced the pipeline needs to be accelerated, insert ``glFlush`` calls in your 392 code before starting blocks of processing that do not generate OpenGL ES 2.0 393 commands. For example, issue a flush before you begin any multithreaded particle 394 work, so that the command buffer will be clear when you start doing OpenGL ES 395 2.0 calls again. Determining where and how often to call ``glFlush`` can be 396 tricky, you will need to experiment to find the sweet spot. 397 398 Rendering and inactive tabs 399 =========================== 400 401 Users will often switch between tabs in a multi-tab browser. A well-behaved 402 application that's performing 3D rendering should pause any real-time processing 403 and yield cycles to other processes when its tab becomes inactive. 404 405 In Chrome, an inactive tab will continue to execute timed functions (such as 406 ``setInterval`` and ``setTimeout``) but the timer interval will be automatically 407 overridden and limited to not less than one second while the tab is inactive. In 408 addition, any callback associated with a ``SwapBuffers`` call will not be sent 409 until the tab is active again. You may receive asynchronous callbacks from 410 functions other than ``SwapBuffers`` while a tab is inactive. Depending on the 411 design of your application, you might choose to handle them as they arrive, or 412 to queue them in a buffer and process them when the tab becomes active. 413 414 The time that passes while a tab is inactive can be considerable. If your main 415 thread pulse is based on the ``SwapBuffers`` callback, your app won't update 416 while a tab is inactive. A Native Client module should be able to detect and 417 respond to the state of the tab in which it's running. For example, when a tab 418 becomes inactive, you can set an atomic flag in the Native Client thread that 419 will skip the 3D rendering and ``SwapBuffers`` calls and continue to call the 420 main thread every 30 msec or so. This provides time to update features that 421 should still run in the background, like audio. It may also be helpful to call 422 ``sched_yield`` or ``usleep`` on any worker threads to release resources and 423 cede cycles to the OS. 424 425 Handling tab activation from the main thread 426 -------------------------------------------- 427 428 You can detect and respond to the activation or deactivation of a tab with 429 JavaScript on your hosting page. Add an EventListener for ``visibilitychange`` 430 that sends a message to the Native Client module, as in this example: 431 432 .. naclcode:: 433 434 document.addEventListener('visibilitychange', function(){ 435 if (document.hidden) { 436 // PostMessage to your Native Client module 437 document.nacl_module.postMessage('INACTIVE'); 438 } else { 439 // PostMessage to your Native Client module 440 document.nacl_module.postMessage('ACTIVE'); 441 } 442 443 }, false); 444 445 Handling tab activation from the Native Client thread 446 ----------------------------------------------------- 447 448 You can also detect and respond to the activation or deactivation of a tab 449 directly from your Native Client module by including code in the function 450 ``pp::Instance::DidChangeView``, which is called whenever a change in the 451 module's view occurs. The code can call ``ppb::View::IsPageVisible`` to 452 determine if the page is visible or not. The most common cause of invisible 453 pages is that the page is in a background tab. 454 455 Tips and best practices 456 ======================= 457 458 Here are some suggestions for writing safe code and getting the maximum 459 performance with the Pepper 3D API. 460 461 Do's 462 ---- 463 464 * **Make sure to enable attrib 0.** OpenGL requires that you enable attrib 0, 465 but OpenGL ES 2.0 does not. For example, you can define a vertex shader with 2 466 attributes, numbered like this: 467 468 .. naclcode:: 469 470 glBindAttribLocation(program, "positions", 1); 471 glBindAttribLocation(program, "normals", 2); 472 473 In this case the shader is not using attrib 0 and Chrome may have to perform 474 some additional work if it is emulating OpenGL ES 2.0 on top of OpenGL. It's 475 always more efficient to enable attrib 0, even if you do not use it. 476 477 * **Check how shaders compile.** Shaders can compile differently on different 478 systems, which can result in ``glGetAttrib*`` functions returning different 479 results. Be sure that the vertex attribute indices match the corresponding 480 name each time you recompile a shader. 481 482 * **Update indices sparingly.** For security reasons, all indices must be 483 validated. If you change indices, Native Client will validate them 484 again. Structure your code so indices are not updated often. 485 486 * **Use a smaller plugin and let CSS scale it.** If you're running into fillrate 487 issues, it may be beneficial to perform scaling via CSS. The size your plugin 488 renders is determined by the width and height attributes of the ``<embed>`` 489 element for the module. The actual size displayed on the web page is 490 controlled by the CSS styles applied to the element. 491 492 * **Avoid matrix-to-matrix conversions.** With some versions of Mac OS, there is 493 a driver problem when compiling shaders. If you get compiler errors for matrix 494 transforms, avoid matrix-to-matrix conversions. For instance, upres a vec3 to 495 a vec4 before transforming it by a mat4, rather than converting the mat4 to a 496 mat3. 497 498 Don'ts 499 ------ 500 501 * **Don't use client side buffers.** OpenGL ES 2.0 can use client side data with 502 ``glVertexAttribPointer`` and ``glDrawElements``, but this is really slow. Try 503 to avoid client side buffers. Use Vertex Buffer Objects (VBOs) instead. 504 505 * **Don't mix vertex data and index data.** By default, Pepper 3D binds buffers 506 to a single point. You could create a buffer and bind it to both 507 ``GL_ARRAY_BUFFER`` and ``GL_ELEMENT_ARRAY_BUFFER``, but that would be 508 expensive overhead and it is not recommended. 509 510 * **Don't call ``glGet*`` or ``glCheck*`` during rendering.** This is normal 511 advice for OpenGL programs, but is particularly important for 3D on 512 Chrome. Calls to any OpenGL ES 2.0 function whose name begins with these 513 strings blocks the Native Client thread. This includes ``glGetError``; avoid 514 calling it in release builds. 515 516 * **Don't use fixed point (``GL_FIXED``) vertex attributes.** Fixed point 517 attributes are not supported in OpenGL ES 2.0, so emulating them in OpenGL ES 518 2.0 is slow. By default, ``GL_FIXED`` support is turned off in the Pepper 3D 519 API. 520 521 * **Don't read data from the GPU.** Don't call ``glReadPixels``, as it is slow. 522 523 * **Don't update a small portion of a large buffer.** In the current OpenGL ES 524 2.0 implementation when you update a portion of a buffer (with 525 ``glSubBufferData`` for example) the entire buffer must be reprocessed. To 526 avoid this problem, keep static and dynamic data in different buffers. 527 528 * **Don't call ``glDisable(GL_TEXTURE_2D)``.** This is an OpenGL ES 2.0 529 error. Each time it is called, an error messages will appear in Chrome's 530 ``about:gpu`` tab. 531