1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 2 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 3 4 <html xmlns="http://www.w3.org/1999/xhtml"> 5 6 <head> 7 <title>OpenSL ES for Android</title> 8 </head> 9 10 <body> 11 12 <h1 id="title">OpenSL ES for Android</h1> 13 14 <p> 15 This article describes the Android native audio APIs based on the 16 Khronos Group OpenSL ES™ 1.0.1 standard. 17 </p> 18 <p> 19 Unless otherwise noted, 20 all features are available at Android API level 9 (Android platform 21 version 2.3) and higher. 22 Some features are only available at Android API level 14 (Android 23 platform version 4.0) and higher; these are noted. 24 </p> 25 <p> 26 OpenSL ES provides a C language interface that is also callable from C++. It 27 exposes features similar to the audio portions of these Android Java APIs: 28 </p> 29 <ul> 30 <li><a href="http://developer.android.com/reference/android/media/MediaPlayer.html"> 31 android.media.MediaPlayer</a> 32 </li> 33 <li><a href="http://developer.android.com/reference/android/media/MediaRecorder.html"> 34 android.media.MediaRecorder</a> 35 </li> 36 </ul> 37 38 <p> 39 As with all of the Android Native Development Kit (NDK), the primary 40 purpose of OpenSL ES for Android is to facilitate the implementation 41 of shared libraries to be called using the Java Native 42 Interface (JNI). NDK is not intended for writing pure C/C++ 43 applications. That said, OpenSL ES is a full-featured API, and we 44 expect that you should be able to accomplish most of your audio 45 needs using only this API, without up-calls to code running in the Android runtime. 46 </p> 47 48 <p> 49 Note: though based on OpenSL ES, the Android native audio API 50 is <i>not</i> a conforming implementation of any OpenSL ES 1.0.1 51 profile (game, music, or phone). This is because Android does not 52 implement all of the features required by any one of the profiles. 53 Any known cases where Android behaves differently than the specification 54 are described in section "Android extensions" below. 55 </p> 56 57 <h2 id="gettingStarted">Getting started</h2> 58 59 <h3 id="exampleCode">Example code</h3> 60 61 <h4 id="recommended">Recommended</h4> 62 63 <p> 64 Supported and tested example code, usable as a model 65 for your own code, is located in NDK folder 66 <code>platforms/android-9/samples/native-audio/</code> and the 67 <a href="https://github.com/googlesamples/android-ndk/tree/master/audio-echo">audio-echo</a> 68 and 69 <a href="https://github.com/googlesamples/android-ndk/tree/master/native-audio">native-audio</a> 70 folders of the repository 71 <a href="https://github.com/googlesamples/android-ndk">https://github.com/googlesamples/android-ndk</a>. 72 </p> 73 74 <h4 id="notRecommended">Not recommended</h4> 75 76 <p> 77 The OpenSL ES 1.0.1 specification contains example code in the 78 appendices (see section "References" below for the link to this 79 specification). However, the examples in Appendix B: Sample Code 80 and Appendix C: Use Case Sample Code use features 81 not supported by Android. Some examples also contain 82 typographical errors, or use APIs that are likely to change. 83 Proceed with caution in referring to these; 84 though the code may be helpful in understanding the full OpenSL ES 85 standard, it should not be used as is with Android. 86 </p> 87 88 <h3 id="adding">Adding OpenSL ES to your application source code</h3> 89 90 <p> 91 OpenSL ES is a C API, but is callable from both C and C++ code. 92 </p> 93 <p> 94 At a minimum, add the following line to your code: 95 </p> 96 <pre> 97 #include <SLES/OpenSLES.h> 98 </pre> 99 100 <p> 101 If you use Android extensions, also include this header: 102 </p> 103 <pre> 104 #include <SLES/OpenSLES_Android.h> 105 </pre> 106 <p> 107 which automatically includes these headers as well (you don't need to 108 include these, they are shown as an aid in learning the API): 109 </p> 110 <pre> 111 #include <SLES/OpenSLES_AndroidConfiguration.h> 112 #include <SLES/OpenSLES_AndroidMetadata.h> 113 </pre> 114 115 <h3 id="makefile">Makefile</h3> 116 117 <p> 118 Modify your Android.mk as follows: 119 </p> 120 <pre> 121 LOCAL_LDLIBS += -lOpenSLES 122 </pre> 123 124 <h3 id="audioContent">Audio content</h3> 125 126 <p> 127 There are many ways to package audio content for your 128 application, including: 129 </p> 130 131 <dl> 132 133 <dt>Resources</dt> 134 <dd> 135 By placing your audio files into the <code>res/raw/</code> folder, 136 they can be accessed easily by the associated APIs for 137 <a href="http://developer.android.com/reference/android/content/res/Resources.html"> 138 Resources</a>. However there is no direct native access to resources, 139 so you will need to write Java programming language code to copy them out before use. 140 </dd> 141 142 <dt>Assets</dt> 143 <dd> 144 By placing your audio files into the <code>assets/</code> folder, 145 they will be directly accessible by the Android native asset manager 146 APIs. See the header files <code>android/asset_manager.h</code> 147 and <code>android/asset_manager_jni.h</code> for more information 148 on these APIs. The example code 149 located in NDK folder 150 <code>platforms/android-9/samples/native-audio/</code> uses these 151 native asset manager APIs in conjunction with the Android file 152 descriptor data locator. 153 </dd> 154 155 <dt>Network</dt> 156 <dd> 157 You can use the URI data locator to play audio content directly from the 158 network. However, be sure to read section "Security and permissions" below. 159 </dd> 160 161 <dt>Local filesystem</dt> 162 <dd> 163 The URI data locator supports the <code>file:</code> scheme for local files, 164 provided the files are accessible by the application. 165 Note that the Android security framework restricts file access via 166 the Linux user ID and group ID mechanism. 167 </dd> 168 169 <dt>Recorded</dt> 170 <dd>Your application can record audio data from the microphone input, 171 store this content, and then play it back later. 172 The example code uses this method for the "Playback" clip. 173 </dd> 174 175 <dt>Compiled and linked inline</dt> 176 <dd> 177 You can link your audio content directly into the shared library, 178 and then play it using an audio player with buffer queue data locator. This is most 179 suitable for short PCM format clips. The example code uses this 180 technique for the "Hello" and "Android" clips. The PCM data was 181 converted to hex strings using a <code>bin2c</code> tool (not supplied). 182 </dd> 183 184 <dt>Real-time synthesis</dt> 185 <dd> 186 Your application can synthesize PCM data on the fly and then play it 187 using an audio player with buffer queue data locator. This is a 188 relatively advanced technique, and the details of audio synthesis 189 are beyond the scope of this article. 190 </dd> 191 192 </dl> 193 194 <p> 195 Finding or creating useful audio content for your application is 196 beyond the scope of this article, but see the "References" section 197 below for some suggested web search terms. 198 </p> 199 <p> 200 Note that it is your responsibility to ensure that you are legally 201 permitted to play or record content, and that there may be privacy 202 considerations for recording content. 203 </p> 204 205 <h3 id="debugging">Debugging</h3> 206 207 <p> 208 For robustness, we recommend that you examine the <code>SLresult</code> 209 value which is returned by most APIs. Use of <code>assert</code> 210 vs. more advanced error handling logic is a matter of coding style 211 and the particular API; see the Wikipedia article on 212 <a href="http://en.wikipedia.org/wiki/Assertion_(computing)">assert</a> 213 for more information. In the supplied example, we have used <code>assert</code> 214 for "impossible" conditions which would indicate a coding error, and 215 explicit error handling for others which are more likely to occur 216 in production. 217 </p> 218 <p> 219 Many API errors result in a log entry, in addition to the non-zero 220 result code. These log entries provide additional detail which can 221 be especially useful for the more complex APIs such as 222 <code>Engine::CreateAudioPlayer</code>. 223 </p> 224 <p> 225 Use <a href="http://developer.android.com/guide/developing/tools/adb.html"> 226 adb logcat</a>, the 227 <a href="http://developer.android.com/tools/help/adt.html"> 228 Eclipse ADT plugin</a> LogCat pane, or 229 <a href="http://developer.android.com/guide/developing/tools/ddms.html#logcat"> 230 ddms logcat</a> to see the log. 231 </p> 232 233 <h2 id="supportedFeatures">Supported features from OpenSL ES 1.0.1</h2> 234 235 <p> 236 This section summarizes available features. In some 237 cases, there are limitations which are described in the next 238 sub-section. 239 </p> 240 241 <h3 id="globalEntry">Global entry points</h3> 242 243 <p> 244 Supported global entry points: 245 </p> 246 <ul> 247 <li><code>slCreateEngine</code> 248 </li> 249 <li><code>slQueryNumSupportedEngineInterfaces</code> 250 </li> 251 <li><code>slQuerySupportedEngineInterfaces</code> 252 </li> 253 </ul> 254 255 <h3 id="objectsInterfaces">Objects and interfaces</h3> 256 257 <p> 258 The following figure indicates objects and interfaces supported by 259 Android's OpenSL ES implementation. A green cell means the feature 260 is supported. 261 </p> 262 263 <p> 264 <img src="chart1.png" alt="Supported objects and interfaces" /> 265 </p> 266 267 <h3 id="limitations">Limitations</h3> 268 269 <p> 270 This section details limitations with respect to the supported 271 objects and interfaces from the previous section. 272 </p> 273 274 <h4 id="bufferQueueDataLocator">Buffer queue data locator</h4> 275 276 <p> 277 An audio player or recorder with buffer queue data locator supports 278 PCM data format only. 279 </p> 280 281 <h4 id="deviceDataLocator">Device data locator</h4> 282 283 <p> 284 The only supported use of an I/O device data locator is when it is 285 specified as the data source for <code>Engine::CreateAudioRecorder</code>. 286 It should be initialized using these values, as shown in the example: 287 </p> 288 <pre> 289 SLDataLocator_IODevice loc_dev = 290 {SL_DATALOCATOR_IODEVICE, SL_IODEVICE_AUDIOINPUT, 291 SL_DEFAULTDEVICEID_AUDIOINPUT, NULL}; 292 </pre> 293 294 <h4 id="dynamicInterfaceManagementSupported">Dynamic interface management</h4> 295 296 <p> 297 <code>RemoveInterface</code> and <code>ResumeInterface</code> are not supported. 298 </p> 299 300 <h4 id="effectCombinations">Effect combinations</h4> 301 302 <p> 303 It is meaningless to have both environmental reverb and preset 304 reverb on the same output mix. 305 </p> 306 <p> 307 The platform may ignore effect requests if it estimates that the 308 CPU load would be too high. 309 </p> 310 311 <h4 id="effectSend">Effect send</h4> 312 313 <p> 314 <code>SetSendLevel</code> supports a single send level per audio player. 315 </p> 316 317 <h4 id="environmentalReverb">Environmental reverb</h4> 318 319 <p> 320 Environmental reverb does not support the <code>reflectionsDelay</code>, 321 <code>reflectionsLevel</code>, or <code>reverbDelay</code> fields of 322 <code>struct SLEnvironmentalReverbSettings</code>. 323 </p> 324 325 <h4 id="mimeDataFormat">MIME data format</h4> 326 327 <p> 328 The MIME data format can be used with URI data locator only, and only 329 for player (not recorder). 330 </p> 331 <p> 332 The Android implementation of OpenSL ES requires that <code>mimeType</code> 333 be initialized to either <code>NULL</code> or a valid UTF-8 string, 334 and that <code>containerType</code> be initialized to a valid value. 335 In the absence of other considerations, such as portability to other 336 implementations, or content format which cannot be identified by header, 337 we recommend that you 338 set the <code>mimeType</code> to <code>NULL</code> and <code>containerType</code> 339 to <code>SL_CONTAINERTYPE_UNSPECIFIED</code>. 340 </p> 341 <p> 342 Supported formats include WAV PCM, WAV alaw, WAV ulaw, MP3, Ogg 343 Vorbis, AAC LC, HE-AACv1 (aacPlus), HE-AACv2 (enhanced aacPlus), 344 AMR, and FLAC [provided these are supported by the overall platform, 345 and AAC formats must be located within an MP4 or ADTS container]. 346 MIDI is not supported. 347 WMA is not part of the open source release, and compatibility 348 with Android OpenSL ES has not been verified. 349 </p> 350 <p> 351 The Android implementation of OpenSL ES does not support direct 352 playback of DRM or encrypted content; if you want to play this, you 353 will need to convert to cleartext in your application before playing, 354 and enforce any DRM restrictions in your application. 355 </p> 356 357 <h4 id="object">Object</h4> 358 359 <p> 360 <code>Resume</code>, <code>RegisterCallback</code>, 361 <code>AbortAsyncOperation</code>, <code>SetPriority</code>, 362 <code>GetPriority</code>, and <code>SetLossOfControlInterfaces</code> 363 are not supported. 364 </p> 365 366 <h4 id="pcmDataFormat">PCM data format</h4> 367 368 <p> 369 The PCM data format can be used with buffer queues only. Supported PCM 370 playback configurations are 8-bit unsigned or 16-bit signed, mono 371 or stereo, little endian byte ordering, and these sample rates: 372 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, or 48000 Hz. 373 For recording, the supported configurations are device-dependent, 374 however generally 16000 Hz mono 16-bit signed is usually available. 375 </p> 376 <p> 377 Note that the field <code>samplesPerSec</code> is actually in 378 units of milliHz, despite the misleading name. To avoid accidentally 379 using the wrong value, you should initialize this field using one 380 of the symbolic constants defined for this purpose (such as 381 <code>SL_SAMPLINGRATE_44_1</code> etc.) 382 </p> 383 <p> 384 For API level 21 and above, see section "Floating-point data" below. 385 </p> 386 387 <h4 id="playbackRate">Playback rate</h4> 388 389 <p> 390 <b>Note:</b> 391 An OpenSL ES <em>playback rate</em> indicates the speed at which an 392 object presents data, expressed in 393 <a href="https://en.wikipedia.org/wiki/Per_mille">per mille</a> 394 units. A <em>rate range</em> is a closed interval that expresses 395 possible rate ranges. 396 </p> 397 <p> 398 The supported playback rate range(s) and capabilities may vary depending 399 on the platform version and implementation, and so should be determined 400 at runtime by querying with <code>PlaybackRate::GetRateRange</code> 401 or <code>PlaybackRate::GetCapabilitiesOfRate</code>. 402 </p> 403 <p> 404 That said, some guidance on typical rate ranges may be useful: 405 In Android 2.3 a single playback rate range from 500 per mille to 2000 per mille 406 inclusive is typically supported, with property 407 <code>SL_RATEPROP_NOPITCHCORAUDIO</code>. 408 In Android 4.0 the same rate range is typically supported for a data source 409 in PCM format, and a unity rate range of 1000 per mille to 1000 per mille for other formats. 410 </p> 411 412 <h4 id="record">Record</h4> 413 414 <p> 415 The <code>SL_RECORDEVENT_HEADATLIMIT</code> and 416 <code>SL_RECORDEVENT_HEADMOVING</code> events are not supported. 417 </p> 418 419 <h4 id="seek">Seek</h4> 420 421 <p> 422 <code>SetLoop</code> can only loop the whole file and not a portion of it; 423 the parameters should be set as follows: the <code>startPos</code> 424 parameter should be zero and the <code>endPos</code> parameter should 425 be <code>SL_TIME_UNKNOWN</code>. 426 </p> 427 428 <h4 id="uriDataLocator">URI data locator</h4> 429 430 <p> 431 The URI data locator can be used with MIME data format only, and 432 only for an audio player (not audio recorder). Supported schemes 433 are <code>http:</code> and <code>file:</code>. 434 A missing scheme defaults to the <code>file:</code> scheme. Other 435 schemes such as <code>https:</code>, <code>ftp:</code>, and 436 <code>content:</code> are not supported. 437 <code>rtsp:</code> is not verified. 438 </p> 439 440 <h3 id="dataStructures">Data structures</h3> 441 442 <p> 443 Android supports these OpenSL ES 1.0.1 data structures: 444 </p> 445 <ul> 446 <li>SLDataFormat_MIME 447 </li> 448 <li>SLDataFormat_PCM 449 </li> 450 <li>SLDataLocator_BufferQueue 451 </li> 452 <li>SLDataLocator_IODevice 453 </li> 454 <li>SLDataLocator_OutputMix 455 </li> 456 <li>SLDataLocator_URI 457 </li> 458 <li>SLDataSink 459 </li> 460 <li>SLDataSource 461 </li> 462 <li>SLEngineOption 463 </li> 464 <li>SLEnvironmentalReverbSettings 465 </li> 466 <li>SLInterfaceID 467 </li> 468 </ul> 469 470 <h3 id="platformConfiguration">Platform configuration</h3> 471 472 <p> 473 OpenSL ES for Android is designed for multi-threaded applications, 474 and is thread-safe. 475 </p> 476 <p> 477 OpenSL ES for Android supports a single engine per application, and 478 up to 32 objects per engine. Available device memory and CPU may further 479 restrict the usable number of objects. 480 </p> 481 <p> 482 <code>slCreateEngine</code> recognizes, but ignores, these engine options: 483 </p> 484 <ul> 485 <li><code>SL_ENGINEOPTION_THREADSAFE</code> 486 </li> 487 <li><code>SL_ENGINEOPTION_LOSSOFCONTROL</code> 488 </li> 489 </ul> 490 491 <p> 492 OpenMAX AL and OpenSL ES may be used together in the same application. 493 In this case, there is internally a single shared engine object, 494 and the 32 object limit is shared between OpenMAX AL and OpenSL ES. 495 The application should first create both engines, then use both engines, 496 and finally destroy both engines. The implementation maintains a 497 reference count on the shared engine, so that it is correctly destroyed 498 at the second destroy. 499 </p> 500 501 <h2 id="planningFutureVersions">Planning for future versions of OpenSL ES</h2> 502 503 <p> 504 The Android native audio APIs are based on Khronos 505 Group OpenSL ES 1.0.1 (see section "References" below). 506 Khronos has released 507 a revised version 1.1 of the standard. The revised version 508 includes new features, clarifications, correction of 509 typographical errors, and some incompatibilities. Most of the expected 510 incompatibilities are relatively minor, or are in areas of OpenSL ES 511 not supported by Android. However, even a small change 512 can be significant for an application developer, so it is important 513 to prepare for this. 514 </p> 515 <p> 516 The Android team is committed to preserving future API binary 517 compatibility for developers to the extent feasible. It is our 518 intention to continue to support future binary compatibility of the 519 1.0.1-based API, even as we add support for later versions of the 520 standard. An application developed with this version should 521 work on future versions of the Android platform, provided that 522 you follow the guidelines listed in section "Planning for 523 binary compatibility" below. 524 </p> 525 <p> 526 Note that future source compatibility will <i>not</i> be a goal. That is, 527 if you upgrade to a newer version of the NDK, you may need to modify 528 your application source code to conform to the new API. We expect 529 that most such changes will be minor; see details below. 530 </p> 531 532 <h3 id="planningBinary">Planning for binary compatibility</h3> 533 534 <p> 535 We recommend that your application follow these guidelines, 536 to improve future binary compatibility: 537 </p> 538 <ul> 539 <li> 540 Use only the documented subset of Android-supported features from 541 OpenSL ES 1.0.1. 542 </li> 543 <li> 544 Do not depend on a particular result code for an unsuccessful 545 operation; be prepared to deal with a different result code. 546 </li> 547 <li> 548 Application callback handlers generally run in a restricted context, 549 and should be written to perform their work quickly and then return 550 as soon as possible. Do not do complex operations within a callback 551 handler. For example, within a buffer queue completion callback, 552 you can enqueue another buffer, but do not create an audio player. 553 </li> 554 <li> 555 Callback handlers should be prepared to be called more or less 556 frequently, to receive additional event types, and should ignore 557 event types that they do not recognize. Callbacks that are configured 558 with an event mask of enabled event types should be prepared to be 559 called with multiple event type bits set simultaneously. 560 Use "&" to test for each event bit rather than a switch case. 561 </li> 562 <li> 563 Use prefetch status and callbacks as a general indication of progress, but do 564 not depend on specific hard-coded fill levels or callback sequence. 565 The meaning of the prefetch status fill level, and the behavior for 566 errors that are detected during prefetch, may change. 567 </li> 568 <li> 569 See section "Buffer queue behavior" below. 570 </li> 571 </ul> 572 573 <h3 id="planningSource">Planning for source compatibility</h3> 574 575 <p> 576 As mentioned, source code incompatibilities are expected in the next 577 version of OpenSL ES from Khronos Group. Likely areas of change include: 578 </p> 579 580 <ul> 581 <li>The buffer queue interface is expected to have significant changes, 582 especially in the areas of <code>BufferQueue::Enqueue</code>, the parameter 583 list for <code>slBufferQueueCallback</code>, 584 and the name of field <code>SLBufferQueueState.playIndex</code>. 585 We recommend that your application code use Android simple buffer 586 queues instead, because we do not plan to change that API. 587 In the example code supplied with the NDK, we have used 588 Android simple buffer queues for playback for this reason. 589 (We also use Android simple buffer queue for recording and decode to PCM, but 590 that is because standard OpenSL ES 1.0.1 does not support record or decode to 591 a buffer queue data sink.) 592 </li> 593 <li>Addition of <code>const</code> to input parameters passed by reference, 594 and to <code>SLchar *</code> struct fields used as input values. 595 This should not require any changes to your code. 596 </li> 597 <li>Substitution of unsigned types for some parameters that are 598 currently signed. You may need to change a parameter type from 599 <code>SLint32</code> to <code>SLuint32</code> or similar, or add a cast. 600 </li> 601 <li><code>Equalizer::GetPresetName</code> will copy the string to 602 application memory instead of returning a pointer to implementation 603 memory. This will be a significant change, so we recommend that you 604 either avoid calling this method, or isolate your use of it. 605 </li> 606 <li>Additional fields in struct types. For output parameters, these 607 new fields can be ignored, but for input parameters the new fields 608 will need to be initialized. Fortunately, these are expected to all 609 be in areas not supported by Android. 610 </li> 611 <li>Interface 612 <a href="http://en.wikipedia.org/wiki/Globally_unique_identifier"> 613 GUIDs</a> will change. Refer to interfaces by symbolic name rather than GUID 614 to avoid a dependency. 615 </li> 616 <li><code>SLchar</code> will change from <code>unsigned char</code> 617 to <code>char</code>. This primarily affects the URI data locator 618 and MIME data format. 619 </li> 620 <li><code>SLDataFormat_MIME.mimeType</code> will be renamed to <code>pMimeType</code>, 621 and <code>SLDataLocator_URI.URI</code> will be renamed to <code>pURI</code>. 622 We recommend that you initialize the <code>SLDataFormat_MIME</code> 623 and <code>SLDataLocator_URI</code> 624 data structures using a brace-enclosed comma-separated list of values, 625 rather than by field name, to isolate your code from this change. 626 In the example code we have used this technique. 627 </li> 628 <li><code>SL_DATAFORMAT_PCM</code> does not permit the application 629 to specify the representation of the data as signed integer, unsigned 630 integer, or floating-point. The Android implementation assumes that 631 8-bit data is unsigned integer and 16-bit is signed integer. In 632 addition, the field <code>samplesPerSec</code> is a misnomer, as 633 the actual units are milliHz. These issues are expected to be 634 addressed in the next OpenSL ES version, which will introduce a new 635 extended PCM data format that permits the application to explicitly 636 specify the representation, and corrects the field name. As this 637 will be a new data format, and the current PCM data format will 638 still be available (though deprecated), it should not require any 639 immediate changes to your code. 640 </li> 641 </ul> 642 643 <h2 id="androidExtensions">Android extensions</h2> 644 645 <p> 646 The API for Android extensions is defined in <code>SLES/OpenSLES_Android.h</code> 647 and the header files that it includes. 648 Consult that file for details on these extensions. Unless otherwise 649 noted, all interfaces are "explicit". 650 </p> 651 <p> 652 Note that use these extensions will limit your application's 653 portability to other OpenSL ES implementations. If this is a concern, 654 we advise that you avoid using them, or isolate your use of these 655 with <code>#ifdef</code> etc. 656 </p> 657 <p> 658 The following figure shows which Android-specific interfaces and 659 data locators are available for each object type. 660 </p> 661 662 <p> 663 <img src="chart2.png" alt="Android extensions" /> 664 </p> 665 666 <h3 id="androidConfiguration">Android configuration interface</h3> 667 668 <p> 669 The Android configuration interface provides a means to set 670 platform-specific parameters for objects. Unlike other OpenSL ES 671 1.0.1 interfaces, the Android configuration interface is available 672 prior to object realization. This permits the object to be configured 673 and then realized. Header file <code>SLES/OpenSLES_AndroidConfiguration.h</code> 674 documents the available configuration keys and values: 675 </p> 676 <ul> 677 <li>stream type for audio players (default <code>SL_ANDROID_STREAM_MEDIA</code>) 678 </li> 679 <li>record profile for audio recorders (default <code>SL_ANDROID_RECORDING_PRESET_GENERIC</code>) 680 </li> 681 </ul> 682 <p> 683 Here is an example code fragment that sets the Android audio stream type on an audio player: 684 </p> 685 <pre> 686 // CreateAudioPlayer and specify SL_IID_ANDROIDCONFIGURATION 687 // in the required interface ID array. Do not realize player yet. 688 // ... 689 SLAndroidConfigurationItf playerConfig; 690 result = (*playerObject)->GetInterface(playerObject, 691 SL_IID_ANDROIDCONFIGURATION, &playerConfig); 692 assert(SL_RESULT_SUCCESS == result); 693 SLint32 streamType = SL_ANDROID_STREAM_ALARM; 694 result = (*playerConfig)->SetConfiguration(playerConfig, 695 SL_ANDROID_KEY_STREAM_TYPE, &streamType, sizeof(SLint32)); 696 assert(SL_RESULT_SUCCESS == result); 697 // ... 698 // Now realize the player here. 699 </pre> 700 <p> 701 Similar code can be used to configure the preset for an audio recorder. 702 </p> 703 704 <h3 id="androidEffects">Android effects interfaces</h3> 705 706 <p> 707 The Android effect, effect send, and effect capabilities interfaces provide 708 a generic mechanism for an application to query and use device-specific 709 audio effects. A device manufacturer should document any available 710 device-specific audio effects. 711 </p> 712 <p> 713 Portable applications should use the OpenSL ES 1.0.1 APIs 714 for audio effects instead of the Android effect extensions. 715 </p> 716 717 <h3 id="androidFile">Android file descriptor data locator</h3> 718 719 <p> 720 The Android file descriptor data locator permits the source for an 721 audio player to be specified as an open file descriptor with read 722 access. The data format must be MIME. 723 </p> 724 <p> 725 This is especially useful in conjunction with the native asset manager. 726 </p> 727 728 <h3 id="androidSimple">Android simple buffer queue data locator and interface</h3> 729 730 <p> 731 The Android simple buffer queue data locator and interface are 732 identical to the OpenSL ES 1.0.1 buffer queue locator and interface, 733 except that Android simple buffer queues may be used with both audio 734 players and audio recorders, and are limited to PCM data format. 735 [OpenSL ES 1.0.1 buffer queues are for audio players only, and are not 736 restricted to PCM data format.] 737 </p> 738 <p> 739 For recording, the application should enqueue empty buffers. Upon 740 notification of completion via a registered callback, the filled 741 buffer is available for the application to read. 742 </p> 743 <p> 744 For playback there is no difference. But for future source code 745 compatibility, we suggest that applications use Android simple 746 buffer queues instead of OpenSL ES 1.0.1 buffer queues. 747 </p> 748 749 <h3 id="dynamicInterfaceObjectCreation">Dynamic interfaces at object creation</h3> 750 751 <p> 752 For convenience, the Android implementation of OpenSL ES 1.0.1 753 permits dynamic interfaces to be specified at object creation time, 754 as an alternative to adding these interfaces after object creation 755 with <code>DynamicInterfaceManagement::AddInterface</code>. 756 </p> 757 758 <h3 id="bufferQueueBehavior">Buffer queue behavior</h3> 759 760 <p> 761 The OpenSL ES 1.0.1 specification requires that "On transition to 762 the <code>SL_PLAYSTATE_STOPPED</code> state the play cursor is 763 returned to the beginning of the currently playing buffer." The 764 Android implementation does not necessarily conform to this 765 requirement. For Android, it is unspecified whether a transition 766 to <code>SL_PLAYSTATE_STOPPED</code> operates as described, or 767 leaves the play cursor unchanged. 768 </p> 769 <p> 770 We recommend that you do not rely on either behavior; after a 771 transition to <code>SL_PLAYSTATE_STOPPED</code>, you should explicitly 772 call <code>BufferQueue::Clear</code>. This will place the buffer 773 queue into a known state. 774 </p> 775 <p> 776 A corollary is that it is unspecified whether buffer queue callbacks 777 are called upon transition to <code>SL_PLAYSTATE_STOPPED</code> or by 778 <code>BufferQueue::Clear</code>. 779 We recommend that you do not rely on either behavior; be prepared 780 to receive a callback in these cases, but also do not depend on 781 receiving one. 782 </p> 783 <p> 784 It is expected that a future version of OpenSL ES will clarify these 785 issues. However, upgrading to that version would result in source 786 code incompatibilities (see section "Planning for source compatibility" 787 above). 788 </p> 789 790 <h3 id="reportingExtensions">Reporting of extensions</h3> 791 792 <p> 793 <code>Engine::QueryNumSupportedExtensions</code>, 794 <code>Engine::QuerySupportedExtension</code>, 795 <code>Engine::IsExtensionSupported</code> report these extensions: 796 </p> 797 <ul> 798 <li><code>ANDROID_SDK_LEVEL_#</code> 799 where # is the platform API level, 9 or higher 800 </li> 801 </ul> 802 803 <h3 id="decodeAudio">Decode audio to PCM</h3> 804 805 <p> 806 This section describes a deprecated Android-specific extension to OpenSL ES 1.0.1 807 for decoding an encoded stream to PCM without immediate playback. 808 The table below gives recommendations for use of this extension and alternatives. 809 </p> 810 <table> 811 <tr> 812 <th>API level</th> 813 <th>Extension<br /> is available</th> 814 <th>Extension<br /> is recommended</th> 815 <th>Alternatives</th> 816 </tr> 817 <tr> 818 <td>13<br /> and below</td> 819 <td>no</td> 820 <td>N/A</td> 821 <td>open source codec with suitable license</td> 822 </tr> 823 <tr> 824 <td>14 to 15</td> 825 <td>yes</td> 826 <td>no</td> 827 <td>open source codec with suitable license</td> 828 </tr> 829 <tr> 830 <td>16 to 20</td> 831 <td>yes</td> 832 <td>no</td> 833 <td> 834 <a href="http://developer.android.com/reference/android/media/MediaCodec.html">android.media.MediaCodec</a><br /> 835 or open source codec with suitable license 836 </td> 837 </tr> 838 <tr> 839 <td>21<br /> and above</td> 840 <td>yes</td> 841 <td>no</td> 842 <td> 843 NDK MediaCodec in <media/NdkMedia*.h><br /> 844 or <a href="http://developer.android.com/reference/android/media/MediaCodec.html">android.media.MediaCodec</a><br /> 845 or open source codec with suitable license 846 </td> 847 </tr> 848 </table> 849 850 <p> 851 A standard audio player plays back to an audio device, and the data sink 852 is specified as an output mix. 853 However, as an Android extension, an audio player instead 854 acts as a decoder if the data source is specified as a URI or Android 855 file descriptor data locator with MIME data format, and the data sink is 856 an Android simple buffer queue data locator with PCM data format. 857 </p> 858 <p> 859 This feature is primarily intended for games to pre-load their 860 audio assets when changing to a new game level, similar to 861 <code>android.media.SoundPool</code>. 862 </p> 863 <p> 864 The application should initially enqueue a set of empty buffers to the Android simple 865 buffer queue. These buffers will be filled with the decoded PCM data. The Android simple 866 buffer queue callback is invoked after each buffer is filled. The 867 callback handler should process the PCM data, re-enqueue the 868 now-empty buffer, and then return. The application is responsible for 869 keeping track of decoded buffers; the callback parameter list does not include 870 sufficient information to indicate which buffer was filled or which buffer to enqueue next. 871 </p> 872 <p> 873 The end of stream is determined implicitly by the data source. 874 At the end of stream a <code>SL_PLAYEVENT_HEADATEND</code> event is 875 delivered. The Android simple buffer queue callback will no longer 876 be called after all consumed data is decoded. 877 </p> 878 <p> 879 The sink's PCM data format typically matches that of the encoded data source 880 with respect to sample rate, channel count, and bit depth. However, the platform 881 implementation is permitted to decode to a different sample rate, channel count, or bit depth. 882 There is a provision to detect the actual PCM format; see section "Determining 883 the format of decoded PCM data via metadata" below. 884 </p> 885 <p> 886 Decode to PCM supports pause and initial seek. Volume control, effects, 887 looping, and playback rate are not supported. 888 </p> 889 <p> 890 Depending on the platform implementation, decoding may require resources 891 that cannot be left idle. Therefore it is not recommended to starve the 892 decoder by failing to provide a sufficient number of empty PCM buffers, 893 e.g. by returning from the Android simple buffer queue callback without 894 enqueueing another empty buffer. The result of decoder starvation is 895 unspecified; the implementation may choose to either drop the decoded 896 PCM data, pause the decoding process, or in severe cases terminate 897 the decoder. 898 </p> 899 900 <h3 id="decodeStreaming">Decode streaming ADTS AAC to PCM</h3> 901 902 <p> 903 Note: this feature is available at API level 14 and higher. 904 </p> 905 <p> 906 An audio player acts as a streaming decoder if the data source is an 907 Android buffer queue data locator with MIME data format, and the data 908 sink is an Android simple buffer queue data locator with PCM data format. 909 The MIME data format should be configured as: 910 </p> 911 <dl> 912 <dt>container</dt> 913 <dd><code>SL_CONTAINERTYPE_RAW</code> 914 </dd> 915 <dt>MIME type string 916 </dt> 917 <dd><code>"audio/vnd.android.aac-adts"</code> (macro <code>SL_ANDROID_MIME_AACADTS</code>) 918 </dd> 919 </dl> 920 <p> 921 This feature is primarily intended for streaming media applications that 922 deal with AAC audio, but need to apply custom processing of the audio 923 prior to playback. Most applications that need to decode audio to PCM 924 should use the method of the previous section "Decode audio to PCM", 925 as it is simpler and handles more audio formats. The technique described 926 here is a more specialized approach, to be used only if both of these 927 conditions apply: 928 </p> 929 <ul> 930 <li>the compressed audio source is a stream of AAC frames contained by ADTS headers 931 </li> 932 <li>the application manages this stream, that is the data is <i>not</i> located within 933 a network resource identified by URI or within a local file identified by file descriptor. 934 </li> 935 </ul> 936 <p> 937 The application should initially enqueue a set of filled buffers to the Android buffer queue. 938 Each buffer contains one or more complete ADTS AAC frames. 939 The Android buffer queue callback is invoked after each buffer is emptied. 940 The callback handler should re-fill and re-enqueue the buffer, and then return. 941 The application need not keep track of encoded buffers; the callback parameter 942 list does include sufficient information to indicate which buffer to enqueue next. 943 The end of stream is explicitly marked by enqueuing an EOS item. 944 After EOS, no more enqueues are permitted. 945 </p> 946 <p> 947 It is not recommended to starve the decoder by failing to provide full 948 ADTS AAC buffers, e.g. by returning from the Android buffer queue callback 949 without enqueueing another full buffer. The result of decoder starvation 950 is unspecified. 951 </p> 952 <p> 953 In all respects except for the data source, the streaming decode method is similar 954 to that of the previous section: 955 </p> 956 <ul> 957 <li>initially enqueue a set of empty buffers to the Android simple buffer queue 958 </li> 959 <li>the Android simple buffer queue callback is invoked after each buffer is filled with PCM data; 960 the callback handler should process the PCM data and then re-enqueue another empty buffer 961 </li> 962 <li>the <code>SL_PLAYEVENT_HEADATEND</code> event is delivered at end of stream 963 </li> 964 <li>the actual PCM format should be detected using metadata rather than by making an assumption 965 </li> 966 <li>the same limitations apply with respect to volume control, effects, etc. 967 </li> 968 <li>starvation for lack of empty PCM buffers is not recommended 969 </li> 970 </ul> 971 <p> 972 Despite the similarity in names, an Android buffer queue is <i>not</i> 973 the same as an Android simple buffer queue. The streaming decoder 974 uses both kinds of buffer queues: an Android buffer queue for the ADTS 975 AAC data source, and an Android simple buffer queue for the PCM data 976 sink. The Android simple buffer queue API is described in this document 977 in section "Android simple buffer queue data locator and interface". 978 The Android buffer queue API is described in the Android native media 979 API documentation, located in $NDK_ROOT/docs/Additional_library_docs/openmaxal/index.html 980 </p> 981 982 <h3 id="determiningFormat">Determining the format of decoded PCM data via metadata</h3> 983 984 <p> 985 The metadata extraction interface <code>SLMetadataExtractionItf</code> 986 is a standard OpenSL ES 1.0.1 interface, not an Android extension. 987 However, the particular metadata keys that 988 indicate the actual format of decoded PCM data are specific to Android, 989 and are defined in header <code>SLES/OpenSLES_AndroidMetadata.h</code>. 990 </p> 991 <p> 992 The metadata key indices are available immediately after 993 <code>Object::Realize</code>. Yet the associated values are not 994 available until after the first encoded data has been decoded. A good 995 practice is to query for the key indices in the main thread after Realize, 996 and to read the PCM format metadata values in the Android simple 997 buffer queue callback handler the first time it is called. 998 </p> 999 <p> 1000 The OpenSL ES 1.0.1 metadata extraction interface 1001 <code>SLMetadataExtractionItf</code> is admittedly cumbersome, as it 1002 requires a multi-step process to first determine key indices and then 1003 to get the key values. Consult the example code for snippets showing 1004 how to work with this interface. 1005 </p> 1006 <p> 1007 Metadata key names are stable. But the key indices are not documented 1008 and are subject to change. An application should not assume that indices 1009 are persistent across different execution runs, and should not assume that 1010 indices are shared for different object instances within the same run. 1011 </p> 1012 1013 <h3 id="floatingPoint">Floating-point data</h3> 1014 1015 <p> 1016 As of API level 21 and above, data can be supplied to an AudioPlayer in 1017 single-precision floating-point format. 1018 </p> 1019 <p> 1020 Example code fragment, to be used during the Engine::CreateAudioPlayer process: 1021 </p> 1022 <pre> 1023 #include <SLES/OpenSLES_Android.h> 1024 ... 1025 SLAndroidDataFormat_PCM_EX pcm; 1026 pcm.formatType = SL_ANDROID_DATAFORMAT_PCM_EX; 1027 pcm.numChannels = 2; 1028 pcm.sampleRate = SL_SAMPLINGRATE_44_1; 1029 pcm.bitsPerSample = 32; 1030 pcm.containerSize = 32; 1031 pcm.channelMask = SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT; 1032 pcm.endianness = SL_BYTEORDER_LITTLEENDIAN; 1033 pcm.representation = SL_ANDROID_PCM_REPRESENTATION_FLOAT; 1034 ... 1035 SLDataSource audiosrc; 1036 audiosrc.pLocator = ... 1037 audiosrc.pFormat = &pcm; 1038 </pre> 1039 1040 <h2 id="programmingNotes">Programming notes</h2> 1041 1042 <p> 1043 These notes supplement the OpenSL ES 1.0.1 specification, 1044 available in the "References" section below. 1045 </p> 1046 1047 <h3 id="objectsInitialization">Objects and interface initialization</h3> 1048 1049 <p> 1050 Two aspects of the OpenSL ES programming model that may be unfamiliar 1051 to new developers are the distinction between objects and interfaces, 1052 and the initialization sequence. 1053 </p> 1054 <p> 1055 Briefly, an OpenSL ES object is similar to the object concept 1056 in programming languages such as Java and C++, except an OpenSL ES 1057 object is <i>only</i> visible via its associated interfaces. This 1058 includes the initial interface for all objects, called 1059 <code>SLObjectItf</code>. There is no handle for an object itself, 1060 only a handle to the <code>SLObjectItf</code> interface of the object. 1061 </p> 1062 <p> 1063 An OpenSL ES object is first "created", which returns an 1064 <code>SLObjectItf</code>, then "realized". This is similar to the 1065 common programming pattern of first constructing an object (which 1066 should never fail other than for lack of memory or invalid parameters), 1067 and then completing initialization (which may fail due to lack of 1068 resources). The realize step gives the implementation a 1069 logical place to allocate additional resources if needed. 1070 </p> 1071 <p> 1072 As part of the API to create an object, an application specifies 1073 an array of desired interfaces that it plans to acquire later. Note 1074 that this array does <i>not</i> automatically acquire the interfaces; 1075 it merely indicates a future intention to acquire them. Interfaces 1076 are distinguished as "implicit" or "explicit". An explicit interface 1077 <i>must</i> be listed in the array if it will be acquired later. 1078 An implicit interface need not be listed in the object create array, 1079 but there is no harm in listing it there. OpenSL ES has one more 1080 kind of interface called "dynamic", which does not need to be 1081 specified in the object create array, and can be added later after 1082 the object is created. The Android implementation provides a 1083 convenience feature to avoid this complexity; see section "Dynamic 1084 interfaces at object creation" above. 1085 </p> 1086 <p> 1087 After the object is created and realized, the application should 1088 acquire interfaces for each feature it needs, using 1089 <code>GetInterface</code> on the initial <code>SLObjectItf</code>. 1090 </p> 1091 <p> 1092 Finally, the object is available for use via its interfaces, though 1093 note that some objects require further setup. In particular, an 1094 audio player with URI data source needs a bit more preparation in 1095 order to detect connection errors. See the next section 1096 "Audio player prefetch" for details. 1097 </p> 1098 <p> 1099 After your application is done with the object, you should explicitly 1100 destroy it; see section "Destroy" below. 1101 </p> 1102 1103 <h3 id="audioPlayerPrefetch">Audio player prefetch</h3> 1104 1105 <p> 1106 For an audio player with URI data source, <code>Object::Realize</code> allocates resources 1107 but does not connect to the data source (i.e. "prepare") or begin 1108 pre-fetching data. These occur once the player state is set to 1109 either <code>SL_PLAYSTATE_PAUSED</code> or <code>SL_PLAYSTATE_PLAYING</code>. 1110 </p> 1111 <p> 1112 Note that some information may still be unknown until relatively 1113 late in this sequence. In particular, initially 1114 <code>Player::GetDuration</code> will return <code>SL_TIME_UNKNOWN</code> 1115 and <code>MuteSolo::GetChannelCount</code> will either return successfully 1116 with channel count zero 1117 or the error result <code>SL_RESULT_PRECONDITIONS_VIOLATED</code>. 1118 These APIs will return the proper values once they are known. 1119 </p> 1120 <p> 1121 Other properties that are initially unknown include the sample rate 1122 and actual media content type based on examining the content's header 1123 (as opposed to the application-specified MIME type and container type). 1124 These too, are determined later during prepare / prefetch, but there are 1125 no APIs to retrieve them. 1126 </p> 1127 <p> 1128 The prefetch status interface is useful for detecting when all 1129 information is available. Or, your application can poll periodically. 1130 Note that some information may <i>never</i> be known, for example, 1131 the duration of a streaming MP3. 1132 </p> 1133 <p> 1134 The prefetch status interface is also useful for detecting errors. 1135 Register a callback and enable at least the 1136 <code>SL_PREFETCHEVENT_FILLLEVELCHANGE</code> and 1137 <code>SL_PREFETCHEVENT_STATUSCHANGE</code> events. If both of these 1138 events are delivered simultaneously, and 1139 <code>PrefetchStatus::GetFillLevel</code> reports a zero level, and 1140 <code>PrefetchStatus::GetPrefetchStatus</code> reports 1141 <code>SL_PREFETCHSTATUS_UNDERFLOW</code>, then this indicates a 1142 non-recoverable error in the data source. 1143 This includes the inability to connect to the data source because 1144 the local filename does not exist or the network URI is invalid. 1145 </p> 1146 <p> 1147 The next version of OpenSL ES is expected to add more explicit 1148 support for handling errors in the data source. However, for future 1149 binary compatibility, we intend to continue to support the current 1150 method for reporting a non-recoverable error. 1151 </p> 1152 <p> 1153 In summary, a recommended code sequence is: 1154 </p> 1155 <ol> 1156 <li>Engine::CreateAudioPlayer 1157 </li> 1158 <li>Object:Realize 1159 </li> 1160 <li>Object::GetInterface for SL_IID_PREFETCHSTATUS 1161 </li> 1162 <li>PrefetchStatus::SetCallbackEventsMask 1163 </li> 1164 <li>PrefetchStatus::SetFillUpdatePeriod 1165 </li> 1166 <li>PrefetchStatus::RegisterCallback 1167 </li> 1168 <li>Object::GetInterface for SL_IID_PLAY 1169 </li> 1170 <li>Play::SetPlayState to SL_PLAYSTATE_PAUSED or SL_PLAYSTATE_PLAYING 1171 </li> 1172 <li>preparation and prefetching occur here; during this time your 1173 callback will be called with periodic status updates 1174 </li> 1175 </ol> 1176 1177 <h3 id="destroy">Destroy</h3> 1178 1179 <p> 1180 Be sure to destroy all objects on exit from your application. Objects 1181 should be destroyed in reverse order of their creation, as it is 1182 not safe to destroy an object that has any dependent objects. 1183 For example, destroy in this order: audio players and recorders, 1184 output mix, then finally the engine. 1185 </p> 1186 <p> 1187 OpenSL ES does not support automatic garbage collection or 1188 <a href="http://en.wikipedia.org/wiki/Reference_counting">reference counting</a> 1189 of interfaces. After you call <code>Object::Destroy</code>, all extant 1190 interfaces derived from the associated object become <i>undefined</i>. 1191 </p> 1192 <p> 1193 The Android OpenSL ES implementation does not detect the incorrect 1194 use of such interfaces. 1195 Continuing to use such interfaces after the object is destroyed will 1196 cause your application to crash or behave in unpredictable ways. 1197 </p> 1198 <p> 1199 We recommend that you explicitly set both the primary object interface 1200 and all associated interfaces to NULL as part of your object 1201 destruction sequence, to prevent the accidental misuse of a stale 1202 interface handle. 1203 </p> 1204 1205 <h3 id="stereoPanning">Stereo panning</h3> 1206 1207 <p> 1208 When <code>Volume::EnableStereoPosition</code> is used to enable 1209 stereo panning of a mono source, there is a 3 dB reduction in total 1210 <a href="http://en.wikipedia.org/wiki/Sound_power_level"> 1211 sound power level</a>. This is needed to permit the total sound 1212 power level to remain constant as the source is panned from one 1213 channel to the other. Therefore, don't enable stereo positioning 1214 if you don't need it. See the Wikipedia article on 1215 <a href="http://en.wikipedia.org/wiki/Panning_(audio)">audio panning</a> 1216 for more information. 1217 </p> 1218 1219 <h3 id="callbacksThreads">Callbacks and threads</h3> 1220 1221 <p> 1222 Callback handlers are generally called <i>synchronously</i> with 1223 respect to the event, that is, at the moment and location where the 1224 event is detected by the implementation. But this point is 1225 <i>asynchronous</i> with respect to the application. Thus you should 1226 use a non-blocking synchronization mechanism to control access 1227 to any variables shared between the application and the callback 1228 handler. In the example code, such as for buffer queues, we have 1229 either omitted this synchronization or used blocking synchronization in 1230 the interest of simplicity. However, proper non-blocking synchronization 1231 would be critical for any production code. 1232 </p> 1233 <p> 1234 Callback handlers are called from internal 1235 non-application thread(s) which are not attached to the Android runtime and thus 1236 are ineligible to use JNI. Because these internal threads are 1237 critical to the integrity of the OpenSL ES implementation, a callback 1238 handler should also not block or perform excessive work. 1239 </p> 1240 <p> 1241 If your callback handler needs to use JNI, or execute work that is 1242 not proportional to the callback, the handler should instead post an 1243 event for another thread to process. Examples of acceptable callback 1244 workload include rendering and enqueuing the next output buffer (for an 1245 AudioPlayer), processing the just-filled input buffer and enqueueing the 1246 next empty buffer (for an AudioRecorder), or simple APIs such as most 1247 of the "Get" family. See section "Performance" below regarding the workload. 1248 </p> 1249 <p> 1250 Note that the converse is safe: an Android application thread which has 1251 entered JNI is allowed to directly call OpenSL ES APIs, including 1252 those which block. However, blocking calls are not recommended from 1253 the main thread, as they may result in "Application Not 1254 Responding" (ANR). 1255 </p> 1256 <p> 1257 The choice of which thread calls a callback handler is largely left up 1258 to the implementation. The reason for this flexibility is to permit 1259 future optimizations, especially on multi-core devices. 1260 </p> 1261 <p> 1262 The thread on which the callback handler runs is not guaranteed to have 1263 the same identity across different calls. Therefore do not rely on the 1264 <code>pthread_t</code> returned by <code>pthread_self()</code>, or the 1265 <code>pid_t</code> returned by <code>gettid()</code>, to be consistent 1266 across calls. Don't use the thread local storage (TLS) APIs such as 1267 <code>pthread_setspecific()</code> and <code>pthread_getspecific()</code> 1268 from a callback, for the same reason. 1269 </p> 1270 <p> 1271 The implementation guarantees that concurrent callbacks of the same kind, 1272 for the same object, will not occur. However, concurrent callbacks of 1273 <i>different</i> kinds for the same object are possible, on different threads. 1274 </p> 1275 1276 <h3 id="performance">Performance</h3> 1277 1278 <p> 1279 As OpenSL ES is a native C API, non-runtime application threads which 1280 call OpenSL ES have no runtime-related overhead such as garbage 1281 collection pauses. With one exception described below, there is no additional performance 1282 benefit to the use of OpenSL ES other than this. In particular, use 1283 of OpenSL ES does not guarantee a lower audio latency, higher scheduling 1284 priority, etc. than what the platform generally provides. 1285 On the other hand, as the Android platform and specific device 1286 implementations continue to evolve, an OpenSL ES application can 1287 expect to benefit from any future system performance improvements. 1288 </p> 1289 <p> 1290 One such evolution is support for reduced audio output latency. 1291 The underpinnings for reduced output latency were first included in 1292 the Android 4.1 platform release ("Jellybean"), and then continued 1293 progress occurred in the Android 4.2 platform. These improvements 1294 are available via OpenSL ES for device implementations that claim feature 1295 "android.hardware.audio.low_latency". If the device doesn't claim this 1296 feature but supports API level 9 (Android platform version 2.3) or later, 1297 then you can still use the OpenSL ES APIs but the output latency may be higher. 1298 The lower output latency path is used only if the application requests a 1299 buffer size and sample rate that are 1300 compatible with the device's native output configuration. 1301 These parameters are device-specific and should be obtained as follows. 1302 </p> 1303 <p> 1304 Beginning with API level 17 (Android platform version 4.2), an application 1305 can query for the platform native or optimal output sample rate and buffer size 1306 for the device's primary output stream. When combined with the feature 1307 test just mentioned, an app can now configure itself appropriately for 1308 lower latency output on devices that claim support. 1309 </p> 1310 <p> 1311 For API level 17 (Android platform version 4.2.2) and earlier, 1312 a buffer count of 2 or more is required for lower latency. 1313 Beginning with API level 18 (Android platform version 4.3), 1314 a buffer count of 1 is sufficient for lower latency. 1315 </p> 1316 <p> 1317 All 1318 OpenSL ES interfaces for output effects preclude the lower latency path. 1319 </p> 1320 <p> 1321 The recommended sequence is: 1322 </p> 1323 <ol> 1324 <li>Check for API level 9 or higher, to confirm use of OpenSL ES. 1325 </li> 1326 <li>Check for feature "android.hardware.audio.low_latency" using code such as this: 1327 <pre> 1328 import android.content.pm.PackageManager; 1329 ... 1330 PackageManager pm = getContext().getPackageManager(); 1331 boolean claimsFeature = pm.hasSystemFeature(PackageManager.FEATURE_AUDIO_LOW_LATENCY); 1332 </pre> 1333 </li> 1334 <li>Check for API level 17 or higher, to confirm use of 1335 <code>android.media.AudioManager.getProperty()</code>. 1336 </li> 1337 <li>Get the native or optimal output sample rate and buffer size for this device's primary output 1338 stream, using code such as this: 1339 <pre> 1340 import android.media.AudioManager; 1341 ... 1342 AudioManager am = (AudioManager) getSystemService(Context.AUDIO_SERVICE); 1343 String sampleRate = am.getProperty(AudioManager.PROPERTY_OUTPUT_SAMPLE_RATE)); 1344 String framesPerBuffer = am.getProperty(AudioManager.PROPERTY_OUTPUT_FRAMES_PER_BUFFER)); 1345 </pre> 1346 Note that <code>sampleRate</code> and <code>framesPerBuffer</code> 1347 are <code>String</code>s. First check for <code>null</code> 1348 and then convert to <code>int</code> using <code>Integer.parseInt()</code>. 1349 </li> 1350 <li>Now use OpenSL ES to create an AudioPlayer with PCM buffer queue data locator. 1351 </li> 1352 </ol> 1353 <p> 1354 The number of lower latency audio players is limited. If your application 1355 requires more than a few audio sources, consider mixing your audio at 1356 application level. Be sure to destroy your audio players when your 1357 activity is paused, as they are a global resource shared with other apps. 1358 </p> 1359 <p> 1360 To avoid audible glitches, the buffer queue callback handler must execute 1361 within a small and predictable time window. This typically implies no 1362 unbounded blocking on mutexes, conditions, or I/O operations. Instead consider 1363 "try locks", locks and waits with timeouts, and non-blocking algorithms. 1364 </p> 1365 <p> 1366 The computation required to render the next buffer (for AudioPlayer) or 1367 consume the previous buffer (for AudioRecord) should take approximately 1368 the same amount of time for each callback. Avoid algorithms that execute in 1369 a non-deterministic amount of time, or are "bursty" in their computations. 1370 A callback computation is bursty if the CPU time spent in any 1371 given callback is significantly larger than the average. 1372 In summary, the ideal is for the CPU execution time of the handler to have 1373 variance near zero, and for the handler to not block for unbounded times. 1374 </p> 1375 <p> 1376 Lower latency audio is possible for these outputs only: on-device speaker, wired 1377 headphones, wired headset, line out, and USB digital audio. 1378 On some devices, speaker latency is higher than other paths due to 1379 digital signal processing for speaker correction and protection. 1380 </p> 1381 <p> 1382 As of API level 21, lower latency audio input is supported on select 1383 devices. To take advantage of this feature, first confirm that lower 1384 latency output is available as described above. The capability for lower 1385 latency output is a prerequisite for the lower latency input feature. 1386 Then create an AudioRecorder with the same sample rate and buffer size 1387 as would be used for output. 1388 OpenSL ES interfaces for input effects preclude the lower latency path. 1389 The record preset SL_ANDROID_RECORDING_PRESET_VOICE_RECOGNITION 1390 must be used for lower latency; this preset disables device-specific 1391 digital signal processing which may add latency to the input path. 1392 For more information on record presets, 1393 see section "Android configuration interface" above. 1394 </p> 1395 <p> 1396 For simultaneous input and output, separate buffer queue 1397 completion handlers are used for each side. There is no guarantee of 1398 the relative order of these callbacks, or the synchronization of the 1399 audio clocks, even when both sides use the same sample rate. 1400 Your application should buffer the data with proper buffer synchronization. 1401 </p> 1402 <p> 1403 One consequence of potentially independent audio clocks is the need 1404 for asynchronous sample rate conversion. A simple (though not ideal 1405 for audio quality) technique for asynchronous sample rate conversion 1406 is to duplicate or drop samples as needed near a zero-crossing point. 1407 More sophisticated conversions are possible. 1408 </p> 1409 1410 <h3 id="securityPermissions">Security and permissions</h3> 1411 1412 <p> 1413 As far as who can do what, security in Android is done at the 1414 process level. Java programming language code can't do anything more than native code, nor 1415 can native code do anything more than Java programming language code. The only differences 1416 between them are what APIs are available. 1417 </p> 1418 <p> 1419 Applications using OpenSL ES must request whatever permissions they 1420 would need for similar non-native APIs. For example, if your application 1421 records audio, then it needs the <code>android.permission.RECORD_AUDIO</code> 1422 permission. Applications that use audio effects need 1423 <code>android.permission.MODIFY_AUDIO_SETTINGS</code>. Applications that play 1424 network URI resources need <code>android.permission.NETWORK</code>. 1425 See <a href="https://developer.android.com/training/permissions/index.html">Working with System Permissions</a> 1426 for more information. 1427 </p> 1428 <p> 1429 Depending on the platform version and implementation, 1430 media content parsers and software codecs may run within the context 1431 of the Android application that calls OpenSL ES (hardware codecs 1432 are abstracted, but are device-dependent). Malformed content 1433 designed to exploit parser and codec vulnerabilities is a known attack 1434 vector. We recommend that you play media only from trustworthy 1435 sources, or that you partition your application such that code that 1436 handles media from untrustworthy sources runs in a relatively 1437 sandboxed environment. For example you could process media from 1438 untrustworthy sources in a separate process. Though both processes 1439 would still run under the same UID, this separation does make an 1440 attack more difficult. 1441 </p> 1442 1443 <h2 id="platformIssues">Platform issues</h2> 1444 1445 <p> 1446 This section describes known issues in the initial platform 1447 release which supports these APIs. 1448 </p> 1449 1450 <h3 id="dynamicInterfaceManagementIssues">Dynamic interface management</h3> 1451 1452 <p> 1453 <code>DynamicInterfaceManagement::AddInterface</code> does not work. 1454 Instead, specify the interface in the array passed to Create, as 1455 shown in the example code for environmental reverb. 1456 </p> 1457 1458 <h2 id="references">References and resources</h2> 1459 1460 <p> 1461 Android: 1462 </p> 1463 <ul> 1464 <li><a href="http://developer.android.com/resources/index.html"> 1465 Android developer resources</a> 1466 </li> 1467 <li><a href="http://groups.google.com/group/android-developers"> 1468 Android developers discussion group</a> 1469 </li> 1470 <li><a href="http://developer.android.com/sdk/ndk/index.html">Android NDK</a> 1471 </li> 1472 <li><a href="http://groups.google.com/group/android-ndk"> 1473 Android NDK discussion group</a> (for developers of native code, including OpenSL ES) 1474 </li> 1475 <li><a href="http://code.google.com/p/android/issues/"> 1476 Android open source bug database</a> 1477 </li> 1478 <li><a href="https://github.com/googlesamples/android-ndk">Android Studio NDK Samples</a> 1479 </li> 1480 <li><a href="http://developer.android.com/samples/index.html">Android Studio Samples</a> 1481 </li> 1482 </ul> 1483 1484 <p> 1485 Khronos Group: 1486 </p> 1487 <ul> 1488 <li><a href="http://www.khronos.org/opensles/"> 1489 Khronos Group OpenSL ES Overview</a> 1490 </li> 1491 <li><a href="http://www.khronos.org/registry/sles/"> 1492 Khronos Group OpenSL ES 1.0.1 specification</a> 1493 </li> 1494 <li><a href="https://forums.khronos.org/forumdisplay.php/91-OpenSL-ES-embedded-audio-acceleration"> 1495 Khronos Group public message board for OpenSL ES</a> 1496 (please limit to non-Android questions) 1497 </li> 1498 </ul> 1499 <p> 1500 For convenience, we have included a copy of the OpenSL ES 1.0.1 1501 specification with the NDK in 1502 <code>docs/opensles/OpenSL_ES_Specification_1.0.1.pdf</code>. 1503 </p> 1504 1505 <p> 1506 Miscellaneous: 1507 </p> 1508 <ul> 1509 <li><a href="http://en.wikipedia.org/wiki/Java_Native_Interface">JNI</a> 1510 </li> 1511 <li><a href="http://stackoverflow.com/search?q=android+audio"> 1512 Stack Overflow</a> 1513 </li> 1514 <li>web search for "interactive audio", "game audio", "sound design", 1515 "audio programming", "audio content", "audio formats", etc. 1516 </li> 1517 <li><a href="http://en.wikipedia.org/wiki/Advanced_Audio_Coding">AAC</a> 1518 </li> 1519 </ul> 1520 1521 </body> 1522 </html> 1523
