1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" 2 "http://www.w3.org/TR/html4/loose.dtd"> 3 <html> 4 5 <head> 6 <title>OpenSL ES for Android</title> 7 </head> 8 9 <body> 10 11 <h1>OpenSL ES for Android</h1> 12 13 This article describes the Android native audio APIs based on the 14 Khronos Group OpenSL ES™ 1.0.1 standard, as of Android API level 9 (Android 15 version 2.3). 16 <p> 17 OpenSL ES provides a C language interface that is also callable from C++, and 18 exposes features similar to the audio portions of these Android APIs 19 callable from Java programming language code: 20 <ul> 21 <li><a href="http://developer.android.com/reference/android/media/MediaPlayer.html"> 22 android.media.MediaPlayer</a> 23 <li><a href="http://developer.android.com/reference/android/media/MediaRecorder.html"> 24 android.media.MediaRecorder</a> 25 </ul> 26 27 As with all of the Android Native Development Kit (NDK), the primary 28 purpose of OpenSL ES for Android is to facilitate the implementation 29 of shared libraries to be called from Java programming language code via Java Native 30 Interface (JNI). NDK is not intended for writing pure C/C++ 31 applications. That said, OpenSL ES is a full-featured API, and we 32 expect that you should be able to accomplish most of your audio 33 needs using only this API, without up-calls to Java. 34 35 <p> 36 Note: though based on OpenSL ES, Android native audio at API level 9 37 is <i>not</i> a conforming implementation of any OpenSL ES 1.0.1 38 profile (game, music, or phone). This is because Android does not 39 implement all of the features required by any one of the profiles. 40 Any known cases where Android behaves differently than the specification 41 are described in section "Android extensions" below. 42 43 <h2>Getting started</h2> 44 45 <h3>Example code</h3> 46 47 <h4>Recommended</h4> 48 49 Supported and tested example code, usable as a model 50 for your own code, is located in NDK folder 51 <code>platforms/android-9/samples/native-audio/</code>. 52 53 <h4>Not recommended</h4> 54 55 The OpenSL ES 1.0.1 specification contains example code in the 56 appendices (see section "References" below for the link to this 57 specification). However, the examples in Appendix B: Sample Code 58 and Appendix C: Use Case Sample Code use features 59 not supported by Android API level 9. Some examples also contain 60 typographical errors, or use APIs that are likely to change. 61 Proceed with caution in referring to these; 62 though the code may be helpful in understanding the full OpenSL ES 63 standard, it should not be used as is with Android. 64 65 <h3>Adding OpenSL ES to your application source code</h3> 66 67 OpenSL ES is a C API, but is callable from both C and C++ code. 68 <p> 69 At a minimum, add the following line to your code: 70 <pre> 71 #include <SLES/OpenSLES.h> 72 </pre> 73 74 If you use Android extensions, also include these headers: 75 <pre> 76 #include <SLES/OpenSLES_Android.h> 77 #include <SLES/OpenSLES_AndroidConfiguration.h> 78 </pre> 79 80 <h3>Makefile</h3> 81 82 Modify your Android.mk as follows: 83 <pre> 84 LOCAL_LDLIBS += libOpenSLES 85 </pre> 86 87 <h3>Audio content</h3> 88 89 There are many ways to package audio content for your 90 application, including: 91 92 <dl> 93 94 <dt>Resources</dt> 95 <dd> 96 By placing your audio files into the <code>res/raw/</code> folder, 97 they can be accessed easily by the associated APIs for 98 <a href="http://developer.android.com/reference/android/content/res/Resources.html"> 99 Resources</a>. However there is no direct native access to resources, 100 so you will need to write Java programming language code to copy them out before use. 101 </dd> 102 103 <dt>Assets</dt> 104 <dd> 105 By placing your audio files into the <code>assets/</code> folder, 106 they will be directly accessible by the Android native asset manager 107 APIs. See the header files <code>android/asset_manager.h</code> 108 and <code>android/asset_manager_jni.h</code> for more information 109 on these APIs, which are new for API level 9. The example code 110 located in NDK folder 111 <code>platforms/android-9/samples/native-audio/</code> uses these 112 native asset manager APIs in conjunction with the Android file 113 descriptor data locator. 114 </dd> 115 116 <dt>Network</dt> 117 <dd> 118 You can use the URI data locator to play audio content directly from the 119 network. However, be sure to read section "Security and permissions" below. 120 </dd> 121 122 <dt>Local filesystem</dt> 123 <dd> 124 The URI data locator supports the <code>file:</code> scheme for local files, 125 provided the files are accessible by the application. 126 Note that the Android security framework restricts file access via 127 the Linux user ID and group ID mechanism. 128 </dd> 129 130 <dt>Recorded</dt> 131 <dd>Your application can record audio data from the microphone input, 132 store this content, and then play it back later. 133 The example code uses this method for the "Playback" clip. 134 </dd> 135 136 <dt>Compiled and linked inline</dt> 137 <dd> 138 You can link your audio content directly into the shared library, 139 and then play it using an audio player with buffer queue data locator. This is most 140 suitable for short PCM format clips. The example code uses this 141 technique for the "Hello" and "Android" clips. The PCM data was 142 converted to hex strings using a <code>bin2c</code> tool (not supplied). 143 </dd> 144 145 <dt>Synthesis</dt> 146 <dd> 147 Your application can synthesize PCM data on the fly and then play it 148 using an audio player with buffer queue data locator. This is a 149 relatively advanced technique, and the details of audio synthesis 150 are beyond the scope of this article. 151 </dd> 152 153 </dl> 154 155 Finding or creating useful audio content for your application is 156 beyond the scope of this article, but see the "References" section 157 below for some suggested web search terms. 158 <p> 159 Note that it is your responsibility to ensure that you are legally 160 permitted to play or record content, and that there may be privacy 161 considerations for recording content. 162 163 <h3>Debugging</h3> 164 165 For robustness, we recommend that you examine the <code>SLresult</code> 166 value which is returned by most APIs. Use of <code>assert</code> 167 vs. more advanced error handling logic is a matter of coding style 168 and the particular API; see the Wikipedia article on 169 <a href="http://en.wikipedia.org/wiki/Assertion_(computing)">assert</a> 170 for more information. In the supplied example, we have used <code>assert</code> 171 for "impossible" conditions which would indicate a coding error, and 172 explicit error handling for others which are more likely to occur 173 in production. 174 <p> 175 Many API errors result in a log entry, in addition to the non-zero 176 result code. These log entries provide additional detail which can 177 be especially useful for the more complex APIs such as 178 <code>Engine::CreateAudioPlayer</code>. 179 <p> 180 Use <a href="http://developer.android.com/guide/developing/tools/adb.html"> 181 adb logcat</a>, the 182 <a href="http://developer.android.com/guide/developing/eclipse-adt.html"> 183 Eclipse ADT plugin</a> LogCat pane, or 184 <a href="http://developer.android.com/guide/developing/tools/ddms.html#logcat"> 185 ddms logcat</a> to see the log. 186 187 <h2>Supported features from OpenSL ES 1.0.1</h2> 188 189 This section summarizes available features in this API level. In some 190 cases, there are limitations which are described in the next 191 sub-section. 192 193 <h3>Global entry points</h3> 194 195 Supported global entry points: 196 <ul> 197 <li><code>slCreateEngine</code> 198 <li><code>slQueryNumSupportedEngineInterfaces</code> 199 <li><code>slQuerySupportedEngineInterfaces</code> 200 </ul> 201 202 <h3>Objects and interfaces</h3> 203 204 The following figure indicates objects and interfaces supported by 205 Android's OpenSL ES implementation. A green cell means the feature 206 is supported. 207 208 <p> 209 <img src="chart1.png" alt="Supported objects and interfaces"> 210 211 <h3>Limitations</h3> 212 213 This section details limitations with respect to the supported 214 objects and interfaces from the previous section. 215 216 <h4>Buffer queue data locator</h4> 217 218 An audio player or recorder with buffer queue data locator supports 219 PCM data format only. 220 221 <h4>Device data locator</h4> 222 223 The only supported use of an I/O device data locator is when it is 224 specified as the data source for <code>Engine::CreateAudioRecorder</code>. 225 It should be initialized using these values, as shown in the example: 226 <pre> 227 SLDataLocator_IODevice loc_dev = 228 {SL_DATALOCATOR_IODEVICE, SL_IODEVICE_AUDIOINPUT, 229 SL_DEFAULTDEVICEID_AUDIOINPUT, NULL}; 230 </pre> 231 232 <h4>Dynamic interface management</h4> 233 234 <code>RemoveInterface</code> and <code>ResumeInterface</code> are not supported. 235 236 <h4>Effect combinations</h4> 237 238 It is meaningless to have both environmental reverb and preset 239 reverb on the same output mix. 240 <p> 241 The platform may ignore effect requests if it estimates that the 242 CPU load would be too high. 243 244 <h4>Effect send</h4> 245 246 <code>SetSendLevel</code> supports a single send level per audio player. 247 248 <h4>Environmental reverb</h4> 249 250 Environmental reverb does not support the <code>reflectionsDelay</code>, 251 <code>reflectionsLevel</code>, or <code>reverbDelay</code> fields of 252 <code>struct SLEnvironmentalReverbSettings</code>. 253 254 <h4>MIME data format</h4> 255 256 The MIME data format can be used with URI data locator only, and only 257 for player (not recorder). 258 <p> 259 The Android implementation of OpenSL ES requires that <code>mimeType</code> 260 be initialized to either <code>NULL</code> or a valid UTF-8 string, 261 and that <code>containerType</code> be initialized to a valid value. 262 In the absence of other considerations, such as portability to other 263 implementations, or content format which cannot be identified by header, 264 we recommend that you 265 set the <code>mimeType</code> to <code>NULL</code> and <code>containerType</code> 266 to <code>SL_CONTAINERTYPE_UNSPECIFIED</code>. 267 <p> 268 Supported formats include WAV PCM, WAV alaw, WAV ulaw, MP3, Ogg 269 Vorbis, AAC LC, HE-AACv1 (aacPlus), HE-AACv2 (enhanced aacPlus), 270 and AMR [provided these are supported by the overall platform, 271 and AAC formats must be located within an MP4 container]. 272 MIDI is not supported. 273 WMA is not part of the open source release, and compatibility 274 with Android OpenSL ES has not been verified. 275 <p> 276 The Android implementation of OpenSL ES does not support direct 277 playback of DRM or encrypted content; if you want to play this, you 278 will need to convert to cleartext in your application before playing, 279 and enforce any DRM restrictions in your application. 280 281 <h4>Object</h4> 282 283 <code>Resume</code>, <code>RegisterCallback</code>, 284 <code>AbortAsyncOperation</code>, <code>SetPriority</code>, 285 <code>GetPriority</code>, and <code>SetLossOfControlInterfaces</code> 286 are not supported. 287 288 <h4>PCM data format</h4> 289 290 The PCM data format can be used with buffer queues only. Supported PCM 291 playback configurations are 8-bit unsigned or 16-bit signed, mono 292 or stereo, little endian byte ordering, and these sample rates: 293 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, or 48000 Hz. 294 For recording, the supported configurations are device-dependent, 295 however generally 16000 Hz mono 16-bit signed is usually available. 296 <p> 297 Note that the field <code>samplesPerSec</code> is actually in 298 units of milliHz, despite the misleading name. To avoid accidentally 299 using the wrong value, you should initialize this field using one 300 of the symbolic constants defined for this purpose (such as 301 <code>SL_SAMPLINGRATE_44_1</code> etc.) 302 303 <h4>Playback rate</h4> 304 305 A single playback rate range from 500 per mille to 2000 per mille 306 inclusive is supported, with property 307 <code>SL_RATEPROP_NOPITCHCORAUDIO</code>. 308 309 <h4>Record</h4> 310 311 The <code>SL_RECORDEVENT_HEADATLIMIT</code> and 312 <code>SL_RECORDEVENT_HEADMOVING</code> events are not supported. 313 314 <h4>Seek</h4> 315 316 <code>SetLoop</code> enables whole file looping. The <code>startPos</code> 317 and <code>endPos</code> parameters are ignored. 318 319 <h4>URI data locator</h4> 320 321 The URI data locator can be used with MIME data format only, and 322 only for an audio player (not audio recorder). Supported schemes 323 are <code>http:</code> and <code>file:</code>. 324 A missing scheme defaults to the <code>file:</code> scheme. Other 325 schemes such as <code>https:</code>, <code>ftp:</code>, and 326 <code>content:</code> are not supported. 327 <code>rtsp:</code> is not verified. 328 329 <h3>Data structures</h3> 330 331 Android API level 9 supports these OpenSL ES 1.0.1 data structures: 332 <ul> 333 <li>SLDataFormat_MIME 334 <li>SLDataFormat_PCM 335 <li>SLDataLocator_BufferQueue 336 <li>SLDataLocator_IODevice 337 <li>SLDataLocator_OutputMix 338 <li>SLDataLocator_URI 339 <li>SLDataSink 340 <li>SLDataSource 341 <li>SLEngineOption 342 <li>SLEnvironmentalReverbSettings 343 <li>SLInterfaceID 344 </ul> 345 346 <h3>Platform configuration</h3> 347 348 OpenSL ES for Android is designed for multi-threaded applications, 349 and is thread-safe. 350 <p> 351 OpenSL ES for Android supports a single engine per application, and 352 up to 32 objects. Available device memory and CPU may further 353 restrict the usable number of objects. 354 <p> 355 <code>slCreateEngine</code> recognizes, but ignores, these engine options: 356 <ul> 357 <li><code>SL_ENGINEOPTION_THREADSAFE</code> 358 <li><code>SL_ENGINEOPTION_LOSSOFCONTROL</code> 359 </ul> 360 361 <h2>Planning for future versions of OpenSL ES</h2> 362 363 The Android native audio APIs at level 9 are based on Khronos 364 Group OpenSL ES 1.0.1 (see section "References" below). 365 As of the time of this writing, the OpenSL ES working group 366 is preparing a revised version of the standard. The revised version 367 will likely include new features, clarifications, correction of 368 typographical errors, and some incompatibilities. Most of the expected 369 incompatibilities are relatively minor, or are in areas of OpenSL ES 370 not supported by Android API level 9. However, even a small change 371 can be significant for an application developer, so it important 372 to prepare for this. 373 <p> 374 The Android team is committed to preserving future API binary 375 compatibility for developers to the extent feasible. It is our 376 intention to continue to support future binary compatibility of the 377 1.0.1-based API, even as we add support for later versions of the 378 standard. An application developed with this version should 379 work on future versions of the Android platform, provided that 380 you follow the guidelines listed in section "Planning for 381 binary compatibility" below. 382 <p> 383 Note that future source compatibility will <i>not</i> be a goal. That is, 384 if you upgrade to a newer version of the NDK, you may need to modify 385 your application source code to conform to the new API. We expect 386 that most such changes will be minor; see details below. 387 388 <h3>Planning for binary compatibility</h3> 389 390 We recommend that your application follow these guidelines, 391 to improve future binary compatibility: 392 <ul> 393 <li> 394 Use only the documented subset of Android-supported features from 395 OpenSL ES 1.0.1. 396 <li> 397 Do not depend on a particular result code for an unsuccessful 398 operation; be prepared to deal with a different result code. 399 <li> 400 Application callback handlers generally run in a restricted context, 401 and should be written to perform their work quickly and then return 402 as soon as possible. Do not do complex operations within a callback 403 handler. For example, within a buffer queue completion callback, 404 you can enqueue another buffer, but do not create an audio player. 405 <li> 406 Callback handlers should be prepared to be called more or less 407 frequently, to receive additional event types, and should ignore 408 event types that they do not recognize. Callbacks that are configured 409 with an event mask of enabled event types should be prepared to be 410 called with multiple event type bits set simultaneously. 411 Use "&" to test for each event bit rather than a switch case. 412 <li> 413 Use prefetch status and callbacks as a general indication of progress, but do 414 not depend on specific hard-coded fill levels or callback sequence. 415 The meaning of the prefetch status fill level, and the behavior for 416 errors that are detected during prefetch, may change. 417 <li> 418 See section "Buffer queue behavior" below. 419 </ul> 420 421 <h3>Planning for source compatibility</h3> 422 423 As mentioned, source code incompatibilities are expected in the next 424 version of OpenSL ES from Khronos Group. Likely areas of change include: 425 426 <ul> 427 <li>The buffer queue interface is expected to have significant changes, 428 especially in the areas of <code>BufferQueue::Enqueue</code>, the parameter 429 list for <code>slBufferQueueCallback</code>, 430 and the name of field <code>SLBufferQueueState.playIndex</code>. 431 We recommend that your application code use Android simple buffer 432 queues instead, because we do not plan to change that API. 433 In the example code supplied with the NDK, we have used 434 Android simple buffer queues for playback for this reason. 435 (We also use Android simple buffer queue for recording, but 436 that is because standard OpenSL ES 1.0.1 does not support record to 437 a buffer queue data sink.) 438 <li>Addition of <code>const</code> to input parameters passed by reference, 439 and to <code>SLchar *</code> struct fields used as input values. 440 This should not require any changes to your code. 441 <li>Substitution of unsigned types for some parameters that are 442 currently signed. You may need to change a parameter type from 443 <code>SLint32</code> to <code>SLuint32</code> or similar, or add a cast. 444 <li><code>Equalizer::GetPresetName</code> will copy the string to 445 application memory instead of returning a pointer to implementation 446 memory. This will be a significant change, so we recommend that you 447 either avoid calling this method, or isolate your use of it. 448 <li>Additional fields in struct types. For output parameters, these 449 new fields can be ignored, but for input parameters the new fields 450 will need to be initialized. Fortunately, these are expected to all 451 be in areas not supported by API level 9. 452 <li>Interface 453 <a href="http://en.wikipedia.org/wiki/Globally_unique_identifier"> 454 GUIDs</a> will change. Refer to interfaces by symbolic name rather than GUID 455 to avoid a dependency. 456 <li><code>SLchar</code> will change from <code>unsigned char</code> 457 to <code>char</code>. This primarily affects the URI data locator 458 and MIME data format. 459 <li><code>SLDataFormat_MIME.mimeType</code> will be renamed to <code>pMimeType</code>, 460 and <code>SLDataLocator_URI.URI</code> will be renamed to <code>pURI</code>. 461 We recommend that you initialize the <code>SLDataFormat_MIME</code> 462 and <code>SLDataLocator_URI</code> 463 data structures using a brace-enclosed comma-separated list of values, 464 rather than by field name, to isolate your code from this change. 465 In the example code we have used this technique. 466 <li><code>SL_DATAFORMAT_PCM</code> does not permit the application 467 to specify the representation of the data as signed integer, unsigned 468 integer, or floating-point. The Android implementation assumes that 469 8-bit data is unsigned integer and 16-bit is signed integer. In 470 addition, the field <code>samplesPerSec</code> is a misnomer, as 471 the actual units are milliHz. These issues are expected to be 472 addressed in the next OpenSL ES version, which will introduce a new 473 extended PCM data format that permits the application to explicitly 474 specify the representation, and corrects the field name. As this 475 will be a new data format, and the current PCM data format will 476 still be available (though deprecated), it should not require any 477 immediate changes to your code. 478 </ul> 479 480 Any actual source code incompatibilities will be explained in detail 481 when the time comes. 482 483 <h2>Android extensions</h2> 484 485 The API for Android extensions is defined in <code>SLES/OpenSLES_Android.h</code>. 486 Consult that file for details on these extensions. Unless otherwise 487 noted, all interfaces are "explicit". 488 <p> 489 Note that use these extensions will limit your application's 490 portability to other OpenSL ES implementations. If this is a concern, 491 we advise that you avoid using them, or isolate your use of these 492 with <code>#ifdef</code> etc. 493 <p> 494 The following figure shows which Android-specific interfaces and 495 data locators are available for each object type. 496 497 <p> 498 <img src="chart2.png" alt="Android extensions"> 499 500 <h3>Android configuration interface</h3> 501 502 The Android configuration interface provides a means to set 503 platform-specific parameters for objects. Unlike other OpenSL ES 504 1.0.1 interfaces, the Android configuration interface is available 505 prior to object realization. This permits the object to be configured 506 and then realized. Header file <code>SLES/OpenSLES_AndroidConfiguration.h</code> 507 documents the available configuration keys and values: 508 <ul> 509 <li>stream type for audio players (default <code>SL_ANDROID_STREAM_MEDIA</code>) 510 <li>record profile for audio recorders (default <code>SL_ANDROID_RECORDING_PRESET_GENERIC</code>) 511 </ul> 512 Here is an example code fragment that sets the Android audio stream type on an audio player: 513 <pre> 514 // CreateAudioPlayer and specify SL_IID_ANDROIDCONFIGURATION 515 // in the required interface ID array. Do not realize player yet. 516 // ... 517 SLAndroidConfigurationItf playerConfig; 518 result = (*playerObject)->GetInterface(playerObject, 519 SL_IID_ANDROIDCONFIGURATION, &playerConfig); 520 assert(SL_RESULT_SUCCESS == result); 521 SLint32 streamType = SL_ANDROID_STREAM_ALARM; 522 result = (*playerConfig)->SetConfiguration(playerConfig, 523 SL_ANDROID_KEY_STREAM_TYPE, &streamType, sizeof(SLint32)); 524 assert(SL_RESULT_SUCCESS == result); 525 // ... 526 // Now realize the player here. 527 </pre> 528 Similar code can be used to configure the preset for an audio recorder. 529 530 <h3>Android effects interfaces</h3> 531 532 The Android effect, effect send, and effect capabilities interfaces provide 533 a generic mechanism for an application to query and use device-specific 534 audio effects. A device manufacturer should document any available 535 device-specific audio effects. 536 <p> 537 Portable applications should use the OpenSL ES 1.0.1 APIs 538 for audio effects instead of the Android effect extensions. 539 540 <h3>Android file descriptor data locator</h3> 541 542 The Android file descriptor data locator permits the source for an 543 audio player to be specified as an open file descriptor with read 544 access. The data format must be MIME. 545 <p> 546 This is especially useful in conjunction with the native asset manager. 547 548 <h3>Android simple buffer queue data locator and interface</h3> 549 550 The Android simple buffer queue data locator and interface are 551 identical to the OpenSL ES 1.0.1 buffer queue locator and interface, 552 except that Android simple buffer queues may be used with both audio 553 players and audio recorders, and are limited to PCM data format. 554 [OpenSL ES 1.0.1 buffer queues are for audio players only, and are not 555 restricted to PCM data format.] 556 <p> 557 For recording, the application should enqueue empty buffers. Upon 558 notification of completion via a registered callback, the filled 559 buffer is available for the application to read. 560 <p> 561 For playback there is no difference. But for future source code 562 compatibility, we suggest that applications use Android simple 563 buffer queues instead of OpenSL ES 1.0.1 buffer queues. 564 565 <h3>Dynamic interfaces at object creation</h3> 566 567 For convenience, the Android implementation of OpenSL ES 1.0.1 568 permits dynamic interfaces to be specified at object creation time, 569 as an alternative to adding these interfaces after object creation 570 with <code>DynamicInterfaceManagement::AddInterface</code>. 571 572 <h3>Buffer queue behavior</h3> 573 574 The OpenSL ES 1.0.1 specification requires that "On transition to 575 the <code>SL_PLAYSTATE_STOPPED</code> state the play cursor is 576 returned to the beginning of the currently playing buffer." The 577 Android implementation does not necessarily conform to this 578 requirement. For Android, it is unspecified whether a transition 579 to <code>SL_PLAYSTATE_STOPPED</code> operates as described, or 580 leaves the play cursor unchanged. 581 <p> 582 We recommend that you do not rely on either behavior; after a 583 transition to <code>SL_PLAYSTATE_STOPPED</code>, you should explicitly 584 call <code>BufferQueue::Clear</code>. This will place the buffer 585 queue into a known state. 586 <p> 587 A corollary is that it is unspecified whether buffer queue callbacks 588 are called upon transition to <code>SL_PLAYSTATE_STOPPED</code> or by 589 <code>BufferQueue::Clear</code>. 590 We recommend that you do not rely on either behavior; be prepared 591 to receive a callback in these cases, but also do not depend on 592 receiving one. 593 <p> 594 It is expected that a future version of OpenSL ES will clarify these 595 issues. However, upgrading to that version would result in source 596 code incompatibilities (see section "Planning for source compatibility" 597 above). 598 599 <h3>Reporting of extensions</h3> 600 601 <code>Engine::QueryNumSupportedExtensions</code>, 602 <code>Engine::QuerySupportedExtension</code>, 603 <code>Engine::IsExtensionSupported</code> report these extensions: 604 <ul> 605 <li><code>ANDROID_SDK_LEVEL_9</code> 606 (platforms that support a higher API level will report an extension of 607 <code>ANDROID_SDK_LEVEL_#</code> where # is that API level) 608 </ul> 609 610 <h2>Programming notes</h2> 611 612 These notes supplement the OpenSL ES 1.0.1 specification, 613 available in the "References" section below. 614 615 <h3>Objects and interface initialization</h3> 616 617 Two aspects of the OpenSL ES programming model that may be unfamiliar 618 to new developers are the distinction between objects and interfaces, 619 and the initialization sequence. 620 <p> 621 Briefly, an OpenSL ES object is similar to the object concept 622 in programming languages such as Java and C++, except an OpenSL ES 623 object is <i>only</i> visible via its associated interfaces. This 624 includes the initial interface for all objects, called 625 <code>SLObjectItf</code>. There is no handle for an object itself, 626 only a handle to the <code>SLObjectItf</code> interface of the object. 627 <p> 628 An OpenSL ES object is first "created", which returns an 629 <code>SLObjectItf</code>, then "realized". This is similar to the 630 common programming pattern of first constructing an object (which 631 should never fail other than for lack of memory or invalid parameters), 632 and then completing initialization (which may fail due to lack of 633 resources). The realize step gives the implementation a 634 logical place to allocate additional resources if needed. 635 <p> 636 As part of the API to create an object, an application specifies 637 an array of desired interfaces that it plans to acquire later. Note 638 that this array does <i>not</i> automatically acquire the interfaces; 639 it merely indicates a future intention to acquire them. Interfaces 640 are distinguished as "implicit" or "explicit". An explicit interface 641 <i>must</i> be listed in the array if it will be acquired later. 642 An implicit interface need not be listed in the object create array, 643 but there is no harm in listing it there. OpenSL ES has one more 644 kind of interface called "dynamic", which does not need to be 645 specified in the object create array, and can be added later after 646 the object is created. The Android implementation provides a 647 convenience feature to avoid this complexity; see section "Dynamic 648 interfaces at object creation" above. 649 <p> 650 After the object is created and realized, the application should 651 acquire interfaces for each feature it needs, using 652 <code>GetInterface</code> on the initial <code>SLObjectItf</code>. 653 <p> 654 Finally, the object is available for use via its interfaces, though 655 note that some objects require further setup. In particular, an 656 audio player with URI data source needs a bit more preparation in 657 order to detect connection errors. See the next section 658 "Audio player prefetch" for details. 659 <p> 660 After your application is done with the object, you should explicitly 661 destroy it; see section "Destroy" below. 662 663 <h3>Audio player prefetch</h3> 664 665 For an audio player with URI data source, <code>Object::Realize</code> allocates resources 666 but does not connect to the data source (i.e. "prepare") or begin 667 pre-fetching data. These occur once the player state is set to 668 either <code>SL_PLAYSTATE_PAUSED</code> or <code>SL_PLAYSTATE_PLAYING</code>. 669 <p> 670 Note that some information may still be unknown until relatively 671 late in this sequence. In particular, initially 672 <code>Player::GetDuration</code> will return <code>SL_TIME_UNKNOWN</code> 673 and <code>MuteSolo::GetChannelCount</code> will return zero. 674 These APIs will return the proper values once they are known. 675 <p> 676 Other properties that are initially unknown include the sample rate 677 and actual media content type based on examining the content's header 678 (as opposed to the application-specified MIME type and container type). 679 These too, are determined later during prepare / prefetch, but there are 680 no APIs to retrieve them. 681 <p> 682 The prefetch status interface is useful for detecting when all 683 information is available. Or, your application can poll periodically. 684 Note that some information may <i>never</i> be known, for example, 685 the duration of a streaming MP3. 686 <p> 687 The prefetch status interface is also useful for detecting errors. 688 Register a callback and enable at least the 689 <code>SL_PREFETCHEVENT_FILLLEVELCHANGE</code> and 690 <code>SL_PREFETCHEVENT_STATUSCHANGE</code> events. If both of these 691 events are delivered simultaneously, and 692 <code>PrefetchStatus::GetFillLevel</code> reports a zero level, and 693 <code>PrefetchStatus::GetPrefetchStatus</code> reports 694 <code>SL_PREFETCHSTATUS_UNDERFLOW</code>, then this indicates a 695 non-recoverable error in the data source. 696 This includes the inability to connect to the data source because 697 the local filename does not exist or the network URI is invalid. 698 <p> 699 The next version of OpenSL ES is expected to add more explicit 700 support for handling errors in the data source. However, for future 701 binary compatibility, we intend to continue to support the current 702 method for reporting a non-recoverable error. 703 <p> 704 In summary, a recommended code sequence is: 705 <ul> 706 <li>Engine::CreateAudioPlayer 707 <li>Object:Realize 708 <li>Object::GetInterface for SL_IID_PREFETCHSTATUS 709 <li>PrefetchStatus::SetCallbackEventsMask 710 <li>PrefetchStatus::SetFillUpdatePeriod 711 <li>PrefetchStatus::RegisterCallback 712 <li>Object::GetInterface for SL_IID_PLAY 713 <li>Play::SetPlayState to SL_PLAYSTATE_PAUSED or SL_PLAYSTATE_PLAYING 714 <li>preparation and prefetching occur here; during this time your 715 callback will be called with periodic status updates 716 </ul> 717 718 <h3>Destroy</h3> 719 720 Be sure to destroy all objects on exit from your application. Objects 721 should be destroyed in reverse order of their creation, as it is 722 not safe to destroy an object that has any dependent objects. 723 For example, destroy in this order: audio players and recorders, 724 output mix, then finally the engine. 725 <p> 726 OpenSL ES does not support automatic garbage collection or 727 <a href="http://en.wikipedia.org/wiki/Reference_counting">reference counting</a> 728 of interfaces. After you call <code>Object::Destroy</code>, all extant 729 interfaces derived from the associated object become <i>undefined</i>. 730 <p> 731 The Android OpenSL ES implementation does not detect the incorrect 732 use of such interfaces. 733 Continuing to use such interfaces after the object is destroyed will 734 cause your application to crash or behave in unpredictable ways. 735 <p> 736 We recommend that you explicitly set both the primary object interface 737 and all associated interfaces to NULL as part of your object 738 destruction sequence, to prevent the accidental misuse of a stale 739 interface handle. 740 741 <h3>Stereo panning</h3> 742 743 When <code>Volume::EnableStereoPosition</code> is used to enable 744 stereo panning of a mono source, there is a 3 dB reduction in total 745 <a href="http://en.wikipedia.org/wiki/Sound_power_level"> 746 sound power level</a>. This is needed to permit the total sound 747 power level to remain constant as the source is panned from one 748 channel to the other. Therefore, don't enable stereo positioning 749 if you don't need it. See the Wikipedia article on 750 <a href="http://en.wikipedia.org/wiki/Panning_(audio)">audio panning</a> 751 for more information. 752 753 <h3>Callbacks and threads</h3> 754 755 Callback handlers are generally called <i>synchronously</i> with 756 respect to the event, that is, at the moment and location where the 757 event is detected by the implementation. But this point is 758 <i>asynchronous</i> with respect to the application. Thus you should 759 use a mutex or other synchronization mechanism to control access 760 to any variables shared between the application and the callback 761 handler. In the example code, such as for buffer queues, we have 762 omitted this synchronization in the interest of simplicity. However, 763 proper mutual exclusion would be critical for any production code. 764 <p> 765 Callback handlers are called from internal 766 non-application thread(s) which are not attached to the Dalvik virtual machine and thus 767 are ineligible to use JNI. Because these internal threads are 768 critical to the integrity of the OpenSL ES implementation, a callback 769 handler should also not block or perform excessive work. Therefore, 770 if your callback handler needs to use JNI or do anything significant 771 (e.g. beyond an <code>Enqueue</code> or something else simple such as the "Get" 772 family), the handler should instead post an event for another thread 773 to process. 774 <p> 775 Note that the converse is safe: a Dalvik application thread which has 776 entered JNI is allowed to directly call OpenSL ES APIs, including 777 those which block. However, blocking calls are not recommended from 778 the main thread, as they may result in the dreaded "Application Not 779 Responding" (ANR). 780 781 <h3>Performance</h3> 782 783 As OpenSL ES is a native C API, non-Dalvik application threads which 784 call OpenSL ES have no Dalvik-related overhead such as garbage 785 collection pauses. However, there is no additional performance 786 benefit to the use of OpenSL ES other than this. In particular, use 787 of OpenSL ES does not result in lower audio latency, higher scheduling 788 priority, etc. than what the platform generally provides. 789 On the other hand, as the Android platform and specific device 790 implementations continue to evolve, an OpenSL ES application can 791 expect to benefit from any future system performance improvements. 792 793 <h3>Security and permissions</h3> 794 795 As far as who can do what, security in Android is done at the 796 process level. Java programming language code can't do anything more than native code, nor 797 can native code do anything more than Java code. The only differences 798 between them are what APIs are available that provide functionality 799 that the platform promises to support in the future and across 800 different devices. 801 <p> 802 Applications using OpenSL ES must request whatever permissions they 803 would need for similar non-native APIs. For example, if your application 804 records audio, then it needs the <code>android.permission.RECORD_AUDIO</code> 805 permission. Applications that use audio effects need 806 <code>android.permission.MODIFY_AUDIO_SETTINGS</code>. Applications that play 807 network URI resources need <code>android.permission.NETWORK</code>. 808 <p> 809 Media content parsers and software codecs run within the context 810 of the Android application that calls OpenSL ES (hardware codecs 811 are abstracted, but are device-dependent). Malformed content 812 designed to exploit parser and codec vulnerabilities is a known attack 813 vector. We recommend that you play media only from trustworthy 814 sources, or that you partition your application such that code that 815 handles media from untrustworthy sources runs in a relatively 816 sandboxed environment. For example you could process media from 817 untrustworthy sources in a separate process. Though both processes 818 would still run under the same UID, this separation does make an 819 attack more difficult. 820 821 <h2>Platform issues</h2> 822 823 This section describes known issues in the initial platform 824 release which supports these APIs. 825 826 <h3>Dynamic interface management</h3> 827 828 <code>DynamicInterfaceManagement::AddInterface</code> does not work. 829 Instead, specify the interface in the array passed to Create, as 830 shown in the example code for environmental reverb. 831 832 <h2>References and resources</h2> 833 834 Google Android: 835 <ul> 836 <li><a href="http://developer.android.com/resources/index.html"> 837 Android developer resources</a> 838 <li><a href="http://groups.google.com/group/android-developers"> 839 Android developers discussion group</a> 840 <li><a href="http://developer.android.com/sdk/ndk/index.html">Android NDK</a> 841 <li><a href="http://groups.google.com/group/android-ndk"> 842 Android NDK discussion group</a> (for developers of native code, including OpenSL ES) 843 <li><a href="http://code.google.com/p/android/issues/"> 844 Android open source bug database</a> 845 </ul> 846 847 Khronos Group: 848 <ul> 849 <li><a href="http://www.khronos.org/opensles/"> 850 Khronos Group OpenSL ES Overview</a> 851 <li><a href="http://www.khronos.org/registry/sles/"> 852 Khronos Group OpenSL ES 1.0.1 specification</a> 853 <li><a href="http://www.khronos.org/message_boards/viewforum.php?f=15"> 854 Khronos Group public message board for OpenSL ES</a> 855 (please limit to non-Android questions) 856 </ul> 857 For convenience, we have included a copy of the OpenSL ES 1.0.1 858 specification with the NDK in 859 <code>docs/opensles/OpenSL_ES_Specification_1.0.1.pdf</code>. 860 861 <p> 862 Miscellaneous: 863 <ul> 864 <li><a href="http://en.wikipedia.org/wiki/Java_Native_Interface">JNI</a> 865 <li><a href="http://stackoverflow.com/search?q=android+audio"> 866 Stack Overflow</a> 867 <li>web search for "interactive audio", "game audio", "sound design", 868 "audio programming", "audio content", "audio formats", etc. 869 </ul> 870 871 </body> 872 </html> 873
