1 {{+bindTo:partials.standard_nacl_article}} 2 3 <section id="audio"> 4 <span id="devguide-coding-audio"></span><h1 id="audio"><span id="devguide-coding-audio"></span>Audio</h1> 5 <div class="contents local" id="contents" style="display: none"> 6 <ul class="small-gap"> 7 <li><a class="reference internal" href="#reference-information" id="id1">Reference information</a></li> 8 <li><a class="reference internal" href="#about-the-pepper-audio-api" id="id2">About the Pepper audio API</a></li> 9 <li><a class="reference internal" href="#digital-audio-concepts" id="id3">Digital audio concepts</a></li> 10 <li><a class="reference internal" href="#setting-up-the-module" id="id4">Setting up the module</a></li> 11 <li><p class="first"><a class="reference internal" href="#creating-an-audio-configuration-resource" id="id5">Creating an audio configuration resource</a></p> 12 <ul class="small-gap"> 13 <li><a class="reference internal" href="#resources" id="id6">Resources</a></li> 14 <li><a class="reference internal" href="#sample-frame-count" id="id7">Sample frame count</a></li> 15 <li><a class="reference internal" href="#supported-audio-configurations" id="id8">Supported audio configurations</a></li> 16 </ul> 17 </li> 18 <li><a class="reference internal" href="#creating-an-audio-resource" id="id9">Creating an audio resource</a></li> 19 <li><p class="first"><a class="reference internal" href="#implementing-a-callback-function" id="id10">Implementing a callback function</a></p> 20 <ul class="small-gap"> 21 <li><a class="reference internal" href="#application-threads-and-real-time-requirements" id="id11">Application threads and real-time requirements</a></li> 22 </ul> 23 </li> 24 <li><a class="reference internal" href="#starting-and-stopping-playback" id="id12">Starting and stopping playback</a></li> 25 </ul> 26 27 </div><p>This chapter describes how to use the Pepper audio API to play an audio 28 stream. The Pepper audio API provides a low-level means of playing a stream of 29 audio samples generated by a Native Client module. The API generally works as 30 follows: A Native Client module creates an audio resource that represents an 31 audio stream, and tells the browser to start or stop playing the audio 32 resource. The browser calls a function in the Native Client module to fill a 33 buffer with audio samples every time it needs data to play from the audio 34 stream.</p> 35 <p>The code examples in this chapter describe a simple Native Client module that 36 generates audio samples using a sine wave with a frequency of 440 Hz. The module 37 starts playing the audio samples as soon as it is loaded into the browser. For a 38 slightly more sophisticated example, see the <code>audio</code> example (source code in 39 the SDK directory <code>examples/api/audio</code>), which lets users specify a frequency 40 for the sine wave and click buttons to start and stop audio playback.</p> 41 <section id="reference-information"> 42 <h2 id="reference-information">Reference information</h2> 43 <p>For reference information related to the Pepper audio API, see the following 44 documentation:</p> 45 <ul class="small-gap"> 46 <li><a class="reference external" href="/native-client/pepper_stable/cpp/classpp_1_1_audio_config">pp::AudioConfig class</a></li> 47 <li><a class="reference external" href="/native-client/pepper_stable/cpp/classpp_1_1_audio">pp::Audio class</a></li> 48 <li><a class="reference external" href="/native-client/pepper_cpp/audio__config_8h">audio_config.h</a></li> 49 <li><a class="reference external" href="/native-client/pepper_stable/cpp/audio_8h">audio.h</a></li> 50 <li><a class="reference external" href="/native-client/pepper_stable/c/group___enums#gaee750c350655f2fb0fe04c04029e0ff8">PP_AudioSampleRate</a></li> 51 </ul> 52 </section><section id="about-the-pepper-audio-api"> 53 <h2 id="about-the-pepper-audio-api">About the Pepper audio API</h2> 54 <p>The Pepper audio API lets Native Client modules play audio streams in a 55 browser. To play an audio stream, a module generates audio samples and writes 56 them into a buffer. The browser reads the audio samples from the buffer and 57 plays them using an audio device on the client computer.</p> 58 <img alt="/native-client/images/pepper-audio-buffer.png" src="/native-client/images/pepper-audio-buffer.png" /> 59 <p>This mechanism is simple but low-level. If you want to play plain sound files in 60 a web application, you may want to consider higher-level alternatives such as 61 using the HTML <code><audio></code> tag, JavaScript, or the new <a class="reference external" href="http://chromium.googlecode.com/svn/trunk/samples/audio/index.html">Web Audio API</a>.</p> 62 <p>The Pepper audio API is a good option for playing audio data if you want to do 63 audio processing in your web application. You might use the audio API, for 64 example, if you want to apply audio effects to sounds, synthesize your own 65 sounds, or do any other type of CPU-intensive processing of audio 66 samples. Another likely use case is gaming applications: you might use a gaming 67 library to process audio data, and then simply use the audio API to output the 68 processed data.</p> 69 <p>The Pepper audio API is straightforward to use:</p> 70 <ol class="arabic simple"> 71 <li>Your module creates an audio configuration resource and an audio resource.</li> 72 <li>Your module implements a callback function that fills an audio buffer with 73 data.</li> 74 <li>Your module invokes the StartPlayback and StopPlayback methods of the audio 75 resource (e.g., when certain events occur).</li> 76 <li>The browser invokes your callback function whenever it needs audio data to 77 play. Your callback function can generate the audio data in a number of 78 ways—e.g., it can generate new data, or it can copy pre-mixed data into the 79 audio buffer.</li> 80 </ol> 81 <p>This basic interaction is illustrated below, and described in detail in the 82 sections that follow.</p> 83 <img alt="/native-client/images/pepper-audio-api.png" src="/native-client/images/pepper-audio-api.png" /> 84 </section><section id="digital-audio-concepts"> 85 <h2 id="digital-audio-concepts">Digital audio concepts</h2> 86 <p>Before you use the Pepper audio API, it’s helpful to understand a few concepts 87 that are fundamental to how digital audio is recorded and played back:</p> 88 <dl class="docutils"> 89 <dt>sample rate</dt> 90 <dd>the number of times an input sound source is sampled per second; 91 correspondingly, the number of samples that are played back per second</dd> 92 <dt>bit depth</dt> 93 <dd>the number of bits used to represent a sample</dd> 94 <dt>channels</dt> 95 <dd>the number of input sources recorded in each sampling interval; 96 correspondingly, the number of outputs that are played back simultaneously 97 (typically using different speakers)</dd> 98 </dl> 99 <p>The higher the sample rate and bit depth used to record a sound wave, the more 100 accurately the sound wave can be reproduced, since it will have been sampled 101 more frequently and stored using a higher level of quantization. Common sampling 102 rates include 44,100 Hz (44,100 samples/second, the sample rate used on CDs), 103 and 48,000 Hz (the sample rate used on DVDs and Digital Audio Tapes). A common 104 bit depth is 16 bits per sample, and a common number of channels is 2 (left and 105 right channels for stereo sound).</p> 106 <p id="pepper-audio-configurations">The Pepper audio API currently lets Native Client modules play audio streams 107 with the following configurations:</p> 108 <ul class="small-gap"> 109 <li><strong>sample rate</strong>: 44,100 Hz or 48,000 Hz</li> 110 <li><strong>bit depth</strong>: 16</li> 111 <li><strong>channels</strong>: 2 (stereo)</li> 112 </ul> 113 </section><section id="setting-up-the-module"> 114 <h2 id="setting-up-the-module">Setting up the module</h2> 115 <p>The code examples below describe a simple Native Client module that generates 116 audio samples using a sine wave with a frequency of 440 Hz. The module starts 117 playing the audio samples as soon as it is loaded into the browser.</p> 118 <p>The Native Client module is set up by implementing subclasses of the 119 <code>pp::Module</code> and <code>pp::Instance</code> classes, as normal.</p> 120 <pre class="prettyprint"> 121 class SineSynthInstance : public pp::Instance { 122 public: 123 explicit SineSynthInstance(PP_Instance instance); 124 virtual ~SineSynthInstance() {} 125 126 // Called by the browser once the NaCl module is loaded and ready to 127 // initialize. Creates a Pepper audio context and initializes it. Returns 128 // true on success. Returning false causes the NaCl module to be deleted 129 // and no other functions to be called. 130 virtual bool Init(uint32_t argc, const char* argn[], const char* argv[]); 131 132 private: 133 // Function called by the browser when it needs more audio samples. 134 static void SineWaveCallback(void* samples, 135 uint32_t buffer_size, 136 void* data); 137 138 // Audio resource. 139 pp::Audio audio_; 140 141 ... 142 143 }; 144 145 class SineSynthModule : public pp::Module { 146 public: 147 SineSynthModule() : pp::Module() {} 148 ~SineSynthModule() {} 149 150 // Create and return a SineSynthInstance object. 151 virtual pp::Instance* CreateInstance(PP_Instance instance) { 152 return new SineSynthInstance(instance); 153 } 154 }; 155 </pre> 156 </section><section id="creating-an-audio-configuration-resource"> 157 <h2 id="creating-an-audio-configuration-resource">Creating an audio configuration resource</h2> 158 <section id="resources"> 159 <h3 id="resources">Resources</h3> 160 <p>Before the module can play an audio stream, it must create two resources: an 161 audio configuration resource and an audio resource. Resources are handles to 162 objects that the browser provides to module instances. An audio resource is an 163 object that represents the state of an audio stream, including whether the 164 stream is paused or being played back, and which callback function to invoke 165 when the samples in the stream’s buffer run out. An audio configuration resource 166 is an object that stores configuration data for an audio resource, including the 167 sampling frequency of the audio samples, and the number of samples that the 168 callback function must provide when the browser invokes it.</p> 169 </section><section id="sample-frame-count"> 170 <h3 id="sample-frame-count">Sample frame count</h3> 171 <p>Prior to creating an audio configuration resource, the module should call 172 <code>RecommendSampleFrameCount</code> to obtain a <em>sample frame count</em> from the 173 browser. The sample frame count is the number of samples that the callback 174 function must provide per channel each time the browser invokes the callback 175 function. For example, if the sample frame count is 4096 for a stereo audio 176 stream, the callback function must provide a 8192 samples (4096 for the left 177 channel and 4096 for the right channel).</p> 178 <p>The module can request a specific sample frame count, but the browser may return 179 a different sample frame count depending on the capabilities of the client 180 device. At present, <code>RecommendSampleFrameCount</code> simply bound-checks the 181 requested sample frame count (see <code>include/ppapi/c/ppb_audio_config.h</code> for the 182 minimum and maximum sample frame counts, currently 64 and 32768). In the future, 183 <code>RecommendSampleFrameCount</code> may perform a more sophisticated calculation, 184 particularly if there is an intrinsic buffer size for the client device.</p> 185 <p>Selecting a sample frame count for an audio stream involves a tradeoff between 186 latency and CPU usage. If you want your module to have short audio latency so 187 that it can rapidly change what’s playing in the audio stream, you should 188 request a small sample frame count. That could be useful in gaming applications, 189 for example, where sounds have to change frequently in response to game 190 action. However, a small sample frame count results in higher CPU usage, since 191 the browser must invoke the callback function frequently to refill the audio 192 buffer. Conversely, a large sample frame count results in higher latency but 193 lower CPU usage. You should request a large sample frame count if your module 194 will play long, uninterrupted audio segments.</p> 195 </section><section id="supported-audio-configurations"> 196 <h3 id="supported-audio-configurations">Supported audio configurations</h3> 197 <p>After the module obtains a sample frame count, it can create an audio 198 configuration resource. Currently the Pepper audio API supports audio streams 199 with the configuration settings shown <a class="reference internal" href="#pepper-audio-configurations"><em>above</em></a>. 200 C++ modules can create a configuration resource by instantiating a 201 <code>pp::AudioConfig</code> object. Check <code>audio_config.h</code> for the latest 202 configurations that are supported.</p> 203 <pre class="prettyprint"> 204 bool SineSynthInstance::Init(uint32_t argc, 205 const char* argn[], 206 const char* argv[]) { 207 208 // Ask the browser/device for an appropriate sample frame count size. 209 sample_frame_count_ = 210 pp::AudioConfig::RecommendSampleFrameCount(PP_AUDIOSAMPLERATE_44100, 211 kSampleFrameCount); 212 213 // Create an audio configuration resource. 214 pp::AudioConfig audio_config = pp::AudioConfig(this, 215 PP_AUDIOSAMPLERATE_44100, 216 sample_frame_count_); 217 218 // Create an audio resource. 219 audio_ = pp::Audio(this, 220 audio_config, 221 SineWaveCallback, 222 this); 223 224 // Start playback when the module instance is initialized. 225 return audio_.StartPlayback(); 226 } 227 </pre> 228 </section></section><section id="creating-an-audio-resource"> 229 <h2 id="creating-an-audio-resource">Creating an audio resource</h2> 230 <p>Once the module has created an audio configuration resource, it can create an 231 audio resource. To do so, it instantiates a <code>pp::Audio</code> object, passing in a 232 pointer to the module instance, the audio configuration resource, a callback 233 function, and a pointer to user data (data that is used in the callback 234 function). See the example above.</p> 235 </section><section id="implementing-a-callback-function"> 236 <h2 id="implementing-a-callback-function">Implementing a callback function</h2> 237 <p>The browser calls the callback function associated with an audio resource every 238 time it needs more samples to play. The callback function can generate new 239 samples (e.g., by applying sound effects), or copy pre-mixed samples into the 240 audio buffer. The example below generates new samples by computing values of a 241 sine wave.</p> 242 <p>The last parameter passed to the callback function is generic user data that the 243 function can use in processing samples. In the example below, the user data is a 244 pointer to the module instance, which includes member variables 245 <code>sample_frame_count_</code> (the sample frame count obtained from the browser) and 246 <code>theta_</code> (the last angle that was used to compute a sine value in the previous 247 callback; this lets the function generate a smooth sine wave by starting at that 248 angle plus a small delta).</p> 249 <pre class="prettyprint"> 250 class SineSynthInstance : public pp::Instance { 251 public: 252 ... 253 254 private: 255 static void SineWaveCallback(void* samples, 256 uint32_t buffer_size, 257 void* data) { 258 259 // The user data in this example is a pointer to the module instance. 260 SineSynthInstance* sine_synth_instance = 261 reinterpret_cast<SineSynthInstance*>(data); 262 263 // Delta by which to increase theta_ for each sample. 264 const double delta = kTwoPi * kFrequency / PP_AUDIOSAMPLERATE_44100; 265 // Amount by which to scale up the computed sine value. 266 const int16_t max_int16 = std::numeric_limits<int16_t>::max(); 267 268 int16_t* buff = reinterpret_cast<int16_t*>(samples); 269 270 // Make sure we can't write outside the buffer. 271 assert(buffer_size >= (sizeof(*buff) * kChannels * 272 sine_synth_instance->sample_frame_count_)); 273 274 for (size_t sample_i = 0; 275 sample_i < sine_synth_instance->sample_frame_count_; 276 ++sample_i, sine_synth_instance->theta_ += delta) { 277 278 // Keep theta_ from going beyond 2*Pi. 279 if (sine_synth_instance->theta_ > kTwoPi) { 280 sine_synth_instance->theta_ -= kTwoPi; 281 } 282 283 // Compute the sine value for the current theta_, scale it up, 284 // and write it into the buffer once for each channel. 285 double sin_value(std::sin(sine_synth_instance->theta_)); 286 int16_t scaled_value = static_cast<int16_t>(sin_value * max_int16); 287 for (size_t channel = 0; channel < kChannels; ++channel) { 288 *buff++ = scaled_value; 289 } 290 } 291 } 292 293 ... 294 }; 295 </pre> 296 <section id="application-threads-and-real-time-requirements"> 297 <h3 id="application-threads-and-real-time-requirements">Application threads and real-time requirements</h3> 298 <p>The callback function runs in a background application thread. This allows audio 299 processing to continue even when the application is busy doing something 300 else. If the main application thread and the callback thread access the same 301 data, you may be tempted to use a lock to control access to that data. You 302 should avoid the use of locks in the callback thread, however, as attempting to 303 acquire a lock may cause the thread to get swapped out, resulting in audio 304 dropouts.</p> 305 <p>In general, you must program the callback thread carefully, as the Pepper audio 306 API is a very low level API that needs to meet hard real-time requirements. If 307 the callback thread spends too much time processing, it can easily miss the 308 real-time deadline, resulting in audio dropouts. One way the callback thread can 309 miss the deadline is by taking too much time doing computation. Another way the 310 callback thread can miss the deadline is by executing a function call that swaps 311 out the callback thread. Unfortunately, such function calls include just about 312 all C Run-Time (CRT) library calls and Pepper API calls. The callback thread 313 should therefore avoid calls to malloc, gettimeofday, mutex, condvars, critical 314 sections, and so forth; any such calls could attempt to take a lock and swap out 315 the callback thread, which would be disastrous for audio playback. Similarly, 316 the callback thread should avoid Pepper API calls. Audio dropouts due to thread 317 swapping can be very rare and very hard to track down and debug—it’s best to 318 avoid making system/Pepper calls in the first place. In short, the audio 319 (callback) thread should use “lock-free” techniques and avoid making CRT library 320 calls.</p> 321 <p>One other issue to be aware of is that the <code>StartPlayback</code> function (discussed 322 below) is an asynchronous RPC; i.e., it does not block. That means that the 323 callback function may not be called immediately after the call to 324 <code>StartPlayback</code>. If it’s important to synchronize the callback thread with 325 another thread so that the audio stream starts playing simultaneously with 326 another action in your application, you must handle such synchronization 327 manually.</p> 328 </section></section><section id="starting-and-stopping-playback"> 329 <h2 id="starting-and-stopping-playback">Starting and stopping playback</h2> 330 <p>To start and stop audio playback, the module simply reacts to JavaScript 331 messages.</p> 332 <pre class="prettyprint"> 333 const char* const kPlaySoundId = "playSound"; 334 const char* const kStopSoundId = "stopSound"; 335 336 void SineSynthInstance::HandleMessage(const pp::Var& var_message) { 337 if (!var_message.is_string()) { 338 return; 339 } 340 std::string message = var_message.AsString(); 341 if (message == kPlaySoundId) { 342 audio_.StartPlayback(); 343 } else if (message == kStopSoundId) { 344 audio_.StopPlayback(); 345 } else if (...) { 346 ... 347 } 348 } 349 </pre> 350 </section></section> 351 352 {{/partials.standard_nacl_article}} 353