1 <html devsite> 2 <head> 3 <title>TV Audio</title> 4 <meta name="project_path" value="/_project.yaml" /> 5 <meta name="book_path" value="/_book.yaml" /> 6 </head> 7 <body> 8 <!-- 9 Copyright 2017 The Android Open Source Project 10 11 Licensed under the Apache License, Version 2.0 (the "License"); 12 you may not use this file except in compliance with the License. 13 You may obtain a copy of the License at 14 15 http://www.apache.org/licenses/LICENSE-2.0 16 17 Unless required by applicable law or agreed to in writing, software 18 distributed under the License is distributed on an "AS IS" BASIS, 19 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 20 See the License for the specific language governing permissions and 21 limitations under the License. 22 --> 23 24 25 26 <p>The TV Input Framework (TIF) manager works with the audio routing API to support flexible audio 27 path changes. When a System on Chip (SoC) implements the TV hardware abstraction layer (HAL), each 28 TV input (HDMI IN, Tuner, etc.) provides <code>TvInputHardwareInfo</code> that specifies AudioPort information for audio type and address.</p> 29 30 <ul> 31 <li><b>Physical</b> audio input/output devices have a corresponding AudioPort.</li> 32 <li><b>Software</b> audio output/input streams are represented as AudioMixPort (child class of 33 AudioPort).</li> 34 </ul> 35 36 <p>The TIF then uses AudioPort information for the audio routing API.</p> 37 38 <p><img src="images/ape_audio_tv_tif.png" alt="Android TV Input Framework (TIF)" /></p> 39 <p class="img-caption"><strong>Figure 1.</strong> TV Input Framework (TIF)</p> 40 41 <h2 id="requirements">Requirements</h2> 42 43 <p>A SoC must implement the audio HAL with the following audio routing API support:</p> 44 45 <table> 46 <tbody> 47 <tr> 48 <th>Audio Ports</th> 49 <td> 50 <ul> 51 <li>TV Audio Input has a corresponding audio source port implementation.</li> 52 <li>TV Audio Output has a corresponding audio sink port implementation.</li> 53 <li>Can create audio patch between any TV input audio port and any TV output audio port.</li> 54 </ul> 55 </td> 56 </tr> 57 <tr> 58 <th>Default Input</th> 59 <td>AudioRecord (created with DEFAULT input source) must seize <i>virtual null input source</i> for 60 AUDIO_DEVICE_IN_DEFAULT acquisition on Android TV.</td> 61 </tr> 62 <tr> 63 <th>Device Loopback</th> 64 <td>Requires supporting an AUDIO_DEVICE_IN_LOOPBACK input that is a complete mix of all audio output 65 of all the TV output (11Khz, 16bit mono or 48Khz, 16bit mono). Used only for audio capture. 66 </td> 67 </tr> 68 </tbody> 69 </table> 70 71 72 <h2 id="audioDevices">TV audio devices</h2> 73 74 <p>Android supports the following audio devices for TV audio input/output.</p> 75 76 <h4><code>system/media/audio/include/system/audio.h</code></h4> 77 78 <p class="note"><strong>Note:</strong> In Android 5.1 and earlier, the path to 79 this file is: <code>system/core/include/system/audio.h</code></p> 80 81 <pre class="devsite-click-to-copy"> 82 /* output devices */ 83 AUDIO_DEVICE_OUT_AUX_DIGITAL = 0x400, 84 AUDIO_DEVICE_OUT_HDMI = AUDIO_DEVICE_OUT_AUX_DIGITAL, 85 /* HDMI Audio Return Channel */ 86 AUDIO_DEVICE_OUT_HDMI_ARC = 0x40000, 87 /* S/PDIF out */ 88 AUDIO_DEVICE_OUT_SPDIF = 0x80000, 89 /* input devices */ 90 AUDIO_DEVICE_IN_AUX_DIGITAL = AUDIO_DEVICE_BIT_IN | 0x20, 91 AUDIO_DEVICE_IN_HDMI = AUDIO_DEVICE_IN_AUX_DIGITAL, 92 /* TV tuner input */ 93 AUDIO_DEVICE_IN_TV_TUNER = AUDIO_DEVICE_BIT_IN | 0x4000, 94 /* S/PDIF in */ 95 AUDIO_DEVICE_IN_SPDIF = AUDIO_DEVICE_BIT_IN | 0x10000, 96 AUDIO_DEVICE_IN_LOOPBACK = AUDIO_DEVICE_BIT_IN | 0x40000, 97 </pre> 98 99 100 <h2 id="halExtension">Audio HAL extension</h2> 101 102 <p>The Audio HAL extension for the audio routing API is defined by following:</p> 103 104 <h4><code>system/media/audio/include/system/audio.h</code></h4> 105 106 <p class="note"><strong>Note:</strong> In Android 5.1 and earlier, the path to 107 this file is: <code>system/core/include/system/audio.h</code></p> 108 109 <pre class="devsite-click-to-copy"> 110 /* audio port configuration structure used to specify a particular configuration of an audio port */ 111 struct audio_port_config { 112 audio_port_handle_t id; /* port unique ID */ 113 audio_port_role_t role; /* sink or source */ 114 audio_port_type_t type; /* device, mix ... */ 115 unsigned int config_mask; /* e.g AUDIO_PORT_CONFIG_ALL */ 116 unsigned int sample_rate; /* sampling rate in Hz */ 117 audio_channel_mask_t channel_mask; /* channel mask if applicable */ 118 audio_format_t format; /* format if applicable */ 119 struct audio_gain_config gain; /* gain to apply if applicable */ 120 union { 121 struct audio_port_config_device_ext device; /* device specific info */ 122 struct audio_port_config_mix_ext mix; /* mix specific info */ 123 struct audio_port_config_session_ext session; /* session specific info */ 124 } ext; 125 }; 126 struct audio_port { 127 audio_port_handle_t id; /* port unique ID */ 128 audio_port_role_t role; /* sink or source */ 129 audio_port_type_t type; /* device, mix ... */ 130 unsigned int num_sample_rates; /* number of sampling rates in following array */ 131 unsigned int sample_rates[AUDIO_PORT_MAX_SAMPLING_RATES]; 132 unsigned int num_channel_masks; /* number of channel masks in following array */ 133 audio_channel_mask_t channel_masks[AUDIO_PORT_MAX_CHANNEL_MASKS]; 134 unsigned int num_formats; /* number of formats in following array */ 135 audio_format_t formats[AUDIO_PORT_MAX_FORMATS]; 136 unsigned int num_gains; /* number of gains in following array */ 137 struct audio_gain gains[AUDIO_PORT_MAX_GAINS]; 138 struct audio_port_config active_config; /* current audio port configuration */ 139 union { 140 struct audio_port_device_ext device; 141 struct audio_port_mix_ext mix; 142 struct audio_port_session_ext session; 143 } ext; 144 }; 145 </pre> 146 147 <h4><code>hardware/libhardware/include/hardware/audio.h</code></h4> 148 149 <pre class="devsite-click-to-copy"> 150 struct audio_hw_device { 151 : 152 /** 153 * Routing control 154 */ 155 156 /* Creates an audio patch between several source and sink ports. 157 * The handle is allocated by the HAL and should be unique for this 158 * audio HAL module. */ 159 int (*create_audio_patch)(struct audio_hw_device *dev, 160 unsigned int num_sources, 161 const struct audio_port_config *sources, 162 unsigned int num_sinks, 163 const struct audio_port_config *sinks, 164 audio_patch_handle_t *handle); 165 166 /* Release an audio patch */ 167 int (*release_audio_patch)(struct audio_hw_device *dev, 168 audio_patch_handle_t handle); 169 170 /* Fills the list of supported attributes for a given audio port. 171 * As input, "port" contains the information (type, role, address etc...) 172 * needed by the HAL to identify the port. 173 * As output, "port" contains possible attributes (sampling rates, formats, 174 * channel masks, gain controllers...) for this port. 175 */ 176 int (*get_audio_port)(struct audio_hw_device *dev, 177 struct audio_port *port); 178 179 /* Set audio port configuration */ 180 int (*set_audio_port_config)(struct audio_hw_device *dev, 181 const struct audio_port_config *config); 182 </pre> 183 184 <h2 id="testing">Testing DEVICE_IN_LOOPBACK</h2> 185 186 <p>To test DEVICE_IN_LOOPBACK for TV monitoring, use the following testing code. After running the 187 test, the captured audio saves to <code>/sdcard/record_loopback.raw</code>, where you can listen to 188 it using <code><a href="https://en.wikipedia.org/wiki/FFmpeg">FFmpeg</a></code>.</p> 189 190 <pre class="devsite-click-to-copy"> 191 <uses-permission android:name="android.permission.MODIFY_AUDIO_ROUTING" /> 192 <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" /> 193 194 AudioRecord mRecorder; 195 Handler mHandler = new Handler(); 196 int mMinBufferSize = AudioRecord.getMinBufferSize(RECORD_SAMPLING_RATE, 197 AudioFormat.CHANNEL_IN_MONO, 198 AudioFormat.ENCODING_PCM_16BIT);; 199 static final int RECORD_SAMPLING_RATE = 48000; 200 public void doCapture() { 201 mRecorder = new AudioRecord(MediaRecorder.AudioSource.DEFAULT, RECORD_SAMPLING_RATE, 202 AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, mMinBufferSize * 10); 203 AudioManager am = (AudioManager) getSystemService(Context.AUDIO_SERVICE); 204 ArrayList<AudioPort> audioPorts = new ArrayList<AudioPort>(); 205 am.listAudioPorts(audioPorts); 206 AudioPortConfig srcPortConfig = null; 207 AudioPortConfig sinkPortConfig = null; 208 for (AudioPort audioPort : audioPorts) { 209 if (srcPortConfig == null 210 && audioPort.role() == AudioPort.ROLE_SOURCE 211 && audioPort instanceof AudioDevicePort) { 212 AudioDevicePort audioDevicePort = (AudioDevicePort) audioPort; 213 if (audioDevicePort.type() == AudioManager.DEVICE_IN_LOOPBACK) { 214 srcPortConfig = audioPort.buildConfig(48000, AudioFormat.CHANNEL_IN_DEFAULT, 215 AudioFormat.ENCODING_DEFAULT, null); 216 Log.d(LOG_TAG, "Found loopback audio source port : " + audioPort); 217 } 218 } 219 else if (sinkPortConfig == null 220 && audioPort.role() == AudioPort.ROLE_SINK 221 && audioPort instanceof AudioMixPort) { 222 sinkPortConfig = audioPort.buildConfig(48000, AudioFormat.CHANNEL_OUT_DEFAULT, 223 AudioFormat.ENCODING_DEFAULT, null); 224 Log.d(LOG_TAG, "Found recorder audio mix port : " + audioPort); 225 } 226 } 227 if (srcPortConfig != null && sinkPortConfig != null) { 228 AudioPatch[] patches = new AudioPatch[] { null }; 229 int status = am.createAudioPatch( 230 patches, 231 new AudioPortConfig[] { srcPortConfig }, 232 new AudioPortConfig[] { sinkPortConfig }); 233 Log.d(LOG_TAG, "Result of createAudioPatch(): " + status); 234 } 235 mRecorder.startRecording(); 236 processAudioData(); 237 mRecorder.stop(); 238 mRecorder.release(); 239 } 240 private void processAudioData() { 241 OutputStream rawFileStream = null; 242 byte data[] = new byte[mMinBufferSize]; 243 try { 244 rawFileStream = new BufferedOutputStream( 245 new FileOutputStream(new File("/sdcard/record_loopback.raw"))); 246 } catch (FileNotFoundException e) { 247 Log.d(LOG_TAG, "Can't open file.", e); 248 } 249 long startTimeMs = System.currentTimeMillis(); 250 while (System.currentTimeMillis() - startTimeMs < 5000) { 251 int nbytes = mRecorder.read(data, 0, mMinBufferSize); 252 if (nbytes <= 0) { 253 continue; 254 } 255 try { 256 rawFileStream.write(data); 257 } catch (IOException e) { 258 Log.e(LOG_TAG, "Error on writing raw file.", e); 259 } 260 } 261 try { 262 rawFileStream.close(); 263 } catch (IOException e) { 264 } 265 Log.d(LOG_TAG, "Exit audio recording."); 266 } 267 </pre> 268 269 <p>Locate the captured audio file in <code>/sdcard/record_loopback.raw</code> and listen to it using 270 <code>FFmpeg</code>:</p> 271 272 <pre class="devsite-click-to-copy"> 273 <code class="devsite-terminal">adb pull /sdcard/record_loopback.raw</code> 274 <code class="devsite-terminal">ffmpeg -f s16le -ar 48k -ac 1 -i record_loopback.raw record_loopback.wav</code> 275 <code class="devsite-terminal">ffplay record_loopback.wav</code> 276 </pre> 277 278 <h2 id="useCases">Use cases</h2> 279 280 <p>This section includes common use cases for TV audio.</p> 281 282 <h3 id="tvSpeaker">TV tuner with speaker output</h3> 283 284 <p>When a TV tuner becomes active, the audio routing API creates an audio patch between the tuner 285 and the default output (e.g. the speaker). The tuner output does not require decoding, but final 286 audio output is mixed with software output_stream.</p> 287 288 <img src="images/ape_audio_tv_tuner.png" alt="Android TV Tuner Audio Patch" /> 289 <p class="img-caption"> 290 <strong>Figure 2.</strong> Audio Patch for TV tuner with speaker output.</p> 291 292 293 <h3 id="hdmiOut">HDMI OUT during live TV</h3> 294 295 <p>A user is watching live TV then switches to the HDMI audio output (Intent.ACTION_HDMI_AUDIO_PLUG) 296 . The output device of all output_streams changes to the HDMI_OUT port, and the TIF manager changes 297 the sink port of the existing tuner audio patch to the HDMI_OUT port.</p> 298 299 <img src="images/ape_audio_tv_hdmi_tuner.png" alt="Android TV HDMI-OUT Audio Patch" /> 300 <p class="img-caption"> 301 <strong>Figure 3.</strong> Audio Patch for HDMI OUT from live TV.</p> 302 303 </body> 304 </html> 305
