Heidi Miller | 54362a6 | 2014-10-24 16:00:04 -0700 | [diff] [blame^] | 1 | page.title=TV Audio |
| 2 | @jd:body |
| 3 | |
| 4 | <!-- |
| 5 | Copyright 2014 The Android Open Source Project |
| 6 | |
| 7 | Licensed under the Apache License, Version 2.0 (the "License"); |
| 8 | you may not use this file except in compliance with the License. |
| 9 | You may obtain a copy of the License at |
| 10 | |
| 11 | http://www.apache.org/licenses/LICENSE-2.0 |
| 12 | |
| 13 | Unless required by applicable law or agreed to in writing, software |
| 14 | distributed under the License is distributed on an "AS IS" BASIS, |
| 15 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 16 | See the License for the specific language governing permissions and |
| 17 | limitations under the License. |
| 18 | --> |
| 19 | <div id="qv-wrapper"> |
| 20 | <div id="qv"> |
| 21 | <h2>In this document</h2> |
| 22 | <ol id="auto-toc"> |
| 23 | </ol> |
| 24 | </div> |
| 25 | </div> |
| 26 | |
| 27 | <p>The TV Input Framework (TIF) manager works with the audio routing API to support flexible audio |
| 28 | path changes. When a System on Chip (SoC) implements the TV hardware abstraction layer (HAL), each |
| 29 | TV input (HDMI IN, Tuner, etc.) provides <code>TvInputHardwareInfo</code> that specifies AudioPort information for audio type and address.</p> |
| 30 | |
| 31 | <ul> |
| 32 | <li><b>Physical</b> audio input/output devices have a corresponding AudioPort.</li> |
| 33 | <li><b>Software</b> audio output/input streams are represented as AudioMixPort (child class of |
| 34 | AudioPort).</li> |
| 35 | </ul> |
| 36 | |
| 37 | <p>The TIF then uses AudioPort information for the audio routing API.</p> |
| 38 | |
| 39 | <p><img src="audio/images/ape_audio_tv_tif.png" alt="Android TV Input Framework (TIF)" /> |
| 40 | <p class="img-caption"><strong>Figure 1.</strong> TV Input Framework (TIF)</p> |
| 41 | |
| 42 | <h2 id="Requirements">Requirements</h2> |
| 43 | |
| 44 | <p>A SoC must implement the audio HAL with the following audio routing API support:</p> |
| 45 | |
| 46 | <table> |
| 47 | <tbody> |
| 48 | <tr> |
| 49 | <th>Audio Ports</th> |
| 50 | <td> |
| 51 | <ul> |
| 52 | <li>TV Audio Input has a corresponding audio source port implementation.</li> |
| 53 | <li>TV Audio Output has a corresponding audio sink port implementation.</li> |
| 54 | <li>Can create audio patch between any TV input audio port and any TV output audio port.</li> |
| 55 | </ul> |
| 56 | </td> |
| 57 | </tr> |
| 58 | <tr> |
| 59 | <th>Default Input</th> |
| 60 | <td>AudioRecord (created with DEFAULT input source) must seize <i>virtual null input source</i> for |
| 61 | AUDIO_DEVICE_IN_DEFAULT acquisition on Android TV.</td> |
| 62 | </tr> |
| 63 | <tr> |
| 64 | <th>Device Loopback</th> |
| 65 | <td>Requires supporting an AUDIO_DEVICE_IN_LOOPBACK input that is a complete mix of all audio output |
| 66 | of all the TV output (11Khz, 16bit mono or 48Khz, 16bit mono). Used only for audio capture. |
| 67 | </td> |
| 68 | </tr> |
| 69 | </tbody> |
| 70 | </table> |
| 71 | |
| 72 | |
| 73 | <h2 id="Audio Devices">TV audio devices</h2> |
| 74 | |
| 75 | <p>Android supports the following audio devices for TV audio input/output.</p> |
| 76 | |
| 77 | <h4>system/core/include/system/audio.h</h4> |
| 78 | |
| 79 | <pre> |
| 80 | /* output devices */ |
| 81 | AUDIO_DEVICE_OUT_AUX_DIGITAL = 0x400, |
| 82 | AUDIO_DEVICE_OUT_HDMI = AUDIO_DEVICE_OUT_AUX_DIGITAL, |
| 83 | /* HDMI Audio Return Channel */ |
| 84 | AUDIO_DEVICE_OUT_HDMI_ARC = 0x40000, |
| 85 | /* S/PDIF out */ |
| 86 | AUDIO_DEVICE_OUT_SPDIF = 0x80000, |
| 87 | /* input devices */ |
| 88 | AUDIO_DEVICE_IN_AUX_DIGITAL = AUDIO_DEVICE_BIT_IN | 0x20, |
| 89 | AUDIO_DEVICE_IN_HDMI = AUDIO_DEVICE_IN_AUX_DIGITAL, |
| 90 | /* TV tuner input */ |
| 91 | AUDIO_DEVICE_IN_TV_TUNER = AUDIO_DEVICE_BIT_IN | 0x4000, |
| 92 | /* S/PDIF in */ |
| 93 | AUDIO_DEVICE_IN_SPDIF = AUDIO_DEVICE_BIT_IN | 0x10000, |
| 94 | AUDIO_DEVICE_IN_LOOPBACK = AUDIO_DEVICE_BIT_IN | 0x40000, |
| 95 | </pre> |
| 96 | |
| 97 | |
| 98 | <h2 id="HAL extension">Audio HAL extension</h2> |
| 99 | |
| 100 | <p>The Audio HAL extension for the audio routing API is defined by following:</p> |
| 101 | |
| 102 | <h4>system/core/include/system/audio.h</h4> |
| 103 | |
| 104 | <pre> |
| 105 | /* audio port configuration structure used to specify a particular configuration of an audio port */ |
| 106 | struct audio_port_config { |
| 107 | audio_port_handle_t id; /* port unique ID */ |
| 108 | audio_port_role_t role; /* sink or source */ |
| 109 | audio_port_type_t type; /* device, mix ... */ |
| 110 | unsigned int config_mask; /* e.g AUDIO_PORT_CONFIG_ALL */ |
| 111 | unsigned int sample_rate; /* sampling rate in Hz */ |
| 112 | audio_channel_mask_t channel_mask; /* channel mask if applicable */ |
| 113 | audio_format_t format; /* format if applicable */ |
| 114 | struct audio_gain_config gain; /* gain to apply if applicable */ |
| 115 | union { |
| 116 | struct audio_port_config_device_ext device; /* device specific info */ |
| 117 | struct audio_port_config_mix_ext mix; /* mix specific info */ |
| 118 | struct audio_port_config_session_ext session; /* session specific info */ |
| 119 | } ext; |
| 120 | }; |
| 121 | struct audio_port { |
| 122 | audio_port_handle_t id; /* port unique ID */ |
| 123 | audio_port_role_t role; /* sink or source */ |
| 124 | audio_port_type_t type; /* device, mix ... */ |
| 125 | unsigned int num_sample_rates; /* number of sampling rates in following array */ |
| 126 | unsigned int sample_rates[AUDIO_PORT_MAX_SAMPLING_RATES]; |
| 127 | unsigned int num_channel_masks; /* number of channel masks in following array */ |
| 128 | audio_channel_mask_t channel_masks[AUDIO_PORT_MAX_CHANNEL_MASKS]; |
| 129 | unsigned int num_formats; /* number of formats in following array */ |
| 130 | audio_format_t formats[AUDIO_PORT_MAX_FORMATS]; |
| 131 | unsigned int num_gains; /* number of gains in following array */ |
| 132 | struct audio_gain gains[AUDIO_PORT_MAX_GAINS]; |
| 133 | struct audio_port_config active_config; /* current audio port configuration */ |
| 134 | union { |
| 135 | struct audio_port_device_ext device; |
| 136 | struct audio_port_mix_ext mix; |
| 137 | struct audio_port_session_ext session; |
| 138 | } ext; |
| 139 | }; |
| 140 | </pre> |
| 141 | |
| 142 | <h4>hardware/libhardware/include/hardware/audio.h</h4> |
| 143 | |
| 144 | <pre> |
| 145 | struct audio_hw_device { |
| 146 | : |
| 147 | /** |
| 148 | * Routing control |
| 149 | */ |
| 150 | |
| 151 | /* Creates an audio patch between several source and sink ports. |
| 152 | * The handle is allocated by the HAL and should be unique for this |
| 153 | * audio HAL module. */ |
| 154 | int (*create_audio_patch)(struct audio_hw_device *dev, |
| 155 | unsigned int num_sources, |
| 156 | const struct audio_port_config *sources, |
| 157 | unsigned int num_sinks, |
| 158 | const struct audio_port_config *sinks, |
| 159 | audio_patch_handle_t *handle); |
| 160 | |
| 161 | /* Release an audio patch */ |
| 162 | int (*release_audio_patch)(struct audio_hw_device *dev, |
| 163 | audio_patch_handle_t handle); |
| 164 | |
| 165 | /* Fills the list of supported attributes for a given audio port. |
| 166 | * As input, "port" contains the information (type, role, address etc...) |
| 167 | * needed by the HAL to identify the port. |
| 168 | * As output, "port" contains possible attributes (sampling rates, formats, |
| 169 | * channel masks, gain controllers...) for this port. |
| 170 | */ |
| 171 | int (*get_audio_port)(struct audio_hw_device *dev, |
| 172 | struct audio_port *port); |
| 173 | |
| 174 | /* Set audio port configuration */ |
| 175 | int (*set_audio_port_config)(struct audio_hw_device *dev, |
| 176 | const struct audio_port_config *config); |
| 177 | </pre> |
| 178 | |
| 179 | <h2 id="Testing">Testing DEVICE_IN_LOOPBACK</h2> |
| 180 | |
| 181 | <p>To test DEVICE_IN_LOOPBACK for TV monitoring, use the following testing code. After running the |
| 182 | test, the captured audio saves to <code>/sdcard/record_loopback.raw</code>, where you can listen to |
| 183 | it using <code>ffmeg</code>.</p> |
| 184 | |
| 185 | <pre> |
| 186 | <uses-permission android:name="android.permission.MODIFY_AUDIO_ROUTING" /> |
| 187 | <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" /> |
| 188 | |
| 189 | AudioRecord mRecorder; |
| 190 | Handler mHandler = new Handler(); |
| 191 | int mMinBufferSize = AudioRecord.getMinBufferSize(RECORD_SAMPLING_RATE, |
| 192 | AudioFormat.CHANNEL_IN_MONO, |
| 193 | AudioFormat.ENCODING_PCM_16BIT);; |
| 194 | static final int RECORD_SAMPLING_RATE = 48000; |
| 195 | public void doCapture() { |
| 196 | mRecorder = new AudioRecord(MediaRecorder.AudioSource.DEFAULT, RECORD_SAMPLING_RATE, |
| 197 | AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, mMinBufferSize * 10); |
| 198 | AudioManager am = (AudioManager) getSystemService(Context.AUDIO_SERVICE); |
| 199 | ArrayList<AudioPort> audioPorts = new ArrayList<AudioPort>(); |
| 200 | am.listAudioPorts(audioPorts); |
| 201 | AudioPortConfig srcPortConfig = null; |
| 202 | AudioPortConfig sinkPortConfig = null; |
| 203 | for (AudioPort audioPort : audioPorts) { |
| 204 | if (srcPortConfig == null |
| 205 | && audioPort.role() == AudioPort.ROLE_SOURCE |
| 206 | && audioPort instanceof AudioDevicePort) { |
| 207 | AudioDevicePort audioDevicePort = (AudioDevicePort) audioPort; |
| 208 | if (audioDevicePort.type() == AudioManager.DEVICE_IN_LOOPBACK) { |
| 209 | srcPortConfig = audioPort.buildConfig(48000, AudioFormat.CHANNEL_IN_DEFAULT, |
| 210 | AudioFormat.ENCODING_DEFAULT, null); |
| 211 | Log.d(LOG_TAG, "Found loopback audio source port : " + audioPort); |
| 212 | } |
| 213 | } |
| 214 | else if (sinkPortConfig == null |
| 215 | && audioPort.role() == AudioPort.ROLE_SINK |
| 216 | && audioPort instanceof AudioMixPort) { |
| 217 | sinkPortConfig = audioPort.buildConfig(48000, AudioFormat.CHANNEL_OUT_DEFAULT, |
| 218 | AudioFormat.ENCODING_DEFAULT, null); |
| 219 | Log.d(LOG_TAG, "Found recorder audio mix port : " + audioPort); |
| 220 | } |
| 221 | } |
| 222 | if (srcPortConfig != null && sinkPortConfig != null) { |
| 223 | AudioPatch[] patches = new AudioPatch[] { null }; |
| 224 | int status = am.createAudioPatch( |
| 225 | patches, |
| 226 | new AudioPortConfig[] { srcPortConfig }, |
| 227 | new AudioPortConfig[] { sinkPortConfig }); |
| 228 | Log.d(LOG_TAG, "Result of createAudioPatch(): " + status); |
| 229 | } |
| 230 | mRecorder.startRecording(); |
| 231 | processAudioData(); |
| 232 | mRecorder.stop(); |
| 233 | mRecorder.release(); |
| 234 | } |
| 235 | private void processAudioData() { |
| 236 | OutputStream rawFileStream = null; |
| 237 | byte data[] = new byte[mMinBufferSize]; |
| 238 | try { |
| 239 | rawFileStream = new BufferedOutputStream( |
| 240 | new FileOutputStream(new File("/sdcard/record_loopback.raw"))); |
| 241 | } catch (FileNotFoundException e) { |
| 242 | Log.d(LOG_TAG, "Can't open file.", e); |
| 243 | } |
| 244 | long startTimeMs = System.currentTimeMillis(); |
| 245 | while (System.currentTimeMillis() - startTimeMs < 5000) { |
| 246 | int nbytes = mRecorder.read(data, 0, mMinBufferSize); |
| 247 | if (nbytes <= 0) { |
| 248 | continue; |
| 249 | } |
| 250 | try { |
| 251 | rawFileStream.write(data); |
| 252 | } catch (IOException e) { |
| 253 | Log.e(LOG_TAG, "Error on writing raw file.", e); |
| 254 | } |
| 255 | } |
| 256 | try { |
| 257 | rawFileStream.close(); |
| 258 | } catch (IOException e) { |
| 259 | } |
| 260 | Log.d(LOG_TAG, "Exit audio recording."); |
| 261 | } |
| 262 | </pre> |
| 263 | |
| 264 | <p>Locate the captured audio file in <code>/sdcard/record_loopback.raw</code> and listen to it using |
| 265 | <code>ffmeg</code>:</p> |
| 266 | |
| 267 | <pre> |
| 268 | adb pull /sdcard/record_loopback.raw |
| 269 | ffmpeg -f s16le -ar 48k -ac 1 -i record_loopback.raw record_loopback.wav |
| 270 | ffplay record_loopback.wav |
| 271 | </pre> |
| 272 | |
| 273 | <h2 id="Use cases">Use cases</h2> |
| 274 | |
| 275 | <p>This section includes common use cases for TV audio.</p> |
| 276 | |
| 277 | <h3>TV tuner with speaker output</h3> |
| 278 | |
| 279 | <p>When a TV tuner becomes active, the audio routing API creates an audio patch between the tuner |
| 280 | and the default output (e.g. the speaker). The tuner output does not require decoding, but final |
| 281 | audio output is mixed with software output_stream.</p> |
| 282 | |
| 283 | <p><img src="audio/images/ape_audio_tv_tuner.png" alt="Android TV Tuner Audio Patch" /> |
| 284 | <p class="img-caption"> |
| 285 | <strong>Figure 2.</strong> Audio Patch for TV tuner with speaker output.</p> |
| 286 | |
| 287 | |
| 288 | <h3>HDMI OUT during live TV</h3> |
| 289 | |
| 290 | <p>A user is watching live TV then switches to the HDMI audio output (Intent.ACTION_HDMI_AUDIO_PLUG) |
| 291 | . The output device of all output_streams changes to the HDMI_OUT port, and the TIF manager changes |
| 292 | the sink port of the existing tuner audio patch to the HDMI_OUT port.</p> |
| 293 | |
| 294 | <p><p><img src="audio/images/ape_audio_tv_hdmi_tuner.png" alt="Android TV HDMI-OUT Audio Patch" /> |
| 295 | <p class="img-caption"> |
| 296 | <strong>Figure 3.</strong> Audio Patch for HDMI OUT from live TV.</p> |