Ambisonic Audio Using the WebAudio API - Mario Guggenberger

360-degree videos are becoming mainstream media and promise an immersive experience. Adjusting the video viewport according to the user or device movements is just half the battle. Not only video but also audio is an important factor for an immersive experience. In this talk I will show how ambisonic audio can be handled in JavaScript using the WebAudio API, to enrich 360 video experiences with true "surround" sound.

Presented at Demuxed 2017 by Mario Guggenberger

Transcript

1. Ambisonic Audio 
 Using the WebAudio API Mario Guggenberger
2. Picture: http://demuxed.com
3. 360° IMMERSIVE VR
4. 360° IMMERSIVE VR Pictures: Facebook, VR Playhouse, Oculus, Beats by Dre, Wikimedia Commons, Shure ?
5. MICROPHONE PATTERNS Pictures: Wikimedia Commons omnidirectional cardioid figure of 8 / bidirectional
6. SPACIAL AUDIO RECORDING W X Y Z Pictures: University of Southern Queensland / Dynamics II, Shure
7. SPACIAL AUDIO RECORDING Picture: SoundField
8. AMBISONICS
9. FIRST-ORDER AMBISONICS (FOA) Pictures: http://pcfarina.eng.unipr.it/TBE-conversion.htm, Wikipedia “Ambisonics”
10. HIGH-ORDER AMBISONICS (HOA) Picture: http://pcfarina.eng.unipr.it/TBE-conversion.htm 0th order 1st order … 3rd order 2nd order
11. 1ST VS 3RD ORDER DIRECTIONALITY Picture: https://blog.audiokinetic.com/ambisonics-as-an-intermediate-spatial-representation-for-vr/
12. DECODING regular irregular headphones
13. source: Ambisonic Surround Sound FAQ http://members.tripod.com/martin_leese/Ambisonic/faq_latest.html
14. SOUND LOCALIZATION
15. SOUND LOCALIZATION: SIDEWARDS t1 t2 Time difference: t1 < t2 Level difference: i1 > i2 i2 i1
16. SOUND LOCALIZATION: MEDIAN
17. HRTF Head-Related Transfer Function
18. HRTF: MEASUREMENT Pictures: https://codeandsound.wordpress.com/tag/hrtf/
19. HRTF DB HRTF DB HRTF: APPLICATION SOUND HRTF DB FILTER FILTER Az, El R L binaural playback
20. 360° IMMERSIVE VR AUDIO ROTATION HRTF HRTF HRTF … Ambisonics + HRTF + Headphones = Binaural 3D Audio
21. WEB AUDIO API
22. Browser WEB AUDIO API Internet Asset HTMLMediaElement Audio Hardware Speakers AudioContext
23. WEB AUDIO API AudioContext Source (AudioNode) Effects (AudioNode) Destination (AudioDestinationNode) Audio Hardware HTMLMediaElement
24. WEB AUDIO API: EXAMPLE MediaElementAudioSourceNode AudioDestinationNode Audio Hardware HTMLMediaElement <video id="myMediaElement" src="videoWithAudio.mp4"></video>
 <script>
 const mediaElement = document.getElementById('myMediaElement');
 const audioContext = new AudioContext();
 const audioSource = audioContext.createMediaElementSource(mediaElement); // Connect audio graph
 audioSource.connect(audioContext.destination);
 </script>
25. WEB AUDIO API: AUDIO NODES OscillatorNode AudioBufferSourceNode AudioDestinationNode ConvolverNode MediaElementAudioSourceNode DelayNode GainNode StereoPannerNode DynamicsCompressorNode MediaStreamAudioSourceNode BiquadFilterNode WaveShaperNode IIRFilterNode MediaStreamAudioDestinationNode AnalyserNode ChannelSplitterNode ChannelMergerNode PannerNode AudioWorkerNode
26. WEB AUDIO API: EXAMPLE Gain <video id="myMediaElement" src="videoWithAudio.mp4"></video>
 <script>
 const mediaElement = document.getElementById('myMediaElement');
 const audioContext = new AudioContext();
 const audioSource = audioContext.createMediaElementSource(mediaElement);
 const gainNode = audioContext.createGain();
 // Connect audio graph
 audioSource.connect(gainNode);
 gainNode.connect(audioContext.destination);
 // Set parameters
 gainNode.gain.value = 0.5;
 </script> MediaElementAudioSourceNode AudioDestinationNode Audio Hardware HTMLMediaElement GainNode
27. WEB AUDIO API: EXAMPLE MediaElementAudioSourceNode AudioDestinationNode Audio Hardware HTMLMediaElement GainNode OscillatorNode <video id="myMediaElement" src="videoWithAudio.mp4"></video>
 <script>
 const mediaElement = document.getElementById('myMediaElement');
 const audioContext = new AudioContext();
 const audioSource = audioContext.createMediaElementSource(mediaElement);
 const gainNode = audioContext.createGain();
 const oscillatorNode = audioContext.createOscillator();
 // Connect audio graph
 audioSource.connect(gainNode);
 oscillatorNode.connect(gainNode.gain);
 gainNode.connect(audioContext.destination);
 // Set parameters
 oscillatorNode.type = 'sine’;
 oscillatorNode.frequency.value = 0.1; // Hz
 oscillatorNode.start();
 </script> Type Frequency
28. PRACTIAL CONSIDERATIONS • Limited number of AudioContext instances • Share instances between multiple video players • Use provided node types as much as possible • They cover many use-cases
29. AMBISONICS WITH THE 
 WEB AUDIO API
30. OPEN SOURCE LIBRARIES JSAmbisonics https://github.com/polarch/JSAmbisonics/ Omnitone https://github.com/GoogleChrome/omnitone/ InputNode OutputNode Source Destination
31. PRACTIAL CONSIDERATIONS • Channel ordering: Furse-Malham, SID, ACN • Channel normalization: maxN, N3D, SN3D • Browser format & channel support • Safari: 6ch AAC • Chrome, Firefox: 6ch AAC, 8ch WAV/OGG/OGG+MP4 • Channel order • E.g. 4ch AAC: • Chrome, Firefox: 0, 1, 2, 3 • Safari: 2, 0, 1, 3 • Test page: https://demo.bitmovin.com/public/mgu/ channeltest/
32. AMBISONICS WITH
 HTML5 VIDEO PLAYERS
33. VIDEO PLAYER REQUIREMENTS • VR video support • Access to viewing direction • Access to media element • API direct access, API container access, DOM query • Must not use MediaElementAudioSourceNod e • or provide API into audio path • Ambisonic track detection Player VR support video.js Yes (plugin) JWPlayer Yes (direction?) hls.js No THEOplayer Yes Bitmovin Yes Shaka No
34. AMBISONICS W/ BITMOVIN PLAYER Load Decoder Select track VR? ON_READY ON_AUDIOTRACK_CHANGED ON_VR_VIEWING_DIRECTION_CHANGE Enable Decoder new Ambisonics(player, config?) Disable Decoder Update rotation Ambisonic
 Track? Ambisonic? Decoder
 loaded?
35. AMBISONICS W/ BITMOVIN PLAYER <AdaptationSet segmentAlignment="true" lang="eng" id="4" bitmovin:label="Ambisonic"> <Role schemeIdUri="https://bitmovin.com/082017/vr" value="ambisonic-fo"/> <SegmentTemplate ... /> <Representation ... /> </AdaptationSet> #EXT-X-MEDIA:TYPE=AUDIO,GROUP- ID="ambisonic",NAME="Ambisonic",LANGUAGE="en",
 AUTOSELECT=YES,DEFAULT=YES,CHANNELS="4", URI="audio/ambisonic/stream.m3u8",
 CHARACTERISTICS="com.bitmovin.vr.ambisonic-fo" DASH MANIFEST HLS MASTER PLAYLIST
36. AMBISONICS W/ BITMOVIN PLAYER • Open Source • Uses Omnitone • 260 LOC • 23 kb • Demo (link in README) (headphones!) https://github.com/bitmovin/
 bitmovin-player-vr-ambisonics
37. THANK YOU :)