Slide 1

Slide 1 text

Ambisonic Audio 
 Using the WebAudio API Mario Guggenberger

Slide 2

Slide 2 text

Picture: http://demuxed.com

Slide 3

Slide 3 text

360° IMMERSIVE VR

Slide 4

Slide 4 text

360° IMMERSIVE VR Pictures: Facebook, VR Playhouse, Oculus, Beats by Dre, Wikimedia Commons, Shure ?

Slide 5

Slide 5 text

MICROPHONE PATTERNS Pictures: Wikimedia Commons omnidirectional cardioid figure of 8 / bidirectional

Slide 6

Slide 6 text

SPACIAL AUDIO RECORDING W X Y Z Pictures: University of Southern Queensland / Dynamics II, Shure

Slide 7

Slide 7 text

SPACIAL AUDIO RECORDING Picture: SoundField

Slide 8

Slide 8 text

AMBISONICS

Slide 9

Slide 9 text

FIRST-ORDER AMBISONICS (FOA) Pictures: http://pcfarina.eng.unipr.it/TBE-conversion.htm, Wikipedia “Ambisonics”

Slide 10

Slide 10 text

HIGH-ORDER AMBISONICS (HOA) Picture: http://pcfarina.eng.unipr.it/TBE-conversion.htm 0th order 1st order … 3rd order 2nd order

Slide 11

Slide 11 text

1ST VS 3RD ORDER DIRECTIONALITY Picture: https://blog.audiokinetic.com/ambisonics-as-an-intermediate-spatial-representation-for-vr/

Slide 12

Slide 12 text

DECODING regular irregular headphones

Slide 13

Slide 13 text

source: Ambisonic Surround Sound FAQ http://members.tripod.com/martin_leese/Ambisonic/faq_latest.html

Slide 14

Slide 14 text

SOUND LOCALIZATION

Slide 15

Slide 15 text

SOUND LOCALIZATION: SIDEWARDS t1 t2 Time difference: t1 < t2 Level difference: i1 > i2 i2 i1

Slide 16

Slide 16 text

SOUND LOCALIZATION: MEDIAN

Slide 17

Slide 17 text

HRTF Head-Related Transfer Function

Slide 18

Slide 18 text

HRTF: MEASUREMENT Pictures: https://codeandsound.wordpress.com/tag/hrtf/

Slide 19

Slide 19 text

HRTF DB HRTF DB HRTF: APPLICATION SOUND HRTF DB FILTER FILTER Az, El R L binaural playback

Slide 20

Slide 20 text

360° IMMERSIVE VR AUDIO ROTATION HRTF HRTF HRTF … Ambisonics + HRTF + Headphones = Binaural 3D Audio

Slide 21

Slide 21 text

WEB AUDIO API

Slide 22

Slide 22 text

Browser WEB AUDIO API Internet Asset HTMLMediaElement Audio Hardware Speakers AudioContext

Slide 23

Slide 23 text

WEB AUDIO API AudioContext Source (AudioNode) Effects (AudioNode) Destination (AudioDestinationNode) Audio Hardware HTMLMediaElement

Slide 24

Slide 24 text

WEB AUDIO API: EXAMPLE MediaElementAudioSourceNode AudioDestinationNode Audio Hardware HTMLMediaElement 
 
 const mediaElement = document.getElementById('myMediaElement');
 const audioContext = new AudioContext();
 const audioSource = audioContext.createMediaElementSource(mediaElement); // Connect audio graph
 audioSource.connect(audioContext.destination);


Slide 25

Slide 25 text

WEB AUDIO API: AUDIO NODES OscillatorNode AudioBufferSourceNode AudioDestinationNode ConvolverNode MediaElementAudioSourceNode DelayNode GainNode StereoPannerNode DynamicsCompressorNode MediaStreamAudioSourceNode BiquadFilterNode WaveShaperNode IIRFilterNode MediaStreamAudioDestinationNode AnalyserNode ChannelSplitterNode ChannelMergerNode PannerNode AudioWorkerNode

Slide 26

Slide 26 text

WEB AUDIO API: EXAMPLE Gain 
 
 const mediaElement = document.getElementById('myMediaElement');
 const audioContext = new AudioContext();
 const audioSource = audioContext.createMediaElementSource(mediaElement);
 const gainNode = audioContext.createGain();
 // Connect audio graph
 audioSource.connect(gainNode);
 gainNode.connect(audioContext.destination);
 // Set parameters
 gainNode.gain.value = 0.5;
 MediaElementAudioSourceNode AudioDestinationNode Audio Hardware HTMLMediaElement GainNode

Slide 27

Slide 27 text

WEB AUDIO API: EXAMPLE MediaElementAudioSourceNode AudioDestinationNode Audio Hardware HTMLMediaElement GainNode OscillatorNode 
 
 const mediaElement = document.getElementById('myMediaElement');
 const audioContext = new AudioContext();
 const audioSource = audioContext.createMediaElementSource(mediaElement);
 const gainNode = audioContext.createGain();
 const oscillatorNode = audioContext.createOscillator();
 // Connect audio graph
 audioSource.connect(gainNode);
 oscillatorNode.connect(gainNode.gain);
 gainNode.connect(audioContext.destination);
 // Set parameters
 oscillatorNode.type = 'sine’;
 oscillatorNode.frequency.value = 0.1; // Hz
 oscillatorNode.start();
 Type Frequency

Slide 28

Slide 28 text

PRACTIAL CONSIDERATIONS • Limited number of AudioContext instances • Share instances between multiple video players • Use provided node types as much as possible • They cover many use-cases

Slide 29

Slide 29 text

AMBISONICS WITH THE 
 WEB AUDIO API

Slide 30

Slide 30 text

OPEN SOURCE LIBRARIES JSAmbisonics https://github.com/polarch/JSAmbisonics/ Omnitone https://github.com/GoogleChrome/omnitone/ InputNode OutputNode Source Destination

Slide 31

Slide 31 text

PRACTIAL CONSIDERATIONS • Channel ordering: Furse-Malham, SID, ACN • Channel normalization: maxN, N3D, SN3D • Browser format & channel support • Safari: 6ch AAC • Chrome, Firefox: 6ch AAC, 8ch WAV/OGG/OGG+MP4 • Channel order • E.g. 4ch AAC: • Chrome, Firefox: 0, 1, 2, 3 • Safari: 2, 0, 1, 3 • Test page: https://demo.bitmovin.com/public/mgu/ channeltest/

Slide 32

Slide 32 text

AMBISONICS WITH
 HTML5 VIDEO PLAYERS

Slide 33

Slide 33 text

VIDEO PLAYER REQUIREMENTS • VR video support • Access to viewing direction • Access to media element • API direct access, API container access, DOM query • Must not use MediaElementAudioSourceNod e • or provide API into audio path • Ambisonic track detection Player VR support video.js Yes (plugin) JWPlayer Yes (direction?) hls.js No THEOplayer Yes Bitmovin Yes Shaka No

Slide 34

Slide 34 text

AMBISONICS W/ BITMOVIN PLAYER Load Decoder Select track VR? ON_READY ON_AUDIOTRACK_CHANGED ON_VR_VIEWING_DIRECTION_CHANGE Enable Decoder new Ambisonics(player, config?) Disable Decoder Update rotation Ambisonic
 Track? Ambisonic? Decoder
 loaded?

Slide 35

Slide 35 text

AMBISONICS W/ BITMOVIN PLAYER #EXT-X-MEDIA:TYPE=AUDIO,GROUP- ID="ambisonic",NAME="Ambisonic",LANGUAGE="en",
 AUTOSELECT=YES,DEFAULT=YES,CHANNELS="4", URI="audio/ambisonic/stream.m3u8",
 CHARACTERISTICS="com.bitmovin.vr.ambisonic-fo" DASH MANIFEST HLS MASTER PLAYLIST

Slide 36

Slide 36 text

AMBISONICS W/ BITMOVIN PLAYER • Open Source • Uses Omnitone • 260 LOC • 23 kb • Demo (link in README) (headphones!) https://github.com/bitmovin/
 bitmovin-player-vr-ambisonics

Slide 37

Slide 37 text

THANK YOU :)