audio Audio Spatialization Effects in the Browser Janessa Det - WEBHACKER @TWITTERNYC experiential summary nodeS2noTE & PANNING START & resources THE WEB AUDIO API APPLICATIONS MOTIVATION sound&space CONVOLUTION DSP COMPONENTS SPATIALIZATION experiential audio primer SIMULATING ENVIRONMENTS

whoami. experientialaudio janessa det (@jandet) @twitternyc Software engineer - web Discovery Front-End thefuture.fm lead front-end dev DJ Defined Radio columbia university M.S. Computer science Graphics & Vision > UX/HCI summary

sound&space. sound in the digital age experientialaudio janessa det (@jandet) time vs. frequency digitalization browsers are 2d sound has become flat summary

sound&space. sound defines a space experientialaudio janessa det (@jandet) summary Sound interacts with surrounding space 3d space environment Environments perceived through sound human perception One of the key ways we perceive our surroundings Janet Cardiff The Forty Part Motet [1]

primer. alas! the web audio api experientialaudio janessa det (@jandet) compatibility features • audio signals processing for your browser • easy to create, process, and control audio within web applications • complements webgl • chrome 10.0-24.0 • safari 6.0 • safari ios 6.0 • firefox NOOOOPE (BUT PLANNED) • AUDIO DATA API summary

audio context primer. audio routing graph experientialaudio janessa det (@jandet) summary audio DESTINATION gain (volume) audio source convolver filter(s) audio source if  ('AudioContext'  in  window)  {    context  =  new  AudioContext(); }  else  if  ('webkitAudioContext'  in  window)  {    context  =  new  webkitAudioContext(); }  else  {    throw  new  Error('AudioContext  not  supported!'); }

nodes2note. types of audio nodes experientialaudio janessa det (@jandet) • raw audio data into the system • final node to connect to • outputs to lineout • speakers/headphones AUDIO SOURCE AUDIO DESTINATION var  request  =  new  XMLHttpRequest();        request.open("GET",  options.path,  true);        request.responseType  =  "arraybuffer"; request.onload  =  function()  { createSoundSource(request.response); }; request.send(); soundSource.connect(context.destination); function  createSoundSource(audioData)  { soundSource  =  context.createBufferSource(); soundSource.buffer  =   context.createBuffer(audioData,  false); soundSource.loop  =  true; }; summary

nodes2note. types of audio nodes experientialaudio janessa det (@jandet) • apply gain to change volume • 0: lowpass • 1: highpass • 2: bandpass • 3: lowshelf • 4: highshelf • 5: peaking • 6: notch • 7: allpass filterNode  =  context.createBiquadFilter(); filterNode.type  =  0; filterNode.frequency.value  =  220; volumeNode  =  context.createGainNode(); volumeNode.gain.value  =  0.5; summary gain (volume) filters

types of audio nodes experientialaudio janessa det (@jandet) summary • delay audio for specified duration nodes2note. delay oscillator delayNode  =  context.createDelayNode(); delayNode.delayTime.value  =  5; • 0: sine • 1: square • 2: sawtooth • 3: triangle • 4: custom oscillatorNode  =  context.createOscillatorNode(); oscillatorNode.type  =  0;

types of audio nodes experientialaudio janessa det (@jandet) summary • scriptprocessornode • synthesize and process audio directly with javascript • custom DSP effects • exposing internals of dsp to web applications nodes2note. fancypants js JS http://www.w3.org/TR/webaudio/

audio context audio routing graph experientialaudio janessa det (@jandet) summary audio DESTINATION gain (volume) filter audio source nodes2note. //  initialize  nodes ... sourceNode.connect(filterNode); filterNode.connect(volumeNode); volumeNode.connect(context.destination);

spatialization. INTRODUCTION TO panning experientialaudio janessa det (@jandet) panning algorithm • distance effects • sounds softer farther away • orientation • what direction are you facing • channels • adjust volume to l & R outputs • mimick human perception of space [1] summary

USING PANNING nodes experientialaudio janessa det (@jandet) summary • connect to an audio source • makes sound source positional • sound cones • inner angle • outer angle • outer gain • note: velocity! • one listener per context • position (x, y, z) • orientation • front vector • up vector spatialization. panning node listener : ) pannerNode  =  context.createPanner(); pannerNode.setPosition(x,  y,  z); pannerNode.coneInnerAngle  =  30; pannerNode.coneOuterAngle  =  40; pannerNode.coneOuterGain  =  0.5; context.listener.setPosition(x,  y,  z); var  front  =  getFrontVector(angle); var  up  =  [0,  0,  1]; context.listener.setOrientation( front.x,  front.y,  front.z,   up.x,  up.y,  up.z);

convolution. INTRODUCTION TO convolution experientialaudio janessa det (@jandet) what is convolution? h(t) x(t) y(t) = h(t) x(t) • “cross-multiplying” 2 signals • LTI system theory • impulse response summary

USING convolver nodes experientialaudio janessa det (@jandet) summary • set buffer as impulse response • connect sound source to convolver node • convolves input through impulse response • emulate sound in any environment convolution. convolver node var  request  =  new  XMLHttpRequest();        request.open("GET",  "sounds/ir/church.m4a",  true);        request.responseType  =  "arraybuffer";       request.onload  =  function  ()  { createConvolverNode(request.response); } request.send(); function  createConvolverNode(audioData)  { convolverNode  =  context.createConvolver(); convolverNode.buffer   =  context.createBuffer(audioData,  false); }

position orientation audio context audio routing graph experientialaudio janessa det (@jandet) summary audio DESTINATION convolution. //  initialize  nodes ... sourceNode.connect(convolverNode); convolverNode.connect(pannerNode); pannerNode.connect(context.destination); panner convolver audio source

demotime. experientialaudio janessa det (@jandet) interaction controls START sound&space nodeS2noTE SPATIALIZATION CONVOLUTION APPLICATIONS primer MOTIVATION THE WEB AUDIO API DSP COMPONENTS & PANNING SIMULATING ENVIRONMENTS & resources experiential audio summary • player • listener • panning sound • sound • panning sound • panning sound • panner node • sound node classes • wasd - movement • panning sound • q/e - rotate ccw/cw • x - make tweeting sound • expansions to come panner-demo.jit.su

applications. experientialaudio janessa det (@jandet) gaming enriched ux mobile web (ios6) augmented reality summary

experientialaudio janessa det (@jandet) summary future work areas to explore • VISUAL INTERFACE & TOOLS TO BUILD AUDIO ROUTING GRAPHS • INTERFACING WITH WEBGL explore full 3d orientation & positioning • INVESTIGATION OF AUDIO DATA API closer to the actual dsp implementations • WEB AUDIO API ON IOS6 SAFARI different ui, applications, and event handling resources. • rtfm http://www.w3.org/TR/webaudio/ • safari developer library http://bit.ly/RcVHxY • DAFX (ref [1]) • JOHN WILEY & SONS • MAR 8, 2011 • impulse responses http://fokkie.home.xs4all.nl/IR.htm

SUMMARY PRESENTATION OVERVIEW GOBACK ORCHOOSEAsection applications & resources spatialization & Panning simulating environments convolution nodes2note dsp components START experiential audio sound&space motivation the web audio api primer