Share
Embed
Transcript
20 slides
-
audio Audio Spatialization Effects in the Browser Janessa Det -
WEBHACKER @TWITTERNYC experiential summary nodeS2noTE & PANNING START & resources THE WEB AUDIO API APPLICATIONS MOTIVATION sound&space CONVOLUTION DSP COMPONENTS SPATIALIZATION experiential audio primer SIMULATING ENVIRONMENTS -
whoami. experientialaudio janessa det (@jandet) @twitternyc Software engineer - web
Discovery Front-End thefuture.fm lead front-end dev DJ Defined Radio columbia university M.S. Computer science Graphics & Vision > UX/HCI summary -
sound&space. sound in the digital age experientialaudio janessa det (@jandet)
time vs. frequency digitalization browsers are 2d <audio></audio> sound has become flat summary -
sound&space. sound defines a space experientialaudio janessa det (@jandet) summary
Sound interacts with surrounding space 3d space environment Environments perceived through sound human perception One of the key ways we perceive our surroundings Janet Cardiff The Forty Part Motet [1] -
primer. alas! the web audio api experientialaudio janessa det (@jandet)
compatibility features • audio signals processing for your browser • easy to create, process, and control audio within web applications • complements webgl • chrome 10.0-24.0 • safari 6.0 • safari ios 6.0 • firefox NOOOOPE (BUT PLANNED) • AUDIO DATA API summary -
audio context primer. audio routing graph experientialaudio janessa det (@jandet)
summary audio DESTINATION gain (volume) audio source convolver filter(s) audio source if ('AudioContext' in window) { context = new AudioContext(); } else if ('webkitAudioContext' in window) { context = new webkitAudioContext(); } else { throw new Error('AudioContext not supported!'); } -
nodes2note. types of audio nodes experientialaudio janessa det (@jandet) •
raw audio data into the system • final node to connect to • outputs to lineout • speakers/headphones AUDIO SOURCE AUDIO DESTINATION var request = new XMLHttpRequest(); request.open("GET", options.path, true); request.responseType = "arraybuffer"; request.onload = function() { createSoundSource(request.response); }; request.send(); soundSource.connect(context.destination); function createSoundSource(audioData) { soundSource = context.createBufferSource(); soundSource.buffer = context.createBuffer(audioData, false); soundSource.loop = true; }; summary -
nodes2note. types of audio nodes experientialaudio janessa det (@jandet) •
apply gain to change volume • 0: lowpass • 1: highpass • 2: bandpass • 3: lowshelf • 4: highshelf • 5: peaking • 6: notch • 7: allpass filterNode = context.createBiquadFilter(); filterNode.type = 0; filterNode.frequency.value = 220; volumeNode = context.createGainNode(); volumeNode.gain.value = 0.5; summary gain (volume) filters -
types of audio nodes experientialaudio janessa det (@jandet) summary •
delay audio for specified duration nodes2note. delay oscillator delayNode = context.createDelayNode(); delayNode.delayTime.value = 5; • 0: sine • 1: square • 2: sawtooth • 3: triangle • 4: custom oscillatorNode = context.createOscillatorNode(); oscillatorNode.type = 0; -
types of audio nodes experientialaudio janessa det (@jandet) summary •
scriptprocessornode • synthesize and process audio directly with javascript • custom DSP effects • exposing internals of dsp to web applications nodes2note. fancypants js JS http://www.w3.org/TR/webaudio/ -
audio context audio routing graph experientialaudio janessa det (@jandet) summary
audio DESTINATION gain (volume) filter audio source nodes2note. // initialize nodes ... sourceNode.connect(filterNode); filterNode.connect(volumeNode); volumeNode.connect(context.destination); -
spatialization. INTRODUCTION TO panning experientialaudio janessa det (@jandet) panning algorithm
• distance effects • sounds softer farther away • orientation • what direction are you facing • channels • adjust volume to l & R outputs • mimick human perception of space [1] summary -
USING PANNING nodes experientialaudio janessa det (@jandet) summary • connect
to an audio source • makes sound source positional • sound cones • inner angle • outer angle • outer gain • note: velocity! • one listener per context • position (x, y, z) • orientation • front vector • up vector spatialization. panning node listener : ) pannerNode = context.createPanner(); pannerNode.setPosition(x, y, z); pannerNode.coneInnerAngle = 30; pannerNode.coneOuterAngle = 40; pannerNode.coneOuterGain = 0.5; context.listener.setPosition(x, y, z); var front = getFrontVector(angle); var up = [0, 0, 1]; context.listener.setOrientation( front.x, front.y, front.z, up.x, up.y, up.z); -
convolution. INTRODUCTION TO convolution experientialaudio janessa det (@jandet) what is
convolution? h(t) x(t) y(t) = h(t) x(t) • “cross-multiplying” 2 signals • LTI system theory • impulse response summary -
USING convolver nodes experientialaudio janessa det (@jandet) summary • set
buffer as impulse response • connect sound source to convolver node • convolves input through impulse response • emulate sound in any environment convolution. convolver node var request = new XMLHttpRequest(); request.open("GET", "sounds/ir/church.m4a", true); request.responseType = "arraybuffer"; request.onload = function () { createConvolverNode(request.response); } request.send(); function createConvolverNode(audioData) { convolverNode = context.createConvolver(); convolverNode.buffer = context.createBuffer(audioData, false); } -
position orientation audio context audio routing graph experientialaudio janessa det
(@jandet) summary audio DESTINATION convolution. // initialize nodes ... sourceNode.connect(convolverNode); convolverNode.connect(pannerNode); pannerNode.connect(context.destination); panner convolver audio source -
demotime. experientialaudio janessa det (@jandet) interaction controls START sound&space nodeS2noTE
SPATIALIZATION CONVOLUTION APPLICATIONS primer MOTIVATION THE WEB AUDIO API DSP COMPONENTS & PANNING SIMULATING ENVIRONMENTS & resources experiential audio summary • player • listener • panning sound • sound • panning sound • panning sound • panner node • sound node classes • wasd - movement • panning sound • q/e - rotate ccw/cw • x - make tweeting sound • expansions to come panner-demo.jit.su -
applications. experientialaudio janessa det (@jandet) gaming enriched ux mobile web
(ios6) augmented reality summary -
experientialaudio janessa det (@jandet) summary future work areas to explore
• VISUAL INTERFACE & TOOLS TO BUILD AUDIO ROUTING GRAPHS • INTERFACING WITH WEBGL explore full 3d orientation & positioning • INVESTIGATION OF AUDIO DATA API closer to the actual dsp implementations • WEB AUDIO API ON IOS6 SAFARI different ui, applications, and event handling resources. • rtfm http://www.w3.org/TR/webaudio/ • safari developer library http://bit.ly/RcVHxY • DAFX (ref [1]) • JOHN WILEY & SONS • MAR 8, 2011 • impulse responses http://fokkie.home.xs4all.nl/IR.htm -
SUMMARY PRESENTATION OVERVIEW GOBACK ORCHOOSEAsection applications & resources spatialization &
Panning simulating environments convolution nodes2note dsp components START experiential audio sound&space motivation the web audio api primer