Make Your Browser Dance | LondonJS

Talk I gave at LondonJS Conf about audio detection with the Web Audio API

Transcript

1. MAKE YOUR BROWSER DANCE
2. HELLO I'm Ruth UX, Design, Front End Dev @rumyra Work in The Lab at O2
3. HAPPY NEW YEAR!
4. WINAMP 0:05
6. MY NAME IS RUTH AND I AM A VJ (No not a DJ) “VJing is a broad designation for realtime visual performance” - Wikipedia
7. GET TO THE POINT ALREADY! CSS Animations Web Audio API (+ others) Can we recreate my old uni days?
8. TWO THINGS Moving visual Analysed sound wave
10. CREATING CSS ANIMATION @ k e y f r a m e s f l a s h i n g { 0 % { o p a c i t y : 0 ; } 5 0 % { o p a c i t y : 1 ; } 1 0 0 % { o p a c i t y : 0 ; } } . l i g h t s i { a n i m a t i o n : f l a s h i n g 2 s i n f i n i t e ; } YOU'LL NEED VENDOR PREFIXES :) List of animatable properties:
11. GO MAD HTML LESS JS Result Edit on < s c r i p t s r c = " h t t p : / / w w w . j a s o n d a v i e s . c o m / d 3 . m i n . j s " > < / s c r i p t < s c r i p t s r c = " h t t p : / / w w w . j a s o n d a v i e s . c o m / w o r d c l o u d / d 3 . l a y o u t . c l o u d . j s < d i v c l a s s = " o u t e r " > < d i v i d = " c o n t a i n e r " > HTML SCSS JS Result Edit on < d i v c l a s s = " c o n t a i n e r " > < d i v c l a s s = " b o x " > < d i v c l a s s = " b o x " > h t m l { HAML Sass Result Edit on % u l . s t r i p - ( 1 . . 3 6 0 ) . e a c h d o | i | % l i . l i n e HTML LESS Result Edit on < d i v i d = " b a c k g r o u n d " > < d i v c l a s s = " c o n t a i n e r " > < d i v c l a s s = " d i c e a n i m a t e " > < d i v c l a s s = " p a n e l " >
12. I MEAN REALLY
13. EXPANDING ON THE LIGHT
14. NOW FOR SOME SOUND Create new instance Web Audio API v a r a u d i o C o n t e x t = ( w i n d o w . A u d i o C o n t e x t | | w i n d o w . w e b k i t A u d i o C o n t e x t | | w i n d o w . m o z A u d i o C o n t e x t ) ; i f ( c o n t e x t C l a s s ) { / / W e b A u d i o A P I i s a v a i l a b l e . v a r m y A u d i o C o n t e x t = n e w a u d i o C o n t e x t ( ) ; }
15. WEB AUDIO API Based around nodes Source node -> sound you are loading in Output node -> where the sound wants to come out
16. GET INPUT NODE Load audio HTTP request a u d i o element & c r e a t e M e d i a E l e m e n t S o u r c e method
17. VIA HTTP REQUEST f u n c t i o n l o a d S o u n d ( ) { v a r a u d i o F i l e U r l = ' / m y F i l e . o g g ' ; v a r r e q u e s t = n e w X M L H t t p R e q u e s t ( ) ; r e q u e s t . o p e n ( " G E T " , a u d i o F i l e U r l , t r u e ) ; r e q u e s t . r e s p o n s e T y p e = " a r r a y b u f f e r " ; r e q u e s t . o n l o a d = f u n c t i o n ( ) { / / t a k e f r o m h t t p r e q u e s t a n d d e c o d e i n t o b u f f e r m y A u d i o C o n t e x t . d e c o d e A u d i o D a t a ( r e q u e s t . r e s p o n s e , f u n c t i o n a u d i o B u f f e r = b u f f e r ; } ) ; } r e q u e s t . s e n d ( ) ; } v a r s o u r c e = m y A u d i o C o n t e x t . c r e a t e B u f f e r S o u r c e ( ) ; Use d e c o d e A u d i o D a t a to parse audio into buffer for use.
18. VIA a u d i o ELEMENT / / s e l e c t a u d i o e l e m e n t v a r a u d i o E l e m e n t = d o c u m e n t . g e t E l e m e n t B y I d ( ' s o u n d F i l e ' ) ; / / c r e a t i n g s o u r c e n o d e v a r s o u r c e = m y A u d i o C o n t e x t . c r e a t e M e d i a E l e m e n t S o u r c e ( a u d i o E l e m e n t ) ;
19. ALL THE THINGS Control volume / / C o n t r o l v o l u m e m y A u d i o C o n t e x t . c r e a t e G a i n ( ) ; Create filters / / C r e a t e f i l t e r m y A u d i o C o n t e x t . c r e a t e B i q u a d F i l t e r ( ) ; Create sound / / C r e a t e s o u n d m y A u d i o C o n t e x t . c r e a t e O s c i l l a t o r ( ) ; The by Boris Smus is free to read online!
20. ANALYSING We can do that :D / / c r e a t e i n s t a n c e v a r a n a l y s e r = m y A u d i o C o n t e x t . c r e a t e A n a l y s e r ( ) ; / / g e t d a t a a n a l y s e r . g e t B y t e F r e q u e n c y D a t a ( ) ; a n a l y s e r . g e t F l o a t F r e q u e n c y D a t a ( ) ; a n a l y s e r . g e t B y t e T i m e D o m a i n D a t a ( ) ;
21. A LITTLE THEORY Image by University of Utah, http://gslc.genetics.utah.edu
22. COLLECTING DATA Javascript typed array to pass in data Each array item reflects a frequency The gain of that frequncy is the value that's passed into that item v a r f r e q u e n c y D a t a = n e w U i n t 8 A r r a y ( 1 0 2 4 ) ; Know your frequency range by your file sample rate - it's half
23. PUTTING IT ALL TOGETHER Use r e q u e s t A n i m a t i o n F r a m e to call the analyser Get data and control the CSS animation e l e m e n t . s t y l e . a n i m a t i o n P l a y S t a t e = " p a u s e d " ; e l e m e n t . s t y l e . a n i m a t i o n P l a y S t a t e = " r u n n i n g " ; Loop through our lights, match them to a frequency and set them alight / / g e t l i g h t e l e m e n t s v a r l i g h t s = d o c u m e n t . g e t E l e m e n t s B y T a g N a m e ( ' i ' ) ; v a r t o t a l L i g h t s = l i g h t s . l e n g t h ; f o r ( v a r i = 0 ; i < t o t a l l i g h t s ; i + + ) = " " { = " " v a r = " " f r e q d a t a k e y = " i * 8 ; " i f = l i g h t s [ i ] . s t y l e . o p a c i t y = " 1 " ; } e l s e { l i g h t s [ i ] . s t y l e . o p a c i t y = " 0 . 2 " ; } } < / t o t a l l i g h t s ; >
24. REAL SOUND Let's make it bigger! We need environment audio No line in functionality Hello g e t U s e r M e d i a v a r m e d i a S t r e a m S o u r c e = m y A u d i o C o n t e x t . c r e a t e M e d i a S t r e a m S o u r c e ( s t r e a m ) ;
25. SEE FOR YOURSELF
26. WHAT'S NEXT? Different visuals canvas, SVG & 3D Other web API's (e.g. Ambient Light, Vibration)
27. THANK YOU
28. USEFUL LINKS