JSMad - What, Why, How (JSConf EU 2012)

As part of my work at ofmlabs, I co-developed an MP3 decoder in pure JavaScript. I was kindly invited to present it to JSConf.eu.

Transcript

1. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
   jsmad
   What, Why, How ?
   

   View Slide

2. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
   What ?
   A pure JavaScript MP3 decoder
   

   View Slide

3. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
   « How do we play music on the web? »
   

   View Slide

4. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
   The uncle you wish you never had
   

   View Slide

5. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
   Flash 10 penetration
   Source : adobe.com
   

   View Slide

6. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
   « Adobe Flash Player has 'critical' security issue,
   won't be addressed until next week »
   — digitaltrends.com
   

   View Slide

7. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
   « 16 month-old bug continues
   to crash Flash »
   — tuaw.com
   

   View Slide

8. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
   « Flash has […] one of the worst security
   records in 2009. It is the number one reason
   Macs crash. »
   — Steve Jobs
   

   View Slide

9. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
   « The slightly clueless
   kid on the blocks. »
   

   View Slide

10. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « The second best
    reason to buy an i7. »
    

    View Slide

11. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    What ?
    A port of libmad
    

    View Slide

12. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « libmad - a clean-room high-quality
    GPL'ed MP3 decoder »
    

    View Slide

13. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « A port of libmad to JavaScript that
    allows MP3 playback in browsers
    without any external plug-ins. »
    

    View Slide

14. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « libmad has 24-bit PCM output,
    does 32-bit fxed point arithmetic &
    is fully ISO/IEC 11172 compliant »
    — underbit.com
    

    View Slide

15. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « jsmad uses foating-point
    arithmetic, and has not been tested
    for ISO/IEC compliance »
    — @nddrylliog + @jensnockert
    

    View Slide

16. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    Why ?
    The patent situation
    

    View Slide

17. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « We have patents on the psycho-acoustic models
    underlying MPEG-1 Layer III audio encoding. »
    — Fraunhofer & Thomson
    

    View Slide

18. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « You have to pay us money if you use MP3 encoders
    or decoders in for-proft software. »
    — Fraunhofer & Thomson
    

    View Slide

19. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « La la la can't hear you. »
    — Every streaming website
    

    View Slide

20. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    OGG MP3 AAC WAV
    IE 9 ✘ ✔ ✔ ✘
    Firefox 5 ✔ ✘ ✘ ✔
    Chrome 12 ✔ ✔ ✔ ✔
    Safari 5 ✘ ✔ ✔ ✔
    Opera 11.5 ✔ ✘ ✘ ✔
    

    View Slide

21. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « Firefox, Y U NO support MP3? »
    — Every web developer
    

    View Slide

22. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « We don't want to encourage proprietary formats. »
    — Mozilla
    

    View Slide

23. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « There, take a sticker and stop asking questions. »
    — Mozilla
    

    View Slide

24. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « Sorry, we don't do stickers. »
    — Opera
    

    View Slide

25. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « MP3 is rubbish anyway ! »
    — Anonymous
    

    View Slide

26. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « Yeah, yeah, I encode MP3 alright. »
    — iTunes
    

    View Slide

27. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « We developed our own psycho-acoustic models »
    — LAME project
    

    View Slide

28. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    

    View Slide

29. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    Why ?
    Features
    

    View Slide

30. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    

    View Slide

31. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « What do you mean, seeking ? »
    — HTML5 Audio
    

    View Slide

32. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « Seeking beyond the buffer
    has worked for ages. »
    — Adobe Flash
    

    View Slide

33. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    

    View Slide

34. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « What do you mean, live? »
    — HTML5 Audio
    

    View Slide

35. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « RTMP solved this years ago. »
    — Adobe Flash
    

    View Slide

36. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    

    View Slide

37. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « What do you mean, SFX? »
    — HTML5 Audio
    

    View Slide

38. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « Ever played a Flash game ? »
    — Adobe Flash
    

    View Slide

39. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « I decode MP3 streams without making
    assumptions about your container. »
    — jsmad
    

    View Slide

40. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    How ?
    How audio compession works
    

    View Slide

41. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    

    View Slide

42. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    

    View Slide

43. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    

    View Slide

44. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    

    View Slide

45. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    

    View Slide

46. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    

    View Slide

47. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    How ?
    How HTML5 works
    

    View Slide

48. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    HTML5
    
    Oh, wow, what are you using, Lynx?
    
    

    View Slide

49. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    HTML5
    
    
    
    
    

    View Slide

50. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    What HTML5 does for you
    ● Data input (fle://, http://)
    ● Buffering (206 Partial Content)
    ● Demuxing (MPEG, OGG, M4A)
    ● Decoding (MP3, Vorbis, AAC)
    ● Audio playback (CoreAudio, PulseAudio)
    ● User interface
    

    View Slide

51. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    What HTML5 does for you
    ● Data input
    ● Buffering
    ● Demuxing
    ● Decoding
    ● Audio playback
    ● User interface
    

    View Slide

52. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    What jsmad does
    ● Data input
    ● Buffering
    ● Demuxing
    ● Decoding
    ● Audio playback
    ● User interface
    

    View Slide

53. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    What jsmad.org runs
    ● Data input
    ● Buffering
    ● Demuxing
    ● Decoding = jsmad
    ● Audio playback = sink.js
    ● User interface
    

    View Slide

54. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    jsmad — decoding all frames
    var file = fileChooser.files[0];
    new Mad.FileStream(file, function(input) {
    var mp3 = new Mad.MP3File(input);
    var mpeg = mp3.getMpegStream();
    var frame = new Mad.Frame();
    while (mpeg.error != Mad.Error.BUFLEN) {
    frame = Mad.Frame.decode(frame);
    }
    });
    

    View Slide

55. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    sink.js — playing audio
    var channels = 2; // stereo
    var rate = 44100; // 44.1 Khz
    var bufsize = 65536 * 4096;
    function refill(buffer) {
    // …
    }
    var dev = Sink(refill, channels, bufsize, rate);
    

    View Slide

56. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    sink.js + jsmad
    var frame = /* … */;
    var synth = new Mad.Synth();
    function refill(buffer) {
    // if current frame all used up, decode new one
    frame = Mad.Frame.decode(frame);
    // synthesize new one if needed
    synth.frame(frame);
    // copy samples from synth to buffer
    // (TypedArray.set or for loop)
    }
    var dev = Sink(refill, channels, bufsize, rate);
    

    View Slide

57. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    How ?
    Low-level audio APIs
    

    View Slide

58. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    

    View Slide

59. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « JavaScript is fast enough. »
    — jsmad
    

    View Slide

60. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    « Just give us low-level access to devices, we'll
    handle it from there »
    — jsmad
    

    View Slide

61. JSConf EU 2011 github.com/audiocogs/jsmad Amos Wenger
    Thank you !
    github.com/audiocogs
    jsmad.org
    @nddrylliog
    

    View Slide