rMIXr: how we learned to stop worrying and love the graph

In this talk we present a case study in the use of the web audio APIs. Specifically, our use of them for the creation of a rapidly developed
prototype application. The app, called rMIXr (http://rmixr.com), is a simple digital audio workstation (DAW) for fan remix contests. We created rMIXr in 48 hours at the Midem Hack Day in June 2015. We’ll give a brief demo of the app and show multi-channel sync. We'll also show various effects as well as cutting/time-slicing.
During the talk we will go through our solutions to these issues, showing by example a number of practical ways to get things done with the Web Audio APIs. We will walk through our abstractions and object models for building a fully functional DAW in the browser. These abstractions were necessary because we included a number of effects from third party libraries. Some of these effects libraries do not comply with the “source and sink” model that the web audio API predicates. Our abstractions gave us a more flexible graph structure that was critical to making this application work. While not perfect, it was certainly excellent for the rapid prototype scenario of the hackday. We’ll also talk about some potential improvements to our abstractions. We will then show how to use localstorage to communicate across browser windows and how this approach can be harnessed to quickly create flexible UI elements with multiple windows.
Slides as given at the 2016 Web Audio Conference in Atlanta, GA. Full abstract at https://smartech.gatech.edu/handle/1853/54668

Transcript

1. rMIXr:
   how we learned to stop
   worrying and love the graph
   

   View Slide

2. Ben Fields
   @alsothings
   Sam Phippen
   @samphippen
   

   View Slide

3. this talk represents our experience
   using the web audio APIs to go
   from nothing to a reasonably
   complete prototype in 48 hours
   about a year ago.
   actual API usage may vary.
   

   View Slide

4. ⚠⚠⚠⚠⚠⚠⚠⚠⚠⚠
   ACTUAL API
   USAGE MAY VARY
   ⚠⚠⚠⚠⚠⚠⚠⚠⚠⚠
   

   View Slide

5. Midem Hack Day
   

   View Slide

6. [[[[midem branding]]]]
   

   View Slide

7. View Slide

8. DJ Spotify
   festivalBag
   

   View Slide

9. rMIXr: white-labelled
   DAW in the browser
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. Effects in rMIXr
    

    View Slide

14. `
    

    View Slide

15. View Slide

16. Source Sink
    

    View Slide

17. Source Sink
    .connect
    

    View Slide

18. Source Sink
    .connect
    

    View Slide

19. Source Sink
    Fx/
    Analysis
    

    View Slide

20. Source Sink
    Fx/
    Analysis
    .connect
    

    View Slide

21. Source Sink
    Fx/
    Analysis
    .connect .connect
    

    View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. OMG NO
    

    View Slide

27. Source Sink
    Fx/
    Analysis
    .connect .connect
    

    View Slide

28. Converb
    Converb
    .input
    Source Sink
    .connect .connect
    

    View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. View Slide

35. View Slide

36. How to do UI for effects?
    

    View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. View Slide

42. View Slide

43. Icon font didn’t load lol
    

    View Slide

44. View Slide

45. View Slide

46. View Slide

47. View Slide

48. View Slide

49. Ben will explain how this works
    

    View Slide

50. The local storage API is
    totally a message bus
    

    View Slide

51. Last write wins is totally a
    good consistency model
    

    View Slide

52. View Slide

53. server side audio
    rendering is for suckers
    

    View Slide

54. thus: need to serialise
    state and associate it with
    page location
    

    View Slide

55. View Slide

56. every time you change a setting,
    

    View Slide

57. every time you change a setting,
    the new state is serialised to the url
    

    View Slide

58. generalise to *all* UI
    elements
    

    View Slide

59. View Slide

60. View Slide

61. View Slide

62. View Slide

63. a channel’s state
    channel strip’s state
    

    View Slide

64. View Slide

65. hash decoding all the way down
    

    View Slide

66. http://rmixr.com/queen/#{"channels":[{"effects":[],"loopPoints":
    [],"notmuted":true},{"effects":[{"name":"threebandEQ","params":
    {"bassgain":"21","midgain":"-24","treblegain":"16"}},
    {"name":"chorus","params":{"rate":2.41,"feedback":0.2,"delay":
    0.05}}],"loopPoints":[]},{"effects":[],"loopPoints":
    [],"notmuted":true},{"effects":[{"name":"converb","params":
    {"dry":0.5,"wet":0.59}}],"loopPoints":[],"notmuted":false},
    {"effects":[{"name":"stereoPan","params":
    {"pan":-0.1}}],"loopPoints":[],"notmuted":true},{"effects":
    [{"name":"stereoPan","params":{"pan":0.11}}],"loopPoints":
    [],"notmuted":false}]}
    

    View Slide

67. the fragment id
    *is*
    the application state
    engine
    

    View Slide

68. at least tweeting and
    entry submission is easy
    

    View Slide

69. View Slide

70. thanks twitter.
    

    View Slide

71. View Slide

72. at least tweeting and
    entry submission is easy
    

    View Slide

73. at least the back button
    works
    

    View Slide

74. {this is where
    we attempt
    a demo}
    

    View Slide

75. so:
    

    View Slide

76. unifying abstractions
    

    View Slide

77. local storage can be a
    messaging queue
    

    View Slide

78. even single page web
    apps can be RESTful
    

    View Slide

79. Let’s have some
    questions
    a!/samphippen
    !/alsothings
    

    View Slide