Everyone a Rockstar: Using Mobile Technology in Live Musical Performance

Slides from our RejectJS 2015 talk (http://rejectjs.org/speakers/#JocelynHo_PianoKirbySaysHi)

Presenters: Jocelyn Ho and Drew Petersen

Code is here: https://github.com/kirbysayshi/synae-server
Eventually a recording will be made available!

Transcript

1. Everyone a Rockstar Using mobile technology in live musical experience
2. Jocelyn Ho @JocelynHo_Piano Young Steinway Artist Adjunct professor at Adelphi University. Drew Petersen @KirbySaysHi JS Engineer at Spotify
3. SYNAESTHESIA PLAYGROUND A collaborative project (a “playground”) for composers, visual artist, and software developer to explore music and the senses.
4. SYNAESTHESIA PLAYGROUND A collaborative project (a “playground”) for composers, visual artist, and software developer to explore music and the senses.
5. Hearing spatiality, color, and temperature
6. Hearing spatiality, color, and temperature • Low
7. Hearing spatiality, color, and temperature • Low • Bright
8. Hearing spatiality, color, and temperature • Low • Bright • Dull
9. Hearing spatiality, color, and temperature • Low • Bright • Dull • Warm
10. Hearing spatiality, color, and temperature • Low • Bright • Dull • Warm • Icy
11. A blending of the senses Synaesthesia
13. Maurice Merleau-Ponty (1908-1961)
14. Synaesthesia Playground One Part of Five Commissioned Works
17. Example of pianist’s gesture detected by Kinect
18. Example of pianist’s gesture detected by Kinect
19. Example of pianist’s gesture detected by Kinect
20. ? ? ? ? ?
21. Game (involvement) Performance (spectatorship)
27. Preview Performance September 16th, 2015 @ Spectrum NYC
32. But How?
33. “Lots and lots of JavaScript”
35. Challenge 1: Architecture
36. Rhizome https://github.com/sebpiq/rhizome A messaging server, with automatic support for: Websocket Clients OSC Clients Connection Management pub/sub serving static files (optional)
37. Typical Realtime Server App Server (has state & logic) Client Yo, what's going on? Here's your /world-state Here's stuff you missed because I know everything
38. Typical Realtime Server App Server (has state & logic) Client Client Client Client
39. Distributed Messaging Server App Server (has last message state) Client SuperClient (has state & logic) Keep me informed about /world-events OK! Hey tell every body about the new /world-events! New /world-events for ya!
40. Distributed Messaging Server App Server Client Conductor (has state & logic) Client Client Client Performer Windows Laptop Kinect
41. Distributed Messaging Server App Server Client Conductor (has state & logic) Client Client Client Performer Windows Laptop Kinect
42. Distributed Messaging Server App Server Client Conductor (has state & logic) Client Client Client Performer Windows Laptop Kinect
43. Distributed Messaging Server App Server Client Conductor (has state & logic) Client Client Client Performer Windows Laptop Kinect
44. Distributed Messaging Server App Server Client Conductor (has state & logic) Client Client Client Performer Windows Laptop Kinect
45. “Hey, the conductor always outputs state to the clients in one direction…”
46. React must always be mentioned during a JS talk, even in passing. Valid answers also include: one-way state, virtual DOM, etc.
47. React JS No, seriously the one-way data flow worked really well. World state flows from the conductor client… …to each audience client …which statelessly render. Each client is simply a terminal that knows its potential.
48. Also Browserify + Babelify ES7 property initializers are super cool http://babeljs.io/blog/2015/06/07/react-on-es6-plus/
49. Challenge 2: Networking aka "How many clients can my Airport Extreme really handle?"
50. 50.
51. Want more? http://funktion.fm/#post/rhizome-interactive-performances-and-network-topologies
52. Challenge 3: Audience Engineering What the WiFi!? What's an IP Address? Audio testing
53. Physical Cards
54. Be your own DNS https://github.com/tjfontaine/node-dns https://github.com/kirbysayshi/synae- server/blob/master/bin/dns Audience typed: http://playground.go var onMessage = function (request, response) { response.answer.push(dns.A({ name: request.question[0].name, address: RHIZOME_HOST, ttl: 600, })); response.send(); };
55. Challenge 4: The Kinect2 Do I really have to use Windows?
56. –The Windows Start Pane Button Thing “Do you want to program your own body, appendage, and finger tracking?”
57. –Me “Yes, that sounds like a lot of fun.”
58. –The Windows Start Pane Button Thing “Do you think you can do what others have spent entire PhD programs working towards in two weeks?”
59. You win, Windows. I'll use the SDK. (Actually he did —->) @TheOperatore https://github.com/theoperatore https://github.com/wouterverweirder/kinect2
60. You win, Windows. I'll use the SDK. (Actually he did —->) @TheOperatore https://github.com/theoperatore https://github.com/wouterverweirder/kinect2
63. Challenge 5: Accelerometer and Kinect Gestures Surely these are solved by npm install gesture-awesome, right?
64. Truly, they are not. And we all wondered why there aren't many fun gesture-based games… Dynamic Time Warping Hidden Markov Machines Support Vector Machines Signal Processing
65. DTW Sliding window of N accelerometer events. Given a pattern, match the stream against that pattern regardless of time stretching. devicemotion((e) => { let { motions } = this; motions.unshift(e); if (motions.length > 32) motions.pop(); let xs = motions.map(e => e.acceleration.x); let ys = motions.map(e => e.acceleration.y); let zs = motions.map(e => e.acceleration.z); let slashDTWX = new DTW(); let slashDTWY = new DTW(); let slashDTWZ = new DTW(); let slashCostX = slashDTWX.compute(xs, [-0.2486258743215352,-0.2486258743215352, ...]); let slashCostY = slashDTWY.compute(ys, [-0.35158794388510284,-0.35158794388510284, ...]); let slashCostZ = slashDTWZ.compute(zs, [0.14063087035836652,0.14063087035836652, ...]); let stillDTWX = new DTW(); let stillDTWY = new DTW(); let stillDTWZ = new DTW(); let stillCostX = stillDTWX.compute(xs, [0, 0, 0, 0]); let stillCostY = stillDTWY.compute(ys, [0, 0, 0, 0]); let stillCostZ = stillDTWZ.compute(zs, [0, 0, 0, 0]); let slashCost = slashCostX + slashCostY + slashCostZ; let stillCost = stillCostX + stillCostY + slashCostZ; if (slashCost < stillCost) { dbg('dm cost: slash', slashCost); dbg('dm cost: still', stillCost); this.triggerSound(); motions.length = 0; } });
66. DTW Images From: FastDTW: Toward Accurate Dynamic Time Warping in Linear Time and Space Stan Salvador and Philip Chan
67. But I used it completely wrong Scores needed to be normalized. It works by lucky math.
68. Generic Force Thresholds Sliding window of N accelerometer events Super gross. But it worked given the short time to implement. devicemotion((e) => { let { motions } = this; e.timestamp = Date.now(); motions.unshift(e); if (motions.length > 32) { motions.pop(); } let average = motions.reduce((sum, motion) => sum + motion.acceleration.y, 0) / motions.length; if (average > 1.8) { this.triggerSound(); this.motions.length = 0; } })
69. Simplify the Gestures Tug and Pull between tech reality (especially given time) and intention of the composer A slow raise of the arm is really hard to detect. What about a sharp reach instead?
70. Now let's do something that presenters are never supposed to do. (Rely on the WiFi)
71. http://192.168.4.17 DEMO TIME
72. http://temporary.live.server (backup)
73. WiFi: Synaesthesia Playground Pass: 12345678 http://192.168.1.225/ (backup for the backup)
74. What just happened?
78. Thanks! @JocelynHo_Piano @KirbySaysHi @synaeplay
 for Synaesthesia Playground updates https://github.com/ kirbysayshi/synae-server