Make More Than Music with Tiny Computers, JavaScript & MIDI @ OdessaJS 2017, Odessa, Ukraine

Read more about WebMIDI at https://midi.mand.is

Transcript

1. MAKE MORE THAN MUSIC WITH TINY COMPUTERS, JAVASCRIPT & MIDI GEORGE MANDIS ODESSAJS 2017 ODESSA, UKRAINE 1 / 82
2. GEORGE MANDIS WEB — EMAIL — TWITTER — GITHUB — BLOG — WORK — I'm a freelance web developer from the U.S. I've been developing for the web for over a decade. george.mand.is george@mand.is @georgeMandis @georgemandis george.mand.is snaptortoise.com 2 / 82
3. GEORGE MANDIS FUN FACTS! First time visiting Ukraine Lived as digital nomad in 18 countries Born in Saudi Arabia! Long story Once unintentionally cheated running a marathon in North Korea Most known project — — come see my other talk! :-D Konami-JS 3 / 82
4. LET'S START WITH AN EXPERIMENT... 4 / 82
5. WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL PERFORMED (POORLY) BY GEORGE MANDIS HTTP://ODESSAJS.MAND.IS/WS1 5 / 82
6. WEB SYMPHONY II JESU, JOY OF MAN'S DESIRING COURTESY OF BACH HTTP://ODESSAJS.MAND.IS/WS2 6 / 82
7. WHAT WE'RE TALKING ABOUT JAVASCRIPT (Obviously) TINY COMPUTERS Arduinos, Rasperry Pis, Espruinos & more MIDI The Musical Instrument Device Interface 7 / 82
8. WHY JAVASCRIPT? 8 / 82
9. JAVASCRIPT IS EVERYWHERE Clients, servers, websites, software, apps, bots, AIs, IoT, embedded tech... It's become the Lingua franca of the web. 9 / 82
10. IT'S ASYNCHRONOUS NATURE A natural fit with MIDI 10 / 82
11. IT'S WHERE THE PEOPLE ARE In ~8 years Node.JS has made JavaScript a legitimate server-side contender and fundamentally changed how we develop for the web and beyond. 11 / 82
12. WHY TINY COMPUTERS? WHY TINY COMPUTERS? WHY TINY COMPUTERS? WHY TINY COMPUTERS? WHY TINY COMPUTERS? 12 / 82
13. WHY TINY COMPUTERS? What are Tiny Computers exactly? 13 / 82
14. 14 / 82
15. 15 / 82
16. 16 / 82
17. RASPERRY PI 17 / 82
18. 18 / 82
19. 19 / 82
20. They're fast, affordable and easy to use Great for stand-alone projects and creative tinkering They're FUN! 20 / 82
21. WHY MIDI? 21 / 82
22. WHY MIDI? Wait — what is MIDI exactly? 22 / 82
23. MIDI Musical Instrument Device Interface It's a protocol created in 1983 It's a standard It's also a file format 23 / 82
24. IT'S NOT THIS... “Passport” by Passport Designs 24 / 82
25. ...OR THIS... “Canyon” by George Stone of Passport Designs 25 / 82
26. ...OR EVEN THIS “Clouds” by Brian Orr 25 / 82
27. MIDI THE PROTOCOL Allows electronic musical instruments to communicate with one another Describes common instrument performance parameters — noteOn, noteOff, pitchbend, programChange and others 27 / 82
28. MIDI THE STANDARD Maintained by the since 1985. Roland, Oberheim, Sequential Circuits, Yamaha, Korg, Kawai MIDI Association 28 / 82
29. MIDI THE FILE FORMAT Describe MIDI Events Describe the timing surrounding those events 29 / 82
30. 30 / 82
31. 31 / 82
32. BACK TO THE QUESTION... WHY MIDI? MIDI is asynchronous (JavaScript!) MIDI talks to hardware (The IoT!) MIDI is easy to use for non-musical applications (Yay!) 32 / 82
33. CHROME SPEAKS MIDI! :-D AND BY EXTENSION CHROME OS, CHROMIUM AND ELECTRON! And so do lots of other JavaScript projects 33 / 82
34. MIDI CONTROLLERS Pre-existing, USB-ready (plug-and-play) Old hardware can be used with a MIDI-USB adapter You can build your own! 34 / 82
35. KEYBOARDS & CONTROL PADS 35 / 82
36. KEYBOARDS & CONTROL PADS 36 / 82
37. BUTTONS AND SWITCHES 37 / 82
38. LESS COMMON MIDI CONTROLLERS Expression pedals Breathe & bite controllers Accelerometer based controllers Tenori On Pianocade Karlax from Da Fact Bananas... (via Makey Makey) 38 / 82
39. BUILD YOUR OWN! It's pretty easy to build a MIDI in/out device on Arduino. 39 / 82
40. ARDUINO MIDI CONTROLLER 40 / 82
41. ANATOMY OF A MIDI MESSAGE Every message is 3 bytes. There are only 2 types of messages: status and data. A status byte always begins with 1 and data bytes always begin with 0 That leaves 7 bits left per byte for expressing the message For a status message, 3 bits of the first byte describe the type of status message; the remaining 4 describe the channel 41 / 82
42. ANATOMY OF A MIDI MESSAGE Example: 128x0001 | 0x1111111 | 0x0000000 status: "noteOn", channel: 1, data1: 127, data2: 0 [1, 127, 0] 42 / 82
43. ANATOMY OF A MIDI MESSAGE 43 / 82
44. WHAT DOES USING JAVASCRIPT & MIDI LOOK LIKE? 43 / 82
45. JAVASCRIPT + MIDI IN THE BROWSER 43 / 82
46. JAVASCRIPT + MIDI IN THE BROWSER Requesting access. navigator.requestMIDIAccess navigator.requestMIDIAccess().then( onMIDIInit, onMIDIFail ); let onMIDIInit = function (MIDIAccessObject) { // Allows us to start interacting with MIDIAccessObjects ... } let onMIDIFail = function () { // MIDI supported but access denied ... }  46 / 82
47. JAVASCRIPT + MIDI IN THE BROWSER Iterating through the MIDI inputs and attaching listeners to MIDI messages as they're received let onMIDIInit = function (MIDIAccessObject) { for (let input of midi.inputs.values()) { if (input.name === "Logidy UMI3") logidy = input; if (input.name === "PuckCC Bluetooth") puckJS = input; } logidy.onmidimessage = handleLogidyController; puckJS.onmidimessage = handlePuckController; }  47 / 82
48. JAVASCRIPT + MIDI IN THE BROWSER Handling a received MIDI message Our 3-Byte MIDI message shows up as an array [0, 0, 0] let handlePuckController = function(event) { data = event.data; if (data[1] == 80) {...} }  48 / 82
49. JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life a little easier WebMIDI.js WebMidi.enable(function (err) { logidy = WebMidi.getInputByname("Logidy UMI3"); puck = WebMidi.getInputByname("PuckCC Bluetooth"); puck.addListener("noteon", "all", function(event) { let data = e.data; }); logidy.addListener("noteon", "all", function(event) { let data = e.data; }); });  49 / 82
50. JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life a little easier MIDIUtils.js puck.addListener("noteon", "all", function(event) { let frequency = MIDIUtils.noteNumberToFrequency(e.data[1]) let noteName = MIDIUtils.noteNumberToName(e.data[1]) let noteNumberFromName = MIDIUtils.noteNameToNoteNumber("A-4") let noteNumberFromFreq = MIDIUtils.frequencyToNoteNumber(440) });  50 / 82
51. THAT'S IT! 51 / 82
52. JAVASCRIPT + MIDI ON THE SERVER What's different and what's the same We don't need to request permission Latency can be more of an issue in some contexts Reading & writing MIDI files is easier (more packages) Some of the frontend libraries work on the server 52 / 82
53. JAVASCRIPT + MIDI ON THE SERVER Basic setup for talking to and listening to a controller const midi = require("midi"); let input = new midi.input(); input.on('message', function(deltaTime, message) { console.log(message, deltaTime); }); input.openPort(0);  53 / 82
54. DEMO TIME! BEFORE WE GET TOO FAR INTO “MORE” THAN MUSIC WITH MIDI... 54 / 82
55. MIDI CONTROLLER EXPLORATION GETTING ACQUAINTED WITH YOUR HARDWARE 55 / 82
56. MAKING A SIMPLE SYNTH UNDER THE HOOD WebMidi.js MIDIUtils.js Pizzicato.js A 5-MINUTE POLYPHONIC SYNTHESIZER IN YOUR BROWSER 56 / 82
57. WEB MIDI & WEB AUDIO UNDER THE HOOD WebMidi.js MIDIUtils.js Pizzicato.js FUN WITH CONTROL CHANGES 57 / 82
58. WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL UNDER THE HOOD Peer.js (WebRTC) MIDIUtils.js Pizzicato.js HTTP://ODESSAJS.MAND.IS/WS1 58 / 82
59. WEB SYMPHONY II SHARING PARTS OF MIDI FILE UNDER THE HOOD Express Socket.io (WebSockets) MIDIUtils.js midifile midievents HTTP://ODESSAJS.MAND.IS/WS2 59 / 82
60. OKAY, NOW LET'S LOOK AT SOME OF THE “MORE” THAN MUSIC WITH MIDI... 60 / 82
61. MIDI COLOR MIXER UNDER THE HOOD WebMidi.js TinyColor.js BUILD AN RGB AND HSL MIXER/PALETTE 61 / 82
62. THE MIDI POETRY BOX The interactive art project that inspired me to explore MIDI and JavaScript RECREATED HERE 62 / 82
63. THE MIDI POETRY BOX HOW IT WORKED Pick a random video from Pick a random book from Choose 4 random sentences from that book to make a "poem" Display the poem on top of the video as it plays at a randomized playback speed archive.org gutenberg.org 63 / 82
64. THE GOONIES SKELETON ORGAN POWERED BY MIDI? 64 / 82
65. 65 / 82
66. 66 / 82
67. 67 / 82
68. COULD WE MAKE SOMETHING LIKE THAT? KIND OF :) 68 / 82
69. COULD WE MAKE A *REAL* ONE? 69 / 82
70. 70 / 82
71. 71 / 82
72. MIDI & CSS FILTERS UNDER THE HOOD WebMidi.js MANIPULATE BLEEDING-EDGE CSS FEATURES 72 / 82
73. MIDI PIXEL ART UNDER THE HOOD Native WebMIDI implementation! var x = noteNumber & 0x0f; var y = (noteNumber & 0xf0) >> 4; DRAW SIMPLE IMAGES ON A LAUNCHPAD 73 / 82
74. MIDI HOT HAND COLOR MIXER UNDER THE HOOD WebMidi.js MIDIUtils.js ANOTHER COLOR MIXER WITH A MORE ABSTRACT CONTROLLER 74 / 82
75. MIDI HOT HAND COLOR CONTROL GAME UNDER THE HOOD WebMidi.js MIDIUtils.js A TWIST ON THE COLOR MIXER, AS A GAME 75 / 82
76. MIDI WHACK-A-MOLE GAME UNDER THE HOOD WebMidi.js RECREATING A SILLY ARCADE GAME ON THE LPD8 76 / 82
77. MIDI GO UNDER THE HOOD (AlphaGo AI not included) TURNING A NOVATION LAUNCHPAD INTO THE CLASSIC STRATEGY GAME Tenuki.js 77 / 82
78. 78 / 82
79. MUSICAL INTERACTIVITY IN THE PHYSICAL WORLD 79 / 82
80. MUSICAL INTERACTIVITY IN THE PHYSICAL WORLD 80 / 82
81. REPURPOSING THINGS FOR MUSIC WHY NOT REPURPOSE MUSIC FOR OTHER THINGS? 81 / 82
82. СПАСИБО! (THANK YOU!) George Mandis Email: Twitter: GitHub: Website: Demos and more info: george@mand.is @georgeMandis georgemandis george.mand.is midi.mand.is 82 / 82