Make More Than Music with Tiny Computers, JavaScript & MIDI - JSConf Colombia 2017

Slides from my presentation at JSConf Colombia (jsconf.co) 2017.

Read more at https://midi.mand.is

Transcript

1. MAKE MORE THAN MUSIC WITH TINY COMPUTERS, JAVASCRIPT & MIDI GEORGE MANDIS JSConf Colombia #jsconfco — @georgeMandis
2. GEORGE MANDIS TWITTER — EMAIL — GITHUB — WEB — WORK — Freelance web developer from Portland, Oregon. Developing for the web for over 10 years. That's... Maybe 45 in JavaScript years? @georgeMandis george@mand.is @georgemandis george.mand.is snaptortoise.com
3. GEORGE MANDIS FUN FACTS! First time visting Colombia! Once lived as digital nomad in 18 countries Born in Saudi Arabia (Long story!) Most known projects — & Once unintentionally cheated running a marathon in North Korea Konami-JS Jeklly RSS Feeds
4. MORE RELEVANTLY I LIKE TO BREAK MAKE THINGS
5. LET'S START WITH AN EXPERIMENT...
6. WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL PERFORMED (POORLY) BY GEORGE MANDIS HTTP://JSCONFCO.MAND.IS/WS1
7. WHAT I'M TALKING ABOUT JAVASCRIPT (Un poco obvio!) TINY COMPUTERS Arduinos, Espruinos, Rasperry Pis, & more MIDI The Musical Instrument Device Interface
8. WHAT MAKES JAVASCRIPT + MIDI + TINY COMPUTERS EXCITING? JavaScript talking to real-world interfaces Low-barrier to entry towards making interactive hardware / web / IoT stuff A tool for art projects and creative explorations that touch on the physical world
9. ¿POR QUE JAVASCRIPT?
10. JAVASCRIPT IS EVERYWHERE Clients, servers, websites, software, hardware, apps, bots, AIs, IoT, embedded tech... It's basically become the lingua franca of the web.
11. IT'S WHERE THE PEOPLE ARE In ~8 years Node.JS has made JavaScript a legitimate server-side contender and fundamentally changed how we develop for the web and beyond. If you want longevity in this industry you have to at least pay attention to where the crowds are gathering.
12. IT'S ASYNCHRONOUS NATURE A natural fit with MIDI
13. ¿POR QUE TINY COMPUTERS? ¿POR QUE TINY COMPUTERS? ¿POR QUE TINY COMPUTERS? ¿POR QUE TINY COMPUTERS? ¿POR QUE TINY COMPUTERS?
14. ¿POR QUE TINY COMPUTERS? What are “Tiny Computers” exactly? Single-board computers & single-board microcontrollers Commonly called — SBCs
18. RASPERRY PI
21. TINY COMPUTERS ARE... Affordable and (generally) easy to use Powerful-enough to perform a variety of tasks Great for stand-alone projects and creative tinkering Capable of running JavaScript natively (on many of them) FUN!
22. ¿POR QUE MIDI?
23. ¿POR QUE MIDI? What is MIDI exactly?
24. MIDI MUSICAL INSTRUMENT DEVICE INTERFACE Often used interchangeably to describe: The MIDI protocol created in 1983 The MIDI standard The MIDI file format
25. IT'S NOT THIS... “Passport” by Passport Designs
26. ...OR THIS... “Canyon” by George Stone of Passport Designs
27. ...OR EVEN THIS “Clouds” by Brian Orr
28. MIDI THE PROTOCOL Allows electronic musical instruments to communicate with other MIDI-capable devices This is done by sending and receiving 3-byte MIDI Messages
29. MIDI THE STANDARD Gives our MIDI messages meaning Most describe common instrument performance parameters — noteOn, noteOff, pitchbend, and others Think HTTP codes: 404 (Not Found), 301 (Moved Permanently), 418 (I'm a teapot) Maintained by the since 1985. Roland, Oberheim, Sequential Circuits, Yamaha, Korg, Kawai MIDI Association
30. MIDI THE FILE FORMAT Describe a sequence of MIDI events Describe the timing surrounding those events
33. REGRESAR A LA PREGUNTA... ¿POR QUE MIDI? MIDI is asynchronous (JavaScript!) MIDI talks to hardware (The IoT!) MIDI is easy to use for non-musical applications (Yay!)
34. CHROME SPEAKS MIDI! This includes Chrome OS, Chromium and Electron! :-D And so do lots of existing JavaScript & Node packages
35. WEB MIDI API
36. MIDI CONTROLLERS Pre-existing, USB-ready (plug-and-play) Old hardware can be used with a MIDI-USB adapter You can even build your own!
37. COMMON CONTROLLERS KEYBOARDS
38. COMMON CONTROLLERS KEYBOARDS
39. COMMON CONTROLLERS PADS & DIALS
40. COMMON CONTROLLERS BUTTONS AND SWITCHES
41. COMMON CONTROLLERS BUTTONS AND SWITCHES
42. LESS COMMON MIDI CONTROLLERS EXPRESSION PEDALS
43. LESS COMMON MIDI CONTROLLERS BREATH CONTROLLER
44. LESS COMMON MIDI CONTROLLERS MOTION-BASED
45. LESS COMMON MIDI CONTROLLERS BUILD YOUR OWN
46. LESS COMMON MIDI CONTROLLERS BUILD YOUR OWN
47. LESS COMMON MIDI CONTROLLERS BUILD YOUR OWN
48. ANATOMY OF A MIDI MESSAGE 128x0001 | 0x1111111 | 0x0000000 Every message is 3 bytes — 1 status, 2 data A status byte always begins with 1 and data bytes always begin with 0 For a status message, 3 bits of the first byte describe the type of status message. The remaining 4 describe the channel (16). For data messages that leaves 7 bits left per byte to describe the event — note, velocity, etc.
49. ANATOMY OF A MIDI MESSAGE 144x0001 | 0x0111100 | 0x1111111
50. ANATOMY OF A MIDI MESSAGE Example: 144x0001 | 0x0111100 | 0x1111111 status: "noteOn" channel: 1 data1: 60 data2: 127 In JavaScript: [144, 60, 127] LET'S TRY IT OUT!
51. WHAT DOES USING JAVASCRIPT & MIDI LOOK LIKE?
52. JAVASCRIPT + MIDI IN THE BROWSER
53. JAVASCRIPT + MIDI IN THE BROWSER Requesting access. navigator.requestMIDIAccess navigator.requestMIDIAccess().then( onMIDIInit, onMIDIFail ); const onMIDIInit = (MIDIAccessObject) => { // Allows us to start interacting with MIDIAccessObjects console.log(MIDIAccessObject); } const onMIDIFail = () => { // MIDI supported but access denied console.log("No MIDI access"); } 
54. JAVASCRIPT + MIDI IN THE BROWSER Iterating through the MIDI inputs and attaching listeners to MIDI messages as they're received const onMIDIInit = (MIDIAccessObject) => { for (let input of MIDIAccessObject.inputs.values()) { if (input.name === "Logidy UMI3") logidy = input; if (input.name === "PuckCC Bluetooth") puckJS = input; } logidy.onmidimessage = handleLogidyController; puckJS.onmidimessage = handlePuckController; } 
55. JAVASCRIPT + MIDI IN THE BROWSER Handling a received MIDI message Our 3-Byte MIDI message shows up as an array [144, 80, 127] noteOn — G# — 100% velocity let handlePuckController = (event) => { data = event.data; if (data[0] === 144 && data[1] === 80 && data[2] === 127) {...} } 
56. JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life a little easier WebMIDI.js WebMidi.enable((err) => { logidy = WebMidi.getInputByname("Logidy UMI3"); puck = WebMidi.getInputByname("PuckCC Bluetooth"); puck.addListener("noteon", "all", (event) => { let data = e.data; }); logidy.addListener("noteon", "all", (event) => { let data = e.data; }); }); 
57. JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life a little easier MIDIUtils.js puck.addListener("noteon", "all", (event) => { let frequency = MIDIUtils.noteNumberToFrequency(e.data[1]) let noteName = MIDIUtils.noteNumberToName(e.data[1]) let noteNumberFromName = MIDIUtils.noteNameToNoteNumber("A-4") let noteNumberFromFreq = MIDIUtils.frequencyToNoteNumber(440) }); 
58. JAVASCRIPT + MIDI ON THE SERVER What's different and what's the same We don't need to request permission Latency can be more of an issue in some contexts Working with MIDI files is easier (more packages) Some of the frontend libraries work on the server Do more interesting, system-level things
59. JAVASCRIPT + MIDI ON THE SERVER Basic setup for talking to and listening to a controller const midi = require("midi"); let input = new midi.input(); input.on('message', function(deltaTime, message) { console.log(message); // [144, 80, 127] }); input.openPort(0); 
60. THAT'S ALL THERE IS TO IT!
61. DEMO TIME! BEFORE WE GET TOO FAR INTO “MORE” THAN MUSIC WITH MIDI...
62. MIDI CONTROLLER EXPLORATION GETTING ACQUAINTED WITH YOUR HARDWARE
63. MAKING A SIMPLE SYNTH A 5-MINUTE POLYPHONIC SYNTHESIZER IN YOUR BROWSER
64. WEB MIDI & WEB AUDIO FUN WITH WEB AUDIO
65. WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL HTTP://JSCONFCO.MAND.IS/WS1
66. WEB SYMPHONY II SHARING PARTS OF MIDI FILE HTTP://JSCONFCO.MAND.IS/WS2
67. OKAY, NOW LET'S LOOK AT SOME OF THE “MORE” THAN MUSIC WITH MIDI...
68. THE MIDI POETRY BOX LET'S TAKE A LOOK!
69. MIDI PIXEL ART DRAW SIMPLE IMAGES ON A LAUNCHPAD
70. MIDI & CSS FILTERS MANIPULATING CSS FILTERS VIA HARDWARE
71. MIDI COLOR MIXING BY HAND ANOTHER COLOR MIXER WITH A MORE ABSTRACT CONTROLLER
72. MIDI BOUNCY BALLS ONE OF MY FIRST PROGRAMS
73. MIDI WHACK-A-MOLE GAME RECREATING A SILLY ARCADE GAME ON THE LPD8
74. MIDI MAZE RUNNER AN INTERACTIVE MAZE
75. MIDI CELLULAR AUTOMATA GENERATIVE PATTERNS + INTERACTIVITY
76. MIDI GO (AlphaGo AI not included) TURNING A NOVATION LAUNCHPAD INTO THE CLASSIC STRATEGY GAME
77. AND MANY MORE! Pitch-to-MIDI MIDI Chess MIDI Kiosk Menu The MIDI Security Stick (Come ask me!)
81. REPURPOSING THINGS FOR MUSIC WHY NOT REPURPOSE MUSIC FOR OTHER THINGS?
82. GRACIAS! George Mandis Demos and more info @ Email: Twitter: GitHub: Website: https://midi.mand.is george@mand.is @georgeMandis georgemandis george.mand.is