Make More Than Music with Tiny Computers, JavaScript & MIDI - JSConf Colombia 2017

Slide 1

Slide 1 text

MAKE MORE THAN MUSIC WITH TINY COMPUTERS, JAVASCRIPT & MIDI GEORGE MANDIS JSConf Colombia #jsconfco — @georgeMandis

Slide 2

Slide 2 text

GEORGE MANDIS TWITTER — EMAIL — GITHUB — WEB — WORK — Freelance web developer from Portland, Oregon. Developing for the web for over 10 years. That's... Maybe 45 in JavaScript years? @georgeMandis [email protected] @georgemandis george.mand.is snaptortoise.com

Slide 3

Slide 3 text

GEORGE MANDIS FUN FACTS! First time visting Colombia! Once lived as digital nomad in 18 countries Born in Saudi Arabia (Long story!) Most known projects — & Once unintentionally cheated running a marathon in North Korea Konami-JS Jeklly RSS Feeds

Slide 4

Slide 4 text

MORE RELEVANTLY I LIKE TO BREAK MAKE THINGS

Slide 5

Slide 5 text

LET'S START WITH AN EXPERIMENT...

Slide 6

Slide 6 text

WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL PERFORMED (POORLY) BY GEORGE MANDIS HTTP://JSCONFCO.MAND.IS/WS1

Slide 7

Slide 7 text

WHAT I'M TALKING ABOUT JAVASCRIPT (Un poco obvio!) TINY COMPUTERS Arduinos, Espruinos, Rasperry Pis, & more MIDI The Musical Instrument Device Interface

Slide 8

Slide 8 text

WHAT MAKES JAVASCRIPT + MIDI + TINY COMPUTERS EXCITING? JavaScript talking to real-world interfaces Low-barrier to entry towards making interactive hardware / web / IoT stuff A tool for art projects and creative explorations that touch on the physical world

Slide 9

Slide 9 text

¿POR QUE JAVASCRIPT?

Slide 10

Slide 10 text

JAVASCRIPT IS EVERYWHERE Clients, servers, websites, software, hardware, apps, bots, AIs, IoT, embedded tech... It's basically become the lingua franca of the web.

Slide 11

Slide 11 text

IT'S WHERE THE PEOPLE ARE In ~8 years Node.JS has made JavaScript a legitimate server-side contender and fundamentally changed how we develop for the web and beyond. If you want longevity in this industry you have to at least pay attention to where the crowds are gathering.

Slide 12

Slide 12 text

IT'S ASYNCHRONOUS NATURE A natural fit with MIDI

Slide 13

Slide 13 text

¿POR QUE TINY COMPUTERS? ¿POR QUE TINY COMPUTERS? ¿POR QUE TINY COMPUTERS? ¿POR QUE TINY COMPUTERS? ¿POR QUE TINY COMPUTERS?

Slide 14

Slide 14 text

¿POR QUE TINY COMPUTERS? What are “Tiny Computers” exactly? Single-board computers & single-board microcontrollers Commonly called — SBCs

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

RASPERRY PI

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

TINY COMPUTERS ARE... Affordable and (generally) easy to use Powerful-enough to perform a variety of tasks Great for stand-alone projects and creative tinkering Capable of running JavaScript natively (on many of them) FUN!

Slide 22

Slide 22 text

¿POR QUE MIDI?

Slide 23

Slide 23 text

¿POR QUE MIDI? What is MIDI exactly?

Slide 24

Slide 24 text

MIDI MUSICAL INSTRUMENT DEVICE INTERFACE Often used interchangeably to describe: The MIDI protocol created in 1983 The MIDI standard The MIDI file format

Slide 25

Slide 25 text

IT'S NOT THIS... “Passport” by Passport Designs

Slide 26

Slide 26 text

...OR THIS... “Canyon” by George Stone of Passport Designs

Slide 27

Slide 27 text

...OR EVEN THIS “Clouds” by Brian Orr

Slide 28

Slide 28 text

MIDI THE PROTOCOL Allows electronic musical instruments to communicate with other MIDI-capable devices This is done by sending and receiving 3-byte MIDI Messages

Slide 29

Slide 29 text

MIDI THE STANDARD Gives our MIDI messages meaning Most describe common instrument performance parameters — noteOn, noteOff, pitchbend, and others Think HTTP codes: 404 (Not Found), 301 (Moved Permanently), 418 (I'm a teapot) Maintained by the since 1985. Roland, Oberheim, Sequential Circuits, Yamaha, Korg, Kawai MIDI Association

Slide 30

Slide 30 text

MIDI THE FILE FORMAT Describe a sequence of MIDI events Describe the timing surrounding those events

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

REGRESAR A LA PREGUNTA... ¿POR QUE MIDI? MIDI is asynchronous (JavaScript!) MIDI talks to hardware (The IoT!) MIDI is easy to use for non-musical applications (Yay!)

Slide 34

Slide 34 text

CHROME SPEAKS MIDI! This includes Chrome OS, Chromium and Electron! :-D And so do lots of existing JavaScript & Node packages

Slide 35

Slide 35 text

WEB MIDI API

Slide 36

Slide 36 text

MIDI CONTROLLERS Pre-existing, USB-ready (plug-and-play) Old hardware can be used with a MIDI-USB adapter You can even build your own!

Slide 37

Slide 37 text

COMMON CONTROLLERS KEYBOARDS

Slide 38

Slide 38 text

COMMON CONTROLLERS KEYBOARDS

Slide 39

Slide 39 text

COMMON CONTROLLERS PADS & DIALS

Slide 40

Slide 40 text

COMMON CONTROLLERS BUTTONS AND SWITCHES

Slide 41

Slide 41 text

COMMON CONTROLLERS BUTTONS AND SWITCHES

Slide 42

Slide 42 text

LESS COMMON MIDI CONTROLLERS EXPRESSION PEDALS

Slide 43

Slide 43 text

LESS COMMON MIDI CONTROLLERS BREATH CONTROLLER

Slide 44

Slide 44 text

LESS COMMON MIDI CONTROLLERS MOTION-BASED

Slide 45

Slide 45 text

LESS COMMON MIDI CONTROLLERS BUILD YOUR OWN

Slide 46

Slide 46 text

LESS COMMON MIDI CONTROLLERS BUILD YOUR OWN

Slide 47

Slide 47 text

LESS COMMON MIDI CONTROLLERS BUILD YOUR OWN

Slide 48

Slide 48 text

ANATOMY OF A MIDI MESSAGE 128x0001 | 0x1111111 | 0x0000000 Every message is 3 bytes — 1 status, 2 data A status byte always begins with 1 and data bytes always begin with 0 For a status message, 3 bits of the first byte describe the type of status message. The remaining 4 describe the channel (16). For data messages that leaves 7 bits left per byte to describe the event — note, velocity, etc.

Slide 49

Slide 49 text

ANATOMY OF A MIDI MESSAGE 144x0001 | 0x0111100 | 0x1111111

Slide 50

Slide 50 text

ANATOMY OF A MIDI MESSAGE Example: 144x0001 | 0x0111100 | 0x1111111 status: "noteOn" channel: 1 data1: 60 data2: 127 In JavaScript: [144, 60, 127] LET'S TRY IT OUT!

Slide 51

Slide 51 text

WHAT DOES USING JAVASCRIPT & MIDI LOOK LIKE?

Slide 52

Slide 52 text

JAVASCRIPT + MIDI IN THE BROWSER

Slide 53

Slide 53 text

JAVASCRIPT + MIDI IN THE BROWSER Requesting access. navigator.requestMIDIAccess navigator.requestMIDIAccess().then( onMIDIInit, onMIDIFail ); const onMIDIInit = (MIDIAccessObject) => { // Allows us to start interacting with MIDIAccessObjects console.log(MIDIAccessObject); } const onMIDIFail = () => { // MIDI supported but access denied console.log("No MIDI access"); } 

Slide 54

Slide 54 text

JAVASCRIPT + MIDI IN THE BROWSER Iterating through the MIDI inputs and attaching listeners to MIDI messages as they're received const onMIDIInit = (MIDIAccessObject) => { for (let input of MIDIAccessObject.inputs.values()) { if (input.name === "Logidy UMI3") logidy = input; if (input.name === "PuckCC Bluetooth") puckJS = input; } logidy.onmidimessage = handleLogidyController; puckJS.onmidimessage = handlePuckController; } 

Slide 55

Slide 55 text

JAVASCRIPT + MIDI IN THE BROWSER Handling a received MIDI message Our 3-Byte MIDI message shows up as an array [144, 80, 127] noteOn — G# — 100% velocity let handlePuckController = (event) => { data = event.data; if (data[0] === 144 && data[1] === 80 && data[2] === 127) {...} } 

Slide 56

Slide 56 text

JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life a little easier WebMIDI.js WebMidi.enable((err) => { logidy = WebMidi.getInputByname("Logidy UMI3"); puck = WebMidi.getInputByname("PuckCC Bluetooth"); puck.addListener("noteon", "all", (event) => { let data = e.data; }); logidy.addListener("noteon", "all", (event) => { let data = e.data; }); }); 

Slide 57

Slide 57 text

JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life a little easier MIDIUtils.js puck.addListener("noteon", "all", (event) => { let frequency = MIDIUtils.noteNumberToFrequency(e.data[1]) let noteName = MIDIUtils.noteNumberToName(e.data[1]) let noteNumberFromName = MIDIUtils.noteNameToNoteNumber("A-4") let noteNumberFromFreq = MIDIUtils.frequencyToNoteNumber(440) }); 

Slide 58

Slide 58 text

JAVASCRIPT + MIDI ON THE SERVER What's different and what's the same We don't need to request permission Latency can be more of an issue in some contexts Working with MIDI files is easier (more packages) Some of the frontend libraries work on the server Do more interesting, system-level things

Slide 59

Slide 59 text

JAVASCRIPT + MIDI ON THE SERVER Basic setup for talking to and listening to a controller const midi = require("midi"); let input = new midi.input(); input.on('message', function(deltaTime, message) { console.log(message); // [144, 80, 127] }); input.openPort(0); 

Slide 60

Slide 60 text

THAT'S ALL THERE IS TO IT!

Slide 61

Slide 61 text

DEMO TIME! BEFORE WE GET TOO FAR INTO “MORE” THAN MUSIC WITH MIDI...

Slide 62

Slide 62 text

MIDI CONTROLLER EXPLORATION GETTING ACQUAINTED WITH YOUR HARDWARE

Slide 63

Slide 63 text

MAKING A SIMPLE SYNTH A 5-MINUTE POLYPHONIC SYNTHESIZER IN YOUR BROWSER

Slide 64

Slide 64 text

WEB MIDI & WEB AUDIO FUN WITH WEB AUDIO

Slide 65

Slide 65 text

WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL HTTP://JSCONFCO.MAND.IS/WS1

Slide 66

Slide 66 text

WEB SYMPHONY II SHARING PARTS OF MIDI FILE HTTP://JSCONFCO.MAND.IS/WS2

Slide 67

Slide 67 text

OKAY, NOW LET'S LOOK AT SOME OF THE “MORE” THAN MUSIC WITH MIDI...

Slide 68

Slide 68 text

THE MIDI POETRY BOX LET'S TAKE A LOOK!

Slide 69

Slide 69 text

MIDI PIXEL ART DRAW SIMPLE IMAGES ON A LAUNCHPAD

Slide 70

Slide 70 text

MIDI & CSS FILTERS MANIPULATING CSS FILTERS VIA HARDWARE

Slide 71

Slide 71 text

MIDI COLOR MIXING BY HAND ANOTHER COLOR MIXER WITH A MORE ABSTRACT CONTROLLER

Slide 72

Slide 72 text

MIDI BOUNCY BALLS ONE OF MY FIRST PROGRAMS

Slide 73

Slide 73 text

MIDI WHACK-A-MOLE GAME RECREATING A SILLY ARCADE GAME ON THE LPD8

Slide 74

Slide 74 text

MIDI MAZE RUNNER AN INTERACTIVE MAZE

Slide 75

Slide 75 text

MIDI CELLULAR AUTOMATA GENERATIVE PATTERNS + INTERACTIVITY

Slide 76

Slide 76 text

MIDI GO (AlphaGo AI not included) TURNING A NOVATION LAUNCHPAD INTO THE CLASSIC STRATEGY GAME

Slide 77

Slide 77 text

AND MANY MORE! Pitch-to-MIDI MIDI Chess MIDI Kiosk Menu The MIDI Security Stick (Come ask me!)

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

REPURPOSING THINGS FOR MUSIC WHY NOT REPURPOSE MUSIC FOR OTHER THINGS?

Slide 82

Slide 82 text

GRACIAS! George Mandis Demos and more info @ Email: Twitter: GitHub: Website: https://midi.mand.is [email protected] @georgeMandis georgemandis george.mand.is