MAKE MORE THAN MUSIC
WITH TINY COMPUTERS,
JAVASCRIPT & MIDI
GEORGE MANDIS
ODESSAJS 2017
ODESSA, UKRAINE
1 / 82
Slide 2
Slide 2 text
GEORGE MANDIS
WEB —
EMAIL —
TWITTER —
GITHUB —
BLOG —
WORK —
I'm a freelance web developer from the U.S. I've been
developing for the web for over a decade.
george.mand.is
george@mand.is
@georgeMandis
@georgemandis
george.mand.is
snaptortoise.com
2 / 82
Slide 3
Slide 3 text
GEORGE MANDIS
FUN FACTS!
First time visiting Ukraine
Lived as digital nomad in 18 countries
Born in Saudi Arabia! Long story
Once unintentionally cheated running a marathon
in North Korea
Most known project — — come see my other
talk! :-D
Konami-JS
3 / 82
Slide 4
Slide 4 text
LET'S START WITH AN
EXPERIMENT...
4 / 82
Slide 5
Slide 5 text
WEB SYMPHONY I
THE INTERACTIVE PIANO RECITAL
PERFORMED (POORLY) BY GEORGE MANDIS
HTTP://ODESSAJS.MAND.IS/WS1
5 / 82
Slide 6
Slide 6 text
WEB SYMPHONY II
JESU, JOY OF MAN'S DESIRING
COURTESY OF BACH
HTTP://ODESSAJS.MAND.IS/WS2
6 / 82
Slide 7
Slide 7 text
WHAT WE'RE TALKING ABOUT
JAVASCRIPT
(Obviously)
TINY COMPUTERS
Arduinos, Rasperry Pis, Espruinos & more
MIDI
The Musical Instrument Device Interface
7 / 82
Slide 8
Slide 8 text
WHY JAVASCRIPT?
8 / 82
Slide 9
Slide 9 text
JAVASCRIPT IS EVERYWHERE
Clients, servers, websites, software, apps, bots, AIs, IoT,
embedded tech... It's become the Lingua franca of the
web.
9 / 82
Slide 10
Slide 10 text
IT'S ASYNCHRONOUS NATURE
A natural fit with MIDI
10 / 82
Slide 11
Slide 11 text
IT'S WHERE THE PEOPLE ARE
In ~8 years Node.JS has made JavaScript a legitimate
server-side contender and fundamentally changed how we
develop for the web and beyond.
11 / 82
WHY TINY COMPUTERS?
What are Tiny Computers exactly?
13 / 82
Slide 14
Slide 14 text
14 / 82
Slide 15
Slide 15 text
15 / 82
Slide 16
Slide 16 text
16 / 82
Slide 17
Slide 17 text
RASPERRY PI
17 / 82
Slide 18
Slide 18 text
18 / 82
Slide 19
Slide 19 text
19 / 82
Slide 20
Slide 20 text
They're fast, affordable and easy to use
Great for stand-alone projects and creative tinkering
They're FUN!
20 / 82
Slide 21
Slide 21 text
WHY MIDI?
21 / 82
Slide 22
Slide 22 text
WHY MIDI?
Wait — what is MIDI exactly?
22 / 82
Slide 23
Slide 23 text
MIDI
Musical Instrument Device Interface
It's a protocol created in 1983
It's a standard
It's also a file format
23 / 82
Slide 24
Slide 24 text
IT'S NOT THIS...
“Passport”
by Passport Designs
24 / 82
Slide 25
Slide 25 text
...OR THIS...
“Canyon”
by George Stone of Passport Designs
25 / 82
Slide 26
Slide 26 text
...OR EVEN THIS
“Clouds”
by Brian Orr
25 / 82
Slide 27
Slide 27 text
MIDI THE PROTOCOL
Allows electronic musical instruments to communicate
with one another
Describes common instrument performance
parameters — noteOn, noteOff, pitchbend,
programChange and others
27 / 82
Slide 28
Slide 28 text
MIDI THE STANDARD
Maintained by the since 1985.
Roland, Oberheim, Sequential Circuits, Yamaha, Korg, Kawai
MIDI Association
28 / 82
Slide 29
Slide 29 text
MIDI THE FILE FORMAT
Describe MIDI Events
Describe the timing surrounding those events
29 / 82
Slide 30
Slide 30 text
30 / 82
Slide 31
Slide 31 text
31 / 82
Slide 32
Slide 32 text
BACK TO THE QUESTION...
WHY MIDI?
MIDI is asynchronous (JavaScript!)
MIDI talks to hardware (The IoT!)
MIDI is easy to use for non-musical applications (Yay!)
32 / 82
Slide 33
Slide 33 text
CHROME SPEAKS MIDI!
:-D
AND BY EXTENSION CHROME OS,
CHROMIUM AND ELECTRON!
And so do lots of other JavaScript projects
33 / 82
Slide 34
Slide 34 text
MIDI CONTROLLERS
Pre-existing, USB-ready (plug-and-play)
Old hardware can be used with a MIDI-USB adapter
You can build your own!
34 / 82
Slide 35
Slide 35 text
KEYBOARDS & CONTROL PADS
35 / 82
Slide 36
Slide 36 text
KEYBOARDS & CONTROL PADS
36 / 82
Slide 37
Slide 37 text
BUTTONS AND SWITCHES
37 / 82
Slide 38
Slide 38 text
LESS COMMON MIDI
CONTROLLERS
Expression pedals
Breathe & bite controllers
Accelerometer based controllers
Tenori On
Pianocade
Karlax from Da Fact
Bananas... (via Makey Makey)
38 / 82
Slide 39
Slide 39 text
BUILD YOUR OWN!
It's pretty easy to build a MIDI in/out device on Arduino.
39 / 82
Slide 40
Slide 40 text
ARDUINO MIDI CONTROLLER
40 / 82
Slide 41
Slide 41 text
ANATOMY OF A MIDI MESSAGE
Every message is 3 bytes.
There are only 2 types of messages: status and data.
A status byte always begins with 1 and data bytes
always begin with 0
That leaves 7 bits left per byte for expressing the
message
For a status message, 3 bits of the first byte describe
the type of status message; the remaining 4 describe
the channel
41 / 82
WHAT DOES USING
JAVASCRIPT & MIDI LOOK LIKE?
43 / 82
Slide 45
Slide 45 text
JAVASCRIPT + MIDI IN THE BROWSER
43 / 82
Slide 46
Slide 46 text
JAVASCRIPT + MIDI IN THE BROWSER
Requesting access.
navigator.requestMIDIAccess
navigator.requestMIDIAccess().then( onMIDIInit, onMIDIFail );
let onMIDIInit = function (MIDIAccessObject) {
// Allows us to start interacting with MIDIAccessObjects
...
}
let onMIDIFail = function () {
// MIDI supported but access denied
...
}
46 / 82
Slide 47
Slide 47 text
JAVASCRIPT + MIDI IN THE BROWSER
Iterating through the MIDI inputs and attaching listeners
to MIDI messages as they're received
let onMIDIInit = function (MIDIAccessObject) {
for (let input of midi.inputs.values()) {
if (input.name === "Logidy UMI3") logidy = input;
if (input.name === "PuckCC Bluetooth") puckJS = input;
}
logidy.onmidimessage = handleLogidyController;
puckJS.onmidimessage = handlePuckController;
}
47 / 82
Slide 48
Slide 48 text
JAVASCRIPT + MIDI IN THE BROWSER
Handling a received MIDI message
Our 3-Byte MIDI message shows up as an array
[0, 0, 0]
let handlePuckController = function(event) {
data = event.data;
if (data[1] == 80) {...}
}
48 / 82
Slide 49
Slide 49 text
JAVASCRIPT + MIDI IN THE BROWSER
Libraries to make life a little easier
WebMIDI.js
WebMidi.enable(function (err) {
logidy = WebMidi.getInputByname("Logidy UMI3");
puck = WebMidi.getInputByname("PuckCC Bluetooth");
puck.addListener("noteon", "all", function(event) {
let data = e.data;
});
logidy.addListener("noteon", "all", function(event) {
let data = e.data;
});
});
49 / 82
Slide 50
Slide 50 text
JAVASCRIPT + MIDI IN THE BROWSER
Libraries to make life a little easier
MIDIUtils.js
puck.addListener("noteon", "all", function(event) {
let frequency = MIDIUtils.noteNumberToFrequency(e.data[1])
let noteName = MIDIUtils.noteNumberToName(e.data[1])
let noteNumberFromName = MIDIUtils.noteNameToNoteNumber("A-4")
let noteNumberFromFreq = MIDIUtils.frequencyToNoteNumber(440)
});
50 / 82
Slide 51
Slide 51 text
THAT'S IT!
51 / 82
Slide 52
Slide 52 text
JAVASCRIPT + MIDI ON THE SERVER
What's different and what's the same
We don't need to request permission
Latency can be more of an issue in some contexts
Reading & writing MIDI files is easier (more packages)
Some of the frontend libraries work on the server
52 / 82
Slide 53
Slide 53 text
JAVASCRIPT + MIDI ON THE SERVER
Basic setup for talking to and listening to a controller
const midi = require("midi");
let input = new midi.input();
input.on('message', function(deltaTime, message) {
console.log(message, deltaTime);
});
input.openPort(0);
53 / 82
Slide 54
Slide 54 text
DEMO TIME!
BEFORE WE GET TOO FAR INTO
“MORE” THAN MUSIC WITH MIDI...
54 / 82
Slide 55
Slide 55 text
MIDI CONTROLLER
EXPLORATION
GETTING ACQUAINTED WITH YOUR
HARDWARE
55 / 82
Slide 56
Slide 56 text
MAKING A SIMPLE SYNTH
UNDER THE HOOD
WebMidi.js
MIDIUtils.js
Pizzicato.js
A 5-MINUTE POLYPHONIC
SYNTHESIZER IN YOUR BROWSER
56 / 82
Slide 57
Slide 57 text
WEB MIDI & WEB AUDIO
UNDER THE HOOD
WebMidi.js
MIDIUtils.js
Pizzicato.js
FUN WITH CONTROL CHANGES
57 / 82
Slide 58
Slide 58 text
WEB SYMPHONY I
THE INTERACTIVE PIANO RECITAL
UNDER THE HOOD
Peer.js (WebRTC)
MIDIUtils.js
Pizzicato.js
HTTP://ODESSAJS.MAND.IS/WS1
58 / 82
Slide 59
Slide 59 text
WEB SYMPHONY II
SHARING PARTS OF MIDI FILE
UNDER THE HOOD
Express
Socket.io (WebSockets)
MIDIUtils.js
midifile
midievents
HTTP://ODESSAJS.MAND.IS/WS2
59 / 82
Slide 60
Slide 60 text
OKAY, NOW LET'S LOOK AT
SOME OF THE “MORE” THAN
MUSIC WITH MIDI...
60 / 82
Slide 61
Slide 61 text
MIDI COLOR MIXER
UNDER THE HOOD
WebMidi.js
TinyColor.js
BUILD AN RGB AND HSL
MIXER/PALETTE
61 / 82
Slide 62
Slide 62 text
THE MIDI POETRY BOX
The interactive art project that inspired me to explore MIDI
and JavaScript
RECREATED HERE
62 / 82
Slide 63
Slide 63 text
THE MIDI POETRY BOX
HOW IT WORKED
Pick a random video from
Pick a random book from
Choose 4 random sentences from that book to make a
"poem"
Display the poem on top of the video as it plays at a
randomized playback speed
archive.org
gutenberg.org
63 / 82
Slide 64
Slide 64 text
THE GOONIES
SKELETON ORGAN
POWERED BY MIDI?
64 / 82
Slide 65
Slide 65 text
65 / 82
Slide 66
Slide 66 text
66 / 82
Slide 67
Slide 67 text
67 / 82
Slide 68
Slide 68 text
COULD WE MAKE SOMETHING
LIKE THAT?
KIND OF :)
68 / 82
Slide 69
Slide 69 text
COULD WE MAKE A *REAL* ONE?
69 / 82
Slide 70
Slide 70 text
70 / 82
Slide 71
Slide 71 text
71 / 82
Slide 72
Slide 72 text
MIDI & CSS FILTERS
UNDER THE HOOD
WebMidi.js
MANIPULATE BLEEDING-EDGE CSS
FEATURES
72 / 82
Slide 73
Slide 73 text
MIDI PIXEL ART
UNDER THE HOOD
Native WebMIDI implementation!
var x = noteNumber & 0x0f;
var y = (noteNumber & 0xf0) >> 4;
DRAW SIMPLE IMAGES ON A
LAUNCHPAD
73 / 82
Slide 74
Slide 74 text
MIDI HOT HAND COLOR MIXER
UNDER THE HOOD
WebMidi.js
MIDIUtils.js
ANOTHER COLOR MIXER WITH A
MORE ABSTRACT CONTROLLER
74 / 82
Slide 75
Slide 75 text
MIDI HOT HAND COLOR
CONTROL GAME
UNDER THE HOOD
WebMidi.js
MIDIUtils.js
A TWIST ON THE COLOR MIXER, AS A
GAME
75 / 82
Slide 76
Slide 76 text
MIDI WHACK-A-MOLE GAME
UNDER THE HOOD
WebMidi.js
RECREATING A SILLY ARCADE GAME
ON THE LPD8
76 / 82
Slide 77
Slide 77 text
MIDI GO
UNDER THE HOOD
(AlphaGo AI not included)
TURNING A NOVATION LAUNCHPAD
INTO THE CLASSIC STRATEGY GAME
Tenuki.js
77 / 82
Slide 78
Slide 78 text
78 / 82
Slide 79
Slide 79 text
MUSICAL INTERACTIVITY IN THE
PHYSICAL WORLD
79 / 82
Slide 80
Slide 80 text
MUSICAL INTERACTIVITY IN THE
PHYSICAL WORLD
80 / 82
Slide 81
Slide 81 text
REPURPOSING THINGS FOR MUSIC
WHY NOT REPURPOSE MUSIC FOR OTHER
THINGS?
81 / 82
Slide 82
Slide 82 text
СПАСИБО!
(THANK YOU!)
George Mandis
Email: Twitter:
GitHub:
Website:
Demos and more info:
george@mand.is @georgeMandis
georgemandis
george.mand.is
midi.mand.is
82 / 82