MAKE MORE THAN MUSIC
WITH TINY COMPUTERS,
JAVASCRIPT & MIDI
GEORGE MANDIS
HOLYJS 2017 — ST. PETERSBURG
1 / 95
Slide 2
Slide 2 text
GEORGE MANDIS
WEB —
EMAIL —
TWITTER —
GITHUB —
WORK —
george.mand.is
[email protected]
@georgeMandis
@georgemandis
snaptortoise.com
2 / 95
Slide 3
Slide 3 text
GEORGE MANDIS
FUN FACTS!
First time in Russia
Lived as digital nomad in 18 countries
Born in Saudi Arabia! Long story
Once unintentionally cheated running a marathon
in North Korea
Most known project — Konami-JS
3 / 95
Slide 4
Slide 4 text
GEORGE MANDIS
About Konami JS
My history with creating frivolous things.
4 / 95
Slide 5
Slide 5 text
GEORGE MANDIS
About Konami JS
My history with creating frivolous things.
5 / 95
Slide 6
Slide 6 text
LET'S START WITH AN
EXPERIMENT...
6 / 95
Slide 7
Slide 7 text
WEBRTC SYMPHONY I
THE INTERACTIVE PIANO RECITAL
PERFORMED (POORLY) BY GEORGE MANDIS
HTTP://HOLYJS.MAND.IS/WS1
7 / 95
Slide 8
Slide 8 text
WEBRTC SYMPHONY II
JESU, JOY OF MAN'S DESIRING
COURTESY OF BACH
HTTP://HOLYJS.MAND.IS/WS2
8 / 95
Slide 9
Slide 9 text
WHAT WE'RE TALKING ABOUT
JAVASCRIPT
(Obviously)
TINY COMPUTERS
Arduinos, Rasperry Pis, Espruinos & more
MIDI
The Musical Instrument Device Interface
9 / 95
Slide 10
Slide 10 text
WHY JAVASCRIPT?
10 / 95
Slide 11
Slide 11 text
JAVASCRIPT IS EVERYWHERE
Clients, servers, websites, software, apps, bots, AIs, IoT,
embedded tech... It's become the Lingua franca of the
web.
11 / 95
Slide 12
Slide 12 text
IT'S ASYNCHRONOUS NATURE
A natural fit with MIDI
12 / 95
Slide 13
Slide 13 text
IT'S WHERE THE PEOPLE ARE
In ~8 years Node.JS has made JavaScript a legitimate
server-side contender and fundamentally changed how we
develop for the web and beyond.
13 / 95
WHY TINY COMPUTERS?
What are Tiny Computers exactly?
15 / 95
Slide 16
Slide 16 text
16 / 95
Slide 17
Slide 17 text
17 / 95
Slide 18
Slide 18 text
18 / 95
Slide 19
Slide 19 text
RASPERRY PI
19 / 95
Slide 20
Slide 20 text
20 / 95
Slide 21
Slide 21 text
21 / 95
Slide 22
Slide 22 text
They're fast, affordable and easy to use
22 / 95
Slide 23
Slide 23 text
Great for Stand-Alone Projects and Creative Tinkering
23 / 95
Slide 24
Slide 24 text
THEY'RE FUN!
24 / 95
Slide 25
Slide 25 text
WHY MIDI?
25 / 95
Slide 26
Slide 26 text
WHY MIDI?
Wait — what is MIDI exactly?
26 / 95
Slide 27
Slide 27 text
MIDI
Musical Instrument Device Interface
It's a protocol
It's a standard
It's also a file format
Created in 1983
Maintained by the since 1985.
MIDI Association
27 / 95
Slide 28
Slide 28 text
IT'S NOT THIS...
“Passport”
by Passport Designs
28 / 95
Slide 29
Slide 29 text
...OR THIS...
“Canyon”
by George Stone of Passport Designs
29 / 95
Slide 30
Slide 30 text
...OR EVEN THIS
“Clouds”
by Brian Orr
29 / 95
Slide 31
Slide 31 text
31 / 95
Slide 32
Slide 32 text
MIDI THE PROTOCOL
Allowed electronic musical instruments to
communicate one another
Described common instrument performance
parameters — noteOn, noteOff, pitchbend,
programChange and others
32 / 95
Slide 33
Slide 33 text
MIDI THE STANDARD
Roland
Oberheim
Sequential Circuits
Yamaha
Korg
Kawai
33 / 95
Slide 34
Slide 34 text
IKUTARO KAKEHASHI
34 / 95
Slide 35
Slide 35 text
MIDI THE FILE FORMAT
Describe MIDI Events
Describe the timing surrounding those events
35 / 95
Slide 36
Slide 36 text
Playback isn't supported on this device.
0:00 / 3:13
36 / 95
Slide 37
Slide 37 text
EVEN MORE MIDI
MIDI Show Control (stage lighting)
General MIDI 2
MIDI Light Control
It...
Never...
Ends...
Black MIDI
37 / 95
Slide 38
Slide 38 text
BACK TO THE QUESTION...
WHY MIDI?
38 / 95
Slide 39
Slide 39 text
MIDI IS ASYNCHRONOUS
39 / 95
Slide 40
Slide 40 text
MIDI TALKS TO HARDWARE
40 / 95
Slide 41
Slide 41 text
MIDI IS EASY TO REPURPOSE
(I seriuosly wish I could take credit for )
this
41 / 95
Slide 42
Slide 42 text
CHROME SPEAKS MIDI!
:-D
And so do lots of other JavaScript projects
42 / 95
Slide 43
Slide 43 text
MIDI CONTROLLERS
Pre-existing, USB-ready (plug-and-play)
Old hardware can be used with a MIDI-USB adapter
You can build your own!
43 / 95
Slide 44
Slide 44 text
KEYBOARDS & CONTROL PADS
44 / 95
Slide 45
Slide 45 text
KEYBOARDS & CONTROL PADS
45 / 95
Slide 46
Slide 46 text
BUTTONS AND SWITCHES
46 / 95
Slide 47
Slide 47 text
LESS COMMON MIDI
CONTROLLERS
Expression pedals
Breathe & bite controllers
Accelerometer based controllers
Tenori On
Pianocade
Karlax from Da Fact
Bananas... (via Makey Makey)
47 / 95
Slide 48
Slide 48 text
BUILD YOUR OWN!
It's pretty easy to build a MIDI in/out device on Arduino.
48 / 95
Slide 49
Slide 49 text
ARDUINO MIDI CONTROLLER
49 / 95
Slide 50
Slide 50 text
ANATOMY OF A MIDI MESSAGE
Every message is 3 bytes.
There are only 2 types of messages: status and data.
A status byte always begins with 1 and data bytes
always begin with 0
That leaves 7 bits left per byte for expressing the
message
For a status message, 3 bits of the first byte describe
the type of status message; the remaining 4 describe
the channel
50 / 95
ANATOMY OF A MIDI FILE
Broken down into 3-sectioned chunks: type, length and
data
Two types of chunks: header and track
53 / 95
Slide 54
Slide 54 text
BEFORE WE GET TOO FAR INTO
“MORE” THAN MUSIC WITH
MIDI...
54 / 95
Slide 55
Slide 55 text
MAKING A SIMPLE SYNTH
UNDER THE HOOD
WebMidi.js
MIDIUtils.js
Pizzicato.js
A 5-MINUTE POLYPHONIC
SYNTHESIZER IN YOUR BROWSER
55 / 95
Slide 56
Slide 56 text
WEB MIDI & WEB AUDIO
UNDER THE HOOD
WebMidi.js
MIDIUtils.js
Pizzicato.js
FUN WITH CONTROL CHANGES
56 / 95
Slide 57
Slide 57 text
WEBRTC SYMPHONY I
THE INTERACTIVE PIANO RECITAL
UNDER THE HOOD
Peer.js
MIDIUtils.js
Pizzicato.js
57 / 95
Slide 58
Slide 58 text
WEBRTC SYMPHONY II
SHARING PARTS OF MIDI FILE
UNDER THE HOOD CLIENT-SIDE
Express
Socket.io
MIDIUtils.js
midifile
midievents
58 / 95
Slide 59
Slide 59 text
OKAY, NOW LET'S LOOK AT
SOME OF THE “MORE” THAN
MUSIC WITH MIDI...
59 / 95
Slide 60
Slide 60 text
MIDI COLOR MIXER
UNDER THE HOOD
WebMidi.js
TinyColor.js
BUILD AN RGB AND HSL
MIXER/PALETTE
60 / 95
Slide 61
Slide 61 text
MIDI & CSS FILTERS
UNDER THE HOOD
WebMidi.js
MANIPULATE BLEEDING-EDGE CSS
FEATURES
61 / 95
Slide 62
Slide 62 text
MIDI PIXEL ART
UNDER THE HOOD
Native WebMIDI implementation!
var x = noteNumber & 0x0f;
var y = (noteNumber & 0xf0) >> 4;
DRAW SIMPLE IMAGES ON A
LAUNCHPAD
62 / 95
Slide 63
Slide 63 text
MIDI HOT HAND COLOR MIXER
UNDER THE HOOD
WebMidi.js
MIDIUtils.js
ANOTHER COLOR MIXER WITH A
MORE ABSTRACT CONTROLLER
63 / 95
Slide 64
Slide 64 text
MIDI HOT HAND COLOR
CONTROL GAME
UNDER THE HOOD
WebMidi.js
MIDIUtils.js
A TWIST ON THE COLOR MIXER, AS A
GAME
64 / 95
Slide 65
Slide 65 text
MIDI WHACK-A-MOLE GAMME
UNDER THE HOOD
WebMidi.js
RECREATING A SILLY ARCADE GAME
ON THE LPD8
65 / 95
Slide 66
Slide 66 text
MIDI GO
UNDER THE HOOD
(AlphaGo AI not included)
TURNING A NOVATION LAUNCHPAD
INTO THE CLASSIC STRATEGY GAME
Tenuki.js
66 / 95
THE SLIDE DECK
GOALS
Use MIDI controllers to...
Go forward and backwards through the slide deck
Trigger “interactions” in active slide for media
Switch program modes to alter these controller
behaviors
69 / 95
Slide 70
Slide 70 text
THE SLIDE DECK
CHALLENGES
Recognize devices regardless of port
Map MIDI messages to Reveal.js API
Store and trigger program changes
70 / 95
Slide 71
Slide 71 text
THE SLIDE DECK
REFERENCING DEVICES BY NUMBER
The default WebMidi.inputs array isn't ideal.
console.log(WebMidi.inputs);
71 / 95
Slide 72
Slide 72 text
72 / 95
Slide 73
Slide 73 text
THE SLIDE DECK
REFERENCING DEVICES BY NUMBER
If I put this down for a couple weeks I will probably forget
which controller WebMidi.inputs[0] was supposed to be.
if (WebMidi.inputs[0]) {
WebMidi.inputs[0].addListener('noteon', '1', function(e){
if (e.data[0] == 144 && e.data[2] == 64 ) {
var value = e.data[1];
if (value === 60) Reveal.left();
if (value === 64) Reveal.right()
if (value === 62) triggerMedia();
}
});
73 / 95
Slide 74
Slide 74 text
THE SLIDE DECK
REFERENCING DEVICES BY NAME
The getInputByName method helps clarify what device
we're talking about.
WebMidi.getInputByName['LPD8']
74 / 95
Slide 75
Slide 75 text
THE SLIDE DECK
REFERENCING DEVICES BY NAME
We can make this a little nicer.
var midiInputs = {};
WebMidi.inputs.forEach(function(input) {
console.log("%cFound " + input.name, "color:green");
midiInputs[input.name] = input;
});
75 / 95
Slide 76
Slide 76 text
76 / 95
Slide 77
Slide 77 text
THE SLIDE DECK
REFERENCING DEVICES BY NAME
In practice this is easier when juggling multiple devices.
if (midiInputs['Logidy UMI3']) {
midiInputs['Logidy UMI3'].addListener('noteon', '1', function(e) {
if (e.data[0] == 144 && e.data[2] == 64 ) {
var value = e.data[1];
if (value === 60) Reveal.left();
if (value === 64) Reveal.right()
if (value === 62) triggerMedia();
}
});
77 / 95
Slide 78
Slide 78 text
THE SLIDE DECK
MAPPING MESSAGES TO REVEAL.JS
Learn each controller on a case-by-case basis
Figure out what makes sense for that device
Sometimes you can change the hardware behavior
78 / 95
Slide 79
Slide 79 text
THE SLIDE DECK
MAPPING MESSAGES TO REVEAL.JS
Using the , and provides:
3 on/off buttons on the floor
1 expression pedal on the floor
8 pads + 8 knobs on desk
1 program change signal via button
1 Wirelss button with 3 types of control-changes
Logidy UMI3 Akai LPDB8 Puck.js
79 / 95