Make More Than Music with Tiny Computers, JavaScript & MIDI @ OdessaJS 2017, Odessa, Ukraine

Make More Than Music with Tiny Computers, JavaScript & MIDI @ OdessaJS 2017, Odessa, Ukraine

Read more about WebMIDI at https://midi.mand.is

52648880c2f9a3136ab83d399efe129d?s=128

George Mandis

July 02, 2017
Tweet

Transcript

  1. 1.

    MAKE MORE THAN MUSIC WITH TINY COMPUTERS, JAVASCRIPT & MIDI

    GEORGE MANDIS ODESSAJS 2017 ODESSA, UKRAINE 1 / 82
  2. 2.

    GEORGE MANDIS WEB — EMAIL — TWITTER — GITHUB —

    BLOG — WORK — I'm a freelance web developer from the U.S. I've been developing for the web for over a decade. george.mand.is george@mand.is @georgeMandis @georgemandis george.mand.is snaptortoise.com 2 / 82
  3. 3.

    GEORGE MANDIS FUN FACTS! First time visiting Ukraine Lived as

    digital nomad in 18 countries Born in Saudi Arabia! Long story Once unintentionally cheated running a marathon in North Korea Most known project — — come see my other talk! :-D Konami-JS 3 / 82
  4. 5.

    WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL PERFORMED (POORLY) BY

    GEORGE MANDIS HTTP://ODESSAJS.MAND.IS/WS1 5 / 82
  5. 6.

    WEB SYMPHONY II JESU, JOY OF MAN'S DESIRING COURTESY OF

    BACH HTTP://ODESSAJS.MAND.IS/WS2 6 / 82
  6. 7.

    WHAT WE'RE TALKING ABOUT JAVASCRIPT (Obviously) TINY COMPUTERS Arduinos, Rasperry

    Pis, Espruinos & more MIDI The Musical Instrument Device Interface 7 / 82
  7. 9.

    JAVASCRIPT IS EVERYWHERE Clients, servers, websites, software, apps, bots, AIs,

    IoT, embedded tech... It's become the Lingua franca of the web. 9 / 82
  8. 11.

    IT'S WHERE THE PEOPLE ARE In ~8 years Node.JS has

    made JavaScript a legitimate server-side contender and fundamentally changed how we develop for the web and beyond. 11 / 82
  9. 12.

    WHY TINY COMPUTERS? WHY TINY COMPUTERS? WHY TINY COMPUTERS? WHY

    TINY COMPUTERS? WHY TINY COMPUTERS? 12 / 82
  10. 14.
  11. 15.
  12. 16.
  13. 18.
  14. 19.
  15. 20.

    They're fast, affordable and easy to use Great for stand-alone

    projects and creative tinkering They're FUN! 20 / 82
  16. 23.

    MIDI Musical Instrument Device Interface It's a protocol created in

    1983 It's a standard It's also a file format 23 / 82
  17. 27.

    MIDI THE PROTOCOL Allows electronic musical instruments to communicate with

    one another Describes common instrument performance parameters — noteOn, noteOff, pitchbend, programChange and others 27 / 82
  18. 28.

    MIDI THE STANDARD Maintained by the since 1985. Roland, Oberheim,

    Sequential Circuits, Yamaha, Korg, Kawai MIDI Association 28 / 82
  19. 30.
  20. 31.
  21. 32.

    BACK TO THE QUESTION... WHY MIDI? MIDI is asynchronous (JavaScript!)

    MIDI talks to hardware (The IoT!) MIDI is easy to use for non-musical applications (Yay!) 32 / 82
  22. 33.

    CHROME SPEAKS MIDI! :-D AND BY EXTENSION CHROME OS, CHROMIUM

    AND ELECTRON! And so do lots of other JavaScript projects 33 / 82
  23. 34.

    MIDI CONTROLLERS Pre-existing, USB-ready (plug-and-play) Old hardware can be used

    with a MIDI-USB adapter You can build your own! 34 / 82
  24. 38.

    LESS COMMON MIDI CONTROLLERS Expression pedals Breathe & bite controllers

    Accelerometer based controllers Tenori On Pianocade Karlax from Da Fact Bananas... (via Makey Makey) 38 / 82
  25. 39.

    BUILD YOUR OWN! It's pretty easy to build a MIDI

    in/out device on Arduino. 39 / 82
  26. 41.

    ANATOMY OF A MIDI MESSAGE Every message is 3 bytes.

    There are only 2 types of messages: status and data. A status byte always begins with 1 and data bytes always begin with 0 That leaves 7 bits left per byte for expressing the message For a status message, 3 bits of the first byte describe the type of status message; the remaining 4 describe the channel 41 / 82
  27. 42.

    ANATOMY OF A MIDI MESSAGE Example: 128x0001 | 0x1111111 |

    0x0000000 status: "noteOn", channel: 1, data1: 127, data2: 0 [1, 127, 0] 42 / 82
  28. 46.

    JAVASCRIPT + MIDI IN THE BROWSER Requesting access. navigator.requestMIDIAccess navigator.requestMIDIAccess().then(

    onMIDIInit, onMIDIFail ); let onMIDIInit = function (MIDIAccessObject) { // Allows us to start interacting with MIDIAccessObjects ... } let onMIDIFail = function () { // MIDI supported but access denied ... }  46 / 82
  29. 47.

    JAVASCRIPT + MIDI IN THE BROWSER Iterating through the MIDI

    inputs and attaching listeners to MIDI messages as they're received let onMIDIInit = function (MIDIAccessObject) { for (let input of midi.inputs.values()) { if (input.name === "Logidy UMI3") logidy = input; if (input.name === "PuckCC Bluetooth") puckJS = input; } logidy.onmidimessage = handleLogidyController; puckJS.onmidimessage = handlePuckController; }  47 / 82
  30. 48.

    JAVASCRIPT + MIDI IN THE BROWSER Handling a received MIDI

    message Our 3-Byte MIDI message shows up as an array [0, 0, 0] let handlePuckController = function(event) { data = event.data; if (data[1] == 80) {...} }  48 / 82
  31. 49.

    JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life

    a little easier WebMIDI.js WebMidi.enable(function (err) { logidy = WebMidi.getInputByname("Logidy UMI3"); puck = WebMidi.getInputByname("PuckCC Bluetooth"); puck.addListener("noteon", "all", function(event) { let data = e.data; }); logidy.addListener("noteon", "all", function(event) { let data = e.data; }); });  49 / 82
  32. 50.

    JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life

    a little easier MIDIUtils.js puck.addListener("noteon", "all", function(event) { let frequency = MIDIUtils.noteNumberToFrequency(e.data[1]) let noteName = MIDIUtils.noteNumberToName(e.data[1]) let noteNumberFromName = MIDIUtils.noteNameToNoteNumber("A-4") let noteNumberFromFreq = MIDIUtils.frequencyToNoteNumber(440) });  50 / 82
  33. 52.

    JAVASCRIPT + MIDI ON THE SERVER What's different and what's

    the same We don't need to request permission Latency can be more of an issue in some contexts Reading & writing MIDI files is easier (more packages) Some of the frontend libraries work on the server 52 / 82
  34. 53.

    JAVASCRIPT + MIDI ON THE SERVER Basic setup for talking

    to and listening to a controller const midi = require("midi"); let input = new midi.input(); input.on('message', function(deltaTime, message) { console.log(message, deltaTime); }); input.openPort(0);  53 / 82
  35. 56.

    MAKING A SIMPLE SYNTH UNDER THE HOOD WebMidi.js MIDIUtils.js Pizzicato.js

    A 5-MINUTE POLYPHONIC SYNTHESIZER IN YOUR BROWSER 56 / 82
  36. 57.

    WEB MIDI & WEB AUDIO UNDER THE HOOD WebMidi.js MIDIUtils.js

    Pizzicato.js FUN WITH CONTROL CHANGES 57 / 82
  37. 58.

    WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL UNDER THE HOOD

    Peer.js (WebRTC) MIDIUtils.js Pizzicato.js HTTP://ODESSAJS.MAND.IS/WS1 58 / 82
  38. 59.

    WEB SYMPHONY II SHARING PARTS OF MIDI FILE UNDER THE

    HOOD Express Socket.io (WebSockets) MIDIUtils.js midifile midievents HTTP://ODESSAJS.MAND.IS/WS2 59 / 82
  39. 62.

    THE MIDI POETRY BOX The interactive art project that inspired

    me to explore MIDI and JavaScript RECREATED HERE 62 / 82
  40. 63.

    THE MIDI POETRY BOX HOW IT WORKED Pick a random

    video from Pick a random book from Choose 4 random sentences from that book to make a "poem" Display the poem on top of the video as it plays at a randomized playback speed archive.org gutenberg.org 63 / 82
  41. 65.
  42. 66.
  43. 67.
  44. 70.
  45. 71.
  46. 73.

    MIDI PIXEL ART UNDER THE HOOD Native WebMIDI implementation! var

    x = noteNumber & 0x0f; var y = (noteNumber & 0xf0) >> 4; DRAW SIMPLE IMAGES ON A LAUNCHPAD 73 / 82
  47. 74.

    MIDI HOT HAND COLOR MIXER UNDER THE HOOD WebMidi.js MIDIUtils.js

    ANOTHER COLOR MIXER WITH A MORE ABSTRACT CONTROLLER 74 / 82
  48. 75.

    MIDI HOT HAND COLOR CONTROL GAME UNDER THE HOOD WebMidi.js

    MIDIUtils.js A TWIST ON THE COLOR MIXER, AS A GAME 75 / 82
  49. 77.

    MIDI GO UNDER THE HOOD (AlphaGo AI not included) TURNING

    A NOVATION LAUNCHPAD INTO THE CLASSIC STRATEGY GAME Tenuki.js 77 / 82
  50. 78.
  51. 82.

    СПАСИБО! (THANK YOU!) George Mandis Email: Twitter: GitHub: Website: Demos

    and more info: george@mand.is @georgeMandis georgemandis george.mand.is midi.mand.is 82 / 82