$30 off During Our Annual Pro Sale. View Details »

Make More Than Music with Tiny Computers, JavaScript & MIDI @ OdessaJS 2017, Odessa, Ukraine

Make More Than Music with Tiny Computers, JavaScript & MIDI @ OdessaJS 2017, Odessa, Ukraine

Read more about WebMIDI at https://midi.mand.is

George Mandis

July 02, 2017
Tweet

More Decks by George Mandis

Other Decks in Programming

Transcript

  1. MAKE MORE THAN MUSIC
    WITH TINY COMPUTERS,
    JAVASCRIPT & MIDI
    GEORGE MANDIS
    ODESSAJS 2017
    ODESSA, UKRAINE
    1 / 82

    View Slide

  2. GEORGE MANDIS
    WEB —
    EMAIL —
    TWITTER —
    GITHUB —
    BLOG —
    WORK —
    I'm a freelance web developer from the U.S. I've been
    developing for the web for over a decade.
    george.mand.is
    [email protected]
    @georgeMandis
    @georgemandis
    george.mand.is
    snaptortoise.com
    2 / 82

    View Slide

  3. GEORGE MANDIS
    FUN FACTS!
    First time visiting Ukraine
    Lived as digital nomad in 18 countries
    Born in Saudi Arabia! Long story
    Once unintentionally cheated running a marathon
    in North Korea
    Most known project — — come see my other
    talk! :-D
    Konami-JS
    3 / 82

    View Slide

  4. LET'S START WITH AN
    EXPERIMENT...
    4 / 82

    View Slide

  5. WEB SYMPHONY I
    THE INTERACTIVE PIANO RECITAL
    PERFORMED (POORLY) BY GEORGE MANDIS
    HTTP://ODESSAJS.MAND.IS/WS1
    5 / 82

    View Slide

  6. WEB SYMPHONY II
    JESU, JOY OF MAN'S DESIRING
    COURTESY OF BACH
    HTTP://ODESSAJS.MAND.IS/WS2
    6 / 82

    View Slide

  7. WHAT WE'RE TALKING ABOUT
    JAVASCRIPT
    (Obviously)
    TINY COMPUTERS
    Arduinos, Rasperry Pis, Espruinos & more
    MIDI
    The Musical Instrument Device Interface
    7 / 82

    View Slide

  8. WHY JAVASCRIPT?
    8 / 82

    View Slide

  9. JAVASCRIPT IS EVERYWHERE
    Clients, servers, websites, software, apps, bots, AIs, IoT,
    embedded tech... It's become the Lingua franca of the
    web.
    9 / 82

    View Slide

  10. IT'S ASYNCHRONOUS NATURE
    A natural fit with MIDI
    10 / 82

    View Slide

  11. IT'S WHERE THE PEOPLE ARE
    In ~8 years Node.JS has made JavaScript a legitimate
    server-side contender and fundamentally changed how we
    develop for the web and beyond.
    11 / 82

    View Slide

  12. WHY TINY COMPUTERS?
    WHY TINY COMPUTERS?
    WHY TINY COMPUTERS?
    WHY TINY COMPUTERS?
    WHY TINY COMPUTERS?
    12 / 82

    View Slide

  13. WHY TINY COMPUTERS?
    What are Tiny Computers exactly?
    13 / 82

    View Slide

  14. 14 / 82

    View Slide

  15. 15 / 82

    View Slide

  16. 16 / 82

    View Slide

  17. RASPERRY PI
    17 / 82

    View Slide

  18. 18 / 82

    View Slide

  19. 19 / 82

    View Slide

  20. They're fast, affordable and easy to use
    Great for stand-alone projects and creative tinkering
    They're FUN!
    20 / 82

    View Slide

  21. WHY MIDI?
    21 / 82

    View Slide

  22. WHY MIDI?
    Wait — what is MIDI exactly?
    22 / 82

    View Slide

  23. MIDI
    Musical Instrument Device Interface
    It's a protocol created in 1983
    It's a standard
    It's also a file format
    23 / 82

    View Slide

  24. IT'S NOT THIS...
    “Passport”
    by Passport Designs
    24 / 82

    View Slide

  25. ...OR THIS...
    “Canyon”
    by George Stone of Passport Designs
    25 / 82

    View Slide

  26. ...OR EVEN THIS
    “Clouds”
    by Brian Orr
    25 / 82

    View Slide

  27. MIDI THE PROTOCOL
    Allows electronic musical instruments to communicate
    with one another
    Describes common instrument performance
    parameters — noteOn, noteOff, pitchbend,
    programChange and others
    27 / 82

    View Slide

  28. MIDI THE STANDARD
    Maintained by the since 1985.
    Roland, Oberheim, Sequential Circuits, Yamaha, Korg, Kawai
    MIDI Association
    28 / 82

    View Slide

  29. MIDI THE FILE FORMAT
    Describe MIDI Events
    Describe the timing surrounding those events
    29 / 82

    View Slide

  30. 30 / 82

    View Slide

  31. 31 / 82

    View Slide

  32. BACK TO THE QUESTION...
    WHY MIDI?
    MIDI is asynchronous (JavaScript!)
    MIDI talks to hardware (The IoT!)
    MIDI is easy to use for non-musical applications (Yay!)
    32 / 82

    View Slide

  33. CHROME SPEAKS MIDI!
    :-D
    AND BY EXTENSION CHROME OS,
    CHROMIUM AND ELECTRON!
    And so do lots of other JavaScript projects
    33 / 82

    View Slide

  34. MIDI CONTROLLERS
    Pre-existing, USB-ready (plug-and-play)
    Old hardware can be used with a MIDI-USB adapter
    You can build your own!
    34 / 82

    View Slide

  35. KEYBOARDS & CONTROL PADS
    35 / 82

    View Slide

  36. KEYBOARDS & CONTROL PADS
    36 / 82

    View Slide

  37. BUTTONS AND SWITCHES
    37 / 82

    View Slide

  38. LESS COMMON MIDI
    CONTROLLERS
    Expression pedals
    Breathe & bite controllers
    Accelerometer based controllers
    Tenori On
    Pianocade
    Karlax from Da Fact
    Bananas... (via Makey Makey)
    38 / 82

    View Slide

  39. BUILD YOUR OWN!
    It's pretty easy to build a MIDI in/out device on Arduino.
    39 / 82

    View Slide

  40. ARDUINO MIDI CONTROLLER
    40 / 82

    View Slide

  41. ANATOMY OF A MIDI MESSAGE
    Every message is 3 bytes.
    There are only 2 types of messages: status and data.
    A status byte always begins with 1 and data bytes
    always begin with 0
    That leaves 7 bits left per byte for expressing the
    message
    For a status message, 3 bits of the first byte describe
    the type of status message; the remaining 4 describe
    the channel
    41 / 82

    View Slide

  42. ANATOMY OF A MIDI MESSAGE
    Example:
    128x0001 | 0x1111111 | 0x0000000
    status: "noteOn", channel: 1, data1: 127, data2: 0
    [1, 127, 0]
    42 / 82

    View Slide

  43. ANATOMY OF A MIDI MESSAGE
    43 / 82

    View Slide

  44. WHAT DOES USING
    JAVASCRIPT & MIDI LOOK LIKE?
    43 / 82

    View Slide

  45. JAVASCRIPT + MIDI IN THE BROWSER
    43 / 82

    View Slide

  46. JAVASCRIPT + MIDI IN THE BROWSER
    Requesting access.
    navigator.requestMIDIAccess
    navigator.requestMIDIAccess().then( onMIDIInit, onMIDIFail );
    let onMIDIInit = function (MIDIAccessObject) {
    // Allows us to start interacting with MIDIAccessObjects
    ...
    }
    let onMIDIFail = function () {
    // MIDI supported but access denied
    ...
    }

    46 / 82

    View Slide

  47. JAVASCRIPT + MIDI IN THE BROWSER
    Iterating through the MIDI inputs and attaching listeners
    to MIDI messages as they're received
    let onMIDIInit = function (MIDIAccessObject) {
    for (let input of midi.inputs.values()) {
    if (input.name === "Logidy UMI3") logidy = input;
    if (input.name === "PuckCC Bluetooth") puckJS = input;
    }
    logidy.onmidimessage = handleLogidyController;
    puckJS.onmidimessage = handlePuckController;
    }

    47 / 82

    View Slide

  48. JAVASCRIPT + MIDI IN THE BROWSER
    Handling a received MIDI message
    Our 3-Byte MIDI message shows up as an array
    [0, 0, 0]
    let handlePuckController = function(event) {
    data = event.data;
    if (data[1] == 80) {...}
    }

    48 / 82

    View Slide

  49. JAVASCRIPT + MIDI IN THE BROWSER
    Libraries to make life a little easier
    WebMIDI.js
    WebMidi.enable(function (err) {
    logidy = WebMidi.getInputByname("Logidy UMI3");
    puck = WebMidi.getInputByname("PuckCC Bluetooth");
    puck.addListener("noteon", "all", function(event) {
    let data = e.data;
    });
    logidy.addListener("noteon", "all", function(event) {
    let data = e.data;
    });
    });

    49 / 82

    View Slide

  50. JAVASCRIPT + MIDI IN THE BROWSER
    Libraries to make life a little easier
    MIDIUtils.js
    puck.addListener("noteon", "all", function(event) {
    let frequency = MIDIUtils.noteNumberToFrequency(e.data[1])
    let noteName = MIDIUtils.noteNumberToName(e.data[1])
    let noteNumberFromName = MIDIUtils.noteNameToNoteNumber("A-4")
    let noteNumberFromFreq = MIDIUtils.frequencyToNoteNumber(440)
    });

    50 / 82

    View Slide

  51. THAT'S IT!
    51 / 82

    View Slide

  52. JAVASCRIPT + MIDI ON THE SERVER
    What's different and what's the same
    We don't need to request permission
    Latency can be more of an issue in some contexts
    Reading & writing MIDI files is easier (more packages)
    Some of the frontend libraries work on the server
    52 / 82

    View Slide

  53. JAVASCRIPT + MIDI ON THE SERVER
    Basic setup for talking to and listening to a controller
    const midi = require("midi");
    let input = new midi.input();
    input.on('message', function(deltaTime, message) {
    console.log(message, deltaTime);
    });
    input.openPort(0);

    53 / 82

    View Slide

  54. DEMO TIME!
    BEFORE WE GET TOO FAR INTO
    “MORE” THAN MUSIC WITH MIDI...
    54 / 82

    View Slide

  55. MIDI CONTROLLER
    EXPLORATION
    GETTING ACQUAINTED WITH YOUR
    HARDWARE
    55 / 82

    View Slide

  56. MAKING A SIMPLE SYNTH
    UNDER THE HOOD
    WebMidi.js
    MIDIUtils.js
    Pizzicato.js
    A 5-MINUTE POLYPHONIC
    SYNTHESIZER IN YOUR BROWSER
    56 / 82

    View Slide

  57. WEB MIDI & WEB AUDIO
    UNDER THE HOOD
    WebMidi.js
    MIDIUtils.js
    Pizzicato.js
    FUN WITH CONTROL CHANGES
    57 / 82

    View Slide

  58. WEB SYMPHONY I
    THE INTERACTIVE PIANO RECITAL
    UNDER THE HOOD
    Peer.js (WebRTC)
    MIDIUtils.js
    Pizzicato.js
    HTTP://ODESSAJS.MAND.IS/WS1
    58 / 82

    View Slide

  59. WEB SYMPHONY II
    SHARING PARTS OF MIDI FILE
    UNDER THE HOOD
    Express
    Socket.io (WebSockets)
    MIDIUtils.js
    midifile
    midievents
    HTTP://ODESSAJS.MAND.IS/WS2
    59 / 82

    View Slide

  60. OKAY, NOW LET'S LOOK AT
    SOME OF THE “MORE” THAN
    MUSIC WITH MIDI...
    60 / 82

    View Slide

  61. MIDI COLOR MIXER
    UNDER THE HOOD
    WebMidi.js
    TinyColor.js
    BUILD AN RGB AND HSL
    MIXER/PALETTE
    61 / 82

    View Slide

  62. THE MIDI POETRY BOX
    The interactive art project that inspired me to explore MIDI
    and JavaScript
    RECREATED HERE
    62 / 82

    View Slide

  63. THE MIDI POETRY BOX
    HOW IT WORKED
    Pick a random video from
    Pick a random book from
    Choose 4 random sentences from that book to make a
    "poem"
    Display the poem on top of the video as it plays at a
    randomized playback speed
    archive.org
    gutenberg.org
    63 / 82

    View Slide

  64. THE GOONIES
    SKELETON ORGAN
    POWERED BY MIDI?
    64 / 82

    View Slide

  65. 65 / 82

    View Slide

  66. 66 / 82

    View Slide

  67. 67 / 82

    View Slide

  68. COULD WE MAKE SOMETHING
    LIKE THAT?
    KIND OF :)
    68 / 82

    View Slide

  69. COULD WE MAKE A *REAL* ONE?
    69 / 82

    View Slide

  70. 70 / 82

    View Slide

  71. 71 / 82

    View Slide

  72. MIDI & CSS FILTERS
    UNDER THE HOOD
    WebMidi.js
    MANIPULATE BLEEDING-EDGE CSS
    FEATURES
    72 / 82

    View Slide

  73. MIDI PIXEL ART
    UNDER THE HOOD
    Native WebMIDI implementation!
    var x = noteNumber & 0x0f;
    var y = (noteNumber & 0xf0) >> 4;
    DRAW SIMPLE IMAGES ON A
    LAUNCHPAD
    73 / 82

    View Slide

  74. MIDI HOT HAND COLOR MIXER
    UNDER THE HOOD
    WebMidi.js
    MIDIUtils.js
    ANOTHER COLOR MIXER WITH A
    MORE ABSTRACT CONTROLLER
    74 / 82

    View Slide

  75. MIDI HOT HAND COLOR
    CONTROL GAME
    UNDER THE HOOD
    WebMidi.js
    MIDIUtils.js
    A TWIST ON THE COLOR MIXER, AS A
    GAME
    75 / 82

    View Slide

  76. MIDI WHACK-A-MOLE GAME
    UNDER THE HOOD
    WebMidi.js
    RECREATING A SILLY ARCADE GAME
    ON THE LPD8
    76 / 82

    View Slide

  77. MIDI GO
    UNDER THE HOOD
    (AlphaGo AI not included)
    TURNING A NOVATION LAUNCHPAD
    INTO THE CLASSIC STRATEGY GAME
    Tenuki.js
    77 / 82

    View Slide

  78. 78 / 82

    View Slide

  79. MUSICAL INTERACTIVITY IN THE
    PHYSICAL WORLD
    79 / 82

    View Slide

  80. MUSICAL INTERACTIVITY IN THE
    PHYSICAL WORLD
    80 / 82

    View Slide

  81. REPURPOSING THINGS FOR MUSIC
    WHY NOT REPURPOSE MUSIC FOR OTHER
    THINGS?
    81 / 82

    View Slide

  82. СПАСИБО!
    (THANK YOU!)
    George Mandis
    Email: Twitter:
    GitHub:
    Website:
    Demos and more info:
    [email protected] @georgeMandis
    georgemandis
    george.mand.is
    midi.mand.is
    82 / 82

    View Slide