$30 off During Our Annual Pro Sale. View Details »

Make More Than Music with Tiny Computers, JavaScript & MIDI - JSConf Colombia 2017

Make More Than Music with Tiny Computers, JavaScript & MIDI - JSConf Colombia 2017

Slides from my presentation at JSConf Colombia (jsconf.co) 2017.

Read more at https://midi.mand.is

George Mandis

November 06, 2017
Tweet

More Decks by George Mandis

Other Decks in Programming

Transcript

  1. MAKE MORE THAN MUSIC
    WITH TINY COMPUTERS,
    JAVASCRIPT & MIDI
    GEORGE MANDIS
    JSConf Colombia
    #jsconfco — @georgeMandis

    View Slide

  2. GEORGE MANDIS
    TWITTER —
    EMAIL —
    GITHUB —
    WEB —
    WORK —
    Freelance web developer from Portland, Oregon.
    Developing for the web for over 10 years.
    That's... Maybe 45 in JavaScript years?
    @georgeMandis
    [email protected]
    @georgemandis
    george.mand.is
    snaptortoise.com

    View Slide

  3. GEORGE MANDIS
    FUN FACTS!
    First time visting Colombia!
    Once lived as digital nomad in 18 countries
    Born in Saudi Arabia (Long story!)
    Most known projects — &
    Once unintentionally cheated running a marathon
    in North Korea
    Konami-JS Jeklly RSS Feeds

    View Slide

  4. MORE RELEVANTLY
    I LIKE TO BREAK MAKE THINGS

    View Slide

  5. LET'S START WITH AN
    EXPERIMENT...

    View Slide

  6. WEB SYMPHONY I
    THE INTERACTIVE PIANO RECITAL
    PERFORMED (POORLY) BY GEORGE MANDIS
    HTTP://JSCONFCO.MAND.IS/WS1

    View Slide

  7. WHAT I'M TALKING ABOUT
    JAVASCRIPT
    (Un poco obvio!)
    TINY COMPUTERS
    Arduinos, Espruinos, Rasperry Pis, & more
    MIDI
    The Musical Instrument Device Interface

    View Slide

  8. WHAT MAKES JAVASCRIPT + MIDI +
    TINY COMPUTERS EXCITING?
    JavaScript talking to real-world interfaces
    Low-barrier to entry towards making interactive
    hardware / web / IoT stuff
    A tool for art projects and creative explorations that
    touch on the physical world

    View Slide

  9. ¿POR QUE JAVASCRIPT?

    View Slide

  10. JAVASCRIPT IS EVERYWHERE
    Clients, servers, websites, software, hardware, apps,
    bots, AIs, IoT, embedded tech...
    It's basically become the lingua franca of the web.

    View Slide

  11. IT'S WHERE THE PEOPLE ARE
    In ~8 years Node.JS has made JavaScript a legitimate
    server-side contender and fundamentally changed how we
    develop for the web and beyond.
    If you want longevity in this industry you have to at least
    pay attention to where the crowds are gathering.

    View Slide

  12. IT'S ASYNCHRONOUS NATURE
    A natural fit with MIDI

    View Slide

  13. ¿POR QUE TINY COMPUTERS?
    ¿POR QUE TINY COMPUTERS?
    ¿POR QUE TINY COMPUTERS?
    ¿POR QUE TINY COMPUTERS?
    ¿POR QUE TINY COMPUTERS?

    View Slide

  14. ¿POR QUE TINY COMPUTERS?
    What are “Tiny Computers” exactly?
    Single-board computers & single-board microcontrollers
    Commonly called — SBCs

    View Slide

  15. View Slide

  16. View Slide

  17. View Slide

  18. RASPERRY PI

    View Slide

  19. View Slide

  20. View Slide

  21. TINY COMPUTERS ARE...
    Affordable and (generally) easy to use
    Powerful-enough to perform a variety of tasks
    Great for stand-alone projects and creative tinkering
    Capable of running JavaScript natively (on many of them)
    FUN!

    View Slide

  22. ¿POR QUE MIDI?

    View Slide

  23. ¿POR QUE MIDI?
    What is MIDI exactly?

    View Slide

  24. MIDI
    MUSICAL INSTRUMENT DEVICE INTERFACE
    Often used interchangeably to describe:
    The MIDI protocol created in 1983
    The MIDI standard
    The MIDI file format

    View Slide

  25. IT'S NOT THIS...
    “Passport”
    by Passport Designs

    View Slide

  26. ...OR THIS...
    “Canyon”
    by George Stone of Passport Designs

    View Slide

  27. ...OR EVEN THIS
    “Clouds”
    by Brian Orr

    View Slide

  28. MIDI THE PROTOCOL
    Allows electronic musical instruments to communicate
    with other MIDI-capable devices
    This is done by sending and receiving 3-byte MIDI
    Messages

    View Slide

  29. MIDI THE STANDARD
    Gives our MIDI messages meaning
    Most describe common instrument performance
    parameters — noteOn, noteOff, pitchbend, and others
    Think HTTP codes: 404 (Not Found), 301 (Moved
    Permanently), 418 (I'm a teapot)
    Maintained by the since 1985.
    Roland, Oberheim, Sequential Circuits, Yamaha, Korg, Kawai
    MIDI Association

    View Slide

  30. MIDI THE FILE FORMAT
    Describe a sequence of MIDI events
    Describe the timing surrounding those events

    View Slide

  31. View Slide

  32. View Slide

  33. REGRESAR A LA PREGUNTA...
    ¿POR QUE MIDI?
    MIDI is asynchronous (JavaScript!)
    MIDI talks to hardware (The IoT!)
    MIDI is easy to use for non-musical applications (Yay!)

    View Slide

  34. CHROME SPEAKS MIDI!
    This includes Chrome OS, Chromium and Electron!
    :-D
    And so do lots of existing JavaScript & Node packages

    View Slide

  35. WEB MIDI API

    View Slide

  36. MIDI CONTROLLERS
    Pre-existing, USB-ready (plug-and-play)
    Old hardware can be used with a MIDI-USB adapter
    You can even build your own!

    View Slide

  37. COMMON CONTROLLERS
    KEYBOARDS

    View Slide

  38. COMMON CONTROLLERS
    KEYBOARDS

    View Slide

  39. COMMON CONTROLLERS
    PADS & DIALS

    View Slide

  40. COMMON CONTROLLERS
    BUTTONS AND SWITCHES

    View Slide

  41. COMMON CONTROLLERS
    BUTTONS AND SWITCHES

    View Slide

  42. LESS COMMON MIDI CONTROLLERS
    EXPRESSION PEDALS

    View Slide

  43. LESS COMMON MIDI CONTROLLERS
    BREATH CONTROLLER

    View Slide

  44. LESS COMMON MIDI CONTROLLERS
    MOTION-BASED

    View Slide

  45. LESS COMMON MIDI CONTROLLERS
    BUILD YOUR OWN

    View Slide

  46. LESS COMMON MIDI CONTROLLERS
    BUILD YOUR OWN

    View Slide

  47. LESS COMMON MIDI CONTROLLERS
    BUILD YOUR OWN

    View Slide

  48. ANATOMY OF A MIDI MESSAGE
    128x0001 | 0x1111111 | 0x0000000
    Every message is 3 bytes — 1 status, 2 data
    A status byte always begins with 1 and data bytes
    always begin with 0
    For a status message, 3 bits of the first byte describe
    the type of status message. The remaining 4 describe
    the channel (16).
    For data messages that leaves 7 bits left per byte to
    describe the event — note, velocity, etc.

    View Slide

  49. ANATOMY OF A MIDI MESSAGE
    144x0001 | 0x0111100 | 0x1111111

    View Slide

  50. ANATOMY OF A MIDI MESSAGE
    Example:
    144x0001 | 0x0111100 | 0x1111111
    status: "noteOn"
    channel: 1
    data1: 60
    data2: 127
    In JavaScript:
    [144, 60, 127]
    LET'S TRY IT OUT!

    View Slide

  51. WHAT DOES USING
    JAVASCRIPT & MIDI LOOK LIKE?

    View Slide

  52. JAVASCRIPT + MIDI IN THE BROWSER

    View Slide

  53. JAVASCRIPT + MIDI IN THE BROWSER
    Requesting access.
    navigator.requestMIDIAccess
    navigator.requestMIDIAccess().then( onMIDIInit, onMIDIFail );
    const onMIDIInit = (MIDIAccessObject) => {
    // Allows us to start interacting with MIDIAccessObjects
    console.log(MIDIAccessObject);
    }
    const onMIDIFail = () => {
    // MIDI supported but access denied
    console.log("No MIDI access");
    }

    View Slide

  54. JAVASCRIPT + MIDI IN THE BROWSER
    Iterating through the MIDI inputs and attaching listeners
    to MIDI messages as they're received
    const onMIDIInit = (MIDIAccessObject) => {
    for (let input of MIDIAccessObject.inputs.values()) {
    if (input.name === "Logidy UMI3") logidy = input;
    if (input.name === "PuckCC Bluetooth") puckJS = input;
    }
    logidy.onmidimessage = handleLogidyController;
    puckJS.onmidimessage = handlePuckController;
    }

    View Slide

  55. JAVASCRIPT + MIDI IN THE BROWSER
    Handling a received MIDI message
    Our 3-Byte MIDI message shows up as an array
    [144, 80, 127]
    noteOn — G# — 100% velocity
    let handlePuckController = (event) => {
    data = event.data;
    if (data[0] === 144 && data[1] === 80 && data[2] === 127) {...}
    }

    View Slide

  56. JAVASCRIPT + MIDI IN THE BROWSER
    Libraries to make life a little easier
    WebMIDI.js
    WebMidi.enable((err) => {
    logidy = WebMidi.getInputByname("Logidy UMI3");
    puck = WebMidi.getInputByname("PuckCC Bluetooth");
    puck.addListener("noteon", "all", (event) => {
    let data = e.data;
    });
    logidy.addListener("noteon", "all", (event) => {
    let data = e.data;
    });
    });

    View Slide

  57. JAVASCRIPT + MIDI IN THE BROWSER
    Libraries to make life a little easier
    MIDIUtils.js
    puck.addListener("noteon", "all", (event) => {
    let frequency = MIDIUtils.noteNumberToFrequency(e.data[1])
    let noteName = MIDIUtils.noteNumberToName(e.data[1])
    let noteNumberFromName = MIDIUtils.noteNameToNoteNumber("A-4")
    let noteNumberFromFreq = MIDIUtils.frequencyToNoteNumber(440)
    });

    View Slide

  58. JAVASCRIPT + MIDI ON THE SERVER
    What's different and what's the same
    We don't need to request permission
    Latency can be more of an issue in some contexts
    Working with MIDI files is easier (more packages)
    Some of the frontend libraries work on the server
    Do more interesting, system-level things

    View Slide

  59. JAVASCRIPT + MIDI ON THE SERVER
    Basic setup for talking to and listening to a controller
    const midi = require("midi");
    let input = new midi.input();
    input.on('message', function(deltaTime, message) {
    console.log(message);
    // [144, 80, 127]
    });
    input.openPort(0);

    View Slide

  60. THAT'S ALL THERE IS TO IT!

    View Slide

  61. DEMO TIME!
    BEFORE WE GET TOO FAR INTO
    “MORE” THAN MUSIC WITH MIDI...

    View Slide

  62. MIDI CONTROLLER
    EXPLORATION
    GETTING ACQUAINTED WITH YOUR
    HARDWARE

    View Slide

  63. MAKING A SIMPLE SYNTH
    A 5-MINUTE POLYPHONIC
    SYNTHESIZER IN YOUR BROWSER

    View Slide

  64. WEB MIDI & WEB AUDIO
    FUN WITH WEB AUDIO

    View Slide

  65. WEB SYMPHONY I
    THE INTERACTIVE PIANO RECITAL
    HTTP://JSCONFCO.MAND.IS/WS1

    View Slide

  66. WEB SYMPHONY II
    SHARING PARTS OF MIDI FILE
    HTTP://JSCONFCO.MAND.IS/WS2

    View Slide

  67. OKAY, NOW LET'S LOOK AT
    SOME OF THE “MORE” THAN
    MUSIC WITH MIDI...

    View Slide

  68. THE MIDI POETRY BOX
    LET'S TAKE A LOOK!

    View Slide

  69. MIDI PIXEL ART
    DRAW SIMPLE IMAGES ON A
    LAUNCHPAD

    View Slide

  70. MIDI & CSS FILTERS
    MANIPULATING CSS FILTERS VIA
    HARDWARE

    View Slide

  71. MIDI COLOR MIXING BY HAND
    ANOTHER COLOR MIXER WITH A
    MORE ABSTRACT CONTROLLER

    View Slide

  72. MIDI BOUNCY BALLS
    ONE OF MY FIRST PROGRAMS

    View Slide

  73. MIDI WHACK-A-MOLE GAME
    RECREATING A SILLY ARCADE GAME
    ON THE LPD8

    View Slide

  74. MIDI MAZE RUNNER
    AN INTERACTIVE MAZE

    View Slide

  75. MIDI CELLULAR AUTOMATA
    GENERATIVE PATTERNS +
    INTERACTIVITY

    View Slide

  76. MIDI GO
    (AlphaGo AI not included)
    TURNING A NOVATION LAUNCHPAD
    INTO THE CLASSIC STRATEGY GAME

    View Slide

  77. AND MANY MORE!
    Pitch-to-MIDI
    MIDI Chess
    MIDI Kiosk Menu
    The MIDI Security Stick
    (Come ask me!)

    View Slide

  78. View Slide

  79. View Slide

  80. View Slide

  81. REPURPOSING THINGS FOR MUSIC
    WHY NOT REPURPOSE MUSIC FOR OTHER
    THINGS?

    View Slide

  82. GRACIAS!
    George Mandis
    Demos and more info @
    Email:
    Twitter:
    GitHub:
    Website:
    https://midi.mand.is
    [email protected]
    @georgeMandis
    georgemandis
    george.mand.is

    View Slide