Make More Than Music with Tiny Computers, JavaScript & MIDI - JSConf Colombia 2017

Make More Than Music with Tiny Computers, JavaScript & MIDI - JSConf Colombia 2017

Slides from my presentation at JSConf Colombia (jsconf.co) 2017.

Read more at https://midi.mand.is

52648880c2f9a3136ab83d399efe129d?s=128

George Mandis

November 06, 2017
Tweet

Transcript

  1. MAKE MORE THAN MUSIC WITH TINY COMPUTERS, JAVASCRIPT & MIDI

    GEORGE MANDIS JSConf Colombia #jsconfco — @georgeMandis
  2. GEORGE MANDIS TWITTER — EMAIL — GITHUB — WEB —

    WORK — Freelance web developer from Portland, Oregon. Developing for the web for over 10 years. That's... Maybe 45 in JavaScript years? @georgeMandis george@mand.is @georgemandis george.mand.is snaptortoise.com
  3. GEORGE MANDIS FUN FACTS! First time visting Colombia! Once lived

    as digital nomad in 18 countries Born in Saudi Arabia (Long story!) Most known projects — & Once unintentionally cheated running a marathon in North Korea Konami-JS Jeklly RSS Feeds
  4. MORE RELEVANTLY I LIKE TO BREAK MAKE THINGS

  5. LET'S START WITH AN EXPERIMENT...

  6. WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL PERFORMED (POORLY) BY

    GEORGE MANDIS HTTP://JSCONFCO.MAND.IS/WS1
  7. WHAT I'M TALKING ABOUT JAVASCRIPT (Un poco obvio!) TINY COMPUTERS

    Arduinos, Espruinos, Rasperry Pis, & more MIDI The Musical Instrument Device Interface
  8. WHAT MAKES JAVASCRIPT + MIDI + TINY COMPUTERS EXCITING? JavaScript

    talking to real-world interfaces Low-barrier to entry towards making interactive hardware / web / IoT stuff A tool for art projects and creative explorations that touch on the physical world
  9. ¿POR QUE JAVASCRIPT?

  10. JAVASCRIPT IS EVERYWHERE Clients, servers, websites, software, hardware, apps, bots,

    AIs, IoT, embedded tech... It's basically become the lingua franca of the web.
  11. IT'S WHERE THE PEOPLE ARE In ~8 years Node.JS has

    made JavaScript a legitimate server-side contender and fundamentally changed how we develop for the web and beyond. If you want longevity in this industry you have to at least pay attention to where the crowds are gathering.
  12. IT'S ASYNCHRONOUS NATURE A natural fit with MIDI

  13. ¿POR QUE TINY COMPUTERS? ¿POR QUE TINY COMPUTERS? ¿POR QUE

    TINY COMPUTERS? ¿POR QUE TINY COMPUTERS? ¿POR QUE TINY COMPUTERS?
  14. ¿POR QUE TINY COMPUTERS? What are “Tiny Computers” exactly? Single-board

    computers & single-board microcontrollers Commonly called — SBCs
  15. None
  16. None
  17. None
  18. RASPERRY PI

  19. None
  20. None
  21. TINY COMPUTERS ARE... Affordable and (generally) easy to use Powerful-enough

    to perform a variety of tasks Great for stand-alone projects and creative tinkering Capable of running JavaScript natively (on many of them) FUN!
  22. ¿POR QUE MIDI?

  23. ¿POR QUE MIDI? What is MIDI exactly?

  24. MIDI MUSICAL INSTRUMENT DEVICE INTERFACE Often used interchangeably to describe:

    The MIDI protocol created in 1983 The MIDI standard The MIDI file format
  25. IT'S NOT THIS... “Passport” by Passport Designs

  26. ...OR THIS... “Canyon” by George Stone of Passport Designs

  27. ...OR EVEN THIS “Clouds” by Brian Orr

  28. MIDI THE PROTOCOL Allows electronic musical instruments to communicate with

    other MIDI-capable devices This is done by sending and receiving 3-byte MIDI Messages
  29. MIDI THE STANDARD Gives our MIDI messages meaning Most describe

    common instrument performance parameters — noteOn, noteOff, pitchbend, and others Think HTTP codes: 404 (Not Found), 301 (Moved Permanently), 418 (I'm a teapot) Maintained by the since 1985. Roland, Oberheim, Sequential Circuits, Yamaha, Korg, Kawai MIDI Association
  30. MIDI THE FILE FORMAT Describe a sequence of MIDI events

    Describe the timing surrounding those events
  31. None
  32. None
  33. REGRESAR A LA PREGUNTA... ¿POR QUE MIDI? MIDI is asynchronous

    (JavaScript!) MIDI talks to hardware (The IoT!) MIDI is easy to use for non-musical applications (Yay!)
  34. CHROME SPEAKS MIDI! This includes Chrome OS, Chromium and Electron!

    :-D And so do lots of existing JavaScript & Node packages
  35. WEB MIDI API

  36. MIDI CONTROLLERS Pre-existing, USB-ready (plug-and-play) Old hardware can be used

    with a MIDI-USB adapter You can even build your own!
  37. COMMON CONTROLLERS KEYBOARDS

  38. COMMON CONTROLLERS KEYBOARDS

  39. COMMON CONTROLLERS PADS & DIALS

  40. COMMON CONTROLLERS BUTTONS AND SWITCHES

  41. COMMON CONTROLLERS BUTTONS AND SWITCHES

  42. LESS COMMON MIDI CONTROLLERS EXPRESSION PEDALS

  43. LESS COMMON MIDI CONTROLLERS BREATH CONTROLLER

  44. LESS COMMON MIDI CONTROLLERS MOTION-BASED

  45. LESS COMMON MIDI CONTROLLERS BUILD YOUR OWN

  46. LESS COMMON MIDI CONTROLLERS BUILD YOUR OWN

  47. LESS COMMON MIDI CONTROLLERS BUILD YOUR OWN

  48. ANATOMY OF A MIDI MESSAGE 128x0001 | 0x1111111 | 0x0000000

    Every message is 3 bytes — 1 status, 2 data A status byte always begins with 1 and data bytes always begin with 0 For a status message, 3 bits of the first byte describe the type of status message. The remaining 4 describe the channel (16). For data messages that leaves 7 bits left per byte to describe the event — note, velocity, etc.
  49. ANATOMY OF A MIDI MESSAGE 144x0001 | 0x0111100 | 0x1111111

  50. ANATOMY OF A MIDI MESSAGE Example: 144x0001 | 0x0111100 |

    0x1111111 status: "noteOn" channel: 1 data1: 60 data2: 127 In JavaScript: [144, 60, 127] LET'S TRY IT OUT!
  51. WHAT DOES USING JAVASCRIPT & MIDI LOOK LIKE?

  52. JAVASCRIPT + MIDI IN THE BROWSER

  53. JAVASCRIPT + MIDI IN THE BROWSER Requesting access. navigator.requestMIDIAccess navigator.requestMIDIAccess().then(

    onMIDIInit, onMIDIFail ); const onMIDIInit = (MIDIAccessObject) => { // Allows us to start interacting with MIDIAccessObjects console.log(MIDIAccessObject); } const onMIDIFail = () => { // MIDI supported but access denied console.log("No MIDI access"); } 
  54. JAVASCRIPT + MIDI IN THE BROWSER Iterating through the MIDI

    inputs and attaching listeners to MIDI messages as they're received const onMIDIInit = (MIDIAccessObject) => { for (let input of MIDIAccessObject.inputs.values()) { if (input.name === "Logidy UMI3") logidy = input; if (input.name === "PuckCC Bluetooth") puckJS = input; } logidy.onmidimessage = handleLogidyController; puckJS.onmidimessage = handlePuckController; } 
  55. JAVASCRIPT + MIDI IN THE BROWSER Handling a received MIDI

    message Our 3-Byte MIDI message shows up as an array [144, 80, 127] noteOn — G# — 100% velocity let handlePuckController = (event) => { data = event.data; if (data[0] === 144 && data[1] === 80 && data[2] === 127) {...} } 
  56. JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life

    a little easier WebMIDI.js WebMidi.enable((err) => { logidy = WebMidi.getInputByname("Logidy UMI3"); puck = WebMidi.getInputByname("PuckCC Bluetooth"); puck.addListener("noteon", "all", (event) => { let data = e.data; }); logidy.addListener("noteon", "all", (event) => { let data = e.data; }); }); 
  57. JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life

    a little easier MIDIUtils.js puck.addListener("noteon", "all", (event) => { let frequency = MIDIUtils.noteNumberToFrequency(e.data[1]) let noteName = MIDIUtils.noteNumberToName(e.data[1]) let noteNumberFromName = MIDIUtils.noteNameToNoteNumber("A-4") let noteNumberFromFreq = MIDIUtils.frequencyToNoteNumber(440) }); 
  58. JAVASCRIPT + MIDI ON THE SERVER What's different and what's

    the same We don't need to request permission Latency can be more of an issue in some contexts Working with MIDI files is easier (more packages) Some of the frontend libraries work on the server Do more interesting, system-level things
  59. JAVASCRIPT + MIDI ON THE SERVER Basic setup for talking

    to and listening to a controller const midi = require("midi"); let input = new midi.input(); input.on('message', function(deltaTime, message) { console.log(message); // [144, 80, 127] }); input.openPort(0); 
  60. THAT'S ALL THERE IS TO IT!

  61. DEMO TIME! BEFORE WE GET TOO FAR INTO “MORE” THAN

    MUSIC WITH MIDI...
  62. MIDI CONTROLLER EXPLORATION GETTING ACQUAINTED WITH YOUR HARDWARE

  63. MAKING A SIMPLE SYNTH A 5-MINUTE POLYPHONIC SYNTHESIZER IN YOUR

    BROWSER
  64. WEB MIDI & WEB AUDIO FUN WITH WEB AUDIO

  65. WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL HTTP://JSCONFCO.MAND.IS/WS1

  66. WEB SYMPHONY II SHARING PARTS OF MIDI FILE HTTP://JSCONFCO.MAND.IS/WS2

  67. OKAY, NOW LET'S LOOK AT SOME OF THE “MORE” THAN

    MUSIC WITH MIDI...
  68. THE MIDI POETRY BOX LET'S TAKE A LOOK!

  69. MIDI PIXEL ART DRAW SIMPLE IMAGES ON A LAUNCHPAD

  70. MIDI & CSS FILTERS MANIPULATING CSS FILTERS VIA HARDWARE

  71. MIDI COLOR MIXING BY HAND ANOTHER COLOR MIXER WITH A

    MORE ABSTRACT CONTROLLER
  72. MIDI BOUNCY BALLS ONE OF MY FIRST PROGRAMS

  73. MIDI WHACK-A-MOLE GAME RECREATING A SILLY ARCADE GAME ON THE

    LPD8
  74. MIDI MAZE RUNNER AN INTERACTIVE MAZE

  75. MIDI CELLULAR AUTOMATA GENERATIVE PATTERNS + INTERACTIVITY

  76. MIDI GO (AlphaGo AI not included) TURNING A NOVATION LAUNCHPAD

    INTO THE CLASSIC STRATEGY GAME
  77. AND MANY MORE! Pitch-to-MIDI MIDI Chess MIDI Kiosk Menu The

    MIDI Security Stick (Come ask me!)
  78. None
  79. None
  80. None
  81. REPURPOSING THINGS FOR MUSIC WHY NOT REPURPOSE MUSIC FOR OTHER

    THINGS?
  82. GRACIAS! George Mandis Demos and more info @ Email: Twitter:

    GitHub: Website: https://midi.mand.is george@mand.is @georgeMandis georgemandis george.mand.is