Make More Than Music with Tiny Computers, JavaScript & MIDI @ OdessaJS 2017, Odessa, Ukraine

Make More Than Music with Tiny Computers, JavaScript & MIDI @ OdessaJS 2017, Odessa, Ukraine

Read more about WebMIDI at https://midi.mand.is

52648880c2f9a3136ab83d399efe129d?s=128

George Mandis

July 02, 2017
Tweet

Transcript

  1. MAKE MORE THAN MUSIC WITH TINY COMPUTERS, JAVASCRIPT & MIDI

    GEORGE MANDIS ODESSAJS 2017 ODESSA, UKRAINE 1 / 82
  2. GEORGE MANDIS WEB — EMAIL — TWITTER — GITHUB —

    BLOG — WORK — I'm a freelance web developer from the U.S. I've been developing for the web for over a decade. george.mand.is george@mand.is @georgeMandis @georgemandis george.mand.is snaptortoise.com 2 / 82
  3. GEORGE MANDIS FUN FACTS! First time visiting Ukraine Lived as

    digital nomad in 18 countries Born in Saudi Arabia! Long story Once unintentionally cheated running a marathon in North Korea Most known project — — come see my other talk! :-D Konami-JS 3 / 82
  4. LET'S START WITH AN EXPERIMENT... 4 / 82

  5. WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL PERFORMED (POORLY) BY

    GEORGE MANDIS HTTP://ODESSAJS.MAND.IS/WS1 5 / 82
  6. WEB SYMPHONY II JESU, JOY OF MAN'S DESIRING COURTESY OF

    BACH HTTP://ODESSAJS.MAND.IS/WS2 6 / 82
  7. WHAT WE'RE TALKING ABOUT JAVASCRIPT (Obviously) TINY COMPUTERS Arduinos, Rasperry

    Pis, Espruinos & more MIDI The Musical Instrument Device Interface 7 / 82
  8. WHY JAVASCRIPT? 8 / 82

  9. JAVASCRIPT IS EVERYWHERE Clients, servers, websites, software, apps, bots, AIs,

    IoT, embedded tech... It's become the Lingua franca of the web. 9 / 82
  10. IT'S ASYNCHRONOUS NATURE A natural fit with MIDI 10 /

    82
  11. IT'S WHERE THE PEOPLE ARE In ~8 years Node.JS has

    made JavaScript a legitimate server-side contender and fundamentally changed how we develop for the web and beyond. 11 / 82
  12. WHY TINY COMPUTERS? WHY TINY COMPUTERS? WHY TINY COMPUTERS? WHY

    TINY COMPUTERS? WHY TINY COMPUTERS? 12 / 82
  13. WHY TINY COMPUTERS? What are Tiny Computers exactly? 13 /

    82
  14. 14 / 82

  15. 15 / 82

  16. 16 / 82

  17. RASPERRY PI 17 / 82

  18. 18 / 82

  19. 19 / 82

  20. They're fast, affordable and easy to use Great for stand-alone

    projects and creative tinkering They're FUN! 20 / 82
  21. WHY MIDI? 21 / 82

  22. WHY MIDI? Wait — what is MIDI exactly? 22 /

    82
  23. MIDI Musical Instrument Device Interface It's a protocol created in

    1983 It's a standard It's also a file format 23 / 82
  24. IT'S NOT THIS... “Passport” by Passport Designs 24 / 82

  25. ...OR THIS... “Canyon” by George Stone of Passport Designs 25

    / 82
  26. ...OR EVEN THIS “Clouds” by Brian Orr 25 / 82

  27. MIDI THE PROTOCOL Allows electronic musical instruments to communicate with

    one another Describes common instrument performance parameters — noteOn, noteOff, pitchbend, programChange and others 27 / 82
  28. MIDI THE STANDARD Maintained by the since 1985. Roland, Oberheim,

    Sequential Circuits, Yamaha, Korg, Kawai MIDI Association 28 / 82
  29. MIDI THE FILE FORMAT Describe MIDI Events Describe the timing

    surrounding those events 29 / 82
  30. 30 / 82

  31. 31 / 82

  32. BACK TO THE QUESTION... WHY MIDI? MIDI is asynchronous (JavaScript!)

    MIDI talks to hardware (The IoT!) MIDI is easy to use for non-musical applications (Yay!) 32 / 82
  33. CHROME SPEAKS MIDI! :-D AND BY EXTENSION CHROME OS, CHROMIUM

    AND ELECTRON! And so do lots of other JavaScript projects 33 / 82
  34. MIDI CONTROLLERS Pre-existing, USB-ready (plug-and-play) Old hardware can be used

    with a MIDI-USB adapter You can build your own! 34 / 82
  35. KEYBOARDS & CONTROL PADS 35 / 82

  36. KEYBOARDS & CONTROL PADS 36 / 82

  37. BUTTONS AND SWITCHES 37 / 82

  38. LESS COMMON MIDI CONTROLLERS Expression pedals Breathe & bite controllers

    Accelerometer based controllers Tenori On Pianocade Karlax from Da Fact Bananas... (via Makey Makey) 38 / 82
  39. BUILD YOUR OWN! It's pretty easy to build a MIDI

    in/out device on Arduino. 39 / 82
  40. ARDUINO MIDI CONTROLLER 40 / 82

  41. ANATOMY OF A MIDI MESSAGE Every message is 3 bytes.

    There are only 2 types of messages: status and data. A status byte always begins with 1 and data bytes always begin with 0 That leaves 7 bits left per byte for expressing the message For a status message, 3 bits of the first byte describe the type of status message; the remaining 4 describe the channel 41 / 82
  42. ANATOMY OF A MIDI MESSAGE Example: 128x0001 | 0x1111111 |

    0x0000000 status: "noteOn", channel: 1, data1: 127, data2: 0 [1, 127, 0] 42 / 82
  43. ANATOMY OF A MIDI MESSAGE 43 / 82

  44. WHAT DOES USING JAVASCRIPT & MIDI LOOK LIKE? 43 /

    82
  45. JAVASCRIPT + MIDI IN THE BROWSER 43 / 82

  46. JAVASCRIPT + MIDI IN THE BROWSER Requesting access. navigator.requestMIDIAccess navigator.requestMIDIAccess().then(

    onMIDIInit, onMIDIFail ); let onMIDIInit = function (MIDIAccessObject) { // Allows us to start interacting with MIDIAccessObjects ... } let onMIDIFail = function () { // MIDI supported but access denied ... }  46 / 82
  47. JAVASCRIPT + MIDI IN THE BROWSER Iterating through the MIDI

    inputs and attaching listeners to MIDI messages as they're received let onMIDIInit = function (MIDIAccessObject) { for (let input of midi.inputs.values()) { if (input.name === "Logidy UMI3") logidy = input; if (input.name === "PuckCC Bluetooth") puckJS = input; } logidy.onmidimessage = handleLogidyController; puckJS.onmidimessage = handlePuckController; }  47 / 82
  48. JAVASCRIPT + MIDI IN THE BROWSER Handling a received MIDI

    message Our 3-Byte MIDI message shows up as an array [0, 0, 0] let handlePuckController = function(event) { data = event.data; if (data[1] == 80) {...} }  48 / 82
  49. JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life

    a little easier WebMIDI.js WebMidi.enable(function (err) { logidy = WebMidi.getInputByname("Logidy UMI3"); puck = WebMidi.getInputByname("PuckCC Bluetooth"); puck.addListener("noteon", "all", function(event) { let data = e.data; }); logidy.addListener("noteon", "all", function(event) { let data = e.data; }); });  49 / 82
  50. JAVASCRIPT + MIDI IN THE BROWSER Libraries to make life

    a little easier MIDIUtils.js puck.addListener("noteon", "all", function(event) { let frequency = MIDIUtils.noteNumberToFrequency(e.data[1]) let noteName = MIDIUtils.noteNumberToName(e.data[1]) let noteNumberFromName = MIDIUtils.noteNameToNoteNumber("A-4") let noteNumberFromFreq = MIDIUtils.frequencyToNoteNumber(440) });  50 / 82
  51. THAT'S IT! 51 / 82

  52. JAVASCRIPT + MIDI ON THE SERVER What's different and what's

    the same We don't need to request permission Latency can be more of an issue in some contexts Reading & writing MIDI files is easier (more packages) Some of the frontend libraries work on the server 52 / 82
  53. JAVASCRIPT + MIDI ON THE SERVER Basic setup for talking

    to and listening to a controller const midi = require("midi"); let input = new midi.input(); input.on('message', function(deltaTime, message) { console.log(message, deltaTime); }); input.openPort(0);  53 / 82
  54. DEMO TIME! BEFORE WE GET TOO FAR INTO “MORE” THAN

    MUSIC WITH MIDI... 54 / 82
  55. MIDI CONTROLLER EXPLORATION GETTING ACQUAINTED WITH YOUR HARDWARE 55 /

    82
  56. MAKING A SIMPLE SYNTH UNDER THE HOOD WebMidi.js MIDIUtils.js Pizzicato.js

    A 5-MINUTE POLYPHONIC SYNTHESIZER IN YOUR BROWSER 56 / 82
  57. WEB MIDI & WEB AUDIO UNDER THE HOOD WebMidi.js MIDIUtils.js

    Pizzicato.js FUN WITH CONTROL CHANGES 57 / 82
  58. WEB SYMPHONY I THE INTERACTIVE PIANO RECITAL UNDER THE HOOD

    Peer.js (WebRTC) MIDIUtils.js Pizzicato.js HTTP://ODESSAJS.MAND.IS/WS1 58 / 82
  59. WEB SYMPHONY II SHARING PARTS OF MIDI FILE UNDER THE

    HOOD Express Socket.io (WebSockets) MIDIUtils.js midifile midievents HTTP://ODESSAJS.MAND.IS/WS2 59 / 82
  60. OKAY, NOW LET'S LOOK AT SOME OF THE “MORE” THAN

    MUSIC WITH MIDI... 60 / 82
  61. MIDI COLOR MIXER UNDER THE HOOD WebMidi.js TinyColor.js BUILD AN

    RGB AND HSL MIXER/PALETTE 61 / 82
  62. THE MIDI POETRY BOX The interactive art project that inspired

    me to explore MIDI and JavaScript RECREATED HERE 62 / 82
  63. THE MIDI POETRY BOX HOW IT WORKED Pick a random

    video from Pick a random book from Choose 4 random sentences from that book to make a "poem" Display the poem on top of the video as it plays at a randomized playback speed archive.org gutenberg.org 63 / 82
  64. THE GOONIES SKELETON ORGAN POWERED BY MIDI? 64 / 82

  65. 65 / 82

  66. 66 / 82

  67. 67 / 82

  68. COULD WE MAKE SOMETHING LIKE THAT? KIND OF :) 68

    / 82
  69. COULD WE MAKE A *REAL* ONE? 69 / 82

  70. 70 / 82

  71. 71 / 82

  72. MIDI & CSS FILTERS UNDER THE HOOD WebMidi.js MANIPULATE BLEEDING-EDGE

    CSS FEATURES 72 / 82
  73. MIDI PIXEL ART UNDER THE HOOD Native WebMIDI implementation! var

    x = noteNumber & 0x0f; var y = (noteNumber & 0xf0) >> 4; DRAW SIMPLE IMAGES ON A LAUNCHPAD 73 / 82
  74. MIDI HOT HAND COLOR MIXER UNDER THE HOOD WebMidi.js MIDIUtils.js

    ANOTHER COLOR MIXER WITH A MORE ABSTRACT CONTROLLER 74 / 82
  75. MIDI HOT HAND COLOR CONTROL GAME UNDER THE HOOD WebMidi.js

    MIDIUtils.js A TWIST ON THE COLOR MIXER, AS A GAME 75 / 82
  76. MIDI WHACK-A-MOLE GAME UNDER THE HOOD WebMidi.js RECREATING A SILLY

    ARCADE GAME ON THE LPD8 76 / 82
  77. MIDI GO UNDER THE HOOD (AlphaGo AI not included) TURNING

    A NOVATION LAUNCHPAD INTO THE CLASSIC STRATEGY GAME Tenuki.js 77 / 82
  78. 78 / 82

  79. MUSICAL INTERACTIVITY IN THE PHYSICAL WORLD 79 / 82

  80. MUSICAL INTERACTIVITY IN THE PHYSICAL WORLD 80 / 82

  81. REPURPOSING THINGS FOR MUSIC WHY NOT REPURPOSE MUSIC FOR OTHER

    THINGS? 81 / 82
  82. СПАСИБО! (THANK YOU!) George Mandis Email: Twitter: GitHub: Website: Demos

    and more info: george@mand.is @georgeMandis georgemandis george.mand.is midi.mand.is 82 / 82