The Lost Art of MIDI – Bringing <bgsound> Back to the Web

In the days of Geocities and Angelfire, a quirky HTML tag called ⟨bgsound⟩ enabled sound files to play in the background of webpages. Usually, these files were in the MIDI format. What a glorious era that was! Sadly, ⟨bgsound⟩ has been removed from browsers and MIDI is obscure and hard to play back. In this talk, we'll bring MIDI and ⟨bgsound⟩ back from the dead using WebAssembly, Emscripten, Web Audio, and Web Components. When we're finished, you'll be able to give your webpages the 90's treatment in a modern, standards-compliant way!

Links from the talk:

Timidity – Play MIDI files in the browser w/ Web Audio, WebAssembly, and libtimidity
https://github.com/feross/timidity

bg-sound – Web Component to emulate the old-school HTML element
https://github.com/feross/bg-sound

BitMidi – Listen to free MIDI songs, download the best MIDI files, and share the best MIDIs on the web
https://bitmidi.com

Feross's Blog
https://feross.org

Feross's Twitter
https://twitter.com/feross

Transcript

1. The Lost Art of MIDI
   Bringing Back to the Web
   Image Credit: hexeosis.tumblr.com
   

   View Slide

2. View Slide

3. Source: cameronsworld.net
   

   View Slide

4. Source: toastytech.com/evil/
   

   View Slide

5. Source: spacejam.com
   

   View Slide

6. Source: fogcam.org
   

   View Slide

7. Source: zombo.com
   

   View Slide

8. 
   

   View Slide

9. Image Credit: Legend of Zelda: Ocarina of Time 3D
   

   View Slide

10. View Slide

11. View Slide

12. What even is a MIDI file?
    Image Credit: hexeosis.tumblr.com
    

    View Slide

13. Musical Instrument Digital Interface
    

    View Slide

14. Prophet-600
    

    View Slide

15. • 1971 – FTP
    • 1974 – TCP
    • 1982 – MIDI
    • 1991 – HTTP
    • 1995 – SSH
    • 2001 – BitTorrent
    • 2009 – Bitcoin
    • 2015 – HTTP/2
    

    View Slide

16. MIDI Messages
    Command Meaning
    0x80 Note-off
    0x90 Note-on
    ... ...
    

    View Slide

17. 90 3C 40
    -----------------> time
    • 90 = Note on
    • 3C = Which key?
    • 40 = How hard was it pressed?
    

    View Slide

18. General MIDI
    • 128 notes (~10 octaves)
    • 16 channels
    • 128 programs (instrument sounds)
    

    View Slide

19. 1-8 Piano
    9-16 Chromatic Percussion
    17-24 Organ
    25-32 Guitar
    33-40 Bass
    41-48 Strings
    49-56 Ensemble
    57-64 Brass
    65-72 Reed
    73-80 Pipe
    81-88 Synth Lead
    89-96 Synth Pad
    97-104 Synth Effects
    105-112 Ethnic
    113-120 Percussive
    121-128 Sound Effects
    

    View Slide

20. Pianos
    1. Acoustic Grand Piano
    2. Bright Acoustic Piano
    3. Electric Grand Piano
    4. Honky-tonk Piano
    5. Electric Piano 1
    6. Electric Piano 2
    7. Harpsichord
    8. Clavi
    

    View Slide

21. View Slide

22. View Slide

23. Where else is MIDI used?
    Image Credit: hexeosis.tumblr.com
    

    View Slide

24. View Slide

25. View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. WebAssembly
    Image Credit: hexeosis.tumblr.com
    

    View Slide

34. View Slide

35. static void load_instrument(MidSong *song, ...) {
    ...
    if (song->ifp == NULL) {
    DEBUG_MSG("Instrument `%s' can't be found.\n", name);
    // Added for JavaScript port
    song->load_requests[song->load_request_count] = strdup(name);
    song->load_request_count += 1;
    ...
    }
    

    View Slide

36. // Added for JavaScript port
    extern int mid_get_load_request_count(MidSong *song) {
    return song->load_request_count;
    }
    extern char *mid_get_load_request(MidSong *song, int index) {
    return song->load_requests[index];
    }
    

    View Slide

37. File system
    // Make directory
    lib.FS.mkdir('new-directory')
    // Write files
    lib.FS.writeFile('myfile.txt', buffer)
    

    View Slide

38. Pointers
    // Allocate memory
    const ptr = lib._malloc(byteLength)
    // Free memory
    lib._free(ptr)
    // Convert a `char*` to a JavaScript string
    const str = lib.Pointer_stringify(ptr)
    

    View Slide

39. View Slide

40. function loadSong (buffer) {
    const bufferPtr = lib._malloc(buffer.byteLength)
    lib.HEAPU8.set(buffer, bufferPtr)
    const iStreamPtr =
    lib._mid_istream_open_mem(bufferPtr, buffer.byteLength)
    const songPtr = lib._mid_song_load(iStreamPtr)
    return songPtr
    }
    

    View Slide

41. let songPtr = loadSong(buffer)
    const instruments = getMissingInstruments(songPtr)
    if (instruments.length > 0) {
    await Promise.all(
    instruments.map(instrument => fetchInstrument(instrument))
    )
    lib._mid_song_free(songPtr)
    songPtr = loadSong(buffer)
    }
    lib._mid_song_start(songPtr)
    

    View Slide

42. function getMissingInstruments (songPtr) {
    const missingCount = lib._mid_get_load_request_count(songPtr)
    const missingInstruments = []
    for (let i = 0; i < missingCount; i++) {
    const instrumentPtr = lib._mid_get_load_request(songPtr, i)
    const instrument = lib.Pointer_stringify(instrumentPtr)
    missingInstruments.push(instrument)
    }
    return missingInstruments
    }
    

    View Slide

43. async function fetchInstrument (instrument) {
    const response = await fetch(`${urlPrefix}/${instrument}`)
    const buf = await response.arrayBuffer()
    lib.FS.writeFile(instrument, buf)
    }
    

    View Slide

44. Web Audio
    Image Credit: hexeosis.tumblr.com
    

    View Slide

45. const audioContext = new AudioContext()
    const node = audioContext.createScriptProcessor(...)
    node.addEventListener('audioprocess', onAudioProcess)
    node.connect(audioContext.destination)
    

    View Slide

46. function onAudioProcess (event) {
    const sampleCount = readMidiData()
    const output0 = event.outputBuffer.getChannelData(0)
    const output1 = event.outputBuffer.getChannelData(1)
    for (let i = 0; i < sampleCount; i++) {
    output0[i] = array[i * 2] / 0x7FFF
    output1[i] = array[i * 2 + 1] / 0x7FFF
    }
    for (let i = sampleCount; i < BUFFER_SIZE; i++) {
    output0[i] = 0
    output1[i] = 0
    }
    }
    

    View Slide

47. function readMidiData () {
    const byteCount = lib._mid_song_read_wave(songPtr, buffer, ...)
    array.set(
    lib.HEAP16.subarray(bufferPtr / 2, (bufferPtr + byteCount) / 2)
    )
    return byteCount / BYTES_PER_SAMPLE // sampleCount
    }
    

    View Slide

48. npm install timidity
    

    View Slide

49. const Timidity = require('timidity')
    const player = new Timidity()
    player.load('/my-file.mid')
    player.play()
    player.on('timeupdate', seconds => {
    console.log(seconds)
    })
    

    View Slide

50. Web Components
    Image Credit: hexeosis.tumblr.com
    

    View Slide

51. Make it a Web Component
    Before
    
    After
    
    
    

    View Slide

52. class BgSound extends HTMLElement {
    connectedCallback () {
    this.player = new Timidity()
    this.player.load(this.src)
    this.player.play()
    }
    disconnectedCallback () {
    this.player.destroy()
    }
    }
    window.customElements.define('bg-sound', BgSound)
    

    View Slide

53. npm install bg-sound
    

    View Slide

54. DEMO
    

    View Slide

55. View Slide

56. View Slide

57. The Mythical Man-Month
    The programmer, like the poet, works only
    slightly removed from pure thought-stuff.
    He builds castles in the air, from air,
    creating by exertion of the imagination.
    Few media of creation are so flexible, so
    easy to polish and rework, so readily
    capable of realizing grand conceptual
    structures.
    — Fred Brooks
    

    View Slide

58. View Slide

59. Thanks!
    @feross
    feross.org
    

    View Slide