Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Crafting Instruments In Code

Crafting Instruments In Code

Joseph Wilk

May 11, 2018
Tweet

More Decks by Joseph Wilk

Other Decks in Technology

Transcript

  1. Crafting
    Instruments
    in
    code
    @josephwilk
    https://en.wikipedia.org/wiki/Silver_Apples

    View full-size slide

  2. @repl_electric http://www.repl-electric.com

    View full-size slide

  3. http://www.repl-electric.com
    @repl_electric

    View full-size slide

  4. http://www.repl-electric.com
    @repl_electric

    View full-size slide

  5. Noise
    Silver Apples

    View full-size slide

  6. Samples
    “Real-world noise of any sort has a very wide frequency
    range, far wider and more complex than any electronically
    generated sound.”

    Jon Hopkins
    https://en.wikipedia.org/wiki/Jon_Hopkins

    View full-size slide

  7. Channel
    (Right)
    Channel
    (Left)
    Frames
    44100
    1 second
    Sample
    16bits
    Frame
    n+1
    Frame
    n+3
    Frame
    n+4
    Frame n
    Frame
    n+2
    176k
    Sample
    16bits
    Amplitude of

    sound wave
    Stereo
    http://soundfile.sapp.org/doc/WaveFormat/

    View full-size slide

  8. Samples as DATA
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.4 0.5 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.3 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.3 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.1 0.6 0.3 0.5 0.3 0.2 0.4 0.3 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3] [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.5 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.5 0.1 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.2 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.5 0.1 0.5 0.3 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.1 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3] [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3] [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2
    [0.5 0.2 0.3 0.5 0.3 0.2
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    .3 0.2 0.4 0.5 0.5 0.2 0.3]
    .3 0.2 0.5 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3]
    [0.5 0.2 0.3 0.5 0.3 0.2 0.4 0.5 0.5 0.2 0.3] [0.5 0.2 0.3 0.5 0.
    [0.5 0.2 0.3 0.5 0.
    [0.5 0.2 0.3 0.5 0.
    0.4 0.5 0.5 0.2 0.3]
    0.4 0.5 0.5 0.2 0.3]

    View full-size slide

  9. Fundamental frequency
    Midi note
    Root mean squared amplitude
    Spectral-centroid
    Spectral-irregularity
    Spectral-inharmonicity
    Spectral skewness
    Spectral kurtosis
    http://jamiebullock.github.io/LibXtract/documentation/

    View full-size slide

  10. Fundamental frequency
    Midi note
    Volume
    Brightness of Sound
    Noisiness of a sound
    How harmonic
    Spectrum skew
    Pitchiness
    http://jamiebullock.github.io/LibXtract/documentation/

    View full-size slide

  11. Query and use it demo
    T-sne
    “magical ability to create compelling two-dimensional “maps”
    from data with hundreds or even thousands of dimensions”
    https://lvdmaaten.github.io/tsne/

    View full-size slide

  12. https://www.instagram.com/p/Bft_6KVHa9l/?taken-by=livecodez
    https://twitter.com/xululululuuum

    View full-size slide

  13. Patching
    https://en.wikipedia.org/wiki/Wendy_Carlos

    View full-size slide

  14. https://vcvrack.com/
    https://vcvrack.com/

    View full-size slide

  15. Code
    https://supercollider.github.io/

    View full-size slide

  16. Procedural
    Property based testing
    (Quickcheck)
    Genetic
    &
    https://en.wikipedia.org/wiki/%C3%89liane_Radigue

    View full-size slide

  17. Generators Properties
    All things that are a square

    should have 4 sides of equal length
    (fn []
    (let [color (gen/rgb-color)
    size (gen/int)]
    (build-a-square color size)))
    Property Based Testing
    https://en.wikipedia.org/wiki/QuickCheck

    View full-size slide

  18. Property Based Testing
    Generators
    reverb echo
    tube
    distort
    delay
    Sine
    Wave
    Square
    Wave
    Pulse
    Wave
    Sound
    Fx
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave

    View full-size slide

  19. Property Based Testing
    Generators
    reverb echo
    tube
    distort
    delay
    Sine
    Wave
    Square
    Wave
    Pulse
    Wave
    Sound
    Fx
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    Tube
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    reverb
    reverb
    Pulse
    Wave
    reverb reverb reverb
    Square
    Wave
    delay reverb
    Pulse
    Wave
    Pulse
    Wave
    reverb
    Pulse
    Wave
    reverb
    reverb
    Generators
    reverb echo
    tube
    distort
    delay
    Sine
    Wave
    Square
    Wave
    Pulse
    Wave
    Sound
    Fx
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    Tube
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    reverb
    reverb
    Pulse
    Wave
    reverb reverb reverb
    Square
    Wave
    delay reverb
    Pulse
    Wave
    Pulse
    Wave
    reverb
    Pulse
    Wave
    reverb
    reverb
    Generators
    reverb echo
    tube
    distort
    delay
    Sine
    Wave
    Square
    Wave
    Pulse
    Wave
    Sound
    Fx
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    Tube
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    reverb
    reverb
    Pulse
    Wave
    reverb reverb reverb
    Square
    Wave
    delay reverb
    Pulse
    Wave
    Pulse
    Wave
    reverb
    Pulse
    Wave
    reverb
    reverb
    Generators
    reverb echo
    tube
    distort
    delay
    Sine
    Wave
    Square
    Wave
    Pulse
    Wave
    Sound
    Fx
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    Tube
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    Pulse
    Wave
    delay
    Square
    Wave
    reverb reverb
    Pulse
    Wave
    Square
    Wave
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    reverb
    reverb
    Pulse
    Wave
    reverb reverb reverb
    Square
    Wave
    delay reverb
    Pulse
    Wave
    Pulse
    Wave
    reverb
    Pulse
    Wave
    reverb
    reverb
    reverb
    Pulse
    Wave
    Pulse
    Wave
    reverb
    Pulse
    Wave
    reverb
    reverb
    reverb
    Pulse
    Wave
    Pulse
    Wave
    reverb
    Pulse
    Wave
    reverb
    reverb
    reverb
    Pulse
    Wave
    Pulse
    Wave
    reverb
    Pulse
    Wave
    reverb
    reverb
    reverb
    Pulse
    Wave
    Pulse
    Wave
    reverb
    Pulse
    Wave
    reverb
    reverb
    Sound
    Fx
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Sound
    Fx
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Sound
    Fx
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Sound
    Fx
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Sound
    Fx
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Sound
    Fx
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Sound
    Fx
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Sound
    Fx
    delay
    Pulse
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave
    Square
    Wave

    View full-size slide

  20. https://twitter.com/Finger_smudger

    View full-size slide

  21. Playing
    instruments
    in
    odd
    ways
    https://en.wikipedia.org/wiki/Daphne_Oram

    View full-size slide

  22. Emacs Is A Operating System
    d[*_*]b
    /|!|\
    _| |_
    Is A Musical Instrument
    Yes Really

    View full-size slide

  23. Emacs Demo
    https://vimeo.com/265188142

    View full-size slide

  24. d[x_x]b
    /|!|\
    _/ \_
    https://vimeo.com/265189088
    Scratching Samples With Emacs

    View full-size slide

  25. Knob Twiddling
    Demo
    https://vimeo.com/269049832

    View full-size slide

  26. Machine
    Learning
    https://en.wikipedia.org/wiki/Else_Marie_Pade

    View full-size slide

  27. https://nsynthsuper.withgoogle.com/
    Neural
    Synthesizer

    View full-size slide

  28. Wavenet Decoder
    Amplitude
    of sound
    wave
    16
    temporal
    features
    Amplitude
    of sound
    wave
    Feedforward
    Sample Sample
    [0.5 0.2 0.3
    0.5 0.3 0.2
    0.4 0.5 0.5
    0.2 0.3]
    [0.5 0.2 0.3
    0.5 0.3 0.2
    0.4 0.5 0.5
    0.2 0.3]
    512

    View full-size slide

  29. https://vimeo.com/269039623

    View full-size slide

  30. AI
    https://chrisdonahue.github.io/wavegan/

    View full-size slide

  31. CreativITY
    https://en.wikipedia.org/wiki/Suzanne_Ciani

    View full-size slide