$30 off During Our Annual Pro Sale. View Details »

JavaScript for Artists

JavaScript for Artists

Slides from my presentation at Microsoft Build 2019

More information see: https://javascriptforartists.com

George Mandis

May 06, 2019
Tweet

More Decks by George Mandis

Other Decks in Technology

Transcript

  1. View Slide

  2. George Mandis
    Developer & Consultant
    CFS2019
    @georgeMandis
    #MSBuild
    JavaScript for Artists

    View Slide

  3. George Mandis
    [email protected]
    @georgeMandis
    #MSBuild
    https://bit.ly/msbuild-jsforartists
    Nice to meet you!

    View Slide

  4. Independent developer
    for ~13 years based out
    of Portland, Oregon
    Google Developer Expert
    in Web Technologies
    Once spent a year living
    & working from 18
    different countries
    Remote-working FTW
    Konami-JS
    A frivolous easter-egg
    script
    https://github.com/
    snaptortoise/konami-js
    Ran in a marathon in
    Pyongang, North Korea.
    Accidentally cheated. 

    "
    George Mandis

    View Slide

  5. George Mandis
    I like to break make things.
    Make use of my tools in wrong unexpected
    ways.
    Today we're going talk about making things—
    specifically crap art via JavaScript and some of
    the tools & techniques that allows us to do so.

    View Slide

  6. The Topic • What do we mean by “art” exactly?
    • What is art?
    • Why JavaScript?

    View Slide

  7. Visual
    Mona Lisa
    Leonardo da Vinci

    View Slide

  8. Visual
    Composition with
    Large Red Plane,
    Yellow, Black, Gray
    and Blue
    Piet Mondrian

    View Slide

  9. Music
    Charlie Parker
    Miles Davis

    View Slide

  10. Dance
    https://unsplash.com/@hudsonhintze

    View Slide

  11. Experiential
    https://unsplash.com/@dissii

    View Slide

  12. Virtual
    https://unsplash.com/@dissii

    View Slide

  13. Anything created through joyful (?) exploration of a
    discipline
    An expression of something captured only through
    the work itself
    Embracing one's curiosity in the pursuit to find an
    answer to a question.
    More succinctly:
    Not precisely useful, but not exactly pointless.
    A Definition For
    Today

    View Slide

  14. It's Everywhere!
    Browsers, servers, hardware...
    Low Barrier to Entry
    Taught in many places, forgiving...
    It's Where the People Are
    Conferences, Meetups, GitHub projects...
    Why JavaScript?

    View Slide

  15. JavaScript is uniquely suited for creating
    interactive, expressive works of digital art of
    all kinds
    Why is this?
    • Ubiquity
    • The Browser
    • Industry & Community Support
    Why JavaScript?

    View Slide

  16. Why Is This
    Important?
    • Through play we can become better
    developers
    • Making art forces us to think about
    problems differently
    • Advances in early computing owe a lot to
    this sort of creative exploration
    • Opportunities for dabbling in interesting,
    cutting-edge technologies (AI/ML)

    View Slide

  17. Why Is This
    Important?
    We can't be experts at everything.
    We are guaranteed to be amateurs at
    something.
    Let's revel in our amateurism and make fun
    things!

    View Slide

  18. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    console.log("Hello, World");

    View Slide

  19. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    const helloWorld = [
    'H','e','l','l','o',',',' ',
    'W','o','r','l','d'
    ];
    console.log(helloWorld.join(''));

    View Slide

  20. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    process.stdout.write("H");
    process.stdout.write("e");
    process.stdout.write("l");
    process.stdout.write("l");
    process.stdout.write("o");
    process.stdout.write(",");
    process.stdout.write(" ");
    process.stdout.write("W");
    process.stdout.write("o");
    process.stdout.write("r");
    process.stdout.write("l");
    process.stdout.write("d\n");

    View Slide

  21. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    const helloWorld = [
    'd', 'l', 'r', 'o', 'W', ' ', ',',
    'o', 'l', 'l', 'e', 'H'
    ];
    helloWorld.reverse().forEach((letter) => {
    process.stdout.write(letter);
    })
    console.log()

    View Slide

  22. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    const helloWorld = [
    72, 101, 108, 108,
    111, 44, 32, 87,
    111, 114, 108, 100
    ];
    helloWorld.forEach((unicode) => {
    process.stdout.write(String.fromCharCode(unicode))
    });
    console.log();

    View Slide

  23. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    const helloWorld = "Ifmmp-!Xpsme";
    helloWorld.split("").forEach((value, index, array) =>
    {
    process.stdout.write(String.fromCharCode(
    helloWorld.charCodeAt(index) - 1))
    });
    console.log()

    View Slide

  24. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    const helloWorld = "***H***e***l****l**o****,
    *W***o***r***l**d***";
    console.log(helloWorld.replace(/\*/gi,''));

    View Slide

  25. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    ["Hello + World", "World + Hello"].forEach((word, index, array) => {
    try {
    eval(word)
    }catch(e) {
    process.stdout.write(e.message.replace(/\ is not defined|
    Unexpected token\ /, "") + ((index !== array.length - 1) ? ", " :
    ""))
    }
    })
    console.log();

    View Slide

  26. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    function helloWorld (string="Hello, World") {
    this.string = string;
    process.stdout.write(this.string[0]);
    return () => helloWorld(string.substr(1))
    }
    helloWorld()()()()()()()()()()()();
    console.log()

    View Slide

  27. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    function alphabet(stop = false, uppercase = false)
    {
    this.counter = typeof this.counter !==
    "number" ? 0 : this.counter + 1;
    const letter = new Array(26).fill().map((value,
    index, array) => {
    return uppercase
    ? String.fromCharCode(97 +
    index).toUpperCase()
    : String.fromCharCode(97 + index);
    })[this.counter];
    if (stop) this.counter = -1;
    return stop ? letter : alphabet;
    }
    let helloWorld = [
    alphabet()()()()()()()(true, true),
    alphabet()()()()(true),
    alphabet()()()()()()()()()()()(true),
    alphabet()()()()()()()()()()()(true),
    alphabet()()()()()()()()()()()()()()(true),
    ", ",
    alphabet()()()()()()()()()()()()()()()()()()()()
    ()()(true, true),
    alphabet()()()()()()()()()()()()()()(true),
    alphabet()()()()()()()()()()()()()()()()()
    (true),
    alphabet()()()()()()()()()()()(true),
    alphabet()()()(true)
    ];
    console.log(helloWorld.join(“”);

    View Slide

  28. • Computer Music (1951)
    • Synthesized Speech (1961)
    • Imagery (1956)
    • Animation (1967)
    • 3D Animations + Interactivity (1971)
    • Literature & Poems (1952)
    (A woefully short and incomplete summary)
    Art & Early
    Computing

    View Slide

  29. https://blogs.bl.uk/sound-and-vision/2016/09/restoring-the-first-
    recording-of-computer-music.html
    Computers
    Music (1951)
    Mark II — God
    Save the King

    View Slide

  30. Synthesized
    Speech (1961)
    "Daisy Bell" —
    Bell Labs
    https://www.youtube.com/watch?v=E7WQ1tdxSqI

    View Slide

  31. Imagery (1956)
    SAGE
    Pin-Up Girl
    A $238 million military
    computer in 1956 dollars
    That’s ~$2.24 billion in 2019
    dollars, in case you were
    wondering.
    The Never-Before-Told Story of the
    World's First Computer Art
    https://www.theatlantic.com/technology/archive/2013/01/the-never-
    before-told-story-of-the-worlds-first-computer-art-its-a-sexy-dame/
    267439/

    View Slide

  32. Animation
    (1967)
    “Hummingbird”
    — Bell Labs
    Charles Csuri
    The Museum of Modern Art
    https://www.moma.org/calendar/exhibitions/3903

    View Slide

  33. 3D Animations
    (1967)
    “Mythical
    Creature”

    Charles Csuri
    Charles A. Csuri Project at Ohio State University
    https://csuriproject.osu.edu/index.php/Detail/objects/581

    View Slide

  34. Literature &
    Poems (1952)
    Algorithmic
    Love Letters
    —Christopher
    Straychey A House of Dust, 1967 — Alison Knowles
    https://glitch.com/~house-of-dust
    Originally written in Fortran IV
    Recreated in JavaScript by Chad Weinard

    View Slide

  35. The JavaScript
    Artist’s Palette
    The Browser
    Hardware
    AI & Machine Learning

    View Slide

  36. • The DOM
    • Canvas & WebGL
    • WebAudio
    • MediaCapture
    • WebRTC / WebSockets
    The Browser

    View Slide

  37. • DOM API
    • zepto.js
    • jQuery.js
    The Browser
    The DOM

    View Slide

  38. • Canvas API
    • p5.js
    • three.js
    • babylon.js
    • TWGL.js
    • A-Frame (WebVR)
    The Browser
    Canvas & WebGL

    View Slide

  39. • WebAudio API
    • Tone.js
    • Tonal.js
    • babylon.js
    • TWGL.js
    • A-Frame (WebVR)
    The Browser
    WebAudio

    View Slide

  40. const video = document.querySelector(“video")
    const stream = await navigator.mediaDevices.getUserMedia({video:{}})
    video.srcObject = stream;
    The Browser
    MediaCapture

    View Slide

  41. • WebRTC (https://webrtc.org/)
    • Socket.io
    The Browser
    WebRTC &
    WebSockets

    View Slide

  42. Espruino
    https://espruino.com
    Tessel
    https://tessel.io/
    Johnny-Five
    https://www.w3.org/TR/webmidi/
    WebMIDI
    https://www.w3.org/TR/webmidi/
    Hardware

    View Slide

  43. Espruino
    Embedded JavaScript
    interpreter
    Tessel
    Linux-based
    Runs Node.js
    Johnny-Five
    Runs on a variety of
    hardware
    WebMIDI
    Accessible in the
    browser without special
    plugins (caveats)
    Hardware

    View Slide

  44. Client-side libraries
    Tensorflow.js, ml5.js, magenta.js
    Web Services
    Azure Cognitive Services, Google
    Cloud, AWS
    Browser Built-in
    Web Speech API, Shape Detection
    API (Very Experimental)
    AI & Machine
    Learning

    View Slide

  45. Client-side
    Pre-trained models or train your own
    (Tensorflow.js, face-api.js, pico.js, tracking.js)
    Web-based Services
    (Azure Face API, Google Cloud Vision,
    Amazon Rekognition)
    Shape Detection API
    New(ish) in Chrome (Experimental: chrome://flags/
    #enable-experimental-web-platform-features)
    Facial Recognition in JavaScript
    Different Approaches

    View Slide

  46. face-api.js
    Client-side Facial
    Recognition
    Create a video element we’l use to
    display our media capture and
    include the face-api.js file.


    View Slide

  47. face-api.js
    Client-side Facial
    Recognition
    Load the networks and models so we
    can recognize faces and
    expressions. Get permission to
    access the webcam and display it in
    our video element.
    const video = document.getElementById('inputVideo')
    async function loadModels() {
    await faceapi.nets.ssdMobilenetv1.load(‘face-api.js/weights')
    await faceapi.nets.faceExpressionNet.load('face-api.js/weights')
    await faceapi.loadFaceRecognitionModel('face-api.js/weights')
    const stream = await navigator.mediaDevices.getUserMedia({video:{}})
    video.srcObject = stream
    }

    View Slide

  48. face-api.js
    Client-side Facial
    Recognition
    Pass option to the models to
    determine how strict we want to
    match and pass the video element
    directly. If we get results we act on
    them. Otherwise, we can use
    requestAnimationFrame to continue
    processing new frames.
    video.addEventListener(‘play’, onPlay)
    async function onPlay() {
    const options = new faceapi.SsdMobilenetv1Options({ minConfidence: .5 })
    const result = await faceapi.detectSingleFace(video, options).withFaceExpressions()
    if (result) {
    // do something with face & expressions object
    }
    window.requestAnimationFrame(onPlay)
    }

    View Slide

  49. Face API
    Azure Cognitive
    Services
    Setup parameters to send to the
    service.
    var params = {
    returnFaceId: "true",
    returnFaceLandmarks: "true",
    returnFaceAttributes:
    "age,gender,headPose,smile,facialHair,glasses,emotion," +
    "hair,makeup,occlusion,accessories,blur,exposure,noise"
    };
    let returnValue = [];
    for (param in params) {
    returnValue.push(
    `${encodeURIComponent(param)}=$
    {encodeURIComponent(params[param])}`
    );
    }

    View Slide

  50. Face API
    Azure Cognitive
    Services
    We can’t pass the video element
    directly so we need to take a
    snapshot of the current frame using
    canvas to return a data URL (data://)
    ctx.drawImage(video, 0, 0);
    let data = canvas.toDataURL("image/jpeg");

    View Slide

  51. Face API
    Azure Cognitive
    Services
    We can use fetch on the data URL to
    return a Blob object that we can then
    send along (i.e. upload) to the
    service for processing.
    fetch(data)
    .then(res => res.blob()).then(blobData => {
    let subscriptionKey = "xxxxxx";
    let uriBase = "https://westus2.api.cognitive.microsoft.com/face/v1.0/detect";
    fetch(uriBase + "?" + parameters, {
    method: "POST",
    headers: {
    "Content-Type": "application/octet-stream",
    "Ocp-Apim-Subscription-Key": subscriptionKey
    },
    body: blobData
    })
    .then(response => {
    return response.json();
    })
    .then(response => {
    // act on response data

    View Slide

  52. Shape
    Detection API
    Chrome
    (Experimental)
    chrome://flags/#enable-experimental-
    web-platform-features
    if (window.FaceDetector == undefined) {
    alert("Sorry, your browser does not support Face Detection.”);
    }
    async function findFaces(imageObject) {
    const faceDetector = new window.FaceDetector();
    try {
    const faces = await faceDetector.detect(imageObject);
    return faces;
    } catch (e) {
    console.log(e);
    }
    }

    View Slide

  53. Shape
    Detection API
    Chrome
    (Experimental)
    chrome://flags/#enable-experimental-
    web-platform-features
    faces = await findFaces(canvas);
    faces.forEach(face => {
    // do something
    });

    View Slide

  54. Demos

    View Slide

  55. View Slide

  56. View Slide

  57. Thank You!
    More Information
    javascriptforartists.com
    Contact
    George Mandis
    @georgeMandis
    [email protected]
    JavaScript for
    Artists

    View Slide