JavaScript for Artists

Slides from my presentation at Microsoft Build 2019

More information see: https://javascriptforartists.com

Transcript

1. View Slide

2. George Mandis
   Developer & Consultant
   CFS2019
   @georgeMandis
   #MSBuild
   JavaScript for Artists
   

   View Slide

3. George Mandis
   george@mand.is
   @georgeMandis
   #MSBuild
   https://bit.ly/msbuild-jsforartists
   Nice to meet you!
   

   View Slide

4. Independent developer
   for ~13 years based out
   of Portland, Oregon
   Google Developer Expert
   in Web Technologies
   Once spent a year living
   & working from 18
   different countries
   Remote-working FTW
   Konami-JS
   A frivolous easter-egg
   script
   https://github.com/
   snaptortoise/konami-js
   Ran in a marathon in
   Pyongang, North Korea.
   Accidentally cheated. 

   "
   George Mandis
   

   View Slide

5. George Mandis
   I like to break make things.
   Make use of my tools in wrong unexpected
   ways.
   Today we're going talk about making things—
   specifically crap art via JavaScript and some of
   the tools & techniques that allows us to do so.
   

   View Slide

6. The Topic • What do we mean by “art” exactly?
   • What is art?
   • Why JavaScript?
   

   View Slide

7. Visual
   Mona Lisa
   Leonardo da Vinci
   

   View Slide

8. Visual
   Composition with
   Large Red Plane,
   Yellow, Black, Gray
   and Blue
   Piet Mondrian
   

   View Slide

9. Music
   Charlie Parker
   Miles Davis
   

   View Slide

10. Dance
    https://unsplash.com/@hudsonhintze
    

    View Slide

11. Experiential
    https://unsplash.com/@dissii
    

    View Slide

12. Virtual
    https://unsplash.com/@dissii
    

    View Slide

13. Anything created through joyful (?) exploration of a
    discipline
    An expression of something captured only through
    the work itself
    Embracing one's curiosity in the pursuit to find an
    answer to a question.
    More succinctly:
    Not precisely useful, but not exactly pointless.
    A Definition For
    Today
    

    View Slide

14. It's Everywhere!
    Browsers, servers, hardware...
    Low Barrier to Entry
    Taught in many places, forgiving...
    It's Where the People Are
    Conferences, Meetups, GitHub projects...
    Why JavaScript?
    

    View Slide

15. JavaScript is uniquely suited for creating
    interactive, expressive works of digital art of
    all kinds
    Why is this?
    • Ubiquity
    • The Browser
    • Industry & Community Support
    Why JavaScript?
    

    View Slide

16. Why Is This
    Important?
    • Through play we can become better
    developers
    • Making art forces us to think about
    problems differently
    • Advances in early computing owe a lot to
    this sort of creative exploration
    • Opportunities for dabbling in interesting,
    cutting-edge technologies (AI/ML)
    

    View Slide

17. Why Is This
    Important?
    We can't be experts at everything.
    We are guaranteed to be amateurs at
    something.
    Let's revel in our amateurism and make fun
    things!
    

    View Slide

18. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    console.log("Hello, World");
    

    View Slide

19. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    const helloWorld = [
    'H','e','l','l','o',',',' ',
    'W','o','r','l','d'
    ];
    console.log(helloWorld.join(''));
    

    View Slide

20. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    process.stdout.write("H");
    process.stdout.write("e");
    process.stdout.write("l");
    process.stdout.write("l");
    process.stdout.write("o");
    process.stdout.write(",");
    process.stdout.write(" ");
    process.stdout.write("W");
    process.stdout.write("o");
    process.stdout.write("r");
    process.stdout.write("l");
    process.stdout.write("d\n");
    

    View Slide

21. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    const helloWorld = [
    'd', 'l', 'r', 'o', 'W', ' ', ',',
    'o', 'l', 'l', 'e', 'H'
    ];
    helloWorld.reverse().forEach((letter) => {
    process.stdout.write(letter);
    })
    console.log()
    

    View Slide

22. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    const helloWorld = [
    72, 101, 108, 108,
    111, 44, 32, 87,
    111, 114, 108, 100
    ];
    helloWorld.forEach((unicode) => {
    process.stdout.write(String.fromCharCode(unicode))
    });
    console.log();
    

    View Slide

23. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    const helloWorld = "Ifmmp-!Xpsme";
    helloWorld.split("").forEach((value, index, array) =>
    {
    process.stdout.write(String.fromCharCode(
    helloWorld.charCodeAt(index) - 1))
    });
    console.log()
    

    View Slide

24. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    const helloWorld = "***H***e***l****l**o****,
    *W***o***r***l**d***";
    console.log(helloWorld.replace(/\*/gi,''));
    

    View Slide

25. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    ["Hello + World", "World + Hello"].forEach((word, index, array) => {
    try {
    eval(word)
    }catch(e) {
    process.stdout.write(e.message.replace(/\ is not defined|
    Unexpected token\ /, "") + ((index !== array.length - 1) ? ", " :
    ""))
    }
    })
    console.log();
    

    View Slide

26. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    function helloWorld (string="Hello, World") {
    this.string = string;
    process.stdout.write(this.string[0]);
    return () => helloWorld(string.substr(1))
    }
    helloWorld()()()()()()()()()()()();
    console.log()
    

    View Slide

27. Creative Opportunities in Coding
    101 Ways to Say ‘Hello World’
    function alphabet(stop = false, uppercase = false)
    {
    this.counter = typeof this.counter !==
    "number" ? 0 : this.counter + 1;
    const letter = new Array(26).fill().map((value,
    index, array) => {
    return uppercase
    ? String.fromCharCode(97 +
    index).toUpperCase()
    : String.fromCharCode(97 + index);
    })[this.counter];
    if (stop) this.counter = -1;
    return stop ? letter : alphabet;
    }
    let helloWorld = [
    alphabet()()()()()()()(true, true),
    alphabet()()()()(true),
    alphabet()()()()()()()()()()()(true),
    alphabet()()()()()()()()()()()(true),
    alphabet()()()()()()()()()()()()()()(true),
    ", ",
    alphabet()()()()()()()()()()()()()()()()()()()()
    ()()(true, true),
    alphabet()()()()()()()()()()()()()()(true),
    alphabet()()()()()()()()()()()()()()()()()
    (true),
    alphabet()()()()()()()()()()()(true),
    alphabet()()()(true)
    ];
    console.log(helloWorld.join(“”);
    

    View Slide

28. • Computer Music (1951)
    • Synthesized Speech (1961)
    • Imagery (1956)
    • Animation (1967)
    • 3D Animations + Interactivity (1971)
    • Literature & Poems (1952)
    (A woefully short and incomplete summary)
    Art & Early
    Computing
    

    View Slide

29. https://blogs.bl.uk/sound-and-vision/2016/09/restoring-the-first-
    recording-of-computer-music.html
    Computers
    Music (1951)
    Mark II — God
    Save the King
    

    View Slide

30. Synthesized
    Speech (1961)
    "Daisy Bell" —
    Bell Labs
    https://www.youtube.com/watch?v=E7WQ1tdxSqI
    

    View Slide

31. Imagery (1956)
    SAGE
    Pin-Up Girl
    A $238 million military
    computer in 1956 dollars
    That’s ~$2.24 billion in 2019
    dollars, in case you were
    wondering.
    The Never-Before-Told Story of the
    World's First Computer Art
    https://www.theatlantic.com/technology/archive/2013/01/the-never-
    before-told-story-of-the-worlds-first-computer-art-its-a-sexy-dame/
    267439/
    

    View Slide

32. Animation
    (1967)
    “Hummingbird”
    — Bell Labs
    Charles Csuri
    The Museum of Modern Art
    https://www.moma.org/calendar/exhibitions/3903
    

    View Slide

33. 3D Animations
    (1967)
    “Mythical
    Creature”
    —
    Charles Csuri
    Charles A. Csuri Project at Ohio State University
    https://csuriproject.osu.edu/index.php/Detail/objects/581
    

    View Slide

34. Literature &
    Poems (1952)
    Algorithmic
    Love Letters
    —Christopher
    Straychey A House of Dust, 1967 — Alison Knowles
    https://glitch.com/~house-of-dust
    Originally written in Fortran IV
    Recreated in JavaScript by Chad Weinard
    

    View Slide

35. The JavaScript
    Artist’s Palette
    The Browser
    Hardware
    AI & Machine Learning
    

    View Slide

36. • The DOM
    • Canvas & WebGL
    • WebAudio
    • MediaCapture
    • WebRTC / WebSockets
    The Browser
    

    View Slide

37. • DOM API
    • zepto.js
    • jQuery.js
    The Browser
    The DOM
    

    View Slide

38. • Canvas API
    • p5.js
    • three.js
    • babylon.js
    • TWGL.js
    • A-Frame (WebVR)
    The Browser
    Canvas & WebGL
    

    View Slide

39. • WebAudio API
    • Tone.js
    • Tonal.js
    • babylon.js
    • TWGL.js
    • A-Frame (WebVR)
    The Browser
    WebAudio
    

    View Slide

40. const video = document.querySelector(“video")
    const stream = await navigator.mediaDevices.getUserMedia({video:{}})
    video.srcObject = stream;
    The Browser
    MediaCapture
    

    View Slide

41. • WebRTC (https://webrtc.org/)
    • Socket.io
    The Browser
    WebRTC &
    WebSockets
    

    View Slide

42. Espruino
    https://espruino.com
    Tessel
    https://tessel.io/
    Johnny-Five
    https://www.w3.org/TR/webmidi/
    WebMIDI
    https://www.w3.org/TR/webmidi/
    Hardware
    

    View Slide

43. Espruino
    Embedded JavaScript
    interpreter
    Tessel
    Linux-based
    Runs Node.js
    Johnny-Five
    Runs on a variety of
    hardware
    WebMIDI
    Accessible in the
    browser without special
    plugins (caveats)
    Hardware
    

    View Slide

44. Client-side libraries
    Tensorflow.js, ml5.js, magenta.js
    Web Services
    Azure Cognitive Services, Google
    Cloud, AWS
    Browser Built-in
    Web Speech API, Shape Detection
    API (Very Experimental)
    AI & Machine
    Learning
    

    View Slide

45. Client-side
    Pre-trained models or train your own
    (Tensorflow.js, face-api.js, pico.js, tracking.js)
    Web-based Services
    (Azure Face API, Google Cloud Vision,
    Amazon Rekognition)
    Shape Detection API
    New(ish) in Chrome (Experimental: chrome://flags/
    #enable-experimental-web-platform-features)
    Facial Recognition in JavaScript
    Different Approaches
    

    View Slide

46. face-api.js
    Client-side Facial
    Recognition
    Create a video element we’l use to
    display our media capture and
    include the face-api.js file.
    
    
    

    View Slide

47. face-api.js
    Client-side Facial
    Recognition
    Load the networks and models so we
    can recognize faces and
    expressions. Get permission to
    access the webcam and display it in
    our video element.
    const video = document.getElementById('inputVideo')
    async function loadModels() {
    await faceapi.nets.ssdMobilenetv1.load(‘face-api.js/weights')
    await faceapi.nets.faceExpressionNet.load('face-api.js/weights')
    await faceapi.loadFaceRecognitionModel('face-api.js/weights')
    const stream = await navigator.mediaDevices.getUserMedia({video:{}})
    video.srcObject = stream
    }
    

    View Slide

48. face-api.js
    Client-side Facial
    Recognition
    Pass option to the models to
    determine how strict we want to
    match and pass the video element
    directly. If we get results we act on
    them. Otherwise, we can use
    requestAnimationFrame to continue
    processing new frames.
    video.addEventListener(‘play’, onPlay)
    async function onPlay() {
    const options = new faceapi.SsdMobilenetv1Options({ minConfidence: .5 })
    const result = await faceapi.detectSingleFace(video, options).withFaceExpressions()
    if (result) {
    // do something with face & expressions object
    }
    window.requestAnimationFrame(onPlay)
    }
    

    View Slide

49. Face API
    Azure Cognitive
    Services
    Setup parameters to send to the
    service.
    var params = {
    returnFaceId: "true",
    returnFaceLandmarks: "true",
    returnFaceAttributes:
    "age,gender,headPose,smile,facialHair,glasses,emotion," +
    "hair,makeup,occlusion,accessories,blur,exposure,noise"
    };
    let returnValue = [];
    for (param in params) {
    returnValue.push(
    `${encodeURIComponent(param)}=$
    {encodeURIComponent(params[param])}`
    );
    }
    

    View Slide

50. Face API
    Azure Cognitive
    Services
    We can’t pass the video element
    directly so we need to take a
    snapshot of the current frame using
    canvas to return a data URL (data://)
    ctx.drawImage(video, 0, 0);
    let data = canvas.toDataURL("image/jpeg");
    

    View Slide

51. Face API
    Azure Cognitive
    Services
    We can use fetch on the data URL to
    return a Blob object that we can then
    send along (i.e. upload) to the
    service for processing.
    fetch(data)
    .then(res => res.blob()).then(blobData => {
    let subscriptionKey = "xxxxxx";
    let uriBase = "https://westus2.api.cognitive.microsoft.com/face/v1.0/detect";
    fetch(uriBase + "?" + parameters, {
    method: "POST",
    headers: {
    "Content-Type": "application/octet-stream",
    "Ocp-Apim-Subscription-Key": subscriptionKey
    },
    body: blobData
    })
    .then(response => {
    return response.json();
    })
    .then(response => {
    // act on response data
    

    View Slide

52. Shape
    Detection API
    Chrome
    (Experimental)
    chrome://flags/#enable-experimental-
    web-platform-features
    if (window.FaceDetector == undefined) {
    alert("Sorry, your browser does not support Face Detection.”);
    }
    async function findFaces(imageObject) {
    const faceDetector = new window.FaceDetector();
    try {
    const faces = await faceDetector.detect(imageObject);
    return faces;
    } catch (e) {
    console.log(e);
    }
    }
    

    View Slide

53. Shape
    Detection API
    Chrome
    (Experimental)
    chrome://flags/#enable-experimental-
    web-platform-features
    faces = await findFaces(canvas);
    faces.forEach(face => {
    // do something
    });
    

    View Slide

54. Demos
    

    View Slide

55. View Slide

56. View Slide

57. Thank You!
    More Information
    javascriptforartists.com
    Contact
    George Mandis
    @georgeMandis
    george@mand.is
    JavaScript for
    Artists
    

    View Slide