JavaScript for Artists

JavaScript for Artists

Slides from my presentation at Microsoft Build 2019

More information see: https://javascriptforartists.com

52648880c2f9a3136ab83d399efe129d?s=128

George Mandis

May 06, 2019
Tweet

Transcript

  1. None
  2. George Mandis Developer & Consultant CFS2019 @georgeMandis #MSBuild JavaScript for

    Artists
  3. George Mandis george@mand.is @georgeMandis #MSBuild https://bit.ly/msbuild-jsforartists Nice to meet you!

  4. Independent developer for ~13 years based out of Portland, Oregon

    Google Developer Expert in Web Technologies Once spent a year living & working from 18 different countries Remote-working FTW Konami-JS A frivolous easter-egg script https://github.com/ snaptortoise/konami-js Ran in a marathon in Pyongang, North Korea. Accidentally cheated. 
 " George Mandis
  5. George Mandis I like to break make things. Make use

    of my tools in wrong unexpected ways. Today we're going talk about making things— specifically crap art via JavaScript and some of the tools & techniques that allows us to do so.
  6. The Topic • What do we mean by “art” exactly?

    • What is art? • Why JavaScript?
  7. Visual Mona Lisa Leonardo da Vinci

  8. Visual Composition with Large Red Plane, Yellow, Black, Gray and

    Blue Piet Mondrian
  9. Music Charlie Parker Miles Davis

  10. Dance https://unsplash.com/@hudsonhintze

  11. Experiential https://unsplash.com/@dissii

  12. Virtual https://unsplash.com/@dissii

  13. Anything created through joyful (?) exploration of a discipline An

    expression of something captured only through the work itself Embracing one's curiosity in the pursuit to find an answer to a question. More succinctly: Not precisely useful, but not exactly pointless. A Definition For Today
  14. It's Everywhere! Browsers, servers, hardware... Low Barrier to Entry Taught

    in many places, forgiving... It's Where the People Are Conferences, Meetups, GitHub projects... Why JavaScript?
  15. JavaScript is uniquely suited for creating interactive, expressive works of

    digital art of all kinds Why is this? • Ubiquity • The Browser • Industry & Community Support Why JavaScript?
  16. Why Is This Important? • Through play we can become

    better developers • Making art forces us to think about problems differently • Advances in early computing owe a lot to this sort of creative exploration • Opportunities for dabbling in interesting, cutting-edge technologies (AI/ML)
  17. Why Is This Important? We can't be experts at everything.

    We are guaranteed to be amateurs at something. Let's revel in our amateurism and make fun things!
  18. Creative Opportunities in Coding 101 Ways to Say ‘Hello World’

    console.log("Hello, World");
  19. Creative Opportunities in Coding 101 Ways to Say ‘Hello World’

    const helloWorld = [ 'H','e','l','l','o',',',' ', 'W','o','r','l','d' ]; console.log(helloWorld.join(''));
  20. Creative Opportunities in Coding 101 Ways to Say ‘Hello World’

    process.stdout.write("H"); process.stdout.write("e"); process.stdout.write("l"); process.stdout.write("l"); process.stdout.write("o"); process.stdout.write(","); process.stdout.write(" "); process.stdout.write("W"); process.stdout.write("o"); process.stdout.write("r"); process.stdout.write("l"); process.stdout.write("d\n");
  21. Creative Opportunities in Coding 101 Ways to Say ‘Hello World’

    const helloWorld = [ 'd', 'l', 'r', 'o', 'W', ' ', ',', 'o', 'l', 'l', 'e', 'H' ]; helloWorld.reverse().forEach((letter) => { process.stdout.write(letter); }) console.log()
  22. Creative Opportunities in Coding 101 Ways to Say ‘Hello World’

    const helloWorld = [ 72, 101, 108, 108, 111, 44, 32, 87, 111, 114, 108, 100 ]; helloWorld.forEach((unicode) => { process.stdout.write(String.fromCharCode(unicode)) }); console.log();
  23. Creative Opportunities in Coding 101 Ways to Say ‘Hello World’

    const helloWorld = "Ifmmp-!Xpsme"; helloWorld.split("").forEach((value, index, array) => { process.stdout.write(String.fromCharCode( helloWorld.charCodeAt(index) - 1)) }); console.log()
  24. Creative Opportunities in Coding 101 Ways to Say ‘Hello World’

    const helloWorld = "***H***e***l****l**o****, *W***o***r***l**d***"; console.log(helloWorld.replace(/\*/gi,''));
  25. Creative Opportunities in Coding 101 Ways to Say ‘Hello World’

    ["Hello + World", "World + Hello"].forEach((word, index, array) => { try { eval(word) }catch(e) { process.stdout.write(e.message.replace(/\ is not defined| Unexpected token\ /, "") + ((index !== array.length - 1) ? ", " : "")) } }) console.log();
  26. Creative Opportunities in Coding 101 Ways to Say ‘Hello World’

    function helloWorld (string="Hello, World") { this.string = string; process.stdout.write(this.string[0]); return () => helloWorld(string.substr(1)) } helloWorld()()()()()()()()()()()(); console.log()
  27. Creative Opportunities in Coding 101 Ways to Say ‘Hello World’

    function alphabet(stop = false, uppercase = false) { this.counter = typeof this.counter !== "number" ? 0 : this.counter + 1; const letter = new Array(26).fill().map((value, index, array) => { return uppercase ? String.fromCharCode(97 + index).toUpperCase() : String.fromCharCode(97 + index); })[this.counter]; if (stop) this.counter = -1; return stop ? letter : alphabet; } let helloWorld = [ alphabet()()()()()()()(true, true), alphabet()()()()(true), alphabet()()()()()()()()()()()(true), alphabet()()()()()()()()()()()(true), alphabet()()()()()()()()()()()()()()(true), ", ", alphabet()()()()()()()()()()()()()()()()()()()() ()()(true, true), alphabet()()()()()()()()()()()()()()(true), alphabet()()()()()()()()()()()()()()()()() (true), alphabet()()()()()()()()()()()(true), alphabet()()()(true) ]; console.log(helloWorld.join(“”);
  28. • Computer Music (1951) • Synthesized Speech (1961) • Imagery

    (1956) • Animation (1967) • 3D Animations + Interactivity (1971) • Literature & Poems (1952) (A woefully short and incomplete summary) Art & Early Computing
  29. https://blogs.bl.uk/sound-and-vision/2016/09/restoring-the-first- recording-of-computer-music.html Computers Music (1951) Mark II — God Save

    the King
  30. Synthesized Speech (1961) "Daisy Bell" — Bell Labs https://www.youtube.com/watch?v=E7WQ1tdxSqI

  31. Imagery (1956) SAGE Pin-Up Girl A $238 million military computer

    in 1956 dollars That’s ~$2.24 billion in 2019 dollars, in case you were wondering. The Never-Before-Told Story of the World's First Computer Art https://www.theatlantic.com/technology/archive/2013/01/the-never- before-told-story-of-the-worlds-first-computer-art-its-a-sexy-dame/ 267439/
  32. Animation (1967) “Hummingbird” — Bell Labs Charles Csuri The Museum

    of Modern Art https://www.moma.org/calendar/exhibitions/3903
  33. 3D Animations (1967) “Mythical Creature” — Charles Csuri Charles A.

    Csuri Project at Ohio State University https://csuriproject.osu.edu/index.php/Detail/objects/581
  34. Literature & Poems (1952) Algorithmic Love Letters —Christopher Straychey A

    House of Dust, 1967 — Alison Knowles https://glitch.com/~house-of-dust Originally written in Fortran IV Recreated in JavaScript by Chad Weinard
  35. The JavaScript Artist’s Palette The Browser Hardware AI & Machine

    Learning
  36. • The DOM • Canvas & WebGL • WebAudio •

    MediaCapture • WebRTC / WebSockets The Browser
  37. • DOM API • zepto.js • jQuery.js The Browser The

    DOM
  38. • Canvas API • p5.js • three.js • babylon.js •

    TWGL.js • A-Frame (WebVR) The Browser Canvas & WebGL
  39. • WebAudio API • Tone.js • Tonal.js • babylon.js •

    TWGL.js • A-Frame (WebVR) The Browser WebAudio
  40. const video = document.querySelector(“video") const stream = await navigator.mediaDevices.getUserMedia({video:{}}) video.srcObject

    = stream; The Browser MediaCapture
  41. • WebRTC (https://webrtc.org/) • Socket.io The Browser WebRTC & WebSockets

  42. Espruino https://espruino.com Tessel https://tessel.io/ Johnny-Five https://www.w3.org/TR/webmidi/ WebMIDI https://www.w3.org/TR/webmidi/ Hardware

  43. Espruino Embedded JavaScript interpreter Tessel Linux-based Runs Node.js Johnny-Five Runs

    on a variety of hardware WebMIDI Accessible in the browser without special plugins (caveats) Hardware
  44. Client-side libraries Tensorflow.js, ml5.js, magenta.js Web Services Azure Cognitive Services,

    Google Cloud, AWS Browser Built-in Web Speech API, Shape Detection API (Very Experimental) AI & Machine Learning
  45. Client-side Pre-trained models or train your own (Tensorflow.js, face-api.js, pico.js,

    tracking.js) Web-based Services (Azure Face API, Google Cloud Vision, Amazon Rekognition) Shape Detection API New(ish) in Chrome (Experimental: chrome://flags/ #enable-experimental-web-platform-features) Facial Recognition in JavaScript Different Approaches
  46. face-api.js Client-side Facial Recognition Create a video element we’l use

    to display our media capture and include the face-api.js file. <video id=“inputVideo”> <script src="face-api.js/dist/face-api.min.js"></script>
  47. face-api.js Client-side Facial Recognition Load the networks and models so

    we can recognize faces and expressions. Get permission to access the webcam and display it in our video element. const video = document.getElementById('inputVideo') async function loadModels() { await faceapi.nets.ssdMobilenetv1.load(‘face-api.js/weights') await faceapi.nets.faceExpressionNet.load('face-api.js/weights') await faceapi.loadFaceRecognitionModel('face-api.js/weights') const stream = await navigator.mediaDevices.getUserMedia({video:{}}) video.srcObject = stream }
  48. face-api.js Client-side Facial Recognition Pass option to the models to

    determine how strict we want to match and pass the video element directly. If we get results we act on them. Otherwise, we can use requestAnimationFrame to continue processing new frames. video.addEventListener(‘play’, onPlay) async function onPlay() { const options = new faceapi.SsdMobilenetv1Options({ minConfidence: .5 }) const result = await faceapi.detectSingleFace(video, options).withFaceExpressions() if (result) { // do something with face & expressions object } window.requestAnimationFrame(onPlay) }
  49. Face API Azure Cognitive Services Setup parameters to send to

    the service. var params = { returnFaceId: "true", returnFaceLandmarks: "true", returnFaceAttributes: "age,gender,headPose,smile,facialHair,glasses,emotion," + "hair,makeup,occlusion,accessories,blur,exposure,noise" }; let returnValue = []; for (param in params) { returnValue.push( `${encodeURIComponent(param)}=$ {encodeURIComponent(params[param])}` ); }
  50. Face API Azure Cognitive Services We can’t pass the video

    element directly so we need to take a snapshot of the current frame using canvas to return a data URL (data://) ctx.drawImage(video, 0, 0); let data = canvas.toDataURL("image/jpeg");
  51. Face API Azure Cognitive Services We can use fetch on

    the data URL to return a Blob object that we can then send along (i.e. upload) to the service for processing. fetch(data) .then(res => res.blob()).then(blobData => { let subscriptionKey = "xxxxxx"; let uriBase = "https://westus2.api.cognitive.microsoft.com/face/v1.0/detect"; fetch(uriBase + "?" + parameters, { method: "POST", headers: { "Content-Type": "application/octet-stream", "Ocp-Apim-Subscription-Key": subscriptionKey }, body: blobData }) .then(response => { return response.json(); }) .then(response => { // act on response data
  52. Shape Detection API Chrome (Experimental) chrome://flags/#enable-experimental- web-platform-features if (window.FaceDetector ==

    undefined) { alert("Sorry, your browser does not support Face Detection.”); } async function findFaces(imageObject) { const faceDetector = new window.FaceDetector(); try { const faces = await faceDetector.detect(imageObject); return faces; } catch (e) { console.log(e); } }
  53. Shape Detection API Chrome (Experimental) chrome://flags/#enable-experimental- web-platform-features faces = await

    findFaces(canvas); faces.forEach(face => { // do something });
  54. Demos

  55. None
  56. None
  57. Thank You! More Information javascriptforartists.com Contact George Mandis @georgeMandis george@mand.is

    JavaScript for Artists