Slide 1

Slide 1 text

Deep Learning in the Browser Explorables | Inference | Prototyping Amit Kapoor amitkaps.com

Slide 2

Slide 2 text

Data | Visuals | Story

Slide 3

Slide 3 text

Learning Paradigm

Slide 4

Slide 4 text

Augmenting Intelligence

Slide 5

Slide 5 text

Human in the Loop

Slide 6

Slide 6 text

Users | Creators | Builders

Slide 7

Slide 7 text

Learn | Play | Create

Slide 8

Slide 8 text

In the Browser, Really?

Slide 9

Slide 9 text

Immediate Accees Reduce Friction Wider Audience

Slide 10

Slide 10 text

Historical Challenges ▸ Early JS libraries were too slow - CPU based ▸ Poor support for numerical operations ▸ Underdeveloped ecosystem for handling data and preprocessing, declarative visualisation, and reactive runtime

Slide 11

Slide 11 text

Webgl accelerated Learning Frameworks

Slide 12

Slide 12 text

Webgl accelerated Learning Frameworks ▸ tensorflow.js ▸ tensorfire ▸ webdnn.js [Inference only] ▸ keras.js [Inference only]

Slide 13

Slide 13 text

How do we Learn?

Slide 14

Slide 14 text

Explorable Explanations

Slide 15

Slide 15 text

Discovery by Active Learning

Slide 16

Slide 16 text

Allowing the users to build intuition on Deep Learning

Slide 17

Slide 17 text

Algorithms Data Data-Models

Slide 18

Slide 18 text

Algorithm: Maze Generation1 1 Visualising Algorithms - Mike Bostock

Slide 19

Slide 19 text

Algorithm: Simple NN2 (2014) 2 Demo of Convnet.js by Andrej Karpathy

Slide 20

Slide 20 text

Algorithm: Neural Network (2016)3 3 Tinker with Neural Network

Slide 21

Slide 21 text

Algorithm: Universal Apprx. Concept 4 4 Universal Approximation using Tensorflow.js

Slide 22

Slide 22 text

Data: Multi-Dimensional Visualisation5 5 Facet Dive

Slide 23

Slide 23 text

Data: T-SNE Projector6 6 Embedding Projector

Slide 24

Slide 24 text

Data-Models: Feature Vis7 7 The Building Blocks of Interpretability

Slide 25

Slide 25 text

Data-Model: Fairness8 8 Fairness in Machine Learning

Slide 26

Slide 26 text

Visualisation Reactivity Immediacy

Slide 27

Slide 27 text

Multi-Disciplinary Skills Needed

Slide 28

Slide 28 text

How do we Create?

Slide 29

Slide 29 text

Model Inference

Slide 30

Slide 30 text

Bring your Deep Learning Model to the Data.

Slide 31

Slide 31 text

Abstract | Perceptual

Slide 32

Slide 32 text

Model Inference: Text9 9 IMDB Sentiment with Keras.js

Slide 33

Slide 33 text

Model Inference: Images10 10 Mnist with Keras.js

Slide 34

Slide 34 text

Model Inference: Style Transfer11 11 Fast Neural Transfer with deeplearn.js

Slide 35

Slide 35 text

Model Inference: Image Augmentation

Slide 36

Slide 36 text

Semi-Supervised Learning12 12 Quick Draw Crowd Sourced Data

Slide 37

Slide 37 text

Creative magenta.js | ml5js

Slide 38

Slide 38 text

Data vs. Model privacy

Slide 39

Slide 39 text

MODEL SIZE: quantization EMBEDDING SIZE: word2bit

Slide 40

Slide 40 text

Easier build for Web & Mobile

Slide 41

Slide 41 text

How do we Build?

Slide 42

Slide 42 text

Rapid Prototyping

Slide 43

Slide 43 text

Experiment and try new things

Slide 44

Slide 44 text

UI Based: Object Detection13 13 Model Builder in Deeplearn.js

Slide 45

Slide 45 text

Immediate environment to write code

Slide 46

Slide 46 text

Reactive Notebooks: ObservableHQ14 14 Reactive Notebook in the Browser

Slide 47

Slide 47 text

Challenges with Rapid Prototyping ▸ Training on GPUs (integration with Node.js) ▸ Custom layers not supported ▸ Lack of feature parity with Python / C++ APIs

Slide 48

Slide 48 text

Improved JS Ecosystem ▸ Declarative visualisation: vega-lite.js ▸ Reactive Runtime: observablehq.com ▸ Columnar In-Memory: arrow.js

Slide 49

Slide 49 text

Way Forward

Slide 50

Slide 50 text

Stronger Ecosystem Stable Libraries Deeper Use Cases

Slide 51

Slide 51 text

Deep Learning in the Browser Explorables | Inference | Prototyping Amit Kapoor amitkaps.com