Democratize Deep Learning Models

A perspective on how to make deep learning models more accessible and usable for a wider spectrum of people. Using tensorflow.js

1. Educate: Explorable Explanations
2. Create: Rapid Prototyping
3. Visualise: Model Visualisation
4. Intervene: Decision Making
5. Imagine: Generative Spaces

Transcript

1. Democratize Deep Learning Models
   Using TensorFlow.js
   Perspective & Views
   Amit Kapoor
   Bargava Subramanian
   1
   

   View Slide

2. Amit Kapoor
   Data | Visual | Story
   — Teaching Analytics & Vis
   — Consulting & Mentoring
   — Building DataVis Tools
   amitkaps.com
   2
   

   View Slide

3. Bargava Subramanian
   Data | AI & ML | Productivity
   — Startup @Binaize
   — Data Scientist (ex @Cisco,
   @RedHat)
   — Consulting on ML products
   bargava.com
   3
   

   View Slide

4. Deep Learning Challenges
   — Scale Up: Faster data & models
   — Scale Out: Bigger data & models
   — Scale Wide: Entire data & model pipelines
   — Scale Across: Beyond data scientists & engineers
   How do we get more Human in the Loop approach to
   using Deep Learning?
   4
   

   View Slide

5. Democratize Deep Learning (DL)
   1. Educate: Why will DL work for our use case?
   2. Create: What can we do to build our DL model?
   3. Visualise: So how do we improve our DL models?
   4. Intervene: So what actions can we take using DL?
   5. Imagine: So what else can we use DL for?
   5
   

   View Slide

6. Perspective on Democratize DL
   1. Educate: Explorable Explanations
   2. Create: Rapid Prototyping
   3. Visualise: Model Visualisation
   4. Intervene: Decision Making
   5. Imagine: Generative Spaces
   6
   

   View Slide

7. Structure for Each Perspective
   — Conceptual: Meaning & View
   — In Real Life: Context & challenge
   — Our Approach: Our Solutions & Experiments
   — Exemplars: Open-source Examples & Libraries
   — Learnings: Lessons learned
   7
   

   View Slide

8. Lens – Tensorflow.js
   — Javascript version of Tensorflow Deep Learning Library
   — Core (Linear Algebra low level API)
   — Layers (Keras-type high level API)
   — Runs both on Client-side (JS, WebAssembly, WebGL) and on
   Server-side (Nodejs with C binary for CPU & GPU)
   — Can be used for both inference and training
   — Custom models optimised for Image, Text, Audio etc.
   — Can Import & use models build in Python version
   8
   

   View Slide

9. 1. Educate
   Explorable Explanations
   “For the things we have to learn before we can do
   them, we learn by doing them.”
   ― Aristotle
   9
   

   View Slide

10. Neural Network Playground1
    1 Tinker with Neural Network
    10
    

    View Slide

11. In Real Life: Context
    Client: Leading US Network Service Provider
    Scope: Threat detection using User & Entity
    Behaviour Analytics
    Status: Unsustainable Rule Based Policies
    Mandate: Evaluate shift to ML/DL Models
    Challenge: Domain experts skeptical of ML/DL
    paradigm
    11
    

    View Slide

12. How to Educate?
    — Work with sample or open data
    — Show the potential for ML/DL models
    — Need to showcase visually explorable insights
    — Need for interactive exploration across
    stakeholders (CISO, CTO)
    12
    

    View Slide

13. Explorable Explanations
    — Showcased multiple models: K-Means, One Class SVM,
    Autoencoders
    — Created models in Python & exported to Tensorflow.js
    — Integrated with an interactive dashboard (using AngularJS)
    — Built-in exploration capability on tweaking (and re-
    running) models
    Eventual API & implementation based on K-Means model
    13
    

    View Slide

14. Exemplar: Blip2
    2 Anomaly Detection Dashboard by Fast Forward Labs
    14
    

    View Slide

15. Exemplar: Distill.pub3
    3 How Momentum Works by Distill.pub
    15
    

    View Slide

16. Learning on Educate
    — Discovery by Active Learning is powerful
    — Allows the user to build intuition on DL
    — Not easy! Requires deep thinking on design and
    usability
    16
    

    View Slide

17. 2. Create
    Rapid Prototyping
    "The secret of getting started is breaking your
    complex overwhelming tasks into smaller manageable
    tasks, and then starting on the first one"
    - Mark Twain
    17
    

    View Slide

18. Transfer Learning on Images4
    4 TensorSpace framework
    18
    

    View Slide

19. In Real Life: Context
    Client: Boutique US Software Consulting shop
    Scope: Computer vision use-cases for an Automotive
    client
    Status: Full stack web & mobile expertise
    Mandate: Build & extend capabilities for ML&DL
    Challenge: No experience in python data stack
    19
    

    View Slide

20. What to Prototype
    — Working with sample data
    — Needed to build an image classification model
    — Show how to integrate that model as API
    — Create an interactive dashboard to demo to client
    20
    

    View Slide

21. Create Rapid Prototypes
    — Built CNN models for tra ic sign detection
    — Built models in tensorflow.js using pre-built
    models (transfer learning)
    — Deployed model as API using Node
    — Built a simple single-page application for user
    to upload image and get predictions
    21
    

    View Slide

22. Exemplar: tfjs in Observablehq5
    5 Collection: Deep Learning in the Browser
    22
    

    View Slide

23. Learning on Create
    — Enabling software developers to build ML unlocks
    huge potential
    — Easy to prototype without adding technical debt
    and additional technology costs (eg: Python data
    stack)
    — Not easy! How to build custom models.
    23
    

    View Slide

24. 3. Visualise
    Model Visualisation
    “The most powerful way to gain insight into a
    system is by moving between levels of abstraction.”
    — Bret Victor
    24
    

    View Slide

25. View Slide

26. Model Visualisation
    [0] Visualise the data space
    [1] Visualise the predictions in the data space
    [2] Visualise the errors in model fitting
    [3] Visualise with di erent model parameters
    [4] Visualise with di erent input datasets
    [5] Visualise the entire model space
    [6] Visualise the entire feature space
    [7] Visualise the many models together
    26
    

    View Slide

27. ModelVis & ML/DL
    [0] DATA VIS: the data space
    [1] PREDICTION: the predictions in the data space
    [2] VALIDATION: the errors in model fitting
    [3] TUNING: with di erent model parameters
    [4] BOOTSTRAP: with di erent input datasets
    [5] ENSEMBLE: the entire model space
    [6] FEATURE ENGG: the entire feature space
    [7] N-MODELS: the many models together
    27
    

    View Slide

28. ModelVis Key Concept
    Use visualisation to aid the transition of implicit
    knowledge about the data to explicit knowledge in
    the model.
    — Iterative, not linear
    — Up and Down, not lateral
    — Complementary, not exclusive
    28
    

    View Slide

29. In Real Life: Context
    Client: Large Consumer Product Company in India
    Scope: Network optimisation of secondary freight
    Status: Regional & fragmented network
    Mandate: Evaluate consolidation opportunities
    Challenge: Limited data on new options, Implicit
    domain knowledge
    29
    

    View Slide

30. View Slide

31. Applying Model Visualisation
    — Engage with Supply Chain domain experts with
    limited ML understanding
    — Need to represent models in the same data
    visualisation space
    — Need to showcase visually explorable model options
    — Need to build interactive exploration for di erent
    stakeholders (CSO, Managers)
    31
    

    View Slide

32. View Slide

33. Exemplar: Manifold from Uber6
    6 Model-Agnostic Visual Debugging tool
    33
    

    View Slide

34. Exemplar: Feature Visualisation7
    7 The Building Blocks of Interpretability
    34
    

    View Slide

35. Learning on Model Vis
    — Model improvements require engaging with domain
    through their lens
    — Visual analytics principles to model visualisation
    — Decouple model vis from model building compute &
    pipelines
    — Multivariate visualisation is more challenging
    than larger data!
    35
    

    View Slide

36. 4. Intervene
    Decision Making
    "All models are wrong, but some are useful"
    — George Box
    36
    

    View Slide

37. Using Models for Decisions8
    8 Fairness in Machine Learning
    37
    

    View Slide

38. Context & Use Case
    Client: Small Regional Movies OTT App in India
    Scope: Map movie ROI & acquisition focus
    Status: Popularity & heuristic-based decisions
    Mandate: Build RecSys for engagement & licensing
    Challenge: Limited user insight, no data team
    38
    

    View Slide

39. Challenges
    — User and movie personas were hard-coded
    — No real-time user insight to drive
    recommendations
    — Hard to link user behaviour to movie acquisition
    — Tool should work for business (content
    acquisition team), not only technology team
    39
    

    View Slide

40. Decision Making Tool
    — Built multiple RecSys models to drive app
    recommendations
    — Interactive dashboard of user & movie embeddings
    — Created "What-If" scenario explorer - to link
    changing user behaviour with movie acquisition
    Shifted from Heuristic-based to Data-driven decision
    40
    

    View Slide

41. Exemplar: RecSys & Business9
    9 Understanding Latent Factors by StitchFix
    41
    

    View Slide

42. Exemplar: CounterFactuals10
    10 What-If tool by Google Pair
    42
    

    View Slide

43. 5. Imagine
    Generative Spaces
    "I don't know, what I don't know"
    — The second level of ignorance
    43
    

    View Slide

44. Font Design Space11
    11 Using Artificial Intelligence to Augment Human Intelligence
    44
    

    View Slide

45. Generative Spaces
    — Moving beyond Supervised learning paradigms
    — Using generative models to invent meaningful
    creative operations
    — Designing and exploring latent spaces (with
    constraints)
    — Applications across design, product development,
    service design
    45
    

    View Slide

46. Examples & Experiments using tfjs
    — ML for the web: ml5.js
    — Music & Art: magenta.js
    — Machine Learning for Artists: ml4a
    — AI Google Experiments: AI Collections
    46
    

    View Slide

47. Democratize Deep Learning (DL)
    1. Educate: Explorable Explanations
    2. Create: Rapid Prototyping
    3. Visualise: Model Visualisation
    4. Intervene: Decision Making
    5. Imagine: Generative Spaces
    47
    

    View Slide

48. Access this presentation at:
    https://speakerdeck.com/amitkaps
    Amit Kapoor
    amitkaps.com
    Bargava Subramanian
    bargava.com
    48
    

    View Slide