AI for Art & Design
with Lite
Margaret Maynard-Reid, Sept 4 2020
#TFUGIndiaSummit
@margaretmz
Slide 2
Slide 2 text
#TFUGIndiaSummit | @margaretmz | #TFLite
Topics
● Computer vision tasks
● Generative art & design
● Datasets
2
AI for Art & Design TensorFlow Lite Samples & tutorials
● Samples & tutorials
● Community
● Tools & resources
● What is TFLite?
● How to make apps
Slide 3
Slide 3 text
Computer Vision Tasks
Complexity
Image classification
(single-label)
Classify an image to
a class
Examples
Painting style or
artist
Van Gogh
Image classification
(multi-label)
Classify an image to
multiple classes
Examples
Movie poster genre
action, sci-fi
Feature extraction
Extracting latent
features of an
image with CNN
models
Examples
Visual search
find similar fashion
Object detection
Identify one or
multiple objects
within an image and
their locations with
bounding boxes.
detect UI elements
Segmentation
Classify whether
each pixel of the
image belongs to a
certain class
segment UI elements
Generative models
(GANs)
Two or more models
trained simultaneously
Examples
- Generate new images
- Super res
- Image-to-image
3
Slide 4
Slide 4 text
Generative Art & Design
AI for Art & Design with TFLite
Slide 5
Slide 5 text
#TFUGIndiaSummit | @margaretmz | #TFLite
Deep Dream 2015
5
Slide 6
Slide 6 text
Use CNN neural networks
Content image + style image = styled image Content Style Generated
Neural Style Transfer: 50 Shades of Miaw (link)
A few more samples of generated images…
Style Transfer 2015
Slide 7
Slide 7 text
#TFUGIndiaSummit | @margaretmz | #TFLite
Generative Adversarial Networks 2014
What are GANs?
Generative Adversarial Networks (GANs)
have at least two network models which
compete against each other ...
7
Slide 8
Slide 8 text
#TFUGIndiaSummit | @margaretmz | #TFLite
GANs variations
GANs are fun but difficult to train!
● 2014 Ian Goodfellow’s original GAN’s paper
● 2015 DCGAN (Deep Convolutional GAN) (paper)
● 2014 cGAN (Conditional GAN) - Generate images conditioned on a class
(paper)
● 2016 Pix2Pix – paired image to image translation (paper)
● 2017 CycleGAN – unpaired Image to image translation
● 2018 StyleGAN – Generate images with some control on image characteristics
(paper)
● Many other GAN variations…
8
Slide 9
Slide 9 text
#TFUGIndiaSummit | @margaretmz | #TFLite
● Training datasets must be paired
● A type of conditional GAN
Link to TensorFlow tutorial
Pix2Pix 2016
9
Slide 10
Slide 10 text
#TFUGIndiaSummit | @margaretmz | #TFLite
CycleGAN 2017
● Training datasets are unpaired
● Horses <--> zebras
● Summer <--> winter
● Photos to paintings (Monet, Van Gogh etc.)
Link to TensorFlow tutorial
10
Slide 11
Slide 11 text
#TFUGIndiaSummit | @margaretmz | #TFLite
BigGAN 2018
Large scale GAN that generates high fidelity class-conditional images.
Link to paper | TF Hub biggan512, biggan-256, biggan-128
11
Slide 12
Slide 12 text
#TFUGIndiaSummit | @margaretmz | #TFLite
Control the styles of the generated images
● Pose
● Face shape
● Age
● Gender
● Hair color
4x4 --> 1024x1024
8GPUs trained for 1 week
Current examples: faces, bedrooms and cats
Link to video
StyleGAN 2018
12
Slide 13
Slide 13 text
#TFUGIndiaSummit | @margaretmz | #TFLite
Sketch2Code
Microsoft: Turn your whiteboard sketches to working code in seconds with Sketch2Code
13
Datasets
Art
● The Art Institute of Chicago - 5000 high res images
● Best Artworks of All Time
● Art Images: Drawing/Painting/Sculptures/Engravings
● Overwatch Heroes Recognition
● Chinese Fine Art
● Museum of Modern Art Collection
15
UI/UX Design
● Rico mobile app datasets
● Kaggle
○ Common Mobile/Web App Icons
○ Icons-50
Slide 16
Slide 16 text
TensorFlow Lite
AI for Art & Design with TFLite
Slide 17
Slide 17 text
#TFUGIndiaSummit | @margaretmz | #TFLite
What is TensorFlow Lite?
A framework with a set of tools for
deploying ML to mobile & embedded:
● A converter
● An interpreter
● Ops / kernels
● Interface to hardware
accelerations:
○ NN AIP
○ Edge TPu
● A bunch of other tools...
17
Slide 18
Slide 18 text
#TFUGIndiaSummit | @margaretmz | #TFLite
Why on device?
18
It’s all about that camera!
Slide 19
Slide 19 text
#TFUGIndiaSummit | @margaretmz | #TFLite
How to make a TensorFlow Lite app?
● ML model:
○ Train / find a model
○ Convert to .tflite format
○ Run inference in Python
○ Benchmark (optional)
○ Add model metadata
● Build Android app
○ UI navigation
○ Input image: gallery or camera
○ Run model inference
○ Display result in UI
19
Slide 20
Slide 20 text
#TFUGIndiaSummit | @margaretmz | #TFLite
What is an ML model?
• A complex mathematical function
• A representation of learnings from
training data
20
Slide 21
Slide 21 text
#TFUGIndiaSummit | @margaretmz | #TFLite
How to get an ML model?
● Train your own
● Transfer learning
● Download from TensorFlow Hub
● Use ML Kit
21
Slide 22
Slide 22 text
https://pair.withgoogle.com/
Slide 23
Slide 23 text
Apps and Tutorials
AI for Art & Design with TFLite
Slide 24
Slide 24 text
#TFUGIndiaSummit | @margaretmz | #TFLite
ML Kit Digital Ink Recognition
Draw & recognize emojis
Blog post:
https://developers.googleblog.com/2020/08/digital-ink-
recognition-in-ml-kit.html
Sample app:
https://developers.google.com/ml-kit/vision/digital-ink-r
ecognition
24
Slide 25
Slide 25 text
Part 2: ML Model Binding from Android Studio
TensorFlow Lite Model Maker
Part 1: TensorFlow Lite Model Maker
● Image/Text classification
25
Slide 26
Slide 26 text
#TFUGIndiaSummit | @margaretmz | #TFLite
Style transfer
Google Arts & Culture app:
https://play.google.com/store/apps/details?id=com.google.android.apps.cultural
Blog post:
4/2/2020 - Transform your photo in the style of an iconic artist
26
Slide 27
Slide 27 text
27
Selfie2Anime
● U-GAT-IT paper
● Image to image translation
● Written in TF1 code
● Unpaired selfie and anime images
● 50 & 100 epochs checkpoints
● Collaboration with Sayak & Khanh
● Links: project repo | blog
Slide 28
Slide 28 text
#TFUGIndiaSummit | @margaretmz | #TFLite
Whitebox CartoonGAN
● Faster & smaller
● Not just for selfies
● Link to project repo
● Link to TF Hub models
● Collaboration with Sayak & Khanh
● Links: project repo | blog
28
Slide 29
Slide 29 text
#TFUGIndiaSummit | @margaretmz | #TFLite
Segmentation + Style Transfer
Demo by Khanh LeViet TFLite inference in Python,
to be implemented on mobile... (project repo)
29
Slide 30
Slide 30 text
Community & resources
AI for Art & Design with TFLite
Awesome TensorFlow Lite
A curated list of models, apps, tutorials and
learning resources…
Thanks to the ML GDE and Android community!
Support of TensorFlow Lite team
https://github.com/margaretmz/awesome-tflite
32
#TFUGIndiaSummit | @margaretmz | #TFLite
AI & the Future of UX
https://uxdesign.cc/ai/home
35
Slide 36
Slide 36 text
#TFUGIndiaSummit | @margaretmz | #TFLite
AI for Art & Design
AI will assist and inspire artists and designers
Everyone can now make art and design!
36
Slide 37
Slide 37 text
#TFUGIndiaSummit | @margaretmz | #TFLite
Follow me on Twitter, Medium or GitHub to learn more about
deep learning, TensorFlow and on-device ML.
@margaretmz
@margaretmz
margaretmz
Thank you!
37