AI for Art & Design (with TensorFlow Lite) Margaret Maynard-Reid 3/15/2023 @margaretmz

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign About me 2

margaretmz.art

@margaretmz | #AIArt | #TensorFlow | #TFLite Aspiring 3D Fashion Designer 4 ● Study at Parsons School of Design ● Creating fashion designs with Clo3D

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Topics: generative & on-device ● Deep dream ● Style transfer ● GANs & diffusion models ● Image to image ● Text to image… Generative AI for Art & Design Training TensorFlow 2 / Keras Inference TensorFlow Lite ● What is TF Lite? ● Sample apps ● Awesome TFLite ● E2E TFLite Tutorials ● Sequential API ● Functional API ● Model subclassing - custom train_step 5

Generative Art & Design 6

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Art & Design ● Drawing ● Painting ● Photography ● Music ● Videos ● Product design ● Fashion design ● Graphic design ● UI/UX design ● ... 7

Computer Vision Tasks Complexity Image classification (single-label) Classify an image to a class Examples Painting style or artist Van Gogh Image classification (multi-label) Classify an image to multiple classes Examples Movie poster genre action, sci-fi Feature extraction Extracting latent features of an image with CNN models Examples Visual search find similar fashion Object detection Identify one or multiple objects within an image and their locations with bounding boxes. detect UI elements Segmentation Classify whether each pixel of the image belongs to a certain class segment UI elements Generative models (GANs & Stable Diffusion Two or more models trained simultaneously Examples - Generate new images - Super res - Image-to-image 8

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Deep Dream 2015 9 Source: Deep Dream generator https://deepdreamgenerator.com/#gallery

Image Style Transfer (2015) Use CNN neural networks Content image + style image = styled image Content Style Generated Neural Style Transfer: 50 Shades of Miaw (link) A few more samples of generated images… 10 Style Transfer 2015

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative Models Source: Lilian Weng blog (link) 11 “Generative models: take a machine, observe many samples from a distribution and generate more samples from that same distribution”. - Ian Goodfellow 2016

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative Adversarial Networks (GANs) GANs have at least two network models which compete against each other ... 12

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Diffusion Models 1. Gradually add gaussian noise to training data 2. Learn how to reverse the process to generate images from noise. 13 Source: Nvidia developer blog (link) Forward image diffusion Generative reverse denoise

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Stable Diffusion https://jalammar.github.io/illustrated-stable-diffusion/ 14

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign 15

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Fashion++ Facebook AI: “Fashion++: Minimal Edits for Outfit Improvement” (news | paper) 16

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Icons 8 - AI powered face generation 17 Image Source: icons8.com

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Photoshop - Neural Filters 18 Image Source: Adobe Photoshop tour

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Text-to-image: StyleCLIP Text-Driven Manipulation of StyleGAN Imagery 19

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Text-to-image: VQGAN + CLIP Use a text prompt to generate images ● VQGAN - Vector Quantized GAN ● CLIP Contrastive Language-Image Pre-training model from OpenAI 20 Image Source: @hannahjdotca on Twitter

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Stable Diffusion Link to tweet here 21 Source: stability.ai

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Keras Stable Diffusion Tutorials Excellent Keras tutorials: ● A walk through latent space with Stable Diffusion ● High-performance image generation using Stable Diffusion in KerasCV ● DreamBooth with Keras 22

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign 23 Lexica.art

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative art & design… A lot of sites for trying out generative art and design for free… ● https://dreamstudio.com/create/ ● https://discord.com/invite/midjourney ● https://designer.microsoft.com/ ● https://techcrunch.com/2021/12/23/wombo-dream-app/ 24

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative models: key takeaways ● Diffusion models democratize generative AI ● AI will assist the artists and designers ● Gives artists and designers inspiration ● None engineers will have access to the AI tools ● Everyone can create art and designs 25

Train a GAN model with TensorFlow 2 / Keras 26

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign GANs in Art & Design My tutorial series with Colab notebooks: https://github.com/margaretmz/GANs-in-Art-and-Design 27

Model Inference with TensorFlow Lite 28

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign TensorFlow 2.x 29 Model ● tf.Keras (TensorFlow) ● Python libraries: Numpy, Matplotlib etc SavedModel or Keras model Serving ● Cloud ● Web ● Mobile ● IoT ● Micro controllers ● Edge TPU Training Inference Data

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign What is TensorFlow Lite? A framework with a set of tools for deploying ML to mobile & embedded: ● A converter ● An interpreter ● Ops / kernels ● Interface to hardware accelerations: ○ e.g. NN AIP & Edge TPU ● A bunch of other tools: quantization, pruning, benchmarking... ○ 30

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Android app with TensorFlow Lite ● ML model: ○ Train / find a model ○ Convert to .tflite format ○ Run inference in Python ○ Benchmark (optional) ○ Add model metadata (for using task APIs) ● Build Android app ○ Create UI navigation ○ Input image: canvas, gallery or camera ○ Load model & run inference ○ Display results in UI 31

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign How to implement a GAN model? ● You only need need the generator model for inference 32

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign ESRGAN on Android with TFLite ● Enhanced Super Resolution GAN ● Pre-trained ESRGAN model from TF Hub ● Convert to TFLite model ○ Dynamic range quantization ○ Input image 50x50 ● TensorFLow C API, GPU library ● Android NDK 33 https://blog.tensorflow.org/2020/12/how-to-generate-super -resolution-images-using-tensorflow-lite-on-android.html

@margaretmz | #AIArt | #TensorFlow | #TFLite Selfie2Anime ● U-GAT-IT paper ● Image to image translation ● Written in TF1 code ● Unpaired selfie and anime images ● 50 & 100 epochs checkpoints ● Collaboration with Sayak & Khanh ● Links: project repo | blog 34

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Cartoonizer ● Model: Whitebox CartoonGAN (authors Xinrui Wang & Jinze Yu) ○ Surface, structure and texture representations ○ Faster & smaller ○ Not just for selfies ● Collaboration with Googlers from TFLite team & another ML GDE ● Links: project repo | blog 35

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Segmentation + Style Transfer Demo by Khanh LeViet TFLite inference in Python, Already implemented in Android (and iOS)... (project repo) 36

@margaretmz | #AIArt | #TensorFlow | #TFLite Awesome TensorFlow Lite A curated list of models, apps, tutorials and learning resources… Thanks to the ML GDE and Android community! Support of TensorFlow Lite team https://github.com/margaretmz/awesome-tflite 37

@margaretmz @GDGSeattle #AI #Art #3DFashionDesign Follow me on Twitter, Medium or GitHub to learn more about generative art and design, TensorFlow and on-device ML. @margaretmz @margaretmz @margaretmz Thank you! 38