Museum of Machine Arts

Talk on interesting projects in machine arts.

Transcript

1. Beyond Face Rotation
   

   View Slide

2. MOMA
   (Museum of Machine Arts)
   - Sourabh Bajaj
   

   View Slide

3. Hello
   • I’m Sourabh
   

   View Slide

4. Hello
   • I’m Sourabh
   • I’m a software Engineer
   

   View Slide

5. Hello
   • I’m Sourabh
   • I’m a software Engineer
   • I tweet @sb2nov
   

   View Slide

6. Hello
   • I’m Sourabh
   • I’m a software Engineer
   • I tweet @sb2nov
   • I eat gummy bears
   

   View Slide

7. Generative Neural Networks
   The network learns to map input to output by seeing many examples
   Output
   Text
   Image
   Video
   Music
   Audio
   3D
   Actions
   Other
   Input
   Random Noise
   Topic
   Text
   Image
   Video
   Music
   Audio
   3D
   Actions
   Other
   Generation / Translation Network
   

   View Slide

8. Object Detection
   and Recognition (ImageNet)
   googleresearch.blogspot.com/2014/09/
   building-deeper-understanding-of-image
   s.html (Szegedy et al., GoogLeNet)
   Live:
   • VGG
   • YOLO
   • YOLO v2
   • LeCun
   Concurrence,
   Localization
   Occlusion
   Out of
   context
   Counting
   Tracking
   

   View Slide

9. Automatic Colorization
   • Larsson et al., people.cs.uchicago.edu/~larsson/colorization
   • IIzuka et al., hi.cs.waseda.ac.jp/~iizuka/projects/colorization/en
   • Web interface: demos.algorithmia.com/colorize-photos (richzhang.github.io/colorization)
   Ground Truth
   Input Output
   

   View Slide

10. Denoising, Super-Resolution and Inpainting
    •Denoising auto-encoders
    (e.g. Chollet)
    •Recursive convolutional net
    for super-resolution, Kim et al.
    • Code: github.com/alexjc/neural-enhance
    • Deepsense.ai blogpost
    • letsenhance.io
    •Context encoders for inpainting
    Pathak et al.
    • Globally and Locally Consistent Image Completion
    • Adobe DeepFill
    • Contextual Attention - demo
    

    View Slide

11. Inceptionism: Deep Dreams
    Mordvintsev et al., googleresearch.blogspot.com/2015/06/inceptionism-going-deeper-into-neural.html (GoogLeNet)
    Pareidolia:
    Perceiving
    a familiar
    pattern
    where
    none exists.
    Johnny 5
    

    View Slide

12. Deep Dreams
    • Code: github.com/google/deepdream (see also: bat-county)
    • Online TensorFlow example
    • Reddit: reddit.com/r/deepdream
    • Web interface:
    • dreamdeeply.com • deepdreamgenerator.com
    • psychic-vr-lab.com/deepdream • deepdream.akkez.ru
    • Captions: www.cs.toronto.edu/~rkiros/inceptionism_captions.html
    • Video:
    • Fear and Loathing in Las Vegas
    • Forest Trail
    

    View Slide

13. Gatys et al., arxiv.org/abs/1508.06576 (VGG)
    • Features from object recognition! (*)
    • Web/Mobile interfaces:
    • dreamscopeapp.com, app, paintbrush (app, video) • ostagram.ru
    • deepart.io (app, video, fixed colors) • photopaint.us
    • instapainting.com/assets • aristo (app, video)
    • prisma-ai.com (app, video) • lucid (app, video)
    • deepdreamgenerator.com • deeparteffects.com (app)
    •Kogan (Alice video, Cubist Mirror)
    • 2001: Odyssey
    • Video: 1 2 3
    Artistic Style Transfer
    

    View Slide

14. Style
    Transfer
    Li and Wand, (code)
    arxiv.org/abs/1601.04589
    Luan et al., (code)
    arxiv.org/abs/1703.07511
    

    View Slide

15. View Slide

16. Neural Doodle: Semantic Style Transfer
    Champandard, arxiv.org/abs/1603.01768
    • Online demo: dmitryulyanov.github.io/feed-forward-neural-doodle
    • Code: github.com/alexjc/neural-doodle
    • Blog: nucl.ai/blog/neural-doodles
    • See also: Wentz, github.com/awentzonline/image-analogies
    

    View Slide

17. Image to Image Translation
    With Conditional Adversarial Networks (PatchGAN)
    Isola et al., phillipi.github.io/pix2pix Interactive: affinelayer.com/pixsrv
    Guide: ml4a.github.io/guides/Pix2Pix fotogenerator.npocloud.nl
    

    View Slide

18. View Slide

19. Unsupervised Image to Image Translation
    (DiscoGAN/CycleGAN/DualGAN)
    • Kim et al., arxiv.org/abs/1703.05192
    • Zhu et al., junyanz.github.io/CycleGAN
    • Yi et al., arxiv.org/abs/1704.02510 Face-off
    

    View Slide

20. View Slide

21. Text to Photo-realistic Image Synthesis
    with Stacked Generative Adversarial Networks (StackGAN)
    Zhang et al., arxiv.org/abs/1612.03242
    Code: github.com/hanzhanggit/StackGAN
    Cha et al., arxiv.org/abs/1708.09321
    StackGAN++:
    Zhang et al., arxiv.org/abs/1710.10916
    Code: github.com/hanzhanggit/StackGAN-v2
    

    View Slide

22. AttnGAN
    Xu et al.,
    arxiv.org/abs/1711.10485
    [supp]
    

    View Slide

23. AttnGAN – real results
    (github.com/taoxugit/AttnGAN)
    The sun is setting
    into the sea
    A man riding a Horse
    The cat is sitting on
    the couch
    

    View Slide

24. Video Prediction and Generation
    • Deep multi-scale video prediction beyond mean square error (Mathieu et al.)
    • Generating Videos with Scene Dynamics (Vondrick et al.)
    • Learning to Generate Long-term Future via Hierarchical Prediction (Villegas et al.)
    • Attentive Semantic Video Generation using Captions (Marwah et al.)
    • Video Generation from Text (Li et al.)
    • Visual to Sound: Generating Natural Sound for Videos in the Wild (Zhou et al.)
    • Imagine This! Scripts to Compositions to Videos (Gupta et al.)
    

    View Slide

25. Mikolov et al.,
    arxiv.org/abs/1301.3781
    Word Embeddings (Word2Vec)
    King
    Queen
    Man
    Woman
    King + ( Woman – Man ) = Queen
    King - Man + Woman = Queen
    Y
    X
    Web demos:
    • rare-technologies.com/
    word2vec-tutorial
    • bionlp-www.utu.fi/wv_demo
    Semantically:
    Algebraically:
    

    View Slide

26. Word Embeddings (Word2Vec)
    

    View Slide

27. Sutskever et al., arxiv.org/abs/1409.3215
    Sentence Embeddings (sequence-to-sequence encoder-decoder LSTM)
    

    View Slide

28. Continuous Sentence
    Representation with
    Variational Autoencoders
    •Bowman et al., Generating
    Sentences from a Continuous Space
    (1511.06349)
    •Semeniuta et al., A Hybrid
    Convolutional Variational
    Autoencoder for Text Generation
    (1702.02390)
    •Web demo:
    robinsloan.com/voyages-in-sentenc
    e-space
    

    View Slide

29. Thanks
    

    View Slide