Neural Style and Deepfakes

Transcript

1. Neural artistic style & Deepfakes
2. Neural artistic style • What is the basic idea? • Apply the art style of one image to another • It’s not just another image filter • Use convolutional neural networks • Learn the low level features • Apply them to a new input image • Based on Gatys et al 2015/2016 • From Freiburg (Germany!!)
3. Fan fetchers neural style style content output
4. How do CNNs work?
5. Areas where CNNs are applied • Image Classification & Segmentation • Face Detection • Autonomous Driving • Robotics
6. Areas where CNNs are applied • Image Classification & Segmentation • Face Detection • Autonomous Driving • Robotics
7. Areas where CNNs are applied • Image Classification & Segmentation • Face Detection • Autonomous Driving • Robotics
8. How does neural style transfer work? • We want a function that shows how close in style and content the images are • Minimize Style/Content/Variation loss
9. How does neural style transfer work? • Use a pre-trained network (need much data to learn image features) • Don’t use the classification layer • Use the the style/content loss to update the generated image • Compare to original for each step
10. DEMO …
11. Neural artistic style – Wrap Up • CNNs are cool and can do loads of stuff • One input image can capture the artistic style • Coloring • Strokes • Areas • Even years after the death of an artist his style can be renewed • But this does not capture high level context
12. Can I use this as an Instagram filter? • Very computationally expensive • Process on Server • GPU optimized • Prisma App ca. 15$ subscription • Instagram has face swap • Works on high level knowledge of a face
13. Deep Fakes • What if you could transfer gestures instead of style? • Two things for really convincing fakes • Match face of other person onto image (with gestures) • Generate audio input for the face • This has been here for a long time with very high efforts (movies) • Fast and the furious • Return of the Jedi
14. Deep Fakes - Applications • Get Nicolas Cage in every movie • Shoot movies “without the actors” • ”The camera never lies” • Not true since Photoshop • This can be done in real time • Create Fake news • Driven by adult content industry first
15. Deep Fakes – How does it work (high level) • Learn what a face looks like (many inputs) • E.g. Nicolas Cage’s face features • Reconstruct the face based on a image/video source (one input) • E.g. Raider’s of the lost arc Movie • Get mimic, create a Nicolas Cage gesture image and align it with the actor
16. Deep Fakes – How does it work • Autoencoder (learn features with little data) • Encode features of the face (Learn a representation of features) • Decode and reconstruct the original input … …
17. Deep Fakes – How does it work … … Training Inference
18. Deep Fakes with Audio • Ok, but I still can’t fake the voice • Adobe Voco • 45 minutes of recordings and you can generate every tone • https://www.youtube.com/watch?v=cQ54GDm1eL0
19. Take away’s • CNNs are very powerful • You can capture the art styles of a painting with one image • You can transfer mimic from one image to another • Many people can now create realistic fake videos • Maybe AI can fight AI (if we can’t tell the difference anymore)
20. Thanks • Time for more DEMO?