Neural artistic style & Deepfakes 

Neural artistic style • What is the basic idea? • Apply the art style of one image to another • It’s not just another image filter • Use convolutional neural networks • Learn the low level features • Apply them to a new input image • Based on Gatys et al 2015/2016 • From Freiburg (Germany!!) 

Fan fetchers neural style style content output 

How do CNNs work? 

Areas where CNNs are applied • Image Classification & Segmentation • Face Detection • Autonomous Driving • Robotics 

Areas where CNNs are applied • Image Classification & Segmentation • Face Detection • Autonomous Driving • Robotics 

Areas where CNNs are applied • Image Classification & Segmentation • Face Detection • Autonomous Driving • Robotics 

How does neural style transfer work? • We want a function that shows how close in style and content the images are • Minimize Style/Content/Variation loss 

How does neural style transfer work? • Use a pre-trained network (need much data to learn image features) • Don’t use the classification layer • Use the the style/content loss to update the generated image • Compare to original for each step 

DEMO … 

Neural artistic style – Wrap Up • CNNs are cool and can do loads of stuff • One input image can capture the artistic style • Coloring • Strokes • Areas • Even years after the death of an artist his style can be renewed • But this does not capture high level context 

Can I use this as an Instagram filter? • Very computationally expensive • Process on Server • GPU optimized • Prisma App ca. 15$ subscription • Instagram has face swap • Works on high level knowledge of a face 

Deep Fakes • What if you could transfer gestures instead of style? • Two things for really convincing fakes • Match face of other person onto image (with gestures) • Generate audio input for the face • This has been here for a long time with very high efforts (movies) • Fast and the furious • Return of the Jedi 

Deep Fakes - Applications • Get Nicolas Cage in every movie • Shoot movies “without the actors” • ”The camera never lies” • Not true since Photoshop • This can be done in real time • Create Fake news • Driven by adult content industry first 

Deep Fakes – How does it work (high level) • Learn what a face looks like (many inputs) • E.g. Nicolas Cage’s face features • Reconstruct the face based on a image/video source (one input) • E.g. Raider’s of the lost arc Movie • Get mimic, create a Nicolas Cage gesture image and align it with the actor 

Deep Fakes – How does it work • Autoencoder (learn features with little data) • Encode features of the face (Learn a representation of features) • Decode and reconstruct the original input … … 

Deep Fakes – How does it work … … Training Inference 

Deep Fakes with Audio • Ok, but I still can’t fake the voice • Adobe Voco • 45 minutes of recordings and you can generate every tone • https://www.youtube.com/watch?v=cQ54GDm1eL0 

Take away’s • CNNs are very powerful • You can capture the art styles of a painting with one image • You can transfer mimic from one image to another • Many people can now create realistic fake videos • Maybe AI can fight AI (if we can’t tell the difference anymore) 

Thanks • Time for more DEMO?