OpenTalks.AI 2023 - Dmitry Korobchenko, NVIDIA, AI Technologies for Digital Characters and Avatars

Transcript

1. AI Technologies for Digital Characters and Avatars Dmitry Korobchenko, Director of AI | OpenTalks.AI | 06 March 2023
2. • What are Digital Characters and Avatars? • Applications • Tasks :: Create a Character • Tasks :: Make it Intelligent • Tasks :: Make it Alive • Summary Agenda
3. What are Digital Characters and Avatars?
4. Applications Media & Entertainment Digital Twins Interactive Avatars Communication Games Movies VFX / CGI Digital Factory / City Drive Simulation Synthetic Data for AI Training Digital Assistants Education Healthcare Social Media Avatars Video Conferences Enterprise Telepresence
5. Tasks in Digital Avatars Domain ✏️ Create a Character Using Reconstruction Using Generative AI
6. Tasks in Digital Avatars Domain ✏️ Create a Character 🧠 Make it Intelligent Using Reconstruction Using Generative AI Conversational AI Environment Awareness
7. Tasks in Digital Avatars Domain ✏️ Create a Character 🧠 Make it Intelligent 😃 Make it Alive Using Reconstruction Using Generative AI Conversational AI Environment Awareness Rendering / Materials Animation / Physics
8. ✏️ Creating Digital Avatars Input Photo Reconstructed 3D Avatar Image to 3D Avatar Source: NVIDIA Image2Avatar Preview
9. ✏️ Creating Digital Avatars Source: Chan et al, “Efficient Geometry-aware 3D Generative Adversarial Networks” Efficient Geometry-Aware 3D GAN
10. ✏️ Creating Digital Avatars Source: Chan et al, “Efficient Geometry-aware 3D Generative Adversarial Networks” Input Photo Inverted GAN Output 3D Reconstruction by GAN Inversion
11. ✏️ Creating Digital Avatars Neural Radiance Fields (NeRF) Source: NVIDIA Instance NeRF
12. 🧠 Conversational AI NLP / NLU / Language Models / Dialog Systems Automatic Speech Recognition Text-to-Speech Vision Identity Recognition Emotion / Gestures Eye Contact Riva SDK Riva SDK
13. 🧠 Environment Awareness 3D Scene Understanding 3D Object Classification Scene Parsing / Segmentation Building a Scene Graph laptop bed table chest
14. 🧠 Environment Awareness 3D Scene Understanding Planning and Navigation 3D Object Classification Scene Parsing / Segmentation Building a Scene Graph Affordance Detection Task-Based Semantic Planning Route Construction “Take the cup from the table and put in on the shelf next to the TV” Go To Table #2 Take Cup #5 Go To Shelf #3 Put Shelf #3 1 2 3 4 laptop bed table chest 1 3 4 2
15. 😃 Rendering / Materials Omniverse RTX Real Photo Rendered Rendered Neural Rendering AI Denoising Super-Resolution
16. 😃 Animation Video-Driven Facial Animation Source: Daněček et al, “EMOCA : Emotion-Driven Monocular Face Capture and Animation” Input Video Reconstructed Facial Expressions
17. 😃 Animation Pose Estimation Source: NVIDIA Omniverse Machinima
18. 😃 Animation Audio-Driven Facial Animation FACS Coefficients Skin, Tongue, Eyes, Teeth Full-Face Template Final Character Composition Retargeting Post- Processing Neural Net Emotion / Style Final Character OR Source: NVIDIA Omniverse Audio2Face Speech Audio
19. 😃 Animation Audio-Driven Gesture and Body Animation Generator or Motion Matching Speech Audio Emotion / Style Retargeting Template Skeleton Animation Final Character Final Character Source: NVIDIA Omniverse Audio2Gesture
20. 😃 Animation Retargeting Source Motion Target Skeletons Source: Aberman et al, “Skeleton-Aware Networks for Deep Motion Retargeting”
21. 😃 Animation Animation Representation Spaces Source: Starke et al, “DeepPhase: Periodic Autoencoders for Learning Motion Phase Manifolds”
22. 😃 Animation / Physics Task-Driven Animation and Object Interactions “Swing the sword” Source: Juravsky et al, “PADL: Language-Directed Physics-Based Character Control” Skill Embedding Animation Clip “Strike the pink block”
23. Animation AI Conversational AI Vision AI Recommender AI Audio2Face Audio2Emotion NLP ASR, TTS Computer Vision Video Analytics Recommender Systems NVIDIA Omniverse Avatar Cloud Engine Cloud-Based AI Microservices
24. NVIDIA Omniverse Audio2Face App Audio-Driven Emotional Facial Animation
25. Summary ✏️ Create a Character 🧠 Make it Intelligent 😃 Make it Alive Using Reconstruction Using Generative AI Face / Body / Texture Conversation Speech Vision 3D Scene Understanding Planning and Navigation Rendering Materials Physics Facial Animation Gestures Locomotion Audio / Video-based Interaction with Objects Task-Based Animation