Share
Embed
OpenTalks.AI 2023 - Dmitry Korobchenko, NVIDIA, AI Technologies for Digital Characters and Avatars
opentalks3
March 18, 2023
Transcript
26 slides
-
AI Technologies for Digital Characters and Avatars Dmitry Korobchenko, Director
of AI | OpenTalks.AI | 06 March 2023 -
• What are Digital Characters and Avatars? • Applications •
Tasks :: Create a Character • Tasks :: Make it Intelligent • Tasks :: Make it Alive • Summary Agenda -
Applications Media & Entertainment Digital Twins Interactive Avatars Communication Games
Movies VFX / CGI Digital Factory / City Drive Simulation Synthetic Data for AI Training Digital Assistants Education Healthcare Social Media Avatars Video Conferences Enterprise Telepresence -
Tasks in Digital Avatars Domain ✏️ Create a Character Using
Reconstruction Using Generative AI -
Tasks in Digital Avatars Domain ✏️ Create a Character 🧠
Make it Intelligent Using Reconstruction Using Generative AI Conversational AI Environment Awareness -
Tasks in Digital Avatars Domain ✏️ Create a Character 🧠
Make it Intelligent 😃 Make it Alive Using Reconstruction Using Generative AI Conversational AI Environment Awareness Rendering / Materials Animation / Physics -
✏️ Creating Digital Avatars Input Photo Reconstructed 3D Avatar Image
to 3D Avatar Source: NVIDIA Image2Avatar Preview -
✏️ Creating Digital Avatars Source: Chan et al, “Efficient Geometry-aware
3D Generative Adversarial Networks” Efficient Geometry-Aware 3D GAN -
✏️ Creating Digital Avatars Source: Chan et al, “Efficient Geometry-aware
3D Generative Adversarial Networks” Input Photo Inverted GAN Output 3D Reconstruction by GAN Inversion -
🧠 Conversational AI NLP / NLU / Language Models /
Dialog Systems Automatic Speech Recognition Text-to-Speech Vision Identity Recognition Emotion / Gestures Eye Contact Riva SDK Riva SDK -
🧠 Environment Awareness 3D Scene Understanding 3D Object Classification Scene
Parsing / Segmentation Building a Scene Graph laptop bed table chest -
🧠 Environment Awareness 3D Scene Understanding Planning and Navigation 3D
Object Classification Scene Parsing / Segmentation Building a Scene Graph Affordance Detection Task-Based Semantic Planning Route Construction “Take the cup from the table and put in on the shelf next to the TV” Go To Table #2 Take Cup #5 Go To Shelf #3 Put Shelf #3 1 2 3 4 laptop bed table chest 1 3 4 2 -
😃 Rendering / Materials Omniverse RTX Real Photo Rendered Rendered
Neural Rendering AI Denoising Super-Resolution -
😃 Animation Video-Driven Facial Animation Source: Daněček et al, “EMOCA
: Emotion-Driven Monocular Face Capture and Animation” Input Video Reconstructed Facial Expressions -
😃 Animation Audio-Driven Facial Animation FACS Coefficients Skin, Tongue, Eyes,
Teeth Full-Face Template Final Character Composition Retargeting Post- Processing Neural Net Emotion / Style Final Character OR Source: NVIDIA Omniverse Audio2Face Speech Audio -
😃 Animation Audio-Driven Gesture and Body Animation Generator or Motion
Matching Speech Audio Emotion / Style Retargeting Template Skeleton Animation Final Character Final Character Source: NVIDIA Omniverse Audio2Gesture -
😃 Animation Retargeting Source Motion Target Skeletons Source: Aberman et
al, “Skeleton-Aware Networks for Deep Motion Retargeting” -
😃 Animation Animation Representation Spaces Source: Starke et al, “DeepPhase:
Periodic Autoencoders for Learning Motion Phase Manifolds” -
😃 Animation / Physics Task-Driven Animation and Object Interactions “Swing
the sword” Source: Juravsky et al, “PADL: Language-Directed Physics-Based Character Control” Skill Embedding Animation Clip “Strike the pink block” -
Animation AI Conversational AI Vision AI Recommender AI Audio2Face Audio2Emotion
NLP ASR, TTS Computer Vision Video Analytics Recommender Systems NVIDIA Omniverse Avatar Cloud Engine Cloud-Based AI Microservices -
Summary ✏️ Create a Character 🧠 Make it Intelligent 😃
Make it Alive Using Reconstruction Using Generative AI Face / Body / Texture Conversation Speech Vision 3D Scene Understanding Planning and Navigation Rendering Materials Physics Facial Animation Gestures Locomotion Audio / Video-based Interaction with Objects Task-Based Animation