OpenTalks.AI 2023 - Dmitry Korobchenko, NVIDIA, AI Technologies for Digital Characters and Avatars

Transcript

1. AI Technologies for Digital Characters and Avatars
   Dmitry Korobchenko, Director of AI | OpenTalks.AI | 06 March 2023
   

   View Slide

2. • What are Digital Characters and Avatars?
   • Applications
   • Tasks :: Create a Character
   • Tasks :: Make it Intelligent
   • Tasks :: Make it Alive
   • Summary
   Agenda
   

   View Slide

3. What are Digital Characters and Avatars?
   

   View Slide

4. Applications
   Media & Entertainment Digital Twins Interactive Avatars Communication
   Games
   Movies
   VFX / CGI
   Digital Factory / City
   Drive Simulation
   Synthetic Data for AI
   Training
   Digital Assistants
   Education
   Healthcare
   Social Media Avatars
   Video Conferences
   Enterprise Telepresence
   

   View Slide

5. Tasks in Digital Avatars Domain
   ✏️ Create a Character
   Using Reconstruction
   Using Generative AI
   

   View Slide

6. Tasks in Digital Avatars Domain
   ✏️ Create a Character 🧠 Make it Intelligent
   Using Reconstruction
   Using Generative AI
   Conversational AI
   Environment Awareness
   

   View Slide

7. Tasks in Digital Avatars Domain
   ✏️ Create a Character 🧠 Make it Intelligent 😃 Make it Alive
   Using Reconstruction
   Using Generative AI
   Conversational AI
   Environment Awareness
   Rendering / Materials
   Animation / Physics
   

   View Slide

8. ✏️ Creating Digital Avatars
   Input Photo Reconstructed 3D Avatar
   Image to 3D Avatar
   Source: NVIDIA Image2Avatar Preview
   

   View Slide

9. ✏️ Creating Digital Avatars
   Source: Chan et al, “Efficient Geometry-aware 3D Generative Adversarial Networks”
   Efficient Geometry-Aware 3D GAN
   

   View Slide

10. ✏️ Creating Digital Avatars
    Source: Chan et al, “Efficient Geometry-aware 3D Generative Adversarial Networks”
    Input Photo Inverted GAN Output
    3D Reconstruction by GAN Inversion
    

    View Slide

11. ✏️ Creating Digital Avatars
    Neural Radiance Fields (NeRF)
    Source: NVIDIA Instance NeRF
    

    View Slide

12. 🧠 Conversational AI
    NLP / NLU / Language Models / Dialog Systems
    Automatic Speech Recognition
    Text-to-Speech
    Vision
    Identity Recognition
    Emotion / Gestures
    Eye Contact
    Riva SDK
    Riva SDK
    

    View Slide

13. 🧠 Environment Awareness
    3D Scene Understanding
    3D Object Classification
    Scene Parsing / Segmentation
    Building a Scene Graph
    laptop
    bed
    table
    chest
    

    View Slide

14. 🧠 Environment Awareness
    3D Scene Understanding Planning and Navigation
    3D Object Classification
    Scene Parsing / Segmentation
    Building a Scene Graph
    Affordance Detection
    Task-Based Semantic Planning
    Route Construction
    “Take the cup from the
    table and put in on the
    shelf next to the TV”
    Go To Table #2
    Take Cup #5
    Go To Shelf #3
    Put Shelf #3
    1
    2
    3
    4
    laptop
    bed
    table
    chest
    1
    3 4
    2
    

    View Slide

15. 😃 Rendering / Materials
    Omniverse RTX
    Real Photo Rendered
    Rendered
    Neural Rendering
    AI Denoising
    Super-Resolution
    

    View Slide

16. 😃 Animation
    Video-Driven Facial Animation
    Source: Daněček et al, “EMOCA : Emotion-Driven Monocular Face Capture and Animation”
    Input Video Reconstructed Facial Expressions
    

    View Slide

17. 😃 Animation
    Pose Estimation
    Source: NVIDIA Omniverse Machinima
    

    View Slide

18. 😃 Animation
    Audio-Driven Facial Animation
    FACS
    Coefficients
    Skin, Tongue, Eyes, Teeth
    Full-Face Template
    Final Character
    Composition
    Retargeting
    Post-
    Processing
    Neural Net
    Emotion / Style
    Final Character
    OR
    Source: NVIDIA Omniverse Audio2Face
    Speech Audio
    

    View Slide

19. 😃 Animation
    Audio-Driven Gesture and Body Animation
    Generator or
    Motion Matching
    Speech Audio
    Emotion / Style
    Retargeting
    Template Skeleton
    Animation
    Final Character
    Final Character
    Source: NVIDIA Omniverse Audio2Gesture
    

    View Slide

20. 😃 Animation
    Retargeting
    Source Motion Target Skeletons
    Source: Aberman et al, “Skeleton-Aware Networks for Deep Motion Retargeting”
    

    View Slide

21. 😃 Animation
    Animation Representation Spaces
    Source: Starke et al, “DeepPhase: Periodic Autoencoders for Learning Motion Phase Manifolds”
    

    View Slide

22. 😃 Animation / Physics
    Task-Driven Animation and Object Interactions
    “Swing the
    sword”
    Source: Juravsky et al, “PADL: Language-Directed Physics-Based Character Control”
    Skill
    Embedding
    Animation Clip
    “Strike the pink block”
    

    View Slide

23. Animation AI Conversational AI Vision AI Recommender AI
    Audio2Face
    Audio2Emotion
    NLP
    ASR, TTS
    Computer Vision
    Video Analytics
    Recommender Systems
    NVIDIA Omniverse Avatar Cloud Engine
    Cloud-Based AI Microservices
    

    View Slide

24. NVIDIA Omniverse Audio2Face App
    Audio-Driven Emotional Facial Animation
    

    View Slide

25. Summary
    ✏️ Create a Character
    🧠 Make it Intelligent
    😃 Make it Alive
    Using Reconstruction
    Using Generative AI
    Face / Body / Texture
    Conversation
    Speech
    Vision
    3D Scene Understanding
    Planning and Navigation
    Rendering
    Materials
    Physics
    Facial Animation
    Gestures
    Locomotion
    Audio / Video-based
    Interaction with Objects
    Task-Based Animation
    

    View Slide

26. View Slide