$30 off During Our Annual Pro Sale. View Details »

OpenTalks.AI 2023 - Dmitry Korobchenko, NVIDIA, AI Technologies for Digital Characters and Avatars

OpenTalks.AI 2023 - Dmitry Korobchenko, NVIDIA, AI Technologies for Digital Characters and Avatars

opentalks3

March 18, 2023
Tweet

More Decks by opentalks3

Other Decks in Business

Transcript

  1. AI Technologies for Digital Characters and Avatars
    Dmitry Korobchenko, Director of AI | OpenTalks.AI | 06 March 2023

    View Slide

  2. • What are Digital Characters and Avatars?
    • Applications
    • Tasks :: Create a Character
    • Tasks :: Make it Intelligent
    • Tasks :: Make it Alive
    • Summary
    Agenda

    View Slide

  3. What are Digital Characters and Avatars?

    View Slide

  4. Applications
    Media & Entertainment Digital Twins Interactive Avatars Communication
    Games
    Movies
    VFX / CGI
    Digital Factory / City
    Drive Simulation
    Synthetic Data for AI
    Training
    Digital Assistants
    Education
    Healthcare
    Social Media Avatars
    Video Conferences
    Enterprise Telepresence

    View Slide

  5. Tasks in Digital Avatars Domain
    ✏️ Create a Character
    Using Reconstruction
    Using Generative AI

    View Slide

  6. Tasks in Digital Avatars Domain
    ✏️ Create a Character 🧠 Make it Intelligent
    Using Reconstruction
    Using Generative AI
    Conversational AI
    Environment Awareness

    View Slide

  7. Tasks in Digital Avatars Domain
    ✏️ Create a Character 🧠 Make it Intelligent 😃 Make it Alive
    Using Reconstruction
    Using Generative AI
    Conversational AI
    Environment Awareness
    Rendering / Materials
    Animation / Physics

    View Slide

  8. ✏️ Creating Digital Avatars
    Input Photo Reconstructed 3D Avatar
    Image to 3D Avatar
    Source: NVIDIA Image2Avatar Preview

    View Slide

  9. ✏️ Creating Digital Avatars
    Source: Chan et al, “Efficient Geometry-aware 3D Generative Adversarial Networks”
    Efficient Geometry-Aware 3D GAN

    View Slide

  10. ✏️ Creating Digital Avatars
    Source: Chan et al, “Efficient Geometry-aware 3D Generative Adversarial Networks”
    Input Photo Inverted GAN Output
    3D Reconstruction by GAN Inversion

    View Slide

  11. ✏️ Creating Digital Avatars
    Neural Radiance Fields (NeRF)
    Source: NVIDIA Instance NeRF

    View Slide

  12. 🧠 Conversational AI
    NLP / NLU / Language Models / Dialog Systems
    Automatic Speech Recognition
    Text-to-Speech
    Vision
    Identity Recognition
    Emotion / Gestures
    Eye Contact
    Riva SDK
    Riva SDK

    View Slide

  13. 🧠 Environment Awareness
    3D Scene Understanding
    3D Object Classification
    Scene Parsing / Segmentation
    Building a Scene Graph
    laptop
    bed
    table
    chest

    View Slide

  14. 🧠 Environment Awareness
    3D Scene Understanding Planning and Navigation
    3D Object Classification
    Scene Parsing / Segmentation
    Building a Scene Graph
    Affordance Detection
    Task-Based Semantic Planning
    Route Construction
    “Take the cup from the
    table and put in on the
    shelf next to the TV”
    Go To Table #2
    Take Cup #5
    Go To Shelf #3
    Put Shelf #3
    1
    2
    3
    4
    laptop
    bed
    table
    chest
    1
    3 4
    2

    View Slide

  15. 😃 Rendering / Materials
    Omniverse RTX
    Real Photo Rendered
    Rendered
    Neural Rendering
    AI Denoising
    Super-Resolution

    View Slide

  16. 😃 Animation
    Video-Driven Facial Animation
    Source: Daněček et al, “EMOCA : Emotion-Driven Monocular Face Capture and Animation”
    Input Video Reconstructed Facial Expressions

    View Slide

  17. 😃 Animation
    Pose Estimation
    Source: NVIDIA Omniverse Machinima

    View Slide

  18. 😃 Animation
    Audio-Driven Facial Animation
    FACS
    Coefficients
    Skin, Tongue, Eyes, Teeth
    Full-Face Template
    Final Character
    Composition
    Retargeting
    Post-
    Processing
    Neural Net
    Emotion / Style
    Final Character
    OR
    Source: NVIDIA Omniverse Audio2Face
    Speech Audio

    View Slide

  19. 😃 Animation
    Audio-Driven Gesture and Body Animation
    Generator or
    Motion Matching
    Speech Audio
    Emotion / Style
    Retargeting
    Template Skeleton
    Animation
    Final Character
    Final Character
    Source: NVIDIA Omniverse Audio2Gesture

    View Slide

  20. 😃 Animation
    Retargeting
    Source Motion Target Skeletons
    Source: Aberman et al, “Skeleton-Aware Networks for Deep Motion Retargeting”

    View Slide

  21. 😃 Animation
    Animation Representation Spaces
    Source: Starke et al, “DeepPhase: Periodic Autoencoders for Learning Motion Phase Manifolds”

    View Slide

  22. 😃 Animation / Physics
    Task-Driven Animation and Object Interactions
    “Swing the
    sword”
    Source: Juravsky et al, “PADL: Language-Directed Physics-Based Character Control”
    Skill
    Embedding
    Animation Clip
    “Strike the pink block”

    View Slide

  23. Animation AI Conversational AI Vision AI Recommender AI
    Audio2Face
    Audio2Emotion
    NLP
    ASR, TTS
    Computer Vision
    Video Analytics
    Recommender Systems
    NVIDIA Omniverse Avatar Cloud Engine
    Cloud-Based AI Microservices

    View Slide

  24. NVIDIA Omniverse Audio2Face App
    Audio-Driven Emotional Facial Animation

    View Slide

  25. Summary
    ✏️ Create a Character
    🧠 Make it Intelligent
    😃 Make it Alive
    Using Reconstruction
    Using Generative AI
    Face / Body / Texture
    Conversation
    Speech
    Vision
    3D Scene Understanding
    Planning and Navigation
    Rendering
    Materials
    Physics
    Facial Animation
    Gestures
    Locomotion
    Audio / Video-based
    Interaction with Objects
    Task-Based Animation

    View Slide

  26. View Slide