Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OpenTalks.AI 2023 - Dmitry Korobchenko, NVIDIA, AI Technologies for Digital Characters and Avatars

OpenTalks.AI 2023 - Dmitry Korobchenko, NVIDIA, AI Technologies for Digital Characters and Avatars

opentalks3

March 18, 2023
Tweet

More Decks by opentalks3

Other Decks in Business

Transcript

  1. • What are Digital Characters and Avatars? • Applications •

    Tasks :: Create a Character • Tasks :: Make it Intelligent • Tasks :: Make it Alive • Summary Agenda
  2. Applications Media & Entertainment Digital Twins Interactive Avatars Communication Games

    Movies VFX / CGI Digital Factory / City Drive Simulation Synthetic Data for AI Training Digital Assistants Education Healthcare Social Media Avatars Video Conferences Enterprise Telepresence
  3. Tasks in Digital Avatars Domain ✏️ Create a Character 🧠

    Make it Intelligent Using Reconstruction Using Generative AI Conversational AI Environment Awareness
  4. Tasks in Digital Avatars Domain ✏️ Create a Character 🧠

    Make it Intelligent 😃 Make it Alive Using Reconstruction Using Generative AI Conversational AI Environment Awareness Rendering / Materials Animation / Physics
  5. ✏️ Creating Digital Avatars Input Photo Reconstructed 3D Avatar Image

    to 3D Avatar Source: NVIDIA Image2Avatar Preview
  6. ✏️ Creating Digital Avatars Source: Chan et al, “Efficient Geometry-aware

    3D Generative Adversarial Networks” Efficient Geometry-Aware 3D GAN
  7. ✏️ Creating Digital Avatars Source: Chan et al, “Efficient Geometry-aware

    3D Generative Adversarial Networks” Input Photo Inverted GAN Output 3D Reconstruction by GAN Inversion
  8. 🧠 Conversational AI NLP / NLU / Language Models /

    Dialog Systems Automatic Speech Recognition Text-to-Speech Vision Identity Recognition Emotion / Gestures Eye Contact Riva SDK Riva SDK
  9. 🧠 Environment Awareness 3D Scene Understanding 3D Object Classification Scene

    Parsing / Segmentation Building a Scene Graph laptop bed table chest
  10. 🧠 Environment Awareness 3D Scene Understanding Planning and Navigation 3D

    Object Classification Scene Parsing / Segmentation Building a Scene Graph Affordance Detection Task-Based Semantic Planning Route Construction “Take the cup from the table and put in on the shelf next to the TV” Go To Table #2 Take Cup #5 Go To Shelf #3 Put Shelf #3 1 2 3 4 laptop bed table chest 1 3 4 2
  11. 😃 Rendering / Materials Omniverse RTX Real Photo Rendered Rendered

    Neural Rendering AI Denoising Super-Resolution
  12. 😃 Animation Video-Driven Facial Animation Source: Daněček et al, “EMOCA

    : Emotion-Driven Monocular Face Capture and Animation” Input Video Reconstructed Facial Expressions
  13. 😃 Animation Audio-Driven Facial Animation FACS Coefficients Skin, Tongue, Eyes,

    Teeth Full-Face Template Final Character Composition Retargeting Post- Processing Neural Net Emotion / Style Final Character OR Source: NVIDIA Omniverse Audio2Face Speech Audio
  14. 😃 Animation Audio-Driven Gesture and Body Animation Generator or Motion

    Matching Speech Audio Emotion / Style Retargeting Template Skeleton Animation Final Character Final Character Source: NVIDIA Omniverse Audio2Gesture
  15. 😃 Animation Retargeting Source Motion Target Skeletons Source: Aberman et

    al, “Skeleton-Aware Networks for Deep Motion Retargeting”
  16. 😃 Animation Animation Representation Spaces Source: Starke et al, “DeepPhase:

    Periodic Autoencoders for Learning Motion Phase Manifolds”
  17. 😃 Animation / Physics Task-Driven Animation and Object Interactions “Swing

    the sword” Source: Juravsky et al, “PADL: Language-Directed Physics-Based Character Control” Skill Embedding Animation Clip “Strike the pink block”
  18. Animation AI Conversational AI Vision AI Recommender AI Audio2Face Audio2Emotion

    NLP ASR, TTS Computer Vision Video Analytics Recommender Systems NVIDIA Omniverse Avatar Cloud Engine Cloud-Based AI Microservices
  19. Summary ✏️ Create a Character 🧠 Make it Intelligent 😃

    Make it Alive Using Reconstruction Using Generative AI Face / Body / Texture Conversation Speech Vision 3D Scene Understanding Planning and Navigation Rendering Materials Physics Facial Animation Gestures Locomotion Audio / Video-based Interaction with Objects Task-Based Animation