Evoking Magic Realism with Augmented Reality Technology

QCon.ai 2019 talk https://www.infoq.com/presentations/realism-augmented-reality

Exploring how building a real-world system is more of a software engineering art. It requires making choices among a set of tradeoffs.

Transcript

1. EVOKING MAGIC REALISM
   WITH AUGMENTED REALITY
   Diana Hu
   QCon.ai 2019
   

   View Slide

2. Hello!
   ● Director of Engineering & Head of AR Platform at Niantic
   ● Co-Founder & CTO @Escher Reality, a YC ‘17 company acquired by Niantic
   ● Building large scale ML & CV production systems since 2012
   ● Worked as data scientist @Intel Labs & OnCue TV
   @sdianahu
   

   View Slide

3. View Slide

4. https://mastersreview.com/literary-terms-magical-realism-science-fiction-and-fantasy/
   

   View Slide

5. A Magic Insight into Reality
   Matter-of-fact inclusion of fantastical elements into seemingly realistic fiction
   Magic Reality
   

   View Slide

6. Augmenting Reality
   Matter-of-fact inclusion of fantastical elements into the physical world
   seemingly realistic fiction
   Digital
   Magic
   Physical
   Reality
   

   View Slide

7. Principles For Magical Realism in AR
   Exploration Exercise Social
   Discover the world
   around us
   Most of the time, a simple
   walk will do
   Real-world, meaningful
   Interactions
   

   View Slide

8. Exploration
   

   View Slide

9. Exercise
   

   View Slide

10. Shared & Social
    

    View Slide

11. View Slide

12. MAKING THE DIGITAL BELIEVABLE
    (AKA How do we build this?)
    

    View Slide

13. Niantic Real World Platform
    

    View Slide

14. A Brief on AR Technology
    NEED TO UNDERSTAND
    THE WORLD
    NEED FOR VISUALS NEED TO CREATE
    

    View Slide

15. Understanding The World
    

    View Slide

16. AR Systems for Human Perception
    Rayner K, Smith TJ, Malcolm GL, Henderson JM. Eye Movements
    and Visual Encoding During Scene Perception. Psychological
    science. 2009;20(1):6-10. doi:10.1111/j.1467-
    9280.2008.02243.x.
    The Miller Response-
    Time Test
    ~100ms => instantaneous.
    <~1 second => fast enough
    >10 seconds => lose user.
    

    View Slide

17. NEED FOR SPEED AS A “MATTER-
    OF-FACT”
    Building Believable AR
    

    View Slide

18. AR System Design Principles
    EFFICIENT
    NETWORKING
    CONCURRENT
    PROGRAMMING
    

    View Slide

19. Speeding up Networking
    Life is in real-time
    

    View Slide

20. View Slide

21. Solution Space
    

    View Slide

22. Solution Space
    

    View Slide

23. Solution Space
    

    View Slide

24. Our Ideal Solution
    

    View Slide

25. Real-time P2P AR Technology
    100’s ms
    

    View Slide

26. Real-time P2P AR Technology
    100’s ms 10’s ms
    

    View Slide

27. Speeding up Computation
    Computer Vision is hard
    

    View Slide

28. Traditional AR SLAM Pipeline
    (10,000ft view)
    

    View Slide

29. Inputs
    Images 1080x720 @ 30-60 Hz IMU 3x3 100s-1000s Hz
    

    View Slide

30. Feature Extraction
    

    View Slide

31. Building AR Maps
    

    View Slide

32. Source:https://spectrum.ieee.org/automaton/robotics/home-robots/piaggio-cargo-robot
    

    View Slide

33. It’ll be Hard to Run in Real-Time
    Source: https://doi.org/10.1016/j.robot.2017.10.019
    

    View Slide

34. Can We do Better?
    

    View Slide

35. Actor Models to the Rescue
    This will probably be its own talk in the future
    

    View Slide

36. Actor Models to the Rescue
    This will probably be its own talk in the future
    

    View Slide

37. Summary
    ● Magical realism in AR
    ● Making the digital believable
    ● Building AR systems for Human perception – Need for Speed
    

    View Slide

38. Credits
    Thanks @Pete Turner, and the rest of the AR Eng team
    

    View Slide

39. THANKS!
    Any Questions?
    Find me @sdianahu
    We are hiring!
    https://www.nianticlabs.com/jobs/
    

    View Slide