Evoking Magic Realism with Augmented Reality Technology

QCon.ai 2019 talk https://www.infoq.com/presentations/realism-augmented-reality

Exploring how building a real-world system is more of a software engineering art. It requires making choices among a set of tradeoffs.

Transcript

1. EVOKING MAGIC REALISM WITH AUGMENTED REALITY Diana Hu QCon.ai 2019
2. Hello! • Director of Engineering & Head of AR Platform at Niantic • Co-Founder & CTO @Escher Reality, a YC ‘17 company acquired by Niantic • Building large scale ML & CV production systems since 2012 • Worked as data scientist @Intel Labs & OnCue TV @sdianahu
4. https://mastersreview.com/literary-terms-magical-realism-science-fiction-and-fantasy/
5. A Magic Insight into Reality Matter-of-fact inclusion of fantastical elements into seemingly realistic fiction Magic Reality
6. Augmenting Reality Matter-of-fact inclusion of fantastical elements into the physical world seemingly realistic fiction Digital Magic Physical Reality
7. Principles For Magical Realism in AR Exploration Exercise Social Discover the world around us Most of the time, a simple walk will do Real-world, meaningful Interactions
8. Exploration
9. Exercise
10. Shared & Social
12. MAKING THE DIGITAL BELIEVABLE (AKA How do we build this?)
13. Niantic Real World Platform
14. A Brief on AR Technology NEED TO UNDERSTAND THE WORLD NEED FOR VISUALS NEED TO CREATE
15. Understanding The World
16. AR Systems for Human Perception Rayner K, Smith TJ, Malcolm GL, Henderson JM. Eye Movements and Visual Encoding During Scene Perception. Psychological science. 2009;20(1):6-10. doi:10.1111/j.1467- 9280.2008.02243.x. The Miller Response- Time Test ~100ms => instantaneous. <~1 second => fast enough >10 seconds => lose user.
17. NEED FOR SPEED AS A “MATTER- OF-FACT” Building Believable AR
18. AR System Design Principles EFFICIENT NETWORKING CONCURRENT PROGRAMMING
19. Speeding up Networking Life is in real-time
21. Solution Space
22. Solution Space
23. Solution Space
24. Our Ideal Solution
25. Real-time P2P AR Technology 100’s ms
26. Real-time P2P AR Technology 100’s ms 10’s ms
27. Speeding up Computation Computer Vision is hard
28. Traditional AR SLAM Pipeline (10,000ft view)
29. Inputs Images 1080x720 @ 30-60 Hz IMU 3x3 100s-1000s Hz
30. Feature Extraction
31. Building AR Maps
32. Source:https://spectrum.ieee.org/automaton/robotics/home-robots/piaggio-cargo-robot
33. It’ll be Hard to Run in Real-Time Source: https://doi.org/10.1016/j.robot.2017.10.019
34. Can We do Better?
35. Actor Models to the Rescue This will probably be its own talk in the future
36. Actor Models to the Rescue This will probably be its own talk in the future
37. Summary • Magical realism in AR • Making the digital believable • Building AR systems for Human perception – Need for Speed
38. Credits Thanks @Pete Turner, and the rest of the AR Eng team
39. THANKS! Any Questions? Find me @sdianahu We are hiring! https://www.nianticlabs.com/jobs/