Designing AR Systems For Speed

Strangeloop 2018
https://www.thestrangeloop.com/2018/designing-augmented-reality-systems-for-speed.html

Building Augmented Reality (AR) systems is a software engineering art. It requires making choices among a set of trade-offs. On one hand, you have complex computer vision and machine learning algorithms that burn many CPU cycles; and on the other hand, you have the conflicting goal of getting the system to be as lean as possible since AR runs on resource-limited hardware such as wearables or mobile devices. This talk will explore how designing AR software is very much like the problem of designing distributed systems.

Like distributed systems, in AR: - We also need highly concurrent systems to take advantage of the many cores in cellphones. Concepts like message queues and "producers-consumers" become valuable tools to reason about computation abstractions on images and sensor data. - We also need highly available systems with an instant response due to the requirement of real-time interactions from the users with AR experiences. - We also need caching mechanisms to enable data locality for AR data retrieval.

Transcript

1. Reasoning About the Future
   Source: https://en.wikipedia.org/wiki/Koomey%27s_law
   Koomey’s Law
   

   View Slide

2. Reasoning About the Future
   Source: https://en.wikipedia.org/wiki/Koomey%27s_law
   Koomey’s Law
   In 1985, Feynman calculated an efficiency upper limit of Factor
   100B since then we've only managed to achieve a Factor 40k
   

   View Slide

3. Reasoning About the Future
   Edholm’s Law
   Source: IEEE Spectrum: Edholm's Law of Bandwidth
   

   View Slide

4. Reasoning About the Future
   Edholm’s Law
   Source: IEEE Spectrum: Edholm's Law of Bandwidth
   “we may someday see the end of wireline”
   

   View Slide

5. How should we design
   AR Systems?
   Better Abstractions
   

   View Slide

6. System Design Principles
   Distributed Systems
   ● Decrease processing time
   ○ Co-location
   ○ Caching
   ○ Remoting
   ● Sharding
   ● Scalability via concurrency
   ○ Pooling
   ○ Parallelization
   

   View Slide

7. System Design Principles
   Distributed Systems
   ● Decrease processing time
   ○ Co-location
   ○ Caching
   ○ Remoting
   ● Sharding
   ● Scalability via concurrency
   ○ Pooling
   ○ Parallelization
   AR Systems
   ● Real time by default
   ○ Efficient networking
   ○ Response time
   ● Many devices
   ● Concurrency to take advantage of
   Many cores
   ○ Lock free programming
   ○ Actor models
   

   View Slide

8. Speeding up Networking
   Life is in real-time
   

   View Slide

9. View Slide

10. Solution Space
    

    View Slide

11. Solution Space
    

    View Slide

12. Solution Space
    

    View Slide

13. Our Ideal Solution
    

    View Slide

14. Real-time P2P AR Technology
    100’s ms
    

    View Slide

15. Real-time P2P AR Technology
    100’s ms 10’s ms
    

    View Slide

16. P2P Networking
    

    View Slide

17. Looks Familiar?
    

    View Slide

18. Speeding up Computation
    CV is hard
    

    View Slide

19. Traditional AR SLAM Pipeline (10,000ft view)
    

    View Slide

20. It’ll be Hard to Run in Real-Time
    Source: https://doi.org/10.1016/j.robot.2017.10.019
    

    View Slide

21. Can We do Better?
    

    View Slide

22. Actor Models to the Rescue
    This will probably be its own talk in the future
    

    View Slide

23. Actor Models to the Rescue
    This will probably be its own talk in the future
    

    View Slide