Designing AR Systems For Speed

Strangeloop 2018
https://www.thestrangeloop.com/2018/designing-augmented-reality-systems-for-speed.html

Building Augmented Reality (AR) systems is a software engineering art. It requires making choices among a set of trade-offs. On one hand, you have complex computer vision and machine learning algorithms that burn many CPU cycles; and on the other hand, you have the conflicting goal of getting the system to be as lean as possible since AR runs on resource-limited hardware such as wearables or mobile devices. This talk will explore how designing AR software is very much like the problem of designing distributed systems.

Like distributed systems, in AR: - We also need highly concurrent systems to take advantage of the many cores in cellphones. Concepts like message queues and "producers-consumers" become valuable tools to reason about computation abstractions on images and sensor data. - We also need highly available systems with an instant response due to the requirement of real-time interactions from the users with AR experiences. - We also need caching mechanisms to enable data locality for AR data retrieval.

Transcript

1. Reasoning About the Future Source: https://en.wikipedia.org/wiki/Koomey%27s_law Koomey’s Law
2. Reasoning About the Future Source: https://en.wikipedia.org/wiki/Koomey%27s_law Koomey’s Law In 1985, Feynman calculated an efficiency upper limit of Factor 100B since then we've only managed to achieve a Factor 40k
3. Reasoning About the Future Edholm’s Law Source: IEEE Spectrum: Edholm's Law of Bandwidth
4. Reasoning About the Future Edholm’s Law Source: IEEE Spectrum: Edholm's Law of Bandwidth “we may someday see the end of wireline”
5. How should we design AR Systems? Better Abstractions
6. System Design Principles Distributed Systems • Decrease processing time ◦ Co-location ◦ Caching ◦ Remoting • Sharding • Scalability via concurrency ◦ Pooling ◦ Parallelization
7. System Design Principles Distributed Systems • Decrease processing time ◦ Co-location ◦ Caching ◦ Remoting • Sharding • Scalability via concurrency ◦ Pooling ◦ Parallelization AR Systems • Real time by default ◦ Efficient networking ◦ Response time • Many devices • Concurrency to take advantage of Many cores ◦ Lock free programming ◦ Actor models
8. Speeding up Networking Life is in real-time
10. Solution Space
11. Solution Space
12. Solution Space
13. Our Ideal Solution
14. Real-time P2P AR Technology 100’s ms
15. Real-time P2P AR Technology 100’s ms 10’s ms
16. P2P Networking
17. Looks Familiar?
18. Speeding up Computation CV is hard
19. Traditional AR SLAM Pipeline (10,000ft view)
20. It’ll be Hard to Run in Real-Time Source: https://doi.org/10.1016/j.robot.2017.10.019
21. Can We do Better?
22. Actor Models to the Rescue This will probably be its own talk in the future
23. Actor Models to the Rescue This will probably be its own talk in the future