Lessons from Mobile Networking at Scale

In the last few years, Lyft has invested heavily into the observability of our apps as we've scaled. At first we mostly treated the networking stack as a black box, hoping that the system would do a best effort to execute network requests and that influencing that was mostly out of our hands.

However, over time we discovered a lot more about how and when our network requests were failing, we developed ways to instrument the health of our apps in real time over billions of API requests a day, even triggering alarms when performance regressed. We also discovered how to tune the networking layer to perform best with various usage patterns for our different apps.

Ultimately, this resulted in us creating our own open source cross-platform mobile networking library that has given us unprecedented visibility and control into the performance of our apps at scale. You can find out more about it here:
https://envoymobile.io

Transcript

1. Mobile Networking At Scale LESSONS FROM JP Simard • jp@lyft.com • @simjp
2. About Me ‣ Family: Wife & two kids, living in Montréal, Canada ‣ OSS: SwiftLint, Jazzy, Yams, 
 SourceKitten, Realm ‣ Podcast: Swift Unwrapped ‣ Lyft: Transit, Bikes & Scooters JP Simard • jp@lyft.com • @simjp
4. 2012 2022
5. Billions of requests a day 100+ mobile engineers Millions of daily active users
6. Our Platform Networking connects our riders and drivers to
8. Server Mobile Stable Fast Single Provider Control Unreliable Inconsistent Many Carriers User Con fi gured
9. App Feature API Client Networking Engine Code you write Black box
10. What have we learned?
11. LESSON 1 Server-side observability only shows you half the picture ▸ We found that our server-side stats weren’t accurately re fl ecting our mobile failure rates ▸ Server-side stats don’t capture if the clients were able to successfully receive all the data that was sent ▸ Large payloads in particular have higher failure rates on the receipt end
13. LESSON 2 Measuring performance a ff ects performance ▸ If you need to make networking requests to send telemetry about your networking, that can compete with bandwidth and other system resources if you’re not careful ▸ This can not only a ff ect the user experience, but the measurements you were trying to take in the fi rst place ▸ Similarly, measuring reliability in fl uences reliability ▸ It’s critical to optimize your measurement pipeline
14. LESSON 3 Small shifts to networking performance matter at scale ▸ What you think is a micro-optimization may end up making a tangible impact to the long tail ▸ Conversely, if you think you’re adding a totally innocuous amount of overhead, you may be surprised to see it measurably negatively impacting users at scale ▸ At Lyft, we caught a case of what was at fi rst glance some lightweight runtime re fl ection to get the name of a type actually be responsible for 4-8% of our total app hangs!
15. LESSON 4 Not all cell carriers are equal ▸ The success rate can vary by up to 10% across carriers with the same OS/library con fi guration ▸ The success rate can vary by up to 10% across libraries using the same carrier ▸ The best performing networking library varies across carriers ▸ This is even more true with a global user base
16. Success Rate by Carrier & Networking Library
17. LESSON 5 The networking engine can in fl uence overall app performance ▸ We were somewhat surprised to see that switching from URLSession to an in- process networking engine actually decreased app hangs and OOM crashes ▸ On Android, we saw similar shifts to ANRs
18. LESSON 6 Caching is the solution to, and cause of, many problems ▸ Planning to experiment with DNS caching soon to see if it improves the time to fi rst connect on the long tail
19. LESSON 7 Debugging this stu ff is hard ▸ This led us and others at Lyft to build some truly advanced remote debugging tools, but more on that some other time 😉🤫
20. We wanted to solve this for Lyft* *and for the open 
 source community
22. envoymobile.io
23. eng.lyft.com
24. Rich Observability ▸ Dozens of metrics data and types out of the box ▸ Point to a statsd or gRPC endpoint to ingest with little con fi guration
25. Rich Observability
26. Full Control ▸ Open source ▸ Modern codebase ▸ C++ / Swift / Kotlin ▸ Debuggable
29. Modern Technologies ▸ QUIC/HTTP3 ▸ TLS 1.3 ▸ gRPC streaming ▸ brotli compression ▸ Happy Eyeballs
30. Cross-Platform ▸ Our iOS, Android and server codebases can all use the same networking engine ▸ Deploys back to iOS 13 􀤆
31. Reasons not to use Envoy Mobile ▸ Most apps should use URLSession ▸ For now we don’t recommend Envoy Mobile for general use ▸ Support for proxies on iOS & Android in progress ▸ iOS background requests are enforced by the OS to only work with URLSession ▸ Binary size increase (~5MB)
32. Contributors Mike José Michael Keith Rafał Alyssa and dozens more on Envoy Mobile directly, not to mention over a hundred contributors to the main Envoy project
33. Lessons Learned 1. Mobile observability is necessary to understand the true health of your overall system 2. Measuring performance a ff ects performance 3. Small shifts to networking performance matter at scale 4. Not all cell carriers are equal 5. The networking engine can in fl uence overall app performance 6. Caching is a double-edged sword, use it carefully 7. Debugging this stu ff is hard
34. Thanks for the ride JP Simard • jp@lyft.com • @simjp
35. Thanks for the ride Thanks for the ride JP Simard • jp@lyft.com • @simjp