Your App is a Giant DDOS Botnet and Other Reasons to Manage and Monitor Your Network Traffic

Presented at Droidcon SF 2018

Transcript

1. Sanae Rosen sanae@yelp.com Your App is a Giant DDOS Botnet and Other Reasons to Manage and Monitor Your Network Traffic
2. Sanae Rosen Since early 2017: Software Engineer, Core Android team PhD related to mobile networking About me
3. Yelp’s Mission Connecting people with great local businesses.
6. What if we log all 5xx server errors? What if we retry all failed analytics?
7. /analytics 500
8. /analytics 500
9. /analytics 500
11. There are many more problems with mobile traffic...
13. https://www.theverge.com/2018/10/22/18011028/google-news-app- bug-background-data-overage-charge
14. Android device fragmentation (2014) (https://opensignal.com/reports/2014/android-fragmentation/)
16. I’m totally a VPN and you should definitely install me!
20. What about experiments? Feature 1 Experiment 1 Experiment server: Experiment 1: on Experiment 2: on Feature 2 Experiment 2 The app Main server
21. What about experiments? Feature 1 Experiment 1 Experiment server: Experiment 1: on Experiment 2: off Feature 2 Experiment 2 The app Main server
22. What about experiments? • What if experiments break? • What if you forget to use an experiment? • What about infrastructure problems?
23. Do we even need to fix this problem?
24. Our solution Shut off bad traffic Monitor network traffic
25. Part 1: shut off bad network traffic
26. Feature 1 Experiment 1 Experiment server Feature 2 The app Main server Network monitor
27. OkHttp Interceptors
28. class TrafficMonitorInterceptor( val monitor: NetworkMonitor ) : Interceptor { override fun intercept(chain: Chain): Response { val request = chain.request() monitor.record(request) if (monitor.shouldDropRequest(request)) { return monitor.generateErrorResponse() } val response = chain.proceed(request) monitor.record(response) return response } }
29. class TrafficMonitorInterceptor( val monitor: NetworkMonitor ) : Interceptor { override fun intercept(chain: Chain): Response { val request = chain.request() monitor.record(request) if (monitor.shouldDropRequest(request)) { return monitor.generateErrorResponse() } val response = chain.proceed(request) monitor.record(response) return response } }
30. class TrafficMonitorInterceptor( val monitor: NetworkMonitor ) : Interceptor { override fun intercept(chain: Chain): Response { val request = chain.request() monitor.record(request) if (monitor.shouldDropRequest(request)) { return monitor.generateErrorResponse() } val response = chain.proceed(request) monitor.record(response) return response } }
31. class TrafficMonitorInterceptor( val monitor: NetworkMonitor ) : Interceptor { override fun intercept(chain: Chain): Response { val request = chain.request() monitor.record(request) if (monitor.shouldDropRequest(request)) { return monitor.generateErrorResponse() } val response = chain.proceed(request) monitor.record(response) return response } }
32. class TrafficMonitorInterceptor( val monitor: NetworkMonitor ) : Interceptor { override fun intercept(chain: Chain): Response { val request = chain.request() monitor.record(request) if (monitor.shouldDropRequest(request)) { return monitor.generateErrorResponse() } val response = chain.proceed(request) monitor.record(response) return response } } Look! Kotlin!
33. Idea 1: Do it like experiments How to turn endpoints off? Network monitor server Endpoint 1: on Endpoint 2: off The app Main server Network monitor Network monitor server Endpoint 1: on Endpoint 2: off
34. Idea 1: Do it like experiments • What if the network monitor server goes down? • High complexity How to turn endpoints off? Network monitor server Endpoint 1: on Endpoint 2: off The app Main server Network monitor Network monitor server Endpoint 1: on Endpoint 2: off
35. Our actual solution Feature The app CDN Main server Network monitor
36. Our actual solution Feature The app 555 error code Network monitor CDN Main server
37. Feature The app Network monitor Our actual solution CDN Main server
38. Main server Can be endpoint-specific too Endpoint 1 The app CDN Network monitor 556 error code Network monitor Endpoint 2
39. Network monitor Can be endpoint-specific too Endpoint 1 The app Main server Endpoint 2 CDN
40. Retries Bad timers: 10 min Good timers: ~10 min
41. Persistence 555
42. Persistence 555 No persistence: Persistence: 555
43. Stage 2: the network monitor https://www.networkworld.com/article/3237944/lan-wan/network-verification-predicting-the-future-of-complex-networks.html https://dumielauxepices.net/monitor-lizard-clipart/monitor-lizard-clipart-transparent-background
44. • What about the subtler problems? ◦ Only slightly bad data usage ◦ Problems with only a few devices Stage 2: the network monitor
45. Stage 2: the network monitor Feature 1 Experiment server Feature 2 The app Main server Network monitor
46. Stage 2: the network monitor Network logs Feature 1 Experiment server Feature 2 The app Main server Network monitor
47. Stage 2: the network monitor Network controller and monitor Data: 0 kb Network requests: 0 Time: 0 min
48. Stage 2: the network monitor 2 kb Network controller and monitor Data: 2 kb Network requests: 1 Time: 4 min
49. Stage 2: the network monitor 4 kb Network controller and monitor Data: 6 kb Network requests: 2 Time: 7 min
50. Stage 2: the network monitor Network controller and monitor Data: 6 kb Network requests: 2 Time: 10 min
51. Stage 2: the network monitor Data: 6 kB Network requests: 2 Time: 10 Network controller and monitor Data: 0 kb Network requests: 0 Time: 0 min
52. • Can also flag high traffic in the outgoing logs ◦ Based on a per-endpoint config • “Endpoint” is a fairly flexible concept - can track related endpoints together Other details
53. Problem 2: How cellular networks work Network traffic energy Tail timer #1 Tail timer #2 Time Energy Time Energy Periodic traffic Batched traffic
54. Piggyback logs on other requests 10 minutes Logs not quite accurate, but no problem to detect Network traffic Logs sent 10 minutes Logs accurate, and something’s gone wrong Network traffic Logs sent
55. Use case: too much data (for one user)
56. Release monitoring
57. Alerting
58. • Coming before droidcon soon! • Set up so you can extend it to be compatible with things other than okhttp Next steps: Open source
60. www.yelp.com/careers/ We're Hiring!
61. @YelpEngineering fb.com/YelpEngineers engineeringblog.yelp.com github.com/yelp
62. @YelpEngineering fb.com/YelpEngineers engineeringblog.yelp.com github.com/yelp https://github.com/Yelp/ kotlin-android-workshop