HTTP in a Hostile World (Droidcon Toronto 2019)

Video: https://youtu.be/tPA9n2mgClI

Networking on Android is difficult work. We need to display fresh data and rich media while limiting our network use. Unfortunately cookie-cutter solutions are not acceptable.

In this talk we’ll:

🥌 Verify that you’re using HTTPS securely
🥌 Acknowledge possible connectivity problems
🥌 Get performance tips
🥌 Code for maintainable APIs
🥌 Mock programmers that don’t test their networking

The body of this talk will be using HTTP effectively.

Transcript

1. @jessewilson https://square.github.io/okhttp/ HTTP in a Hostile World
2. HTTP? • Web Pages • Web Resources (images!) • Downloads • Web APIs • Mobile APIs • Server APIs • Core Networking APIs (DNS) • Pretty much all networking ever
3. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x
4. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Request
5. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x URL
6. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Headers
7. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Body
8. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x "time": “30m", "price": 15.00 }x Response
9. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Status
10. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Headers
11. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Body
12. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x
13. $ nc square.com 80 GET /robots.txt HTTP/1.1 host: square.com HTTP/1.1 301 Moved Permanently location: https://square.com/robots.txt date: Thu, 07 Nov 2019 04:05:21 GMT server: envoy content-length: 0
 $
14. HTTP/0.9 HTTP/1.0 HTTP/1.1 HTTP/2 HTTP/3 1991 1996 1997 2015 2019 HTML only headers and status codes persistent connections multiplexing and binary encoding out-of-order packets and IP address changes
15. HTTP in a Hostile World? • Correct • Ready • Secure • New stuff I’m excited about
16. Correct
17. Send Keefer $20 for
 pizza and beer
18. Send Keefer $20 for
 pizza and beer …
19. … SEND KEEFER $20 FOR
 PIZZA AND BEER!
20. SEND KEEFER $20 FOR
 PIZZA AND BEER! Successfully transferred $40 to Keefer
21. Idempotence • Repeating a request won’t repeat the action
22. What Triggers Repeats? • Users double-tapping the send button • Applications retrying • Network code retrying • Gateways retrying
23. Implementing Idempotence: Identify Intention • Generate a universally unique token in the UI:
 d6abe0fcd5d8c51e6c8a9159a873efcc • Write the outbound intention to a persistent queue, like your app’s SQLite database or your cloud queue service (SQS, etc.) • When processing an action, persist the identifier • The easy way: create a UNIQUE INDEX on the token
24. flickr.com/photos/fsse-info/
25. API Contract v. 1 • Client sends a pizza order in JSON • Server sends HTTP 200 and the delivery info in JSON
26. HTTP/1.1 200 OK { "time": "30m", "price": 15.00 } POST /pizza HTTP/1.1 { "size": "large", "toppings": ["pineapple"] }
27. API Contract v. 2 • Client sends a pizza order in JSON • Success: Server sends HTTP 200 and the delivery info in JSON • Failure: Server sends HTTP 500 and the failure info in JSON
28. POST /pizza HTTP/1.1 { "size": "large", "toppings": ["pineapple"] } HTTP/1.1 500 OK { "message": "too many carbs" }
29. Middleboxes!
30. Middleboxes!
31. Designing HTTP APIs • Conceptually, your client talks to your server • In practice, ‘invisible’ infrastructure exists • Sometimes it fails to be invisible
32. Status Codes just a sec thumbs up! response is elsewhere client problem server problem 1xx 2xx 3xx 4xx 5xx
33. Resilient to Middle Boxes • Send Content-Type on outbound requests • Expect Content-Type on inbound responses • HTTP 200? All Good • Other status codes? Be defensive!
34. REST: Retrofit interface PizzaShop { @GET("/{store}/{date}/deliveries") fun deliveries( @Path("store") store: String, @Path("date") date: String ): Call<List<Delivery>> } square.github.io/retrofit
35. service PizzaShop { rpc OrderFeed (stream Update) returns (stream Order) { } } gRPC: Wire 3 square.github.io/wire
36. interface PizzaShopClient : Service { @WireRpc( path = "/yum.pizza.PizzaShop/OrderFeed", requestAdapter = "yum.pizza.Update#ADAPTER”, responseAdapter = "yum.pizza.Order#ADAPTER" ) fun OrderFeed(): GrpcStreamingCall<Update, Order> } gRPC: Wire 3 square.github.io/wire
37. Images: imageView.load("https://pizza.yum/pineapple.png") { crossfade(true) placeholder(R.drawable.image) transformations(CircleCropTransformation()) } Coil coil-kt.github.io/coil
38. Testing: MockWebServer github.com/square/okhttp/mockwebserver val server = MockWebServer() server.start() server.enqueue(MockResponse() .setHeader("Cookie", "fave_topping=pineapple") .setBody("""{"time": "30m", "price": 19.99 }""")) val serverUrl = server.url("/")
39. Debugging: Charles Proxy charlesproxy.com
40. Debugging: Charles Proxy • Observe all your HTTP calls charlesproxy.com
41. Debugging: Charles Proxy • Observe all your HTTP calls • Intercepts HTTPS! charlesproxy.com
42. Ready
43. Redundancy! • Mechanisms to sync across multiple data centers • For availability and operations
44. Connectivity is a Lie • Bell and Rogers fail • Backends fail • Graceful fallbacks are possible
45. Request Avoidance • Push vs. poll • Local models in SQLite, etc.
46. Caching • URLs may represent resources, not calls • Resources can be cached!
47. Cache-Control: no-cache, no-store, max-age=1, s-maxage=2, private, must-revalidate, max-stale=3, min-fresh=4, only-if-cached, no-transform
48. Precise Caching • Just understand all of the above headers • Too difficult!
49. Good-Enough Caching • It may change
 Cache-Control: max-age=0 • It will never change
 Cache-Control: max-age=31536000
50. val cache = Cache( directory = File(context.cacheDir, "http"), maxSize = 100L * 1024 * 1024 ) val client = OkHttpClient.Builder() .cache(cache) .build()
51. Compression • Make sure gzip responses are enabled on client and server • Cuts size by 70% for JSON! • Consider gzip requests too • Use brotli for best compression
52. CDNs • Content Delivery Networks • Easiest way to improve upon speed of light • Great for availability • These can do image resizing!
53. Granularity • HTTP/1’s per-resource overhead is high • More speed?
 Fewer resources! • With HTTP/2 logical resources are best whatsapp.com
54. Secure
55. Data is Dangerous • You’re keeping other people’s secrets • You don’t really know what’s secret and what’s not • Attackers are persistent
56. Other People’s Secrets? • Passwords are obvious • Lots of bigger, less obvious failures
57. Limit Exposure to Data • Are we collecting this field because we need it? • When will we delete it? • GDPR in Europe, CCPA in California • Differential Privacy
59. TLS: Necessary • Transport Layer Security is the s in https:// • Confirms that you’re really talking to https://cash.app/ • Prevents eavesdropping • Prevents interference
60. TLS: Not Sufficient • Doesn’t impact how https://cash.app/ secures the data • Doesn’t prevent URLs like https://cash.app.attacker.com/ • Old implementations of TLS had severe bugs • Nation-state attackers do nasty things
62. Nation-State Attacks • A certificate authority is a company that confirms ownership of https://cash.app/ and signs our TLS certificate • By default browsers trust ~300 certificate authorities (CAs) • Nation states add themselves to the browser • Or compromise a vulnerable certificate authority
64. Defending against
 Nation-State Attacks • Certificate Pinning • Limit which of the 300 certificate authorities you trust
65. CertificatePinner certificatePinner = new CertificatePinner.Builder() .add("pizza.yum", "sha256/9TxoNDzGwoGUV8oDSZI8XpGpwyPz0GY/xAyWRLLjpPc=") .add("pizza.yum", "sha256/36Bv98D5O9bYGYRXnMTZJLQRW8i8fvOiWC/TJ4jOhLw=") .add("pizza.yum", "sha256/++iJU8rCHLXviXyKrJ2yqBWjs/5+LAqWuP04dHVHFb0=") .build(); OkHttpClient okHttpClient = new OkHttpClient.Builder() .certificatePinner(certificatePinner) .build();
66. TLS Evolution first widely-used release renamed from SSL to TLS bug fixes & new crypto bug fixes & new crypto major update with simplifications! 1996 1999 2006 2008 2018 SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3
67. TLS Evolution first widely-used release renamed from SSL to TLS bug fixes & new crypto bug fixes & new crypto major update with simplifications! JAN 2020 1996 1999 2006 2008 2018 SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3
68. TLS Evolution first widely-used release renamed from SSL to TLS bug fixes & new crypto bug fixes & new crypto major update with simplifications! JAN 2020 1996 1999 2006 2008 2018 Android 5+ Or Google Play Security Provider SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3
69. Whitebox Attackers • Our apps are instruction manuals for reverse engineers • There are amazing tools that make this point-and-click easy
70. github.com/sensepost/objection Objection
71. Defending Against
 Whitebox Attackers • A sufficiently-determined attacker will succeed • All we can do is make it really annoying • Perhaps a magic HTTP header?
72. New Stuff I’m Excited About
73. Envoy • Middleboxes are so 2018 • Let’s add side-boxes! envoyproxy.io
76. HTTP/3 • Better worst-case latency • Change your IP address without restarting downloads! quicwg.org
77. TCP streams Requests and Responses TCP packets HTTP/1.1
78. TCP streams Requests and Responses TCP packets HTTP/1.1 TCP streams TLS streams Requests and Responses TCP packets HTTP/1.1 with TLS
79. TCP streams Requests and Responses TCP packets HTTP/1.1 TCP streams TLS streams Requests and Responses TCP packets HTTP/1.1 with TLS TCP streams TLS streams HTTP/2 frames TCP packets HTTP/2 with TLS Requests and Responses
80. TCP streams Requests and Responses TCP packets HTTP/1.1 TCP streams TLS streams Requests and Responses TCP packets HTTP/1.1 with TLS TCP streams TLS streams HTTP/2 frames TCP packets HTTP/2 with TLS Requests and Responses UDP packets HTTP/3 with TLS Requests and Responses QUIC streams
81. • What if Retrofit inspired a server framework? • gRPC + REST in one framework • Early days! Watch this space cashapp.github.io/misk
82. OkHttp • New concurrency internals • Performance Features • Async I/O and coroutines square.github.io/okhttp
83. @jessewilson Thanks Toronto! https://square.github.io/okhttp/