HTTP in a Hostile World (Droidcon Toronto 2019)

69252b3de5cb7f464c09301d9a6b0401?s=47 Jesse Wilson
November 13, 2019

HTTP in a Hostile World (Droidcon Toronto 2019)

Video: https://youtu.be/tPA9n2mgClI

Networking on Android is difficult work. We need to display fresh data and rich media while limiting our network use. Unfortunately cookie-cutter solutions are not acceptable.

In this talk we’ll:

🥌 Verify that you’re using HTTPS securely
🥌 Acknowledge possible connectivity problems
🥌 Get performance tips
🥌 Code for maintainable APIs
🥌 Mock programmers that don’t test their networking

The body of this talk will be using HTTP effectively.

69252b3de5cb7f464c09301d9a6b0401?s=128

Jesse Wilson

November 13, 2019
Tweet

Transcript

  1. @jessewilson https://square.github.io/okhttp/ HTTP in a Hostile World

  2. HTTP? • Web Pages • Web Resources (images!) • Downloads

    • Web APIs • Mobile APIs • Server APIs • Core Networking APIs (DNS) • Pretty much all networking ever
  3. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x
  4. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x Request
  5. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x URL
  6. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x Headers
  7. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x Body
  8. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x "time": “30m", "price": 15.00 }x Response
  9. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x

    "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Status
  10. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x

    "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Headers
  11. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x

    "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Body
  12. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x

    "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x
  13. $ nc square.com 80 GET /robots.txt HTTP/1.1 host: square.com HTTP/1.1

    301 Moved Permanently location: https://square.com/robots.txt date: Thu, 07 Nov 2019 04:05:21 GMT server: envoy content-length: 0
 $
  14. HTTP/0.9 HTTP/1.0 HTTP/1.1 HTTP/2 HTTP/3 1991 1996 1997 2015 2019

    HTML only headers and status codes persistent connections multiplexing and binary encoding out-of-order packets and IP address changes
  15. HTTP in a Hostile World? • Correct • Ready •

    Secure • New stuff I’m excited about
  16. Correct

  17. Send Keefer $20 for
 pizza and beer

  18. Send Keefer $20 for
 pizza and beer …

  19. … SEND KEEFER $20 FOR
 PIZZA AND BEER!

  20. SEND KEEFER $20 FOR
 PIZZA AND BEER! Successfully transferred $40

    to Keefer
  21. Idempotence • Repeating a request won’t repeat the action

  22. What Triggers Repeats? • Users double-tapping the send button •

    Applications retrying • Network code retrying • Gateways retrying
  23. Implementing Idempotence: Identify Intention • Generate a universally unique token

    in the UI:
 d6abe0fcd5d8c51e6c8a9159a873efcc • Write the outbound intention to a persistent queue, like your app’s SQLite database or your cloud queue service (SQS, etc.) • When processing an action, persist the identifier • The easy way: create a UNIQUE INDEX on the token
  24. flickr.com/photos/fsse-info/

  25. API Contract v. 1 • Client sends a pizza order

    in JSON • Server sends HTTP 200 and the delivery info in JSON
  26. HTTP/1.1 200 OK { "time": "30m", "price": 15.00 } POST

    /pizza HTTP/1.1 { "size": "large", "toppings": ["pineapple"] }
  27. API Contract v. 2 • Client sends a pizza order

    in JSON • Success: Server sends HTTP 200 and the delivery info in JSON • Failure: Server sends HTTP 500 and the failure info in JSON
  28. POST /pizza HTTP/1.1 { "size": "large", "toppings": ["pineapple"] } HTTP/1.1

    500 OK { "message": "too many carbs" }
  29. Middleboxes!

  30. Middleboxes!

  31. Designing HTTP APIs • Conceptually, your client talks to your

    server • In practice, ‘invisible’ infrastructure exists • Sometimes it fails to be invisible
  32. Status Codes just a sec thumbs up! response is elsewhere

    client problem server problem 1xx 2xx 3xx 4xx 5xx
  33. Resilient to Middle Boxes • Send Content-Type on outbound requests

    • Expect Content-Type on inbound responses • HTTP 200? All Good • Other status codes? Be defensive!
  34. REST: Retrofit interface PizzaShop { @GET("/{store}/{date}/deliveries") fun deliveries( @Path("store") store:

    String, @Path("date") date: String ): Call<List<Delivery>> } square.github.io/retrofit
  35. service PizzaShop { rpc OrderFeed (stream Update) returns (stream Order)

    { } } gRPC: Wire 3 square.github.io/wire
  36. interface PizzaShopClient : Service { @WireRpc( path = "/yum.pizza.PizzaShop/OrderFeed", requestAdapter

    = "yum.pizza.Update#ADAPTER”, responseAdapter = "yum.pizza.Order#ADAPTER" ) fun OrderFeed(): GrpcStreamingCall<Update, Order> } gRPC: Wire 3 square.github.io/wire
  37. Images: imageView.load("https://pizza.yum/pineapple.png") { crossfade(true) placeholder(R.drawable.image) transformations(CircleCropTransformation()) } Coil coil-kt.github.io/coil

  38. Testing: MockWebServer github.com/square/okhttp/mockwebserver val server = MockWebServer() server.start() server.enqueue(MockResponse() .setHeader("Cookie",

    "fave_topping=pineapple") .setBody("""{"time": "30m", "price": 19.99 }""")) val serverUrl = server.url("/")
  39. Debugging: Charles Proxy charlesproxy.com

  40. Debugging: Charles Proxy • Observe all your HTTP calls charlesproxy.com

  41. Debugging: Charles Proxy • Observe all your HTTP calls •

    Intercepts HTTPS! charlesproxy.com
  42. Ready

  43. Redundancy! • Mechanisms to sync across multiple data centers •

    For availability and operations
  44. Connectivity is a Lie • Bell and Rogers fail •

    Backends fail • Graceful fallbacks are possible
  45. Request Avoidance • Push vs. poll • Local models in

    SQLite, etc.
  46. Caching • URLs may represent resources, not calls • Resources

    can be cached!
  47. Cache-Control: no-cache, no-store, max-age=1, s-maxage=2, private, must-revalidate, max-stale=3, min-fresh=4, only-if-cached,

    no-transform
  48. Precise Caching • Just understand all of the above headers

    • Too difficult!
  49. Good-Enough Caching • It may change
 Cache-Control: max-age=0 • It

    will never change
 Cache-Control: max-age=31536000
  50. val cache = Cache( directory = File(context.cacheDir, "http"), maxSize =

    100L * 1024 * 1024 ) val client = OkHttpClient.Builder() .cache(cache) .build()
  51. Compression • Make sure gzip responses are enabled on client

    and server • Cuts size by 70% for JSON! • Consider gzip requests too • Use brotli for best compression
  52. CDNs • Content Delivery Networks • Easiest way to improve

    upon speed of light • Great for availability • These can do image resizing!
  53. Granularity • HTTP/1’s per-resource overhead is high • More speed?


    Fewer resources! • With HTTP/2 logical resources are best whatsapp.com
  54. Secure

  55. Data is Dangerous • You’re keeping other people’s secrets •

    You don’t really know what’s secret and what’s not • Attackers are persistent
  56. Other People’s Secrets? • Passwords are obvious • Lots of

    bigger, less obvious failures
  57. Limit Exposure to Data • Are we collecting this field

    because we need it? • When will we delete it? • GDPR in Europe, CCPA in California • Differential Privacy
  58. None
  59. TLS: Necessary • Transport Layer Security is the s in

    https:// • Confirms that you’re really talking to https://cash.app/ • Prevents eavesdropping • Prevents interference
  60. TLS: Not Sufficient • Doesn’t impact how https://cash.app/ secures the

    data • Doesn’t prevent URLs like https://cash.app.attacker.com/ • Old implementations of TLS had severe bugs • Nation-state attackers do nasty things
  61. None
  62. Nation-State Attacks • A certificate authority is a company that

    confirms ownership of https://cash.app/ and signs our TLS certificate • By default browsers trust ~300 certificate authorities (CAs) • Nation states add themselves to the browser • Or compromise a vulnerable certificate authority
  63. None
  64. Defending against
 Nation-State Attacks • Certificate Pinning • Limit which

    of the 300 certificate authorities you trust
  65. CertificatePinner certificatePinner = new CertificatePinner.Builder() .add("pizza.yum", "sha256/9TxoNDzGwoGUV8oDSZI8XpGpwyPz0GY/xAyWRLLjpPc=") .add("pizza.yum", "sha256/36Bv98D5O9bYGYRXnMTZJLQRW8i8fvOiWC/TJ4jOhLw=") .add("pizza.yum",

    "sha256/++iJU8rCHLXviXyKrJ2yqBWjs/5+LAqWuP04dHVHFb0=") .build(); OkHttpClient okHttpClient = new OkHttpClient.Builder() .certificatePinner(certificatePinner) .build();
  66. TLS Evolution first widely-used release renamed from SSL to TLS

    bug fixes & new crypto bug fixes & new crypto major update with simplifications! 1996 1999 2006 2008 2018 SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3
  67. TLS Evolution first widely-used release renamed from SSL to TLS

    bug fixes & new crypto bug fixes & new crypto major update with simplifications! JAN 2020 1996 1999 2006 2008 2018 SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3
  68. TLS Evolution first widely-used release renamed from SSL to TLS

    bug fixes & new crypto bug fixes & new crypto major update with simplifications! JAN 2020 1996 1999 2006 2008 2018 Android 5+ Or Google Play Security Provider SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3
  69. Whitebox Attackers • Our apps are instruction manuals for reverse

    engineers • There are amazing tools that make this point-and-click easy
  70. github.com/sensepost/objection Objection

  71. Defending Against
 Whitebox Attackers • A sufficiently-determined attacker will succeed

    • All we can do is make it really annoying • Perhaps a magic HTTP header?
  72. New Stuff I’m Excited About

  73. Envoy • Middleboxes are so 2018 • Let’s add side-boxes!

    envoyproxy.io
  74. None
  75. None
  76. HTTP/3 • Better worst-case latency • Change your IP address

    without restarting downloads! quicwg.org
  77. TCP streams Requests and Responses TCP packets HTTP/1.1

  78. TCP streams Requests and Responses TCP packets HTTP/1.1 TCP streams

    TLS streams Requests and Responses TCP packets HTTP/1.1 with TLS
  79. TCP streams Requests and Responses TCP packets HTTP/1.1 TCP streams

    TLS streams Requests and Responses TCP packets HTTP/1.1 with TLS TCP streams TLS streams HTTP/2 frames TCP packets HTTP/2 with TLS Requests and Responses
  80. TCP streams Requests and Responses TCP packets HTTP/1.1 TCP streams

    TLS streams Requests and Responses TCP packets HTTP/1.1 with TLS TCP streams TLS streams HTTP/2 frames TCP packets HTTP/2 with TLS Requests and Responses UDP packets HTTP/3 with TLS Requests and Responses QUIC streams
  81. • What if Retrofit inspired a server framework? • gRPC

    + REST in one framework • Early days! Watch this space cashapp.github.io/misk
  82. OkHttp • New concurrency internals • Performance Features • Async

    I/O and coroutines square.github.io/okhttp
  83. @jessewilson Thanks Toronto! https://square.github.io/okhttp/