Upgrade to Pro — share decks privately, control downloads, hide ads and more …

HTTP in a Hostile World (Droidcon Toronto 2019)

HTTP in a Hostile World (Droidcon Toronto 2019)

Video: https://youtu.be/tPA9n2mgClI

Networking on Android is difficult work. We need to display fresh data and rich media while limiting our network use. Unfortunately cookie-cutter solutions are not acceptable.

In this talk we’ll:

🥌 Verify that you’re using HTTPS securely
🥌 Acknowledge possible connectivity problems
🥌 Get performance tips
🥌 Code for maintainable APIs
🥌 Mock programmers that don’t test their networking

The body of this talk will be using HTTP effectively.

Jesse Wilson

November 13, 2019
Tweet

More Decks by Jesse Wilson

Other Decks in Programming

Transcript

  1. @jessewilson https://square.github.io/okhttp/ HTTP in a Hostile World

  2. HTTP? • Web Pages • Web Resources (images!) • Downloads

    • Web APIs • Mobile APIs • Server APIs • Core Networking APIs (DNS) • Pretty much all networking ever
  3. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x
  4. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x Request
  5. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x URL
  6. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x Headers
  7. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x Body
  8. POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8

    {x "size": "large", "toppings": ["pineapple"] }x HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x "time": “30m", "price": 15.00 }x Response
  9. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x

    "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Status
  10. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x

    "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Headers
  11. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x

    "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x Body
  12. HTTP/1.1 200 OK content-length: 21 
 content-type: application/json; charset=UTF-8 {x

    "time": “30m", "price": 15.00 }x POST /pizza HTTP/1.1 host: pizza.app content-length: 50 content-type: application/json; charset=UTF-8 {x "size": "large", "toppings": ["pineapple"] }x
  13. $ nc square.com 80 GET /robots.txt HTTP/1.1 host: square.com HTTP/1.1

    301 Moved Permanently location: https://square.com/robots.txt date: Thu, 07 Nov 2019 04:05:21 GMT server: envoy content-length: 0
 $
  14. HTTP/0.9 HTTP/1.0 HTTP/1.1 HTTP/2 HTTP/3 1991 1996 1997 2015 2019

    HTML only headers and status codes persistent connections multiplexing and binary encoding out-of-order packets and IP address changes
  15. HTTP in a Hostile World? • Correct • Ready •

    Secure • New stuff I’m excited about
  16. Correct

  17. Send Keefer $20 for
 pizza and beer

  18. Send Keefer $20 for
 pizza and beer …

  19. … SEND KEEFER $20 FOR
 PIZZA AND BEER!

  20. SEND KEEFER $20 FOR
 PIZZA AND BEER! Successfully transferred $40

    to Keefer
  21. Idempotence • Repeating a request won’t repeat the action

  22. What Triggers Repeats? • Users double-tapping the send button •

    Applications retrying • Network code retrying • Gateways retrying
  23. Implementing Idempotence: Identify Intention • Generate a universally unique token

    in the UI:
 d6abe0fcd5d8c51e6c8a9159a873efcc • Write the outbound intention to a persistent queue, like your app’s SQLite database or your cloud queue service (SQS, etc.) • When processing an action, persist the identifier • The easy way: create a UNIQUE INDEX on the token
  24. flickr.com/photos/fsse-info/

  25. API Contract v. 1 • Client sends a pizza order

    in JSON • Server sends HTTP 200 and the delivery info in JSON
  26. HTTP/1.1 200 OK { "time": "30m", "price": 15.00 } POST

    /pizza HTTP/1.1 { "size": "large", "toppings": ["pineapple"] }
  27. API Contract v. 2 • Client sends a pizza order

    in JSON • Success: Server sends HTTP 200 and the delivery info in JSON • Failure: Server sends HTTP 500 and the failure info in JSON
  28. POST /pizza HTTP/1.1 { "size": "large", "toppings": ["pineapple"] } HTTP/1.1

    500 OK { "message": "too many carbs" }
  29. Middleboxes!

  30. Middleboxes!

  31. Designing HTTP APIs • Conceptually, your client talks to your

    server • In practice, ‘invisible’ infrastructure exists • Sometimes it fails to be invisible
  32. Status Codes just a sec thumbs up! response is elsewhere

    client problem server problem 1xx 2xx 3xx 4xx 5xx
  33. Resilient to Middle Boxes • Send Content-Type on outbound requests

    • Expect Content-Type on inbound responses • HTTP 200? All Good • Other status codes? Be defensive!
  34. REST: Retrofit interface PizzaShop { @GET("/{store}/{date}/deliveries") fun deliveries( @Path("store") store:

    String, @Path("date") date: String ): Call<List<Delivery>> } square.github.io/retrofit
  35. service PizzaShop { rpc OrderFeed (stream Update) returns (stream Order)

    { } } gRPC: Wire 3 square.github.io/wire
  36. interface PizzaShopClient : Service { @WireRpc( path = "/yum.pizza.PizzaShop/OrderFeed", requestAdapter

    = "yum.pizza.Update#ADAPTER”, responseAdapter = "yum.pizza.Order#ADAPTER" ) fun OrderFeed(): GrpcStreamingCall<Update, Order> } gRPC: Wire 3 square.github.io/wire
  37. Images: imageView.load("https://pizza.yum/pineapple.png") { crossfade(true) placeholder(R.drawable.image) transformations(CircleCropTransformation()) } Coil coil-kt.github.io/coil

  38. Testing: MockWebServer github.com/square/okhttp/mockwebserver val server = MockWebServer() server.start() server.enqueue(MockResponse() .setHeader("Cookie",

    "fave_topping=pineapple") .setBody("""{"time": "30m", "price": 19.99 }""")) val serverUrl = server.url("/")
  39. Debugging: Charles Proxy charlesproxy.com

  40. Debugging: Charles Proxy • Observe all your HTTP calls charlesproxy.com

  41. Debugging: Charles Proxy • Observe all your HTTP calls •

    Intercepts HTTPS! charlesproxy.com
  42. Ready

  43. Redundancy! • Mechanisms to sync across multiple data centers •

    For availability and operations
  44. Connectivity is a Lie • Bell and Rogers fail •

    Backends fail • Graceful fallbacks are possible
  45. Request Avoidance • Push vs. poll • Local models in

    SQLite, etc.
  46. Caching • URLs may represent resources, not calls • Resources

    can be cached!
  47. Cache-Control: no-cache, no-store, max-age=1, s-maxage=2, private, must-revalidate, max-stale=3, min-fresh=4, only-if-cached,

    no-transform
  48. Precise Caching • Just understand all of the above headers

    • Too difficult!
  49. Good-Enough Caching • It may change
 Cache-Control: max-age=0 • It

    will never change
 Cache-Control: max-age=31536000
  50. val cache = Cache( directory = File(context.cacheDir, "http"), maxSize =

    100L * 1024 * 1024 ) val client = OkHttpClient.Builder() .cache(cache) .build()
  51. Compression • Make sure gzip responses are enabled on client

    and server • Cuts size by 70% for JSON! • Consider gzip requests too • Use brotli for best compression
  52. CDNs • Content Delivery Networks • Easiest way to improve

    upon speed of light • Great for availability • These can do image resizing!
  53. Granularity • HTTP/1’s per-resource overhead is high • More speed?


    Fewer resources! • With HTTP/2 logical resources are best whatsapp.com
  54. Secure

  55. Data is Dangerous • You’re keeping other people’s secrets •

    You don’t really know what’s secret and what’s not • Attackers are persistent
  56. Other People’s Secrets? • Passwords are obvious • Lots of

    bigger, less obvious failures
  57. Limit Exposure to Data • Are we collecting this field

    because we need it? • When will we delete it? • GDPR in Europe, CCPA in California • Differential Privacy
  58. None
  59. TLS: Necessary • Transport Layer Security is the s in

    https:// • Confirms that you’re really talking to https://cash.app/ • Prevents eavesdropping • Prevents interference
  60. TLS: Not Sufficient • Doesn’t impact how https://cash.app/ secures the

    data • Doesn’t prevent URLs like https://cash.app.attacker.com/ • Old implementations of TLS had severe bugs • Nation-state attackers do nasty things
  61. None
  62. Nation-State Attacks • A certificate authority is a company that

    confirms ownership of https://cash.app/ and signs our TLS certificate • By default browsers trust ~300 certificate authorities (CAs) • Nation states add themselves to the browser • Or compromise a vulnerable certificate authority
  63. None
  64. Defending against
 Nation-State Attacks • Certificate Pinning • Limit which

    of the 300 certificate authorities you trust
  65. CertificatePinner certificatePinner = new CertificatePinner.Builder() .add("pizza.yum", "sha256/9TxoNDzGwoGUV8oDSZI8XpGpwyPz0GY/xAyWRLLjpPc=") .add("pizza.yum", "sha256/36Bv98D5O9bYGYRXnMTZJLQRW8i8fvOiWC/TJ4jOhLw=") .add("pizza.yum",

    "sha256/++iJU8rCHLXviXyKrJ2yqBWjs/5+LAqWuP04dHVHFb0=") .build(); OkHttpClient okHttpClient = new OkHttpClient.Builder() .certificatePinner(certificatePinner) .build();
  66. TLS Evolution first widely-used release renamed from SSL to TLS

    bug fixes & new crypto bug fixes & new crypto major update with simplifications! 1996 1999 2006 2008 2018 SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3
  67. TLS Evolution first widely-used release renamed from SSL to TLS

    bug fixes & new crypto bug fixes & new crypto major update with simplifications! JAN 2020 1996 1999 2006 2008 2018 SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3
  68. TLS Evolution first widely-used release renamed from SSL to TLS

    bug fixes & new crypto bug fixes & new crypto major update with simplifications! JAN 2020 1996 1999 2006 2008 2018 Android 5+ Or Google Play Security Provider SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3
  69. Whitebox Attackers • Our apps are instruction manuals for reverse

    engineers • There are amazing tools that make this point-and-click easy
  70. github.com/sensepost/objection Objection

  71. Defending Against
 Whitebox Attackers • A sufficiently-determined attacker will succeed

    • All we can do is make it really annoying • Perhaps a magic HTTP header?
  72. New Stuff I’m Excited About

  73. Envoy • Middleboxes are so 2018 • Let’s add side-boxes!

    envoyproxy.io
  74. None
  75. None
  76. HTTP/3 • Better worst-case latency • Change your IP address

    without restarting downloads! quicwg.org
  77. TCP streams Requests and Responses TCP packets HTTP/1.1

  78. TCP streams Requests and Responses TCP packets HTTP/1.1 TCP streams

    TLS streams Requests and Responses TCP packets HTTP/1.1 with TLS
  79. TCP streams Requests and Responses TCP packets HTTP/1.1 TCP streams

    TLS streams Requests and Responses TCP packets HTTP/1.1 with TLS TCP streams TLS streams HTTP/2 frames TCP packets HTTP/2 with TLS Requests and Responses
  80. TCP streams Requests and Responses TCP packets HTTP/1.1 TCP streams

    TLS streams Requests and Responses TCP packets HTTP/1.1 with TLS TCP streams TLS streams HTTP/2 frames TCP packets HTTP/2 with TLS Requests and Responses UDP packets HTTP/3 with TLS Requests and Responses QUIC streams
  81. • What if Retrofit inspired a server framework? • gRPC

    + REST in one framework • Early days! Watch this space cashapp.github.io/misk
  82. OkHttp • New concurrency internals • Performance Features • Async

    I/O and coroutines square.github.io/okhttp
  83. @jessewilson Thanks Toronto! https://square.github.io/okhttp/