HTTP in a Hostile World (Droidcon Toronto 2019)

Video: https://youtu.be/tPA9n2mgClI

Networking on Android is difficult work. We need to display fresh data and rich media while limiting our network use. Unfortunately cookie-cutter solutions are not acceptable.

In this talk we’ll:

🥌 Verify that you’re using HTTPS securely
🥌 Acknowledge possible connectivity problems
🥌 Get performance tips
🥌 Code for maintainable APIs
🥌 Mock programmers that don’t test their networking

The body of this talk will be using HTTP effectively.

Transcript

1. @jessewilson
   https://square.github.io/okhttp/
   HTTP in a Hostile World
   

   View Slide

2. HTTP?
   • Web Pages
   • Web Resources (images!)
   • Downloads
   • Web APIs
   • Mobile APIs
   • Server APIs
   • Core Networking APIs (DNS)
   • Pretty much all networking ever
   

   View Slide

3. POST /pizza HTTP/1.1
   host: pizza.app
   content-length: 50
   content-type: application/json; charset=UTF-8
   {x
   "size": "large",
   "toppings": ["pineapple"]
   }x
   

   View Slide

4. POST /pizza HTTP/1.1
   host: pizza.app
   content-length: 50
   content-type: application/json; charset=UTF-8
   {x
   "size": "large",
   "toppings": ["pineapple"]
   }x
   Request
   

   View Slide

5. POST /pizza HTTP/1.1
   host: pizza.app
   content-length: 50
   content-type: application/json; charset=UTF-8
   {x
   "size": "large",
   "toppings": ["pineapple"]
   }x
   URL
   

   View Slide

6. POST /pizza HTTP/1.1
   host: pizza.app
   content-length: 50
   content-type: application/json; charset=UTF-8
   {x
   "size": "large",
   "toppings": ["pineapple"]
   }x
   Headers
   

   View Slide

7. POST /pizza HTTP/1.1
   host: pizza.app
   content-length: 50
   content-type: application/json; charset=UTF-8
   {x
   "size": "large",
   "toppings": ["pineapple"]
   }x
   Body
   

   View Slide

8. POST /pizza HTTP/1.1
   host: pizza.app
   content-length: 50
   content-type: application/json; charset=UTF-8
   {x
   "size": "large",
   "toppings": ["pineapple"]
   }x
   HTTP/1.1 200 OK
   content-length: 21 

   content-type: application/json; charset=UTF-8
   {x
   "time": “30m",
   "price": 15.00
   }x
   Response
   

   View Slide

9. HTTP/1.1 200 OK
   content-length: 21 

   content-type: application/json; charset=UTF-8
   {x
   "time": “30m",
   "price": 15.00
   }x
   POST /pizza HTTP/1.1
   host: pizza.app
   content-length: 50
   content-type: application/json; charset=UTF-8
   {x
   "size": "large",
   "toppings": ["pineapple"]
   }x
   Status
   

   View Slide

10. HTTP/1.1 200 OK
    content-length: 21 

    content-type: application/json; charset=UTF-8
    {x
    "time": “30m",
    "price": 15.00
    }x
    POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x
    Headers
    

    View Slide

11. HTTP/1.1 200 OK
    content-length: 21 

    content-type: application/json; charset=UTF-8
    {x
    "time": “30m",
    "price": 15.00
    }x
    POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x
    Body
    

    View Slide

12. HTTP/1.1 200 OK
    content-length: 21 

    content-type: application/json; charset=UTF-8
    {x
    "time": “30m",
    "price": 15.00
    }x
    POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x
    

    View Slide

13. $ nc square.com 80
    GET /robots.txt HTTP/1.1
    host: square.com
    HTTP/1.1 301 Moved Permanently
    location: https://square.com/robots.txt
    date: Thu, 07 Nov 2019 04:05:21 GMT
    server: envoy
    content-length: 0

    $
    

    View Slide

14. HTTP/0.9
    HTTP/1.0
    HTTP/1.1
    HTTP/2
    HTTP/3
    1991
    1996
    1997
    2015
    2019
    HTML only
    headers and status codes
    persistent connections
    multiplexing and binary encoding
    out-of-order packets and IP address changes
    

    View Slide

15. HTTP in a Hostile World?
    • Correct
    • Ready
    • Secure
    • New stuff I’m excited about
    

    View Slide

16. Correct
    

    View Slide

17. Send Keefer $20 for

    pizza and beer
    

    View Slide

18. Send Keefer $20 for

    pizza and beer
    …
    

    View Slide

19. …
    SEND KEEFER $20 FOR

    PIZZA AND BEER!
    

    View Slide

20. SEND KEEFER $20 FOR

    PIZZA AND BEER!
    Successfully transferred
    $40 to Keefer
    

    View Slide

21. Idempotence
    • Repeating a request won’t repeat the action
    

    View Slide

22. What Triggers Repeats?
    • Users double-tapping the send button
    • Applications retrying
    • Network code retrying
    • Gateways retrying
    

    View Slide

23. Implementing Idempotence:
    Identify Intention
    • Generate a universally unique token in the UI:

    d6abe0fcd5d8c51e6c8a9159a873efcc
    • Write the outbound intention to a persistent queue, like your
    app’s SQLite database or your cloud queue service (SQS, etc.)
    • When processing an action, persist the identifier
    • The easy way: create a UNIQUE INDEX on the token
    

    View Slide

24. flickr.com/photos/fsse-info/
    

    View Slide

25. API Contract v. 1
    • Client sends a pizza order in JSON
    • Server sends HTTP 200 and the delivery info in JSON
    

    View Slide

26. HTTP/1.1 200 OK
    { "time": "30m", "price": 15.00 }
    POST /pizza HTTP/1.1
    { "size": "large", "toppings": ["pineapple"] }
    

    View Slide

27. API Contract v. 2
    • Client sends a pizza order in JSON
    • Success: Server sends HTTP 200 and the delivery info in JSON
    • Failure: Server sends HTTP 500 and the failure info in JSON
    

    View Slide

28. POST /pizza HTTP/1.1
    { "size": "large", "toppings": ["pineapple"] }
    HTTP/1.1 500 OK
    { "message": "too many carbs" }
    

    View Slide

29. Middleboxes!
    

    View Slide

30. Middleboxes!
    

    View Slide

31. Designing HTTP APIs
    • Conceptually, your client talks to your server
    • In practice, ‘invisible’ infrastructure exists
    • Sometimes it fails to be invisible
    

    View Slide

32. Status Codes
    just a sec
    thumbs up!
    response is elsewhere
    client problem
    server problem
    1xx
    2xx
    3xx
    4xx
    5xx
    

    View Slide

33. Resilient to Middle Boxes
    • Send Content-Type on outbound requests
    • Expect Content-Type on inbound responses
    • HTTP 200? All Good
    • Other status codes? Be defensive!
    

    View Slide

34. REST: Retrofit
    interface PizzaShop {
    @GET("/{store}/{date}/deliveries")
    fun deliveries(
    @Path("store") store: String,
    @Path("date") date: String
    ): Call>
    }
    square.github.io/retrofit
    

    View Slide

35. service PizzaShop {
    rpc OrderFeed (stream Update)
    returns (stream Order) {
    }
    }
    gRPC: Wire 3
    square.github.io/wire
    

    View Slide

36. interface PizzaShopClient : Service {
    @WireRpc(
    path = "/yum.pizza.PizzaShop/OrderFeed",
    requestAdapter = "yum.pizza.Update#ADAPTER”,
    responseAdapter = "yum.pizza.Order#ADAPTER"
    )
    fun OrderFeed(): GrpcStreamingCall
    }
    gRPC: Wire 3
    square.github.io/wire
    

    View Slide

37. Images:
    imageView.load("https://pizza.yum/pineapple.png") {
    crossfade(true)
    placeholder(R.drawable.image)
    transformations(CircleCropTransformation())
    }
    Coil
    coil-kt.github.io/coil
    

    View Slide

38. Testing: MockWebServer
    github.com/square/okhttp/mockwebserver
    val server = MockWebServer()
    server.start()
    server.enqueue(MockResponse()
    .setHeader("Cookie", "fave_topping=pineapple")
    .setBody("""{"time": "30m", "price": 19.99 }"""))
    val serverUrl = server.url("/")
    

    View Slide

39. Debugging: Charles Proxy
    charlesproxy.com
    

    View Slide

40. Debugging: Charles Proxy
    • Observe all your HTTP calls
    charlesproxy.com
    

    View Slide

41. Debugging: Charles Proxy
    • Observe all your HTTP calls
    • Intercepts HTTPS!
    charlesproxy.com
    

    View Slide

42. Ready
    

    View Slide

43. Redundancy!
    • Mechanisms to sync across multiple data centers
    • For availability and operations
    

    View Slide

44. Connectivity is a Lie
    • Bell and Rogers fail
    • Backends fail
    • Graceful fallbacks are possible
    

    View Slide

45. Request Avoidance
    • Push vs. poll
    • Local models in SQLite, etc.
    

    View Slide

46. Caching
    • URLs may represent resources, not calls
    • Resources can be cached!
    

    View Slide

47. Cache-Control: no-cache, no-store,
    max-age=1, s-maxage=2, private,
    must-revalidate, max-stale=3,
    min-fresh=4, only-if-cached,
    no-transform
    

    View Slide

48. Precise Caching
    • Just understand all of the above headers
    • Too difficult!
    

    View Slide

49. Good-Enough Caching
    • It may change

    Cache-Control: max-age=0
    • It will never change

    Cache-Control: max-age=31536000
    

    View Slide

50. val cache = Cache(
    directory = File(context.cacheDir, "http"),
    maxSize = 100L * 1024 * 1024
    )
    val client = OkHttpClient.Builder()
    .cache(cache)
    .build()
    

    View Slide

51. Compression
    • Make sure gzip responses are enabled on client and server
    • Cuts size by 70% for JSON!
    • Consider gzip requests too
    • Use brotli for best compression
    

    View Slide

52. CDNs
    • Content Delivery Networks
    • Easiest way to improve upon speed of light
    • Great for availability
    • These can do image resizing!
    

    View Slide

53. Granularity
    • HTTP/1’s per-resource
    overhead is high
    • More speed?

    Fewer resources!
    • With HTTP/2 logical
    resources are best
    whatsapp.com
    

    View Slide

54. Secure
    

    View Slide

55. Data is Dangerous
    • You’re keeping other people’s secrets
    • You don’t really know what’s secret and what’s not
    • Attackers are persistent
    

    View Slide

56. Other People’s Secrets?
    • Passwords are obvious
    • Lots of bigger, less obvious failures
    

    View Slide

57. Limit Exposure to Data
    • Are we collecting this field because we need it?
    • When will we delete it?
    • GDPR in Europe, CCPA in California
    • Differential Privacy
    

    View Slide

58. View Slide

59. TLS: Necessary
    • Transport Layer Security is the s in https://
    • Confirms that you’re really talking to https://cash.app/
    • Prevents eavesdropping
    • Prevents interference
    

    View Slide

60. TLS: Not Sufficient
    • Doesn’t impact how https://cash.app/ secures the data
    • Doesn’t prevent URLs like https://cash.app.attacker.com/
    • Old implementations of TLS had severe bugs
    • Nation-state attackers do nasty things
    

    View Slide

61. View Slide

62. Nation-State Attacks
    • A certificate authority is a company that confirms ownership of
    https://cash.app/ and signs our TLS certificate
    • By default browsers trust ~300 certificate authorities (CAs)
    • Nation states add themselves to the browser
    • Or compromise a vulnerable certificate authority
    

    View Slide

63. View Slide

64. Defending against

    Nation-State Attacks
    • Certificate Pinning
    • Limit which of the 300 certificate authorities you trust
    

    View Slide

65. CertificatePinner certificatePinner = new CertificatePinner.Builder()
    .add("pizza.yum", "sha256/9TxoNDzGwoGUV8oDSZI8XpGpwyPz0GY/xAyWRLLjpPc=")
    .add("pizza.yum", "sha256/36Bv98D5O9bYGYRXnMTZJLQRW8i8fvOiWC/TJ4jOhLw=")
    .add("pizza.yum", "sha256/++iJU8rCHLXviXyKrJ2yqBWjs/5+LAqWuP04dHVHFb0=")
    .build();
    OkHttpClient okHttpClient = new OkHttpClient.Builder()
    .certificatePinner(certificatePinner)
    .build();
    

    View Slide

66. TLS Evolution
    first widely-used release
    renamed from SSL to TLS
    bug fixes & new crypto
    bug fixes & new crypto
    major update with simplifications!
    1996
    1999
    2006
    2008
    2018
    SSLv3
    TLSv1
    TLSv1.1
    TLSv1.2
    TLSv1.3
    

    View Slide

67. TLS Evolution
    first widely-used release
    renamed from SSL to TLS
    bug fixes & new crypto
    bug fixes & new crypto
    major update with simplifications!
    JAN 2020
    1996
    1999
    2006
    2008
    2018
    SSLv3
    TLSv1
    TLSv1.1
    TLSv1.2
    TLSv1.3
    

    View Slide

68. TLS Evolution
    first widely-used release
    renamed from SSL to TLS
    bug fixes & new crypto
    bug fixes & new crypto
    major update with simplifications!
    JAN 2020
    1996
    1999
    2006
    2008
    2018
    Android 5+
    Or Google Play
    Security Provider
    SSLv3
    TLSv1
    TLSv1.1
    TLSv1.2
    TLSv1.3
    

    View Slide

69. Whitebox Attackers
    • Our apps are instruction manuals for reverse engineers
    • There are amazing tools that make this point-and-click easy
    

    View Slide

70. github.com/sensepost/objection
    Objection
    

    View Slide

71. Defending Against

    Whitebox Attackers
    • A sufficiently-determined attacker will succeed
    • All we can do is make it really annoying
    • Perhaps a magic HTTP header?
    

    View Slide

72. New Stuff I’m Excited About
    

    View Slide

73. Envoy
    • Middleboxes are so 2018
    • Let’s add side-boxes!
    envoyproxy.io
    

    View Slide

74. View Slide

75. View Slide

76. HTTP/3
    • Better worst-case latency
    • Change your IP address without restarting downloads!
    quicwg.org
    

    View Slide

77. TCP streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    

    View Slide

78. TCP streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    TCP streams
    TLS streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    with TLS
    

    View Slide

79. TCP streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    TCP streams
    TLS streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    with TLS
    TCP streams
    TLS streams
    HTTP/2 frames
    TCP packets
    HTTP/2
    with TLS
    Requests and
    Responses
    

    View Slide

80. TCP streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    TCP streams
    TLS streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    with TLS
    TCP streams
    TLS streams
    HTTP/2 frames
    TCP packets
    HTTP/2
    with TLS
    Requests and
    Responses
    UDP packets
    HTTP/3
    with TLS
    Requests and
    Responses
    QUIC streams
    

    View Slide

81. • What if Retrofit inspired a server framework?
    • gRPC + REST in one framework
    • Early days! Watch this space
    cashapp.github.io/misk
    

    View Slide

82. OkHttp
    • New concurrency internals
    • Performance Features
    • Async I/O and coroutines
    square.github.io/okhttp
    

    View Slide

83. @jessewilson
    Thanks Toronto!
    https://square.github.io/okhttp/
    

    View Slide