$30 off During Our Annual Pro Sale. View Details »

HTTP in a Hostile World (Droidcon Toronto 2019)

Jesse Wilson
November 13, 2019

HTTP in a Hostile World (Droidcon Toronto 2019)

Video: https://youtu.be/tPA9n2mgClI

Networking on Android is difficult work. We need to display fresh data and rich media while limiting our network use. Unfortunately cookie-cutter solutions are not acceptable.

In this talk we’ll:

🥌 Verify that you’re using HTTPS securely
🥌 Acknowledge possible connectivity problems
🥌 Get performance tips
🥌 Code for maintainable APIs
🥌 Mock programmers that don’t test their networking

The body of this talk will be using HTTP effectively.

Jesse Wilson

November 13, 2019
Tweet

More Decks by Jesse Wilson

Other Decks in Programming

Transcript

  1. @jessewilson
    https://square.github.io/okhttp/
    HTTP in a Hostile World

    View Slide

  2. HTTP?
    • Web Pages
    • Web Resources (images!)
    • Downloads
    • Web APIs
    • Mobile APIs
    • Server APIs
    • Core Networking APIs (DNS)
    • Pretty much all networking ever

    View Slide

  3. POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x

    View Slide

  4. POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x
    Request

    View Slide

  5. POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x
    URL

    View Slide

  6. POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x
    Headers

    View Slide

  7. POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x
    Body

    View Slide

  8. POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x
    HTTP/1.1 200 OK
    content-length: 21 

    content-type: application/json; charset=UTF-8
    {x
    "time": “30m",
    "price": 15.00
    }x
    Response

    View Slide

  9. HTTP/1.1 200 OK
    content-length: 21 

    content-type: application/json; charset=UTF-8
    {x
    "time": “30m",
    "price": 15.00
    }x
    POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x
    Status

    View Slide

  10. HTTP/1.1 200 OK
    content-length: 21 

    content-type: application/json; charset=UTF-8
    {x
    "time": “30m",
    "price": 15.00
    }x
    POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x
    Headers

    View Slide

  11. HTTP/1.1 200 OK
    content-length: 21 

    content-type: application/json; charset=UTF-8
    {x
    "time": “30m",
    "price": 15.00
    }x
    POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x
    Body

    View Slide

  12. HTTP/1.1 200 OK
    content-length: 21 

    content-type: application/json; charset=UTF-8
    {x
    "time": “30m",
    "price": 15.00
    }x
    POST /pizza HTTP/1.1
    host: pizza.app
    content-length: 50
    content-type: application/json; charset=UTF-8
    {x
    "size": "large",
    "toppings": ["pineapple"]
    }x

    View Slide

  13. $ nc square.com 80
    GET /robots.txt HTTP/1.1
    host: square.com
    HTTP/1.1 301 Moved Permanently
    location: https://square.com/robots.txt
    date: Thu, 07 Nov 2019 04:05:21 GMT
    server: envoy
    content-length: 0

    $

    View Slide

  14. HTTP/0.9
    HTTP/1.0
    HTTP/1.1
    HTTP/2
    HTTP/3
    1991
    1996
    1997
    2015
    2019
    HTML only
    headers and status codes
    persistent connections
    multiplexing and binary encoding
    out-of-order packets and IP address changes

    View Slide

  15. HTTP in a Hostile World?
    • Correct
    • Ready
    • Secure
    • New stuff I’m excited about

    View Slide

  16. Correct

    View Slide

  17. Send Keefer $20 for

    pizza and beer

    View Slide

  18. Send Keefer $20 for

    pizza and beer

    View Slide


  19. SEND KEEFER $20 FOR

    PIZZA AND BEER!

    View Slide

  20. SEND KEEFER $20 FOR

    PIZZA AND BEER!
    Successfully transferred
    $40 to Keefer

    View Slide

  21. Idempotence
    • Repeating a request won’t repeat the action

    View Slide

  22. What Triggers Repeats?
    • Users double-tapping the send button
    • Applications retrying
    • Network code retrying
    • Gateways retrying

    View Slide

  23. Implementing Idempotence:
    Identify Intention
    • Generate a universally unique token in the UI:

    d6abe0fcd5d8c51e6c8a9159a873efcc
    • Write the outbound intention to a persistent queue, like your
    app’s SQLite database or your cloud queue service (SQS, etc.)
    • When processing an action, persist the identifier
    • The easy way: create a UNIQUE INDEX on the token

    View Slide

  24. flickr.com/photos/fsse-info/

    View Slide

  25. API Contract v. 1
    • Client sends a pizza order in JSON
    • Server sends HTTP 200 and the delivery info in JSON

    View Slide

  26. HTTP/1.1 200 OK
    { "time": "30m", "price": 15.00 }
    POST /pizza HTTP/1.1
    { "size": "large", "toppings": ["pineapple"] }

    View Slide

  27. API Contract v. 2
    • Client sends a pizza order in JSON
    • Success: Server sends HTTP 200 and the delivery info in JSON
    • Failure: Server sends HTTP 500 and the failure info in JSON

    View Slide

  28. POST /pizza HTTP/1.1
    { "size": "large", "toppings": ["pineapple"] }
    HTTP/1.1 500 OK
    { "message": "too many carbs" }

    View Slide

  29. Middleboxes!

    View Slide

  30. Middleboxes!

    View Slide

  31. Designing HTTP APIs
    • Conceptually, your client talks to your server
    • In practice, ‘invisible’ infrastructure exists
    • Sometimes it fails to be invisible

    View Slide

  32. Status Codes
    just a sec
    thumbs up!
    response is elsewhere
    client problem
    server problem
    1xx
    2xx
    3xx
    4xx
    5xx

    View Slide

  33. Resilient to Middle Boxes
    • Send Content-Type on outbound requests
    • Expect Content-Type on inbound responses
    • HTTP 200? All Good
    • Other status codes? Be defensive!

    View Slide

  34. REST: Retrofit
    interface PizzaShop {
    @GET("/{store}/{date}/deliveries")
    fun deliveries(
    @Path("store") store: String,
    @Path("date") date: String
    ): Call>
    }
    square.github.io/retrofit

    View Slide

  35. service PizzaShop {
    rpc OrderFeed (stream Update)
    returns (stream Order) {
    }
    }
    gRPC: Wire 3
    square.github.io/wire

    View Slide

  36. interface PizzaShopClient : Service {
    @WireRpc(
    path = "/yum.pizza.PizzaShop/OrderFeed",
    requestAdapter = "yum.pizza.Update#ADAPTER”,
    responseAdapter = "yum.pizza.Order#ADAPTER"
    )
    fun OrderFeed(): GrpcStreamingCall
    }
    gRPC: Wire 3
    square.github.io/wire

    View Slide

  37. Images:
    imageView.load("https://pizza.yum/pineapple.png") {
    crossfade(true)
    placeholder(R.drawable.image)
    transformations(CircleCropTransformation())
    }
    Coil
    coil-kt.github.io/coil

    View Slide

  38. Testing: MockWebServer
    github.com/square/okhttp/mockwebserver
    val server = MockWebServer()
    server.start()
    server.enqueue(MockResponse()
    .setHeader("Cookie", "fave_topping=pineapple")
    .setBody("""{"time": "30m", "price": 19.99 }"""))
    val serverUrl = server.url("/")

    View Slide

  39. Debugging: Charles Proxy
    charlesproxy.com

    View Slide

  40. Debugging: Charles Proxy
    • Observe all your HTTP calls
    charlesproxy.com

    View Slide

  41. Debugging: Charles Proxy
    • Observe all your HTTP calls
    • Intercepts HTTPS!
    charlesproxy.com

    View Slide

  42. Ready

    View Slide

  43. Redundancy!
    • Mechanisms to sync across multiple data centers
    • For availability and operations

    View Slide

  44. Connectivity is a Lie
    • Bell and Rogers fail
    • Backends fail
    • Graceful fallbacks are possible

    View Slide

  45. Request Avoidance
    • Push vs. poll
    • Local models in SQLite, etc.

    View Slide

  46. Caching
    • URLs may represent resources, not calls
    • Resources can be cached!

    View Slide

  47. Cache-Control: no-cache, no-store,
    max-age=1, s-maxage=2, private,
    must-revalidate, max-stale=3,
    min-fresh=4, only-if-cached,
    no-transform

    View Slide

  48. Precise Caching
    • Just understand all of the above headers
    • Too difficult!

    View Slide

  49. Good-Enough Caching
    • It may change

    Cache-Control: max-age=0
    • It will never change

    Cache-Control: max-age=31536000

    View Slide

  50. val cache = Cache(
    directory = File(context.cacheDir, "http"),
    maxSize = 100L * 1024 * 1024
    )
    val client = OkHttpClient.Builder()
    .cache(cache)
    .build()

    View Slide

  51. Compression
    • Make sure gzip responses are enabled on client and server
    • Cuts size by 70% for JSON!
    • Consider gzip requests too
    • Use brotli for best compression

    View Slide

  52. CDNs
    • Content Delivery Networks
    • Easiest way to improve upon speed of light
    • Great for availability
    • These can do image resizing!

    View Slide

  53. Granularity
    • HTTP/1’s per-resource
    overhead is high
    • More speed?

    Fewer resources!
    • With HTTP/2 logical
    resources are best
    whatsapp.com

    View Slide

  54. Secure

    View Slide

  55. Data is Dangerous
    • You’re keeping other people’s secrets
    • You don’t really know what’s secret and what’s not
    • Attackers are persistent

    View Slide

  56. Other People’s Secrets?
    • Passwords are obvious
    • Lots of bigger, less obvious failures

    View Slide

  57. Limit Exposure to Data
    • Are we collecting this field because we need it?
    • When will we delete it?
    • GDPR in Europe, CCPA in California
    • Differential Privacy

    View Slide

  58. View Slide

  59. TLS: Necessary
    • Transport Layer Security is the s in https://
    • Confirms that you’re really talking to https://cash.app/
    • Prevents eavesdropping
    • Prevents interference

    View Slide

  60. TLS: Not Sufficient
    • Doesn’t impact how https://cash.app/ secures the data
    • Doesn’t prevent URLs like https://cash.app.attacker.com/
    • Old implementations of TLS had severe bugs
    • Nation-state attackers do nasty things

    View Slide

  61. View Slide

  62. Nation-State Attacks
    • A certificate authority is a company that confirms ownership of
    https://cash.app/ and signs our TLS certificate
    • By default browsers trust ~300 certificate authorities (CAs)
    • Nation states add themselves to the browser
    • Or compromise a vulnerable certificate authority

    View Slide

  63. View Slide

  64. Defending against

    Nation-State Attacks
    • Certificate Pinning
    • Limit which of the 300 certificate authorities you trust

    View Slide

  65. CertificatePinner certificatePinner = new CertificatePinner.Builder()
    .add("pizza.yum", "sha256/9TxoNDzGwoGUV8oDSZI8XpGpwyPz0GY/xAyWRLLjpPc=")
    .add("pizza.yum", "sha256/36Bv98D5O9bYGYRXnMTZJLQRW8i8fvOiWC/TJ4jOhLw=")
    .add("pizza.yum", "sha256/++iJU8rCHLXviXyKrJ2yqBWjs/5+LAqWuP04dHVHFb0=")
    .build();
    OkHttpClient okHttpClient = new OkHttpClient.Builder()
    .certificatePinner(certificatePinner)
    .build();

    View Slide

  66. TLS Evolution
    first widely-used release
    renamed from SSL to TLS
    bug fixes & new crypto
    bug fixes & new crypto
    major update with simplifications!
    1996
    1999
    2006
    2008
    2018
    SSLv3
    TLSv1
    TLSv1.1
    TLSv1.2
    TLSv1.3

    View Slide

  67. TLS Evolution
    first widely-used release
    renamed from SSL to TLS
    bug fixes & new crypto
    bug fixes & new crypto
    major update with simplifications!
    JAN 2020
    1996
    1999
    2006
    2008
    2018
    SSLv3
    TLSv1
    TLSv1.1
    TLSv1.2
    TLSv1.3

    View Slide

  68. TLS Evolution
    first widely-used release
    renamed from SSL to TLS
    bug fixes & new crypto
    bug fixes & new crypto
    major update with simplifications!
    JAN 2020
    1996
    1999
    2006
    2008
    2018
    Android 5+
    Or Google Play
    Security Provider
    SSLv3
    TLSv1
    TLSv1.1
    TLSv1.2
    TLSv1.3

    View Slide

  69. Whitebox Attackers
    • Our apps are instruction manuals for reverse engineers
    • There are amazing tools that make this point-and-click easy

    View Slide

  70. github.com/sensepost/objection
    Objection

    View Slide

  71. Defending Against

    Whitebox Attackers
    • A sufficiently-determined attacker will succeed
    • All we can do is make it really annoying
    • Perhaps a magic HTTP header?

    View Slide

  72. New Stuff I’m Excited About

    View Slide

  73. Envoy
    • Middleboxes are so 2018
    • Let’s add side-boxes!
    envoyproxy.io

    View Slide

  74. View Slide

  75. View Slide

  76. HTTP/3
    • Better worst-case latency
    • Change your IP address without restarting downloads!
    quicwg.org

    View Slide

  77. TCP streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1

    View Slide

  78. TCP streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    TCP streams
    TLS streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    with TLS

    View Slide

  79. TCP streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    TCP streams
    TLS streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    with TLS
    TCP streams
    TLS streams
    HTTP/2 frames
    TCP packets
    HTTP/2
    with TLS
    Requests and
    Responses

    View Slide

  80. TCP streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    TCP streams
    TLS streams
    Requests and
    Responses
    TCP packets
    HTTP/1.1
    with TLS
    TCP streams
    TLS streams
    HTTP/2 frames
    TCP packets
    HTTP/2
    with TLS
    Requests and
    Responses
    UDP packets
    HTTP/3
    with TLS
    Requests and
    Responses
    QUIC streams

    View Slide

  81. • What if Retrofit inspired a server framework?
    • gRPC + REST in one framework
    • Early days! Watch this space
    cashapp.github.io/misk

    View Slide

  82. OkHttp
    • New concurrency internals
    • Performance Features
    • Async I/O and coroutines
    square.github.io/okhttp

    View Slide

  83. @jessewilson
    Thanks Toronto!
    https://square.github.io/okhttp/

    View Slide