Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LINT: Journey to Network Modernization with HTT...

LINT: Journey to Network Modernization with HTTP/2 and TLS

LINE DEVDAY 2021

November 10, 2021
Tweet

More Decks by LINE DEVDAY 2021

Other Decks in Technology

Transcript

  1. Agenda - Background - Connectivity Improvement - Migration to HTTP/2

    - Streaming Push - TLS with Ticket Resumption - Summary
  2. LINE Messaging Architecture LINE Client LEGY (API Gateway) Talk Server

    Storage Collaborative Services * LEGY: LINE Event GatewaY
  3. History of LEGY Protocol HTTP/1.1: Early Days of LINE LINE

    Client LEGY connection connection connection Request Response
  4. History of LEGY Protocol HTTP/1.1: Early Days of LINE Long

    Polling - Mimic real-time event delivery via HTTP
  5. History of LEGY Protocol HTTP/1.1: Early Days of LINE Long

    Polling - Mimic real-time event delivery via HTTP Client LEGY fetchOps {rev: 1, count: 100} Talk fetchOps {rev: 1, count: 100} No Content fetchOps {rev: 1, count: 100} publish Response OP created fetchOps {rev: 101, count: 100} Response OP is the unit of event in LINE
  6. History of LEGY Protocol SPDY: Since Oct. 2012 Header Compression

    - Headers can be compressed using zlib - High server memory usage - Couldn’t support compression for some connections SPDY
  7. Header Compression History of LEGY Protocol SPDY: Since Oct. 2012

    Header Cache - Same HTTP headers are transferred repeatedly - LEGY holds caches for such headers - HTTP request gets smaller - Headers can be compressed using zlib - High server memory usage - Couldn’t support compression for some connections SPDY LINE
  8. Header Compression History of LEGY Protocol SPDY: Since Oct. 2012

    Header Cache - Same HTTP headers are transferred repeatedly - LEGY holds caches for such headers - HTTP request gets smaller - Headers can be compressed using zlib - High server memory usage - Couldn’t support compression for some connections SPDY LINE Client LEGY header1: value1 header2: value2 Server header1: value1 header2: value2 cache key header3: value3 header1: value1 header2: value2 Header3: value3 cache key
  9. History of LEGY Protocol SPDY: Since Oct. 2012 Server Push

    SPDY - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages
  10. History of LEGY Protocol SPDY: Since Oct. 2012 Server Push

    Custom SPDY Push SPDY LINE - LINE’s some features and services follow subscribe/push pattern - The pushes are not associated with a request - Original request is closed before the pushes - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages
  11. History of LEGY Protocol SPDY: Since Oct. 2012 Server Push

    Custom SPDY Push SPDY LINE - LINE’s some features and services follow subscribe/push pattern - The pushes are not associated with a request - Original request is closed before the pushes Client LEGY Subscribe Service Subscribe Push OK OK Push (asso. with stream 0) Push Push (asso. with stream 0) - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages
  12. History of LEGY Protocol SPDY: Since Oct. 2012 Server Push

    Custom SPDY Push SPDY LINE - LINE’s some features and services follow subscribe/push pattern - The pushes are not associated with a request - Original request is closed before the pushes Note that fetchOps still works through long polling - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages Client LEGY Subscribe Service Subscribe Push OK OK Push (asso. with stream 0) Push Push (asso. with stream 0)
  13. History of LEGY Protocol SPDY: Since Oct. 2012 Encryption via

    TLS SPDY - Use TLS for connection-wise encryption - Not mandatory but recommended - Overhead for slow network and clients - Resulted in bad user experience in 3G
  14. Encryption via TLS History of LEGY Protocol SPDY: Since Oct.

    2012 LEGY Encryption - Lightweight in-house encryption method - Encrypt message body and sensitive headers not whole connection - Effective for clients with 3G environment SPDY LINE - Use TLS for connection-wise encryption - Not mandatory but recommended - Overhead for slow network and clients - Resulted in bad user experience in 3G
  15. Encryption via TLS History of LEGY Protocol SPDY: Since Oct.

    2012 LEGY Encryption SPDY LINE - Use TLS for connection-wise encryption - Not mandatory but recommended - Overhead for slow network and clients - Resulted in bad user experience in 3G Original LEGY Encryption Applied Headers Body Header part Body part Sensitive Headers + Body LE Headers Non-sensitive Headers Header part Body part - Lightweight in-house encryption method - Encrypt message body and sensitive headers not whole connection - Effective for clients with 3G environment
  16. Problems - There is no standard library to work with

    our SPDY protocol - It makes code management harder Outdated Protocol Safer but Heavy TLS - TLS is safer than LEGY Encryption and gets evolved - The handshake is still heavy - Our SPDY protocol itself is not conformant to the specification - It’s confusing ⇨ Errors, bugs Standard Inconformity SPDY Header Cache Custom Push LEGY Encryption
  17. LINT: LINE Improvement for Next Ten years Outdated SPDY HTTP/2

    HTTP/2 Conformant Push Streaming Push Security TLS
  18. Bad Cases HTTP/2 servers are buggy HTTP/2 clients are buggy

    Clients must be configured from the external not to use HTTP/2
  19. Bad Cases HTTP/2 servers are buggy HTTP/2 clients are buggy

    Clients must be configured from the external not to use HTTP/2 SPDY must be able to be used, instead of HTTP/2, depending on the situation
  20. Connection Info. Means of Network Configuration from LEGY Define where

    to connect Define client’s network behaviors (feature on/off, feature parameters) Define how to connect
  21. Connection Info. It’s All About Reliability - Must route clients

    to stable LEGY PoPs. LEGY PoP Status - Each client has different application policy. Different Client Types - Each country has different network characteristics. Different Countries - As client is updated, features are added and removed. Different Client Versions
  22. Connection Info. Example: Connection Configuration Config for LEGY Servers Fallback

    country Client type and version Network Environment Specified country
  23. Connection Info. Example: Connection Configuration Config for LEGY Servers Fallback

    country Client type and version Network Environment Specified country
  24. Connection Info. How It Works LEGY Repository Client Repository Default

    Conn-info Repository export import LEGY conn-info default conn-info Periodically fetch conn-info w/ local cache Use HTTP/2 with TLS Use SPDY with LEGY Encryption
  25. Migration Process Conn-info HTTP/2 + SPDY Conn-info HTTP/2 + SPDY

    Dogfooding Domain Client SPDY for normal users Conn-info HTTP/2 + SPDY Normal Domain Client HTTP2 for JP only Conn-info HTTP/2 + SPDY Normal Domain Client Fully respect conn-info Dogfooding JP LINE All Regions Beta/RC production
  26. Migration Process Conn-info HTTP/2 + SPDY Conn-info HTTP/2 + SPDY

    Dogfooding Domain Client SPDY for normal users Conn-info HTTP/2 + SPDY Normal Domain Client HTTP2 for JP only Conn-info HTTP/2 + SPDY Normal Domain Client Fully respect conn-info Dogfooding JP LINE All Regions Beta/RC 85% of connections use HTTP/2 production
  27. Server Push Requirements - It is easy to use with

    any HTTP/2 libraries - The mechanism should be standard conformant - Push can be unsolicited
  28. HTTP/2 Server Push Client LEGY Subscribe Service Subscribe OK Push

    PUSH_PROMISE (B) OK PUSH_PROMISE (C) Push Stream A Stream B
  29. HTTP/2 Server Push RFC7540 §8.2.1 The PUSH_PROMISE frames sent by

    the server are sent on that explicit request's stream. Client LEGY Subscribe Service Subscribe OK Push PUSH_PROMISE (B) OK PUSH_PROMISE (C) Push Stream A Stream B ❌
  30. Control on The Connection Server Push Requires Connection Controllability LINE

    CLIENT LEGY Connection Lib + OS Subscribe OK PUSH_PROMISE
  31. Control on The Connection Server Push Requires Connection Controllability LINE

    CLIENT LEGY Connection Lib + OS Subscribe OK Subscription succeeded. Server will let me know when data is available. PUSH_PROMISE
  32. Control on The Connection Server Push Requires Connection Controllability LINE

    CLIENT LEGY Lib + OS Client terminated the subscription. Connection Request Response Connection Subscription succeeded. Server will let me know when data is available.
  33. Control on The Connection Server Push Requires Connection Controllability LINE

    CLIENT LEGY Lib + OS Client terminated the subscription. Connection Request Response Has DEAD subscription
  34. Control on The Connection Server Push Requires Connection Controllability LINE

    CLIENT LEGY Lib + OS Connection Request Response The push session must be within a stream, not a connection
  35. Streaming in HTTP/1.1 Chunked Transfer Encoding HTTP/1.1 200 OK Content-Type:

    text/plain Transfer-Encoding: chunked 7\r\n Mozilla\r\n 9\r\n Developer\r\n 7\r\n Network\r\n 0\r\n \r\n * Example from https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding#examples Client Server Request Headers (chunked transfer) 7\r\nMozilla\r\n Data Created 9\r\nDeveloper\r\n Data Created 7\r\nNetwork\r\n Data Ceated 0\r\n\r\n Nothing More
  36. Streaming in HTTP/2 HTTP/2 is streaming E = 0 Welcome

    E = 0 to E = 0 DEV E = 0 DAY E = 1 2021 Sender Receiver Welcome to DEVDAY 2021 7\r\n Welcome\r\n 4\r\n 2021\r\n 2\r\n to\r\n 3\r\n DEV\r\n 3\r\n DAY\r\n 0\r\n \r\n
  37. Streaming in HTTP/2 HTTP/2 is streaming LEGY Client E =

    0 PUSH1 E = 0 PUSH2 E = 0 PUSH3 E = 0 PUSH4 E = 1 PUSH5 - No customization on HTTP/2 - Any libraries can be used - Unsolicited push is available
  38. Streaming Push Stream As A Push Session Client LEGY HEADERS

    HEADERS (status:200) DATA (sign-on) Stream 1 Service1 Service2 Subscribe Subscribe DATA (sign-on results) OK OK PUSH DATA (push) DATA (push ack) Push Session Established
  39. Streaming Push Long Polling in Push Session Client LEGY HEADERS

    HEADERS (status:200) DATA (fetchOps sign-on) Stream 1 Talk fetchOps DATA (sign-on result) Empty publish fetchOps Response OP created DATA (fetchOps sign-on) Push Session Established
  40. TLS

  41. Heavy TLS Handshake Client Server SYN SYN+ACK ACK Client Server

    SYN SYN+ACK ClientHello ServerHelloDone Finished Finished Application Data Application Data ACK Plain TCP TLSv1.2
  42. Heavy TLS Handshake Client Server SYN SYN+ACK ACK Client Server

    SYN SYN+ACK ClientHello ServerHelloDone Finished Finished Application Data Application Data ACK Plain TCP TLSv1.2 Additional 2 RTT
  43. TLSv1.3 Less RTT for Handshake Client Server TCP Handshake Client

    Server ClientHello TCP Handshake ServerHello + Finished Finished + Application Data ClientHello + Application Data ServerHello + Finished + ApplicationData Finished + Application Data TLSv1.3 TLSv1.3 0-RTT Additional 1 RTT regardless of Session resumption
  44. TLSv1.3 Less RTT for Handshake Client Server TCP Handshake Client

    Server ClientHello TCP Handshake ServerHello + Finished Finished + Application Data ClientHello + Application Data ServerHello + Finished + ApplicationData Finished + Application Data TLSv1.3 TLSv1.3 0-RTT Additional 1 RTT regardless of Session resumption 20% of connections are using still TLSv1.2
  45. 1 RTT in TLSv1.2 Client Server ClientHello ServerHelloDone Finished Finished

    Application Data TLSv1.2 First Connection TCP Handshake
  46. 1 RTT in TLSv1.2 Client Server ClientHello ServerHelloDone Finished Finished

    Application Data TLSv1.2 First Connection TCP Handshake Client Server TCP Handshake ClientHello ServerHello + Finished Finished + Application Data TLSv1.2 Second Connection Don’t renegotiate
  47. 1 RTT in TLSv1.2 Client Server ClientHello ServerHelloDone Finished Finished

    Application Data TLSv1.2 First Connection TCP Handshake Client Server TCP Handshake ClientHello ServerHello + Finished Finished + Application Data TLSv1.2 Second Connection Session ID & Session Ticket
  48. Session ID LEGY LINE CLIENT LEGY LEGY LEGY First connection

    Store session (ID: 1) Resume (ID: 1) No such session Do the full Handshake
  49. Session ID LEGY LINE CLIENT LEGY LEGY LEGY First connection

    Resume (ID: 1) Redis Redis Redis Redis Redis Store session (ID: 1) Retrieve session (ID: 1) Rendezvous Hashing Session Found Handshake is abbreviated
  50. RFC 5077: Session Ticket Handshake LEGY LINE CLIENT LEGY LEGY

    LEGY First connection Old key Current key Future key Generate ticket
  51. RFC 5077: Session Ticket Handshake LEGY LINE CLIENT LEGY LEGY

    LEGY First connection Resume with ticket Old key Current key Future key Generate ticket Decode ticket
  52. RFC 5077: Session Ticket Key Management LEGY LEGY LEGY LEGY

    Old key Current key Future key KMS CronJob 1. Request key gen. 2. Future Keys 3. Update keys 4. Fetch and rotate keys
  53. LINE 10-year Service and Service Growing For The Next 10

    Years Reliability Extensibility Security User Experience
  54. LINE 10-year Service and Service Growing For The Next 10

    Years Reliability Extensibility Security User Experience LE/TCP SPDY Header Cache Custom Push TLS w/ resumption HTTP/2 Streaming Push 93%
  55. LINE 10-year Service and Service Growing For The Next 10

    Years Reliability Extensibility Security User Experience LE/TCP SPDY Header Cache Custom Push TLS w/ resumption HTTP/2 Streaming Push 85%
  56. LINE 10-year Service and Service Growing For The Next 10

    Years Reliability Extensibility Security User Experience LE/TCP SPDY Header Cache Custom Push TLS w/ resumption HTTP/2 Streaming Push