$30 off During Our Annual Pro Sale. View Details »

LINT: Journey to Network Modernization with HTTP/2 and TLS

LINT: Journey to Network Modernization with HTTP/2 and TLS

LINE DEVDAY 2021
PRO

November 10, 2021
Tweet

More Decks by LINE DEVDAY 2021

Other Decks in Technology

Transcript

  1. None
  2. LINT: Network Modernization with HTTP/2 and TLS Lee Byeoksan LINE+

  3. Agenda - Background - Connectivity Improvement - Migration to HTTP/2

    - Streaming Push - TLS with Ticket Resumption - Summary
  4. Background

  5. LINE Messaging Architecture LINE Client LEGY (API Gateway) Talk Server

    Storage Collaborative Services * LEGY: LINE Event GatewaY
  6. LINE Messaging Architecture LINE Client Talk Server Storage Collaborative Services

    * LEGY: LINE Event GatewaY LEGY (API Gateway)
  7. History of LEGY Protocol HTTP/1.1: Early Days of LINE LINE

    Client LEGY connection connection connection Request Response
  8. History of LEGY Protocol HTTP/1.1: Early Days of LINE Long

    Polling - Mimic real-time event delivery via HTTP
  9. History of LEGY Protocol HTTP/1.1: Early Days of LINE Long

    Polling - Mimic real-time event delivery via HTTP Client LEGY fetchOps {rev: 1, count: 100} Talk fetchOps {rev: 1, count: 100} No Content fetchOps {rev: 1, count: 100} publish Response OP created fetchOps {rev: 101, count: 100} Response OP is the unit of event in LINE
  10. History of LEGY Protocol SPDY: Since Oct. 2012 LINE Client

    LEGY connection Request Response
  11. History of LEGY Protocol SPDY: Since Oct. 2012 Header Compression

    - Headers can be compressed using zlib - High server memory usage - Couldn’t support compression for some connections SPDY
  12. Header Compression History of LEGY Protocol SPDY: Since Oct. 2012

    Header Cache - Same HTTP headers are transferred repeatedly - LEGY holds caches for such headers - HTTP request gets smaller - Headers can be compressed using zlib - High server memory usage - Couldn’t support compression for some connections SPDY LINE
  13. Header Compression History of LEGY Protocol SPDY: Since Oct. 2012

    Header Cache - Same HTTP headers are transferred repeatedly - LEGY holds caches for such headers - HTTP request gets smaller - Headers can be compressed using zlib - High server memory usage - Couldn’t support compression for some connections SPDY LINE Client LEGY header1: value1 header2: value2 Server header1: value1 header2: value2 cache key header3: value3 header1: value1 header2: value2 Header3: value3 cache key
  14. History of LEGY Protocol SPDY: Since Oct. 2012 Server Push

    SPDY - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages
  15. History of LEGY Protocol SPDY: Since Oct. 2012 Server Push

    Custom SPDY Push SPDY LINE - LINE’s some features and services follow subscribe/push pattern - The pushes are not associated with a request - Original request is closed before the pushes - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages
  16. History of LEGY Protocol SPDY: Since Oct. 2012 Server Push

    Custom SPDY Push SPDY LINE - LINE’s some features and services follow subscribe/push pattern - The pushes are not associated with a request - Original request is closed before the pushes Client LEGY Subscribe Service Subscribe Push OK OK Push (asso. with stream 0) Push Push (asso. with stream 0) - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages
  17. History of LEGY Protocol SPDY: Since Oct. 2012 Server Push

    Custom SPDY Push SPDY LINE - LINE’s some features and services follow subscribe/push pattern - The pushes are not associated with a request - Original request is closed before the pushes Note that fetchOps still works through long polling - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages Client LEGY Subscribe Service Subscribe Push OK OK Push (asso. with stream 0) Push Push (asso. with stream 0)
  18. History of LEGY Protocol SPDY: Since Oct. 2012 Encryption via

    TLS SPDY - Use TLS for connection-wise encryption - Not mandatory but recommended - Overhead for slow network and clients - Resulted in bad user experience in 3G
  19. Encryption via TLS History of LEGY Protocol SPDY: Since Oct.

    2012 LEGY Encryption - Lightweight in-house encryption method - Encrypt message body and sensitive headers not whole connection - Effective for clients with 3G environment SPDY LINE - Use TLS for connection-wise encryption - Not mandatory but recommended - Overhead for slow network and clients - Resulted in bad user experience in 3G
  20. Encryption via TLS History of LEGY Protocol SPDY: Since Oct.

    2012 LEGY Encryption SPDY LINE - Use TLS for connection-wise encryption - Not mandatory but recommended - Overhead for slow network and clients - Resulted in bad user experience in 3G Original LEGY Encryption Applied Headers Body Header part Body part Sensitive Headers + Body LE Headers Non-sensitive Headers Header part Body part - Lightweight in-house encryption method - Encrypt message body and sensitive headers not whole connection - Effective for clients with 3G environment
  21. Problems - There is no standard library to work with

    our SPDY protocol - It makes code management harder Outdated Protocol Safer but Heavy TLS - TLS is safer than LEGY Encryption and gets evolved - The handshake is still heavy - Our SPDY protocol itself is not conformant to the specification - It’s confusing ⇨ Errors, bugs Standard Inconformity SPDY Header Cache Custom Push LEGY Encryption
  22. LINT: LINE Improvement for Next Ten years Outdated SPDY HTTP/2

    HTTP/2 Conformant Push Streaming Push Security TLS
  23. Migration to HTTP/2

  24. Bad Cases HTTP/2 servers are buggy HTTP/2 clients are buggy

  25. Bad Cases HTTP/2 servers are buggy HTTP/2 clients are buggy

    Clients must be configured from the external not to use HTTP/2
  26. Bad Cases HTTP/2 servers are buggy HTTP/2 clients are buggy

    Clients must be configured from the external not to use HTTP/2 SPDY must be able to be used, instead of HTTP/2, depending on the situation
  27. Overview LEGY V1 SPDY V2 upgrade HTTP/2 SPDY Network Abstraction

    Layer
  28. Overview LEGY SPDY HTTP/2 SPDY Fallback Network Abstraction Layer V1

    V2 upgrade
  29. Overview LEGY SPDY HTTP/2 SPDY Disable HTTP2 Network Abstraction Layer

    V1 V2 upgrade
  30. Overview LEGY SPDY HTTP/2 SPDY Network Abstraction Layer V1 V2

    upgrade
  31. Connection Info. Means of Network Configuration from LEGY Define where

    to connect Define client’s network behaviors (feature on/off, feature parameters) Define how to connect
  32. Connection Info. It’s All About Reliability - Must route clients

    to stable LEGY PoPs. LEGY PoP Status - Each client has different application policy. Different Client Types - Each country has different network characteristics. Different Countries - As client is updated, features are added and removed. Different Client Versions
  33. Connection Info. Example: Connection Configuration

  34. Connection Info. Example: Connection Configuration Config for LEGY Servers

  35. Connection Info. Example: Connection Configuration Config for LEGY Servers Fallback

    country Specified country
  36. Connection Info. Example: Connection Configuration Config for LEGY Servers Fallback

    country Client type and version Specified country
  37. Connection Info. Example: Connection Configuration Config for LEGY Servers Fallback

    country Client type and version Network Environment Specified country
  38. Connection Info. Example: Connection Configuration Config for LEGY Servers Fallback

    country Client type and version Network Environment Specified country
  39. Connection Info. How It Works LEGY Repository Client Repository Default

    Conn-info Repository export import LEGY conn-info default conn-info Periodically fetch conn-info w/ local cache Use HTTP/2 with TLS Use SPDY with LEGY Encryption
  40. Migration Process Conn-info HTTP/2 + SPDY Conn-info HTTP/2 + SPDY

    Dogfooding Domain Client SPDY for normal users Conn-info HTTP/2 + SPDY Normal Domain Client HTTP2 for JP only Conn-info HTTP/2 + SPDY Normal Domain Client Fully respect conn-info Dogfooding JP LINE All Regions Beta/RC production
  41. Migration Process Conn-info HTTP/2 + SPDY Conn-info HTTP/2 + SPDY

    Dogfooding Domain Client SPDY for normal users Conn-info HTTP/2 + SPDY Normal Domain Client HTTP2 for JP only Conn-info HTTP/2 + SPDY Normal Domain Client Fully respect conn-info Dogfooding JP LINE All Regions Beta/RC 85% of connections use HTTP/2 production
  42. Streaming Push

  43. Server Push Requirements - It is easy to use with

    any HTTP/2 libraries - The mechanism should be standard conformant - Push can be unsolicited
  44. HTTP/2 Server Push Client Server Request PUSH_PROMISE (B) PUSH_PROMISE (C)

    Response Push Push Stream A Stream B Stream C
  45. HTTP/2 Server Push Client LEGY Subscribe Service Subscribe OK Push

    PUSH_PROMISE (B) OK PUSH_PROMISE (C) Push Stream A Stream B
  46. HTTP/2 Server Push RFC7540 §8.2.1 The PUSH_PROMISE frames sent by

    the server are sent on that explicit request's stream. Client LEGY Subscribe Service Subscribe OK Push PUSH_PROMISE (B) OK PUSH_PROMISE (C) Push Stream A Stream B ❌
  47. Control on The Connection Server Push Requires Connection Controllability LINE

    CLIENT LEGY Connection Lib + OS Subscribe OK PUSH_PROMISE
  48. Control on The Connection Server Push Requires Connection Controllability LINE

    CLIENT LEGY Connection Lib + OS Subscribe OK Subscription succeeded. Server will let me know when data is available. PUSH_PROMISE
  49. Control on The Connection Server Push Requires Connection Controllability LINE

    CLIENT LEGY Lib + OS Client terminated the subscription. Connection Request Response Connection Subscription succeeded. Server will let me know when data is available.
  50. Control on The Connection Server Push Requires Connection Controllability LINE

    CLIENT LEGY Lib + OS Client terminated the subscription. Connection Request Response Has DEAD subscription
  51. Control on The Connection Server Push Requires Connection Controllability LINE

    CLIENT LEGY Lib + OS Connection Request Response The push session must be within a stream, not a connection
  52. Streaming in HTTP/1.1 Chunked Transfer Encoding HTTP/1.1 200 OK Content-Type:

    text/plain Transfer-Encoding: chunked 7\r\n Mozilla\r\n 9\r\n Developer\r\n 7\r\n Network\r\n 0\r\n \r\n * Example from https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding#examples Client Server Request Headers (chunked transfer) 7\r\nMozilla\r\n Data Created 9\r\nDeveloper\r\n Data Created 7\r\nNetwork\r\n Data Ceated 0\r\n\r\n Nothing More
  53. Streaming in HTTP/2 HTTP/2 is streaming E = 0 Welcome

    E = 0 to E = 0 DEV E = 0 DAY E = 1 2021 Sender Receiver Welcome to DEVDAY 2021 7\r\n Welcome\r\n 4\r\n 2021\r\n 2\r\n to\r\n 3\r\n DEV\r\n 3\r\n DAY\r\n 0\r\n \r\n
  54. Streaming in HTTP/2 HTTP/2 is streaming LEGY Client E =

    0 PUSH1 E = 0 PUSH2 E = 0 PUSH3 E = 0 PUSH4 E = 1 PUSH5 - No customization on HTTP/2 - Any libraries can be used - Unsolicited push is available
  55. Streaming Push Stream As A Push Session Client LEGY HEADERS

    HEADERS (status:200) DATA (sign-on) Stream 1 Service1 Service2 Subscribe Subscribe DATA (sign-on results) OK OK PUSH DATA (push) DATA (push ack) Push Session Established
  56. Streaming Push Long Polling in Push Session Client LEGY HEADERS

    HEADERS (status:200) DATA (fetchOps sign-on) Stream 1 Talk fetchOps DATA (sign-on result) Empty publish fetchOps Response OP created DATA (fetchOps sign-on) Push Session Established
  57. TLS

  58. Heavy TLS Handshake Client Server SYN SYN+ACK ACK Application Data

    Plain TCP
  59. Heavy TLS Handshake Client Server SYN SYN+ACK ACK Client Server

    SYN SYN+ACK ClientHello ServerHelloDone Finished Finished Application Data Application Data ACK Plain TCP TLSv1.2
  60. Heavy TLS Handshake Client Server SYN SYN+ACK ACK Client Server

    SYN SYN+ACK ClientHello ServerHelloDone Finished Finished Application Data Application Data ACK Plain TCP TLSv1.2 Additional 2 RTT
  61. TLSv1.3 Less RTT for Handshake Client Server TCP Handshake Client

    Server ClientHello TCP Handshake ServerHello + Finished Finished + Application Data ClientHello + Application Data ServerHello + Finished + ApplicationData Finished + Application Data TLSv1.3 TLSv1.3 0-RTT Additional 1 RTT regardless of Session resumption
  62. TLSv1.3 Less RTT for Handshake Client Server TCP Handshake Client

    Server ClientHello TCP Handshake ServerHello + Finished Finished + Application Data ClientHello + Application Data ServerHello + Finished + ApplicationData Finished + Application Data TLSv1.3 TLSv1.3 0-RTT Additional 1 RTT regardless of Session resumption 20% of connections are using still TLSv1.2
  63. 1 RTT in TLSv1.2 Client Server ClientHello ServerHelloDone Finished Finished

    Application Data TLSv1.2 First Connection TCP Handshake
  64. 1 RTT in TLSv1.2 Client Server ClientHello ServerHelloDone Finished Finished

    Application Data TLSv1.2 First Connection TCP Handshake Client Server TCP Handshake ClientHello ServerHello + Finished Finished + Application Data TLSv1.2 Second Connection Don’t renegotiate
  65. 1 RTT in TLSv1.2 Client Server ClientHello ServerHelloDone Finished Finished

    Application Data TLSv1.2 First Connection TCP Handshake Client Server TCP Handshake ClientHello ServerHello + Finished Finished + Application Data TLSv1.2 Second Connection Session ID & Session Ticket
  66. Session ID LEGY LINE CLIENT LEGY LEGY LEGY First connection

    Store session (ID: 1)
  67. Session ID LEGY LINE CLIENT LEGY LEGY LEGY First connection

    Store session (ID: 1) Resume (ID: 1) No such session Do the full Handshake
  68. Session ID LEGY LINE CLIENT LEGY LEGY LEGY First connection

    Resume (ID: 1) Redis Redis Redis Redis Redis Store session (ID: 1) Retrieve session (ID: 1) Rendezvous Hashing Session Found Handshake is abbreviated
  69. RFC 5077: Session Ticket Handshake LEGY LINE CLIENT LEGY LEGY

    LEGY First connection Old key Current key Future key Generate ticket
  70. RFC 5077: Session Ticket Handshake LEGY LINE CLIENT LEGY LEGY

    LEGY First connection Resume with ticket Old key Current key Future key Generate ticket Decode ticket
  71. RFC 5077: Session Ticket Key Management LEGY LEGY LEGY LEGY

    Old key Current key Future key KMS CronJob 1. Request key gen. 2. Future Keys 3. Update keys 4. Fetch and rotate keys
  72. Results TLSv1.2 TLSv1.2 New TLSv1.2 Session ID TLSv1.2 Ticket

  73. Results TLSv1.3 TLSv1.3 New TLSv1.3 Ticket

  74. Result Statistics TLS 93% TLSv1.3 80% TLSv1.2 Resumption 20%

  75. Summary

  76. LINE 10-year Service and Service Growing For The Next 10

    Years
  77. LINE 10-year Service and Service Growing For The Next 10

    Years Reliability Extensibility Security User Experience
  78. LINE 10-year Service and Service Growing For The Next 10

    Years Reliability Extensibility Security User Experience LE/TCP SPDY Header Cache Custom Push TLS w/ resumption HTTP/2 Streaming Push 93%
  79. LINE 10-year Service and Service Growing For The Next 10

    Years Reliability Extensibility Security User Experience LE/TCP SPDY Header Cache Custom Push TLS w/ resumption HTTP/2 Streaming Push 85%
  80. LINE 10-year Service and Service Growing For The Next 10

    Years Reliability Extensibility Security User Experience LE/TCP SPDY Header Cache Custom Push TLS w/ resumption HTTP/2 Streaming Push
  81. Network Changes, LINE Improves

  82. Thank you