Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LINT: Journey to Network Modernization with HTTP/2 and TLS

LINT: Journey to Network Modernization with HTTP/2 and TLS

LINE DEVDAY 2021

November 10, 2021
Tweet

More Decks by LINE DEVDAY 2021

Other Decks in Technology

Transcript

  1. LINT: Network Modernization
    with HTTP/2 and TLS
    Lee Byeoksan
    LINE+

    View full-size slide

  2. Agenda
    - Background
    - Connectivity Improvement
    - Migration to HTTP/2
    - Streaming Push
    - TLS with Ticket Resumption
    - Summary

    View full-size slide

  3. LINE Messaging Architecture
    LINE
    Client
    LEGY
    (API Gateway)
    Talk Server
    Storage
    Collaborative
    Services
    * LEGY: LINE Event GatewaY

    View full-size slide

  4. LINE Messaging Architecture
    LINE
    Client
    Talk Server
    Storage
    Collaborative
    Services
    * LEGY: LINE Event GatewaY
    LEGY
    (API Gateway)

    View full-size slide

  5. History of LEGY Protocol
    HTTP/1.1: Early Days of LINE
    LINE
    Client
    LEGY
    connection
    connection
    connection
    Request
    Response

    View full-size slide

  6. History of LEGY Protocol
    HTTP/1.1: Early Days of LINE
    Long Polling
    - Mimic real-time event delivery via HTTP

    View full-size slide

  7. History of LEGY Protocol
    HTTP/1.1: Early Days of LINE
    Long Polling
    - Mimic real-time event delivery via HTTP
    Client LEGY
    fetchOps {rev: 1, count: 100}
    Talk
    fetchOps {rev: 1, count: 100}
    No Content
    fetchOps {rev: 1, count: 100}
    publish
    Response
    OP created
    fetchOps {rev: 101, count: 100}
    Response
    OP is the unit of event in LINE

    View full-size slide

  8. History of LEGY Protocol
    SPDY: Since Oct. 2012
    LINE
    Client
    LEGY
    connection
    Request
    Response

    View full-size slide

  9. History of LEGY Protocol
    SPDY: Since Oct. 2012
    Header Compression
    - Headers can be compressed using zlib
    - High server memory usage
    - Couldn’t support compression for some
    connections
    SPDY

    View full-size slide

  10. Header Compression
    History of LEGY Protocol
    SPDY: Since Oct. 2012
    Header Cache
    - Same HTTP headers are transferred repeatedly
    - LEGY holds caches for such headers
    - HTTP request gets smaller
    - Headers can be compressed using zlib
    - High server memory usage
    - Couldn’t support compression for some
    connections
    SPDY
    LINE

    View full-size slide

  11. Header Compression
    History of LEGY Protocol
    SPDY: Since Oct. 2012
    Header Cache
    - Same HTTP headers are transferred repeatedly
    - LEGY holds caches for such headers
    - HTTP request gets smaller
    - Headers can be compressed using zlib
    - High server memory usage
    - Couldn’t support compression for some
    connections
    SPDY
    LINE
    Client LEGY
    header1: value1
    header2: value2
    Server
    header1: value1
    header2: value2
    cache key
    header3: value3
    header1: value1
    header2: value2
    Header3: value3
    cache key

    View full-size slide

  12. History of LEGY Protocol
    SPDY: Since Oct. 2012
    Server Push
    SPDY
    - Enables multiple responses for a single request
    - The pushes are associated with a request
    - The pushes MUST be made before closing the
    request
    - Use case is fetching resources to render web
    pages

    View full-size slide

  13. History of LEGY Protocol
    SPDY: Since Oct. 2012
    Server Push
    Custom SPDY Push
    SPDY
    LINE
    - LINE’s some features and services follow
    subscribe/push pattern
    - The pushes are not associated with a request
    - Original request is closed before the pushes
    - Enables multiple responses for a single request
    - The pushes are associated with a request
    - The pushes MUST be made before closing the
    request
    - Use case is fetching resources to render web
    pages

    View full-size slide

  14. History of LEGY Protocol
    SPDY: Since Oct. 2012
    Server Push
    Custom SPDY Push
    SPDY
    LINE
    - LINE’s some features and services follow
    subscribe/push pattern
    - The pushes are not associated with a request
    - Original request is closed before the pushes
    Client LEGY
    Subscribe
    Service
    Subscribe
    Push
    OK
    OK
    Push (asso. with stream 0)
    Push
    Push (asso. with stream 0)
    - Enables multiple responses for a single request
    - The pushes are associated with a request
    - The pushes MUST be made before closing the
    request
    - Use case is fetching resources to render web
    pages

    View full-size slide

  15. History of LEGY Protocol
    SPDY: Since Oct. 2012
    Server Push
    Custom SPDY Push
    SPDY
    LINE
    - LINE’s some features and services follow
    subscribe/push pattern
    - The pushes are not associated with a request
    - Original request is closed before the pushes
    Note that fetchOps still works through long polling
    - Enables multiple responses for a single request
    - The pushes are associated with a request
    - The pushes MUST be made before closing the
    request
    - Use case is fetching resources to render web
    pages
    Client LEGY
    Subscribe
    Service
    Subscribe
    Push
    OK
    OK
    Push (asso. with stream 0)
    Push
    Push (asso. with stream 0)

    View full-size slide

  16. History of LEGY Protocol
    SPDY: Since Oct. 2012
    Encryption via TLS
    SPDY
    - Use TLS for connection-wise encryption
    - Not mandatory but recommended
    - Overhead for slow network and clients
    - Resulted in bad user experience in 3G

    View full-size slide

  17. Encryption via TLS
    History of LEGY Protocol
    SPDY: Since Oct. 2012
    LEGY Encryption
    - Lightweight in-house encryption method
    - Encrypt message body and sensitive headers
    not whole connection
    - Effective for clients with 3G environment
    SPDY
    LINE
    - Use TLS for connection-wise encryption
    - Not mandatory but recommended
    - Overhead for slow network and clients
    - Resulted in bad user experience in 3G

    View full-size slide

  18. Encryption via TLS
    History of LEGY Protocol
    SPDY: Since Oct. 2012
    LEGY Encryption
    SPDY
    LINE
    - Use TLS for connection-wise encryption
    - Not mandatory but recommended
    - Overhead for slow network and clients
    - Resulted in bad user experience in 3G
    Original LEGY Encryption Applied
    Headers
    Body
    Header part
    Body part
    Sensitive Headers
    +
    Body
    LE Headers
    Non-sensitive
    Headers
    Header part
    Body part
    - Lightweight in-house encryption method
    - Encrypt message body and sensitive headers
    not whole connection
    - Effective for clients with 3G environment

    View full-size slide

  19. Problems
    - There is no standard library to work with our SPDY protocol
    - It makes code management harder
    Outdated Protocol
    Safer but Heavy TLS
    - TLS is safer than LEGY Encryption and gets evolved
    - The handshake is still heavy
    - Our SPDY protocol itself is not conformant to the specification
    - It’s confusing ⇨ Errors, bugs
    Standard Inconformity
    SPDY
    Header Cache Custom Push
    LEGY Encryption

    View full-size slide

  20. LINT: LINE Improvement for Next Ten years
    Outdated SPDY
    HTTP/2
    HTTP/2 Conformant Push
    Streaming
    Push
    Security
    TLS

    View full-size slide

  21. Migration to HTTP/2

    View full-size slide

  22. Bad Cases
    HTTP/2 servers are buggy
    HTTP/2 clients are buggy

    View full-size slide

  23. Bad Cases
    HTTP/2 servers are buggy
    HTTP/2 clients are buggy
    Clients must be configured from the external not to use HTTP/2

    View full-size slide

  24. Bad Cases
    HTTP/2 servers are buggy
    HTTP/2 clients are buggy
    Clients must be configured from the external not to use HTTP/2
    SPDY must be able to be used, instead of HTTP/2, depending on the situation

    View full-size slide

  25. Overview
    LEGY
    V1
    SPDY
    V2
    upgrade
    HTTP/2
    SPDY
    Network
    Abstraction
    Layer

    View full-size slide

  26. Overview
    LEGY
    SPDY
    HTTP/2
    SPDY
    Fallback
    Network
    Abstraction
    Layer
    V1 V2
    upgrade

    View full-size slide

  27. Overview
    LEGY
    SPDY
    HTTP/2
    SPDY
    Disable
    HTTP2
    Network
    Abstraction
    Layer
    V1 V2
    upgrade

    View full-size slide

  28. Overview
    LEGY
    SPDY
    HTTP/2
    SPDY
    Network
    Abstraction
    Layer
    V1 V2
    upgrade

    View full-size slide

  29. Connection Info.
    Means of Network Configuration from LEGY
    Define where to connect
    Define client’s network behaviors (feature on/off, feature parameters)
    Define how to connect

    View full-size slide

  30. Connection Info.
    It’s All About Reliability
    - Must route clients to stable LEGY PoPs.
    LEGY PoP Status
    - Each client has different application
    policy.
    Different Client Types
    - Each country has different network
    characteristics.
    Different Countries
    - As client is updated, features are added
    and removed.
    Different Client Versions

    View full-size slide

  31. Connection Info.
    Example: Connection Configuration

    View full-size slide

  32. Connection Info.
    Example: Connection Configuration
    Config for LEGY Servers

    View full-size slide

  33. Connection Info.
    Example: Connection Configuration
    Config for LEGY Servers
    Fallback country
    Specified country

    View full-size slide

  34. Connection Info.
    Example: Connection Configuration
    Config for LEGY Servers
    Fallback country
    Client type and version
    Specified country

    View full-size slide

  35. Connection Info.
    Example: Connection Configuration
    Config for LEGY Servers
    Fallback country
    Client type and version
    Network
    Environment
    Specified country

    View full-size slide

  36. Connection Info.
    Example: Connection Configuration
    Config for LEGY Servers
    Fallback country
    Client type and version
    Network
    Environment
    Specified country

    View full-size slide

  37. Connection Info.
    How It Works
    LEGY
    Repository
    Client
    Repository
    Default Conn-info
    Repository
    export import
    LEGY
    conn-info
    default
    conn-info
    Periodically fetch conn-info w/ local cache
    Use HTTP/2 with TLS
    Use SPDY with LEGY Encryption

    View full-size slide

  38. Migration Process
    Conn-info
    HTTP/2 + SPDY
    Conn-info
    HTTP/2 + SPDY
    Dogfooding Domain
    Client
    SPDY for normal users
    Conn-info
    HTTP/2 + SPDY
    Normal Domain
    Client
    HTTP2 for JP only
    Conn-info
    HTTP/2 + SPDY
    Normal Domain
    Client
    Fully respect conn-info
    Dogfooding JP LINE All Regions
    Beta/RC
    production

    View full-size slide

  39. Migration Process
    Conn-info
    HTTP/2 + SPDY
    Conn-info
    HTTP/2 + SPDY
    Dogfooding Domain
    Client
    SPDY for normal users
    Conn-info
    HTTP/2 + SPDY
    Normal Domain
    Client
    HTTP2 for JP only
    Conn-info
    HTTP/2 + SPDY
    Normal Domain
    Client
    Fully respect conn-info
    Dogfooding JP LINE All Regions
    Beta/RC
    85% of connections use
    HTTP/2
    production

    View full-size slide

  40. Streaming Push

    View full-size slide

  41. Server Push Requirements
    - It is easy to use with any HTTP/2 libraries
    - The mechanism should be standard conformant
    - Push can be unsolicited

    View full-size slide

  42. HTTP/2 Server Push
    Client Server
    Request
    PUSH_PROMISE (B)
    PUSH_PROMISE (C)
    Response
    Push
    Push
    Stream A
    Stream B
    Stream C

    View full-size slide

  43. HTTP/2 Server Push
    Client LEGY
    Subscribe
    Service
    Subscribe
    OK
    Push
    PUSH_PROMISE (B)
    OK
    PUSH_PROMISE (C)
    Push
    Stream A
    Stream B

    View full-size slide

  44. HTTP/2 Server Push
    RFC7540 §8.2.1
    The PUSH_PROMISE frames sent by the server are sent on that explicit request's
    stream.
    Client LEGY
    Subscribe
    Service
    Subscribe
    OK
    Push
    PUSH_PROMISE (B)
    OK
    PUSH_PROMISE (C)
    Push
    Stream A
    Stream B

    View full-size slide

  45. Control on The Connection
    Server Push Requires Connection Controllability
    LINE
    CLIENT
    LEGY
    Connection
    Lib
    +
    OS
    Subscribe
    OK
    PUSH_PROMISE

    View full-size slide

  46. Control on The Connection
    Server Push Requires Connection Controllability
    LINE
    CLIENT
    LEGY
    Connection
    Lib
    +
    OS
    Subscribe
    OK
    Subscription succeeded.
    Server will let me know when data is available.
    PUSH_PROMISE

    View full-size slide

  47. Control on The Connection
    Server Push Requires Connection Controllability
    LINE
    CLIENT
    LEGY
    Lib
    +
    OS
    Client terminated the subscription.
    Connection
    Request
    Response
    Connection
    Subscription succeeded.
    Server will let me know when data is available.

    View full-size slide

  48. Control on The Connection
    Server Push Requires Connection Controllability
    LINE
    CLIENT
    LEGY
    Lib
    +
    OS
    Client terminated the subscription.
    Connection
    Request
    Response
    Has DEAD subscription

    View full-size slide

  49. Control on The Connection
    Server Push Requires Connection Controllability
    LINE
    CLIENT
    LEGY
    Lib
    +
    OS Connection
    Request
    Response
    The push session must be within a stream, not a connection

    View full-size slide

  50. Streaming in HTTP/1.1
    Chunked Transfer Encoding
    HTTP/1.1 200 OK
    Content-Type: text/plain
    Transfer-Encoding: chunked
    7\r\n
    Mozilla\r\n
    9\r\n
    Developer\r\n
    7\r\n
    Network\r\n
    0\r\n
    \r\n
    * Example from https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding#examples
    Client Server
    Request
    Headers (chunked transfer)
    7\r\nMozilla\r\n
    Data Created
    9\r\nDeveloper\r\n
    Data Created
    7\r\nNetwork\r\n
    Data Ceated
    0\r\n\r\n
    Nothing More

    View full-size slide

  51. Streaming in HTTP/2
    HTTP/2 is streaming
    E = 0
    Welcome
    E = 0
    to
    E = 0
    DEV
    E = 0
    DAY
    E = 1
    2021
    Sender Receiver
    Welcome to DEVDAY 2021
    7\r\n
    Welcome\r\n
    4\r\n
    2021\r\n
    2\r\n
    to\r\n
    3\r\n
    DEV\r\n
    3\r\n
    DAY\r\n
    0\r\n
    \r\n

    View full-size slide

  52. Streaming in HTTP/2
    HTTP/2 is streaming
    LEGY
    Client
    E = 0
    PUSH1
    E = 0
    PUSH2
    E = 0
    PUSH3
    E = 0
    PUSH4
    E = 1
    PUSH5
    - No customization on HTTP/2
    - Any libraries can be used
    - Unsolicited push is available

    View full-size slide

  53. Streaming Push
    Stream As A Push Session
    Client LEGY
    HEADERS
    HEADERS (status:200)
    DATA (sign-on)
    Stream 1 Service1 Service2
    Subscribe
    Subscribe
    DATA (sign-on results)
    OK
    OK
    PUSH
    DATA (push)
    DATA (push ack)
    Push Session Established

    View full-size slide

  54. Streaming Push
    Long Polling in Push Session
    Client LEGY
    HEADERS
    HEADERS (status:200)
    DATA (fetchOps sign-on)
    Stream 1 Talk
    fetchOps
    DATA (sign-on result)
    Empty
    publish
    fetchOps
    Response
    OP created
    DATA (fetchOps sign-on)
    Push Session Established

    View full-size slide

  55. Heavy TLS Handshake
    Client Server
    SYN
    SYN+ACK
    ACK
    Application Data
    Plain TCP

    View full-size slide

  56. Heavy TLS Handshake
    Client Server
    SYN
    SYN+ACK
    ACK
    Client Server
    SYN
    SYN+ACK
    ClientHello
    ServerHelloDone
    Finished
    Finished
    Application Data
    Application Data
    ACK
    Plain TCP TLSv1.2

    View full-size slide

  57. Heavy TLS Handshake
    Client Server
    SYN
    SYN+ACK
    ACK
    Client Server
    SYN
    SYN+ACK
    ClientHello
    ServerHelloDone
    Finished
    Finished
    Application Data
    Application Data
    ACK
    Plain TCP TLSv1.2
    Additional 2 RTT

    View full-size slide

  58. TLSv1.3
    Less RTT for Handshake
    Client Server
    TCP Handshake
    Client Server
    ClientHello
    TCP Handshake
    ServerHello + Finished
    Finished + Application Data
    ClientHello + Application Data
    ServerHello + Finished
    + ApplicationData
    Finished + Application Data
    TLSv1.3 TLSv1.3 0-RTT
    Additional 1 RTT
    regardless of
    Session resumption

    View full-size slide

  59. TLSv1.3
    Less RTT for Handshake
    Client Server
    TCP Handshake
    Client Server
    ClientHello
    TCP Handshake
    ServerHello + Finished
    Finished + Application Data
    ClientHello + Application Data
    ServerHello + Finished
    + ApplicationData
    Finished + Application Data
    TLSv1.3 TLSv1.3 0-RTT
    Additional 1 RTT
    regardless of
    Session resumption
    20% of connections are using still TLSv1.2

    View full-size slide

  60. 1 RTT in TLSv1.2
    Client Server
    ClientHello
    ServerHelloDone
    Finished
    Finished
    Application Data
    TLSv1.2
    First Connection
    TCP Handshake

    View full-size slide

  61. 1 RTT in TLSv1.2
    Client Server
    ClientHello
    ServerHelloDone
    Finished
    Finished
    Application Data
    TLSv1.2
    First Connection
    TCP Handshake
    Client Server
    TCP Handshake
    ClientHello
    ServerHello + Finished
    Finished + Application Data
    TLSv1.2
    Second Connection
    Don’t renegotiate

    View full-size slide

  62. 1 RTT in TLSv1.2
    Client Server
    ClientHello
    ServerHelloDone
    Finished
    Finished
    Application Data
    TLSv1.2
    First Connection
    TCP Handshake
    Client Server
    TCP Handshake
    ClientHello
    ServerHello + Finished
    Finished + Application Data
    TLSv1.2
    Second Connection
    Session ID
    &
    Session Ticket

    View full-size slide

  63. Session ID
    LEGY
    LINE
    CLIENT
    LEGY
    LEGY
    LEGY
    First connection
    Store session (ID: 1)

    View full-size slide

  64. Session ID
    LEGY
    LINE
    CLIENT
    LEGY
    LEGY
    LEGY
    First connection
    Store session (ID: 1)
    Resume (ID: 1)
    No such session
    Do the full Handshake

    View full-size slide

  65. Session ID
    LEGY
    LINE
    CLIENT
    LEGY
    LEGY
    LEGY
    First connection
    Resume (ID: 1)
    Redis
    Redis
    Redis
    Redis Redis
    Store session (ID: 1)
    Retrieve session (ID: 1)
    Rendezvous Hashing
    Session Found
    Handshake is abbreviated

    View full-size slide

  66. RFC 5077: Session Ticket
    Handshake
    LEGY
    LINE
    CLIENT
    LEGY
    LEGY
    LEGY
    First connection
    Old key Current key Future key
    Generate ticket

    View full-size slide

  67. RFC 5077: Session Ticket
    Handshake
    LEGY
    LINE
    CLIENT
    LEGY
    LEGY
    LEGY
    First connection
    Resume with ticket
    Old key Current key Future key
    Generate ticket
    Decode ticket

    View full-size slide

  68. RFC 5077: Session Ticket
    Key Management
    LEGY
    LEGY
    LEGY
    LEGY
    Old key Current key Future key
    KMS
    CronJob
    1. Request key gen.
    2. Future Keys
    3. Update keys
    4. Fetch and rotate keys

    View full-size slide

  69. Results
    TLSv1.2
    TLSv1.2 New
    TLSv1.2 Session ID
    TLSv1.2 Ticket

    View full-size slide

  70. Results
    TLSv1.3
    TLSv1.3 New
    TLSv1.3 Ticket

    View full-size slide

  71. Result
    Statistics
    TLS
    93%
    TLSv1.3
    80%
    TLSv1.2 Resumption
    20%

    View full-size slide

  72. LINE
    10-year Service and Service Growing For The Next 10 Years

    View full-size slide

  73. LINE
    10-year Service and Service Growing For The Next 10 Years
    Reliability Extensibility Security User Experience

    View full-size slide

  74. LINE
    10-year Service and Service Growing For The Next 10 Years
    Reliability Extensibility Security User Experience
    LE/TCP
    SPDY
    Header
    Cache
    Custom
    Push
    TLS w/ resumption
    HTTP/2
    Streaming
    Push
    93%

    View full-size slide

  75. LINE
    10-year Service and Service Growing For The Next 10 Years
    Reliability Extensibility Security User Experience
    LE/TCP
    SPDY
    Header
    Cache
    Custom
    Push
    TLS w/ resumption
    HTTP/2
    Streaming
    Push
    85%

    View full-size slide

  76. LINE
    10-year Service and Service Growing For The Next 10 Years
    Reliability Extensibility Security User Experience
    LE/TCP
    SPDY
    Header
    Cache
    Custom
    Push
    TLS w/ resumption
    HTTP/2
    Streaming
    Push

    View full-size slide

  77. Network Changes, LINE Improves

    View full-size slide