LINT: Journey to Network Modernization with HTTP/2 and TLS

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

LINT: Network Modernization with HTTP/2 and TLS Lee Byeoksan LINE+

Slide 3

Slide 3 text

Agenda - Background - Connectivity Improvement - Migration to HTTP/2 - Streaming Push - TLS with Ticket Resumption - Summary

Slide 4

Slide 4 text

Background

Slide 5

Slide 5 text

LINE Messaging Architecture LINE Client LEGY (API Gateway) Talk Server Storage Collaborative Services * LEGY: LINE Event GatewaY

Slide 6

Slide 6 text

LINE Messaging Architecture LINE Client Talk Server Storage Collaborative Services * LEGY: LINE Event GatewaY LEGY (API Gateway)

Slide 7

Slide 7 text

History of LEGY Protocol HTTP/1.1: Early Days of LINE LINE Client LEGY connection connection connection Request Response

Slide 8

Slide 8 text

History of LEGY Protocol HTTP/1.1: Early Days of LINE Long Polling - Mimic real-time event delivery via HTTP

Slide 9

Slide 9 text

History of LEGY Protocol HTTP/1.1: Early Days of LINE Long Polling - Mimic real-time event delivery via HTTP Client LEGY fetchOps {rev: 1, count: 100} Talk fetchOps {rev: 1, count: 100} No Content fetchOps {rev: 1, count: 100} publish Response OP created fetchOps {rev: 101, count: 100} Response OP is the unit of event in LINE

Slide 10

Slide 10 text

History of LEGY Protocol SPDY: Since Oct. 2012 LINE Client LEGY connection Request Response

Slide 11

Slide 11 text

History of LEGY Protocol SPDY: Since Oct. 2012 Header Compression - Headers can be compressed using zlib - High server memory usage - Couldn’t support compression for some connections SPDY

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Header Compression History of LEGY Protocol SPDY: Since Oct. 2012 Header Cache - Same HTTP headers are transferred repeatedly - LEGY holds caches for such headers - HTTP request gets smaller - Headers can be compressed using zlib - High server memory usage - Couldn’t support compression for some connections SPDY LINE Client LEGY header1: value1 header2: value2 Server header1: value1 header2: value2 cache key header3: value3 header1: value1 header2: value2 Header3: value3 cache key

Slide 14

Slide 14 text

History of LEGY Protocol SPDY: Since Oct. 2012 Server Push SPDY - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages

Slide 15

Slide 15 text

History of LEGY Protocol SPDY: Since Oct. 2012 Server Push Custom SPDY Push SPDY LINE - LINE’s some features and services follow subscribe/push pattern - The pushes are not associated with a request - Original request is closed before the pushes - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages

Slide 16

Slide 16 text

History of LEGY Protocol SPDY: Since Oct. 2012 Server Push Custom SPDY Push SPDY LINE - LINE’s some features and services follow subscribe/push pattern - The pushes are not associated with a request - Original request is closed before the pushes Client LEGY Subscribe Service Subscribe Push OK OK Push (asso. with stream 0) Push Push (asso. with stream 0) - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages

Slide 17

Slide 17 text

History of LEGY Protocol SPDY: Since Oct. 2012 Server Push Custom SPDY Push SPDY LINE - LINE’s some features and services follow subscribe/push pattern - The pushes are not associated with a request - Original request is closed before the pushes Note that fetchOps still works through long polling - Enables multiple responses for a single request - The pushes are associated with a request - The pushes MUST be made before closing the request - Use case is fetching resources to render web pages Client LEGY Subscribe Service Subscribe Push OK OK Push (asso. with stream 0) Push Push (asso. with stream 0)

Slide 18

Slide 18 text

History of LEGY Protocol SPDY: Since Oct. 2012 Encryption via TLS SPDY - Use TLS for connection-wise encryption - Not mandatory but recommended - Overhead for slow network and clients - Resulted in bad user experience in 3G

Slide 19

Slide 19 text

Encryption via TLS History of LEGY Protocol SPDY: Since Oct. 2012 LEGY Encryption - Lightweight in-house encryption method - Encrypt message body and sensitive headers not whole connection - Effective for clients with 3G environment SPDY LINE - Use TLS for connection-wise encryption - Not mandatory but recommended - Overhead for slow network and clients - Resulted in bad user experience in 3G

Slide 20

Slide 20 text

Encryption via TLS History of LEGY Protocol SPDY: Since Oct. 2012 LEGY Encryption SPDY LINE - Use TLS for connection-wise encryption - Not mandatory but recommended - Overhead for slow network and clients - Resulted in bad user experience in 3G Original LEGY Encryption Applied Headers Body Header part Body part Sensitive Headers + Body LE Headers Non-sensitive Headers Header part Body part - Lightweight in-house encryption method - Encrypt message body and sensitive headers not whole connection - Effective for clients with 3G environment

Slide 21

Slide 21 text

Problems - There is no standard library to work with our SPDY protocol - It makes code management harder Outdated Protocol Safer but Heavy TLS - TLS is safer than LEGY Encryption and gets evolved - The handshake is still heavy - Our SPDY protocol itself is not conformant to the specification - It’s confusing ⇨ Errors, bugs Standard Inconformity SPDY Header Cache Custom Push LEGY Encryption

Slide 22

Slide 22 text

LINT: LINE Improvement for Next Ten years Outdated SPDY HTTP/2 HTTP/2 Conformant Push Streaming Push Security TLS

Slide 23

Slide 23 text

Migration to HTTP/2

Slide 24

Slide 24 text

Bad Cases HTTP/2 servers are buggy HTTP/2 clients are buggy

Slide 25

Slide 25 text

Bad Cases HTTP/2 servers are buggy HTTP/2 clients are buggy Clients must be configured from the external not to use HTTP/2

Slide 26

Slide 26 text

Bad Cases HTTP/2 servers are buggy HTTP/2 clients are buggy Clients must be configured from the external not to use HTTP/2 SPDY must be able to be used, instead of HTTP/2, depending on the situation

Slide 27

Slide 27 text

Overview LEGY V1 SPDY V2 upgrade HTTP/2 SPDY Network Abstraction Layer

Slide 28

Slide 28 text

Overview LEGY SPDY HTTP/2 SPDY Fallback Network Abstraction Layer V1 V2 upgrade

Slide 29

Slide 29 text

Overview LEGY SPDY HTTP/2 SPDY Disable HTTP2 Network Abstraction Layer V1 V2 upgrade

Slide 30

Slide 30 text

Overview LEGY SPDY HTTP/2 SPDY Network Abstraction Layer V1 V2 upgrade

Slide 31

Slide 31 text

Connection Info. Means of Network Configuration from LEGY Define where to connect Define client’s network behaviors (feature on/off, feature parameters) Define how to connect

Slide 32

Slide 32 text

Connection Info. It’s All About Reliability - Must route clients to stable LEGY PoPs. LEGY PoP Status - Each client has different application policy. Different Client Types - Each country has different network characteristics. Different Countries - As client is updated, features are added and removed. Different Client Versions

Slide 33

Slide 33 text

Connection Info. Example: Connection Configuration

Slide 34

Slide 34 text

Connection Info. Example: Connection Configuration Config for LEGY Servers

Slide 35

Slide 35 text

Connection Info. Example: Connection Configuration Config for LEGY Servers Fallback country Specified country

Slide 36

Slide 36 text

Connection Info. Example: Connection Configuration Config for LEGY Servers Fallback country Client type and version Specified country

Slide 37

Slide 37 text

Connection Info. Example: Connection Configuration Config for LEGY Servers Fallback country Client type and version Network Environment Specified country

Slide 38

Slide 38 text

Connection Info. Example: Connection Configuration Config for LEGY Servers Fallback country Client type and version Network Environment Specified country

Slide 39

Slide 39 text

Connection Info. How It Works LEGY Repository Client Repository Default Conn-info Repository export import LEGY conn-info default conn-info Periodically fetch conn-info w/ local cache Use HTTP/2 with TLS Use SPDY with LEGY Encryption

Slide 40

Slide 40 text

Migration Process Conn-info HTTP/2 + SPDY Conn-info HTTP/2 + SPDY Dogfooding Domain Client SPDY for normal users Conn-info HTTP/2 + SPDY Normal Domain Client HTTP2 for JP only Conn-info HTTP/2 + SPDY Normal Domain Client Fully respect conn-info Dogfooding JP LINE All Regions Beta/RC production

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Streaming Push

Slide 43

Slide 43 text

Server Push Requirements - It is easy to use with any HTTP/2 libraries - The mechanism should be standard conformant - Push can be unsolicited

Slide 44

Slide 44 text

HTTP/2 Server Push Client Server Request PUSH_PROMISE (B) PUSH_PROMISE (C) Response Push Push Stream A Stream B Stream C

Slide 45

Slide 45 text

HTTP/2 Server Push Client LEGY Subscribe Service Subscribe OK Push PUSH_PROMISE (B) OK PUSH_PROMISE (C) Push Stream A Stream B

Slide 46

Slide 46 text

HTTP/2 Server Push RFC7540 §8.2.1 The PUSH_PROMISE frames sent by the server are sent on that explicit request's stream. Client LEGY Subscribe Service Subscribe OK Push PUSH_PROMISE (B) OK PUSH_PROMISE (C) Push Stream A Stream B ❌

Slide 47

Slide 47 text

Control on The Connection Server Push Requires Connection Controllability LINE CLIENT LEGY Connection Lib + OS Subscribe OK PUSH_PROMISE

Slide 48

Slide 48 text

Control on The Connection Server Push Requires Connection Controllability LINE CLIENT LEGY Connection Lib + OS Subscribe OK Subscription succeeded. Server will let me know when data is available. PUSH_PROMISE

Slide 49

Slide 49 text

Control on The Connection Server Push Requires Connection Controllability LINE CLIENT LEGY Lib + OS Client terminated the subscription. Connection Request Response Connection Subscription succeeded. Server will let me know when data is available.

Slide 50

Slide 50 text

Control on The Connection Server Push Requires Connection Controllability LINE CLIENT LEGY Lib + OS Client terminated the subscription. Connection Request Response Has DEAD subscription

Slide 51

Slide 51 text

Control on The Connection Server Push Requires Connection Controllability LINE CLIENT LEGY Lib + OS Connection Request Response The push session must be within a stream, not a connection

Slide 52

Slide 52 text

Streaming in HTTP/1.1 Chunked Transfer Encoding HTTP/1.1 200 OK Content-Type: text/plain Transfer-Encoding: chunked 7\r\n Mozilla\r\n 9\r\n Developer\r\n 7\r\n Network\r\n 0\r\n \r\n * Example from https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding#examples Client Server Request Headers (chunked transfer) 7\r\nMozilla\r\n Data Created 9\r\nDeveloper\r\n Data Created 7\r\nNetwork\r\n Data Ceated 0\r\n\r\n Nothing More

Slide 53

Slide 53 text

Streaming in HTTP/2 HTTP/2 is streaming E = 0 Welcome E = 0 to E = 0 DEV E = 0 DAY E = 1 2021 Sender Receiver Welcome to DEVDAY 2021 7\r\n Welcome\r\n 4\r\n 2021\r\n 2\r\n to\r\n 3\r\n DEV\r\n 3\r\n DAY\r\n 0\r\n \r\n

Slide 54

Slide 54 text

Streaming in HTTP/2 HTTP/2 is streaming LEGY Client E = 0 PUSH1 E = 0 PUSH2 E = 0 PUSH3 E = 0 PUSH4 E = 1 PUSH5 - No customization on HTTP/2 - Any libraries can be used - Unsolicited push is available

Slide 55

Slide 55 text

Streaming Push Stream As A Push Session Client LEGY HEADERS HEADERS (status:200) DATA (sign-on) Stream 1 Service1 Service2 Subscribe Subscribe DATA (sign-on results) OK OK PUSH DATA (push) DATA (push ack) Push Session Established

Slide 56

Slide 56 text

Streaming Push Long Polling in Push Session Client LEGY HEADERS HEADERS (status:200) DATA (fetchOps sign-on) Stream 1 Talk fetchOps DATA (sign-on result) Empty publish fetchOps Response OP created DATA (fetchOps sign-on) Push Session Established

Slide 57

Slide 57 text

TLS

Slide 58

Slide 58 text

Heavy TLS Handshake Client Server SYN SYN+ACK ACK Application Data Plain TCP

Slide 59

Slide 59 text

Heavy TLS Handshake Client Server SYN SYN+ACK ACK Client Server SYN SYN+ACK ClientHello ServerHelloDone Finished Finished Application Data Application Data ACK Plain TCP TLSv1.2

Slide 60

Slide 60 text

Heavy TLS Handshake Client Server SYN SYN+ACK ACK Client Server SYN SYN+ACK ClientHello ServerHelloDone Finished Finished Application Data Application Data ACK Plain TCP TLSv1.2 Additional 2 RTT

Slide 61

Slide 61 text

TLSv1.3 Less RTT for Handshake Client Server TCP Handshake Client Server ClientHello TCP Handshake ServerHello + Finished Finished + Application Data ClientHello + Application Data ServerHello + Finished + ApplicationData Finished + Application Data TLSv1.3 TLSv1.3 0-RTT Additional 1 RTT regardless of Session resumption

Slide 62

Slide 62 text

Slide 63

Slide 63 text

1 RTT in TLSv1.2 Client Server ClientHello ServerHelloDone Finished Finished Application Data TLSv1.2 First Connection TCP Handshake

Slide 64

Slide 64 text

1 RTT in TLSv1.2 Client Server ClientHello ServerHelloDone Finished Finished Application Data TLSv1.2 First Connection TCP Handshake Client Server TCP Handshake ClientHello ServerHello + Finished Finished + Application Data TLSv1.2 Second Connection Don’t renegotiate

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Session ID LEGY LINE CLIENT LEGY LEGY LEGY First connection Store session (ID: 1)

Slide 67

Slide 67 text

Session ID LEGY LINE CLIENT LEGY LEGY LEGY First connection Store session (ID: 1) Resume (ID: 1) No such session Do the full Handshake

Slide 68

Slide 68 text

Session ID LEGY LINE CLIENT LEGY LEGY LEGY First connection Resume (ID: 1) Redis Redis Redis Redis Redis Store session (ID: 1) Retrieve session (ID: 1) Rendezvous Hashing Session Found Handshake is abbreviated

Slide 69

Slide 69 text

RFC 5077: Session Ticket Handshake LEGY LINE CLIENT LEGY LEGY LEGY First connection Old key Current key Future key Generate ticket

Slide 70

Slide 70 text

RFC 5077: Session Ticket Handshake LEGY LINE CLIENT LEGY LEGY LEGY First connection Resume with ticket Old key Current key Future key Generate ticket Decode ticket

Slide 71

Slide 71 text

RFC 5077: Session Ticket Key Management LEGY LEGY LEGY LEGY Old key Current key Future key KMS CronJob 1. Request key gen. 2. Future Keys 3. Update keys 4. Fetch and rotate keys

Slide 72

Slide 72 text

Results TLSv1.2 TLSv1.2 New TLSv1.2 Session ID TLSv1.2 Ticket

Slide 73

Slide 73 text

Results TLSv1.3 TLSv1.3 New TLSv1.3 Ticket

Slide 74

Slide 74 text

Result Statistics TLS 93% TLSv1.3 80% TLSv1.2 Resumption 20%

Slide 75

Slide 75 text

Summary

Slide 76

Slide 76 text

LINE 10-year Service and Service Growing For The Next 10 Years

Slide 77

Slide 77 text

LINE 10-year Service and Service Growing For The Next 10 Years Reliability Extensibility Security User Experience

Slide 78

Slide 78 text

LINE 10-year Service and Service Growing For The Next 10 Years Reliability Extensibility Security User Experience LE/TCP SPDY Header Cache Custom Push TLS w/ resumption HTTP/2 Streaming Push 93%

Slide 79

Slide 79 text

LINE 10-year Service and Service Growing For The Next 10 Years Reliability Extensibility Security User Experience LE/TCP SPDY Header Cache Custom Push TLS w/ resumption HTTP/2 Streaming Push 85%

Slide 80

Slide 80 text

LINE 10-year Service and Service Growing For The Next 10 Years Reliability Extensibility Security User Experience LE/TCP SPDY Header Cache Custom Push TLS w/ resumption HTTP/2 Streaming Push