Mux: How I stopped worrying and learned to love the multiplexing.

From USENIX SREcon15 – March 16–17, 2015 | Santa Clara, CA

At its core, Mux is a generic RPC multiplexing protocol created at Twitter. As a strictly OSI Layer 5 session protocol, it can be used in conjunction with protocols from other layers. We'll discuss the motivation for creating a session protocol as well as gains such as elimination of head-of-line blocking, explicit queue management, and better networking economics.

Video recording of this talk is available at https://www.usenix.org/conference/srecon15/program/presentation/demir

Transcript

1. Mux How I stopped worrying and learned to love the multiplexing.
2. Berk D. Demir @bd
3. Today, we’ll talk about RPC Multiplexing.
4. OSI 7 Layer Model
5. Physical Data Link Network Transport Session Presentation Application IEEE 802.3 Physical IEEE 802.3 MAC IP TCP HTTP TLS (a bit of TLS)
6. What’s the problem?
7. Head of Line Blocking Not every interaction is a single RPC. Calls for discrete resources should not block each other
8. What’s the big deal? Just open more TCP connections!
9. Free as in beer New socket connections are not free as in resources and latency. (Remember TCP 3-way handshake)
10. … also TCP Slow Start In the data center Across data centers From the POPs
11. For every TCP connection You have a separate network queue and the half of separate liveness detection logic.
12. TCP Keepalives. …or a blast from October 1989 Not more frequent than one every 2 hours. It’s the kernel, not the application.
13. What about? Liveness Detection Request Cancellation (cost of tearing down a TCP connection is too damn high!) Availability Advertisement
14. …also Control Plane vs Data Plane Separate these so we can have out-of-band messaging(node-to-node) without affecting data plane.
15. Back to those 7 Layers
16. Physical Data Link Network Transport Session Presentation Application IEEE 802.3 Physical IEEE 802.3 MAC IP TCP Thrift
17. These concerns could be addressed purely in Layer 5. We needed a session protocol.
18. Explicit Queue Management NACKs Leases Proper back pressure signaling
19. Interesting Use Cases Destination Dispatch GC Avoidance Service-to-service Authentication awk for Distributed Systems
20. Destination Dispatch Intelligent routing and load balancing for less intelligent clients.
21. Destination Dispatch unicorn 1 unicorn 3 unicorn 2 muxproxy mux router service x service y service z web host
22. Destination Dispatch Mux routers at the data center edge for load, global incident status and preference aware RPC routing.
23. GC Avoidance We can easily predict a young generation collection. If we can gracefully drain all our clients via leases, why worry about GC pauses?
24. RPC Authentication (Lessons from HTTP) Authenticating every single RPC is expensive. Implementing AAA in application or network layer is disruptive. Let’s address the concern in the session layer.
25. RPC Authentication (Lessons from HTTP) expensive: HTTP Basic/Digest disruptive: Modify L7, IPsec session layer: Implement your own with X.509, Kerberos, etc.
26. Debugging Distributed Systems is HARD. Production Readiness Reviews are tough. Production is real life and it is wild.
27. RPC TAP (awk for Distributed Systems) inject failure simulate latency, backpressure rewrite destinations
28. More Mux is part of Finagle and it’s open source.
29. Related HTTP/2 & QUIC
30. There must be questions! @bd
31. Jeff Dean Numbers L1 cache ref: 1ns Branch mispredict: 3ns L2 cache reference: 4ns Mutex lock/unlock: 17ns Main memory reference: 100ns Send 2000 bytes over the network: 400ns Compress 1K with Zippy: 2,000ns Read 1MB from memory (seq): 12,000ns SSD random read: 16,000ns Read 1M from SSD: 200,000ns RTT in the same DC: 400,000ns RTT from SMF1-to-ATLA: 80,000,000ns RTT from Sacramento-to-Amsterdam: 150,000,000ns http://www.eecs.berkeley.edu/~rcs/research/