Low Latency Systems: How do they do it?

An overview of tips and tricks from low latency system design and the hardware trends that drive them.

Transcript

1. Low Latency Systems How do they do it?
2. What is Low Latency?
3. LMAX Exchange Average trade latency 500 microseconds
4. How Do They Do It?
5. Fundamentals
6. Work asynchronously
7. Work in parallel
8. Keep it all in memory
9. Keep it all on one machine
10. Aside: Modern CPUs
11. Aside: Modern CPUs
12. Hardware Trends
13. “Latency Lags Bandwidth”
14. “Latency Lags Bandwidth”
15. How Do They Deal With It?
16. Prefetch Data • Disk Readahead ◦ Data on disk is prefetched into memory ◦ Masks latency if data is accessed sequentially • Cache Prefetching ◦ Data in memory is prefetched into CPU cache ◦ Masks latency if data is accessed sequentially
17. Sequential I/O • Arrays ◦ Best when used with primitive or value types ◦ Open Addressing - represent a hash table in an array ◦ Ring Buffer - represent a queue in an array • Write Ahead Log ◦ Most databases, many filesystems ◦ Logging for crash recovery
18. Sequential I/O • Log Structured Merge Trees ◦ Riak, Cassandra, HBase ◦ The log is the database
19. Batch Write • Adaptive Flushing ◦ Queue data to be written ◦ Flushing thread loops flushing the queue ◦ Adaptively increases the size of the writes under heavy load • Group Commit / Write Coalescing ◦ MySQL, PostgreSQL, SSD/HDD Controllers ◦ Combine concurrent operations into one write
20. Distributed System in a Box
21. Distributed System in a Box
22. How Do They Deal With It?
23. Shared Nothing Architecture • Logically run one instance of the application per core • Linux provides the ability to ◦ Pin threads to specific cores ◦ Give each core it’s own queue for disk and network I/O • Minimize cross-core communication ◦ Partition data such that processing can occur on one core
24. Shared Nothing Architecture
25. Lightning Round
26. CPU Optimizations
27. CPU Optimizations • Lock Contention ◦ Atomic CAS (Compare and Swap) operations ◦ Lock free data structures • Context Switching ◦ One thread per core • False Sharing ◦ Keep hot data in its own “cache line” • Branch Misprediction ◦ Branchless algorithms (bit manipulation tricks)
28. Fin