Pushing Python: Lessons Learned Building a High Throughput Service in Python by Kevin Ballard

Taba is a distributed metrics aggregator, similar in concept to statsd. Built with Python using Redis, gevent, and Cython, it currently handles over 6M events/sec with strong consistency guarantees. This talk will present an overview of its design, and discuss the challenges and solutions encountered in the process of building a high throughput, low latency distributed service.

Transcript

1. Kevin  Ballard   SFpython.org   2014-­‐03-­‐12  
2. kevin@   tellapart.com   Introductions
3. Taba •  Distributed event aggregation service import taba ... taba.RecordValue(‘winning_bid_price’, wincpm) ... $ taba-cli aggregate winning_bid_price {“name”: “winning_bid_price”, “10m”: {“count”: 14709, “total”: 5836.4}, “percentiles”: [0.07 0.16 0.32 0.84 1.33 8.03]}
4. Taba +10,000,000   events/sec   +50,000   metrics   +1,000   clients   +100   processors  
5. GET THE DATA MODEL RIGHT Lesson #1
6. Data Model
7. Data Model Event:  (‘bid_cpm’,  ‘Counter’,  time(),  0.233)       State:             Aggregate:  {“10m”:  43.9,  “1h”:  592.22}    
8. Data Model
9. Data Model
10. Data Model
11. STATE IS HARD Lesson #2
12. Centralizing State
13. GENERATORS + GREENLETS = AWESOME Lesson #3
14. Asynchronous Iterator •  JIT processing •  Automatically switches through I/O
15. CPYTHON SUFFERS FROM MEMORY FRAGMENTATION Lesson #4
16. Fragmentation •  Fragmentation is when a process’s heap is inefficiently used. •  The GC may report a low memory footprint, but the OS reports a much larger RSS.
17. Fragmentation
18. Fragmentation
19. Fragmentation
20. Fragmentation
21. Fragmentation
22. Hybrid Memory Management •  Use Cython to allocate page-sized blocks of pointers into incoming chunk •  Hand-off the whole thing to the CPython memory manager •  Whole thing gets deallocated at once
23. Hybrid Memory Management
24. Hybrid Memory Management
25. Hybrid Memory Management
26. Ratcheting •  Ratcheting is a pathological case of Fragmentation, caused by the fact that the heap must be contiguous*: •  It’s a limitation of CPython that it cannot compact memory (mostly due to extensions).
27. Ratcheting •  Ratcheting is a pathological case of Fragmentation, caused by the fact that the heap must be contiguous*: •  It’s a limitation of CPython that it cannot compact memory (mostly due to extensions).
28. Ratcheting •  Ratcheting is a pathological case of Fragmentation, caused by the fact that the heap must be contiguous*: •  It’s a limitation of CPython that it cannot compact memory (mostly due to extensions).
29. Ratcheting •  Ratcheting is a pathological case of Fragmentation, caused by the fact that the heap must be contiguous*: •  It’s a limitation of CPython that it cannot compact memory (mostly due to extensions).
30. Ratcheting •  Avoid persistent objects •  Sockets are common offenders •  Anything that has to be persistent should be created at application startup, before processing data •  Avoid letting the heap grow in the first place
31. fin. github.com/tellapart/taba       kevin@tellapart.com      |    @misterkgb     We’re  Hiring!      tellapart.com/careers