Moving to production

Transcript

1. Moving to production Fabian Król Live Chat
2. Live Chat
3. system
4. project • simple proxy: chat window <—> main chat server • http <—> protobuf • rewrite existing component
5. traffic requirements • 6M req/min • 700M MAU • 1.9 Gbps (95 percentile) • 3 Gbps DDoS
6. WHY ERLANG
7. WHY ERLANG pure experiment!!!1111oneone
8. it’s ready • 9 months of development • 1 student • 204 commits • 9260 insertions(+)
9. it’s ready we just need to move it to production
10. it’s ready we just need to move it to production
11. it’s ready we just need to move it to production
12. #1: test • from • no tests • to • few unit test for critical functions • public api integration testing • CI on every commit (travis)
13. #2: release • release quick and release often • independent builds • easy to track • easy to rebuild old versions • one line deployment • master + develop + build from tag
14. #3: debug • make it easy to debug on production
15. #4: monitoring • system level monitoring (erld) • erlang vm is not really unix-friendly • zabbix • os level monitoring • app level alerting • graphite • app level metrics
16. feature requirements • tests • metrics
17. moving to production
18. how to replace existing system • blue green deployment • canary release √
19. how to replace existing system • step 1: 2% of traffic with sticky session • •
20. how to replace existing system • step 1: 2% of traffic with sticky session • step 2: 2% of traffic without sticky session •
21. how to replace existing system • step 1: 2% of traffic with sticky session • step 2: 2% of traffic without sticky session • step 3: 0% of traffic for 1.5 month
22. – unknown artist “It’s hard to rewrite systems.”
23. it’s hard • reproduce all features • reproduce all bugs
24. - PreparedBody = re:replace(lists:droplast(Body), "\"", "\\\\\"", [global]), + Body1 = lists:droplast(Body), + Body2 = re:replace(Body1, "\\\\", "\\\\\\\\", [global]), + PreparedBody = re:replace(Body2, "\"", "\\\\\"", [global]),
25. hosts • hardware √ | cloud • virtualization √ | bare metal • 4 √ | 8 | 16 | 32 cpus
26. hosts: tunning • hardware √ | cloud • virtualization | bare metal √ • 4 | 8 √ | 16 | 32 cpus
27. hardware comparison • 1.3x more cpus to handle traffic • (we can go up to 1.16) • memory is not important
28. code comparison • erlang: 5515 LOC • c++: 15681 LOC
29. 35% ✨
30. 100% • 6 months • 600 commits • 7521 insertions(+) • 5902 deletions(-)
31. – unknown artist “It’s hard to rewrite systems.”
32. erlang pros • it’s easy to write code • small codebase • it’s easy to collect metric (?) • it allows to debug and trace on live production host
33. WHY ERLANG
34. WHY ERLANG *it works under a heavy traffic*
35. schedulers • one per vm thread (one per cpu) • does it owns memory management • reductions: • process is re-scheduled after N reductions • no process can block the system
36. GC • each process have private heap • per proces GC • it's quick
37. ⚡