Benchmarking – lightning.io

My lightning talk about benchmarking.

Transcript

1. BENCHMARKING Paweł Ledwoń @pawel_ledwon Pusher
2. Why?
3. MILLION REASONS Find limits Estimate hardware needs Check high-load stability Test software updates Find optimal hardware Gain confidence in the system Show some numbers to customers
4. …and many more…
5. Performance is a Feature http://www.codinghorror.com/blog/2011/06/performance-is-a-feature.html
6. GOAL give me million connections send tons of messages latency must be below 1s
7. SIMPLE NODE.JS SCRIPT it must work, because it’s: 1. simple 2. node.js 3. script
9. is it node.js? WHATTHEFUCKWENTWRONG? is it amazon? are we io bound? are we cpu bound? are we memory bound? am i rubbish? is it garbage collector? is it ruby? how about kernel? bad day? maybe it’s event machine?... or em-websocket? hmmm... or tcp? or redis? context switching? haproxy? why am i doing this?
10. Why not test it like we test features ?
11. unit testing integration testing
12. unit testing integration testing
13. UNIT TESTING 1. take a single component 2. remove all dependencies 3. test if it works
14. MICROBENCHMARKING 1. take a single component 2. remove all dependencies 3. test if it works
15. is it node.js? WHAT WENT WRONG? is it amazon? are we io bound? are we cpu bound? are we memory bound? am i rubbish? is it garbage collector? is it ruby? how about kernel? bad day? maybe it’s event machine?... or em-websocket? hmmm... or tcp? or redis? context switching? haproxy? why am i doing this?
16. DO MICROBENCHMARKS 1. rule out many parts of the stack 2. find upper boundaries 3. understand the components 4. profile misbehaving parts hell, even start with them
17. don’t do only microbenchmarks!
18. WHAT’S THE ISSUE? • impossible to test glue code • more things running at once • memory/cpu patterns differ • and much more…
19. unit testing integration testing
20. unit testing integration testing
21. INTEGRATION TESTING 1. take all components 2. combine them together 3. test if they work
22. MACROBENCHMARKING 1. take all components 2. combine them together 3. test if they work
23. WHY IS IT DIFFICULT? • too many variables • too many side-effects • hard to isolate anything
24. how do we fight with it?
25. good practices
26. PROPER EXPERIMENTS • know what you are looking for • change one variable at a time • monitor everything • sometimes guess
27. GRAPHS
29. benchmarking framework
30. WHEN IT’S CHEAP • is 1.9.3 faster than 1.9.2? how’s memory? • can we upgrade operating system? • m1.large or c1.xlarge? or maybe dedicated? • how awesome is this em-websocket patch? • did the performance change over the time?
31. …
32. THANKS! @pawel_ledwon