Testing for capacity

How does Spotify plan capacity for their micro service architecture? What are the challenges and the tools Spotify uses? Here's short travel into HTTP part of that.

http://www.meetup.com/Prestandaforum-Stockholm/events/203673172/

Transcript

1. testing for capacity for
2. Sergey Vysotskiy vsp@spotify.com
3. 3ELEMENTS OF PERFORMANCE
4. Client
5. Internet Weather Client
6. Internet Weather Client Your Pretty Servers
7. NEXT 30 MINUTES: HOW TO COUNT SERVERS
8. NEXT 30 MINUTES: HOW TO COUNT SERVERS GOAL INFLUENCE ON TESTS
9. NEXT 30 MINUTES: HOW TO COUNT SERVERS GOAL INFLUENCE ON TESTS ENVIRONMENT INFLUENCE ON TESTS
10. NEXT 30 MINUTES: HOW TO COUNT SERVERS GOAL INFLUENCE ON TESTS ENVIRONMENT INFLUENCE ON TESTS SHINY TOOLS WE LIKE
11. ENVIRONMENT ENVIRONMENT https://www.flickr.com/photos/gcwest/6575768243/
12. Monolithic
13. Micro Services Monolithic
16. 100+ Services
17. 100+ Services Frequent Changes
18. 100+ Services Frequent Changes Simplify!
19. Simplify!
20. Math (pv2/2)+p=const Do the
21. There’s exceptions
22. Weird exceptions
23. MOCK all the things
24. Mock should be FAST
25. Mock should be FLEXIBLE
26. DO THE DECOMPOSITION If problem looks really hard
27. ENVIRONMENT IMPACT ON TESTS https://www.flickr.com/photos/coofdy/12989245854/
28. What do we test for? CAPACITY!
29. We need System limits UX limits Time To Restore
30. Test LESS
31. Test LESS
32. Test LESS
33. Test LESS
34. Test LESS
35. VIRTUAL USERS Let’s define tests with
36. VIRTUAL USERS Remember decomposition?
37. THE ONLY THING THAT SERVER KNOW REQUEST DISTRIBUTION
38. THE ONLY THING THAT SERVER KNOW CONNECTIONS NUMBER
39. HIT-BASED MODEL Such wow! Very rescue!
40. 10% of A 40% of B 50 % of C 10000 RPS
41. 10% of A 40% of B 50 % of C 10000 RPS Test data
42. 10% of A 40% of B 50 % of C 10000 RPS Test itself
43. Input data: Monitoring
44. Input data: Experience
45. WHY HIT-BASED MODEL IS AWESOME? 3 reasons
46. 1. Easy to analyze.
47. 2. Easy to change.
48. 3. Easy to plan with.
49. SMALL SECRET* *Don’t try with stateful protocols.
50. SMALL SECRET* *Still works for stateful applications.
51. HIT-BASED MODEL Such wow! Very rescue!
52. ENVIRONMENT TOOLS https://www.flickr.com/photos/ebarney/3348965637/
53. IN-HOUSETools Protocols
54. IN-HOUSETools Protocols HTTP
55. JMeter: designed for VU
56. JMeter: OK with plugins
57. JMeter: wow RPS shaping
58. JMeter: preformance :-(
59. Yandex.Tank: some history
60. address=203.0.113.1 port=80 rps_schedule=const(10, 10m) header_http = 1.1 headers = [Host: www.target.example.com] [Connection: close] ammofile=/tmp/access.log ammo_type=access
61. address=203.0.113.1 port=80 rps_schedule=const(10, 10m) header_http = 1.1 headers = [Host: www.target.example.com] [Connection: close] ammofile=/tmp/access.log ammo_type=access
62. address=203.0.113.1 port=80 rps_schedule=const(10, 10m) header_http = 1.1 headers = [Host: www.target.example.com] [Connection: close] ammofile=/tmp/access.log ammo_type=access
63. address=203.0.113.1 port=80 rps_schedule=const(10, 10m) header_http = 1.1 headers = [Host: www.target.example.com] [Connection: close] ammofile=/tmp/access.log ammo_type=access
64. address=203.0.113.1 port=80 rps_schedule=const(10, 10m) header_http = 1.1 headers = [Host: www.target.example.com] [Connection: close] ammofile=/tmp/access.log ammo_type=access
65. address=203.0.113.1 port=80 rps_schedule=const(10, 10m) header_http = 1.1 headers = [Host: www.target.example.com] [Connection: close] uris = / /lekjfvnieoqr?fvjgr=werb245
66. Yandex.Tank: in action
67. RAW DATA + R/PANDAS/WHATEVER
68. Yandex.Tank: built-in reports
69. Yandex.Tank: built-in reports
70. Production Monitoring if it’s good enough
71. Automate all the things!
72. Target LoadGen Monitoring Report Jenkins
73. Sergey Vysotskiy vsp@spotify.com Thanks! https://www.flickr.com/photos/30389420@N08/15073997375/