Distributed Load Testing Using Locust and Kubernetes

This deck is presented at PyCon Indonesia 2019.

Transcript

1. Distributed Load Testing Using Locust and Kubernetes Aditya Satrya Head of Engineering at Jabar Digital Service
2. Outline 2 ▷ Load Testing ▷ Locust ▷ Kubernetes
3. Load Testing 3
4. 4 Why?
5. 5 https://www.thinkwithgoogle.com/marketing-resources/data-measurement/mobile-page-speed-new-industry-benchmarks/
6. 6 Latency increases as request grows
7. Common Basic Metrics 7 ▷ Latency (Response Time) ▷ Throughput (Request / second)
8. 8 Percentiles
9. Lead to decisions 9 ▷ Does the server have enough resources (CPU, memory, etc.) to handle the anticipated load? ▷ Does the server respond quickly enough to provide a good user experience? ▷ Is our application running efficiently? ▷ Do we need to scale up our server hardware, or scale out to multiple servers? ▷ Are there any pages or API calls that are particularly resource intensive?
10. Locust 10
11. 11 ▷ Open source (MIT license) ▷ Supports Python 2.7, 3.5, 3.6, 3.7, 3.8 ▷ https://locust.io/ ▷ https://github.com/locustio/locust
12. What I like about Locust 12 ▷ Define user behaviour in Python code ▷ Distributed ▷ Run in command line or from UI ▷ Exportable result to CSV
13. 13 Example
14. Kubernetes 14
15. 15 Why?
16. 16 Generate enough traffic
17. What is Kubernetes? 17 ▷ Open-source system for automating: ◦ deployment ◦ scaling ◦ management of containerized applications
18. Master & Node Architecture 18
19. 19 How Kubernetes works?
20. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers
21. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers
22. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest
23. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest
24. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest frontend: 4/5 X
25. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest frontend: 4/5
26. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest frontend: 5/5
27. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest Run 3 instances of backend:latest
28. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest Run 3 instances of backend:latest Run 10 instances of analytics:latest analytics: 7/10 frontend: 4/5 backend: 3/3
29. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest Run 3 instances of backend:latest Run 10 instances of analytics:latest
30. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest Run 3 instances of backend:latest Run 10 instances of analytics:latest
31. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest Run 3 instances of backend:latest Run 10 instances of analytics:latest analytics: 10/10 frontend: 5/5 backend: 3/3
32. 32 Networking aspect
33. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5
34. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 10.x.x.x label: analytics Services
35. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 10.x.x.x label: analytics 10.x.x.x label: backend 10.x.x.x label: frontend Services
36. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 10.x.x.x label: analytics 10.x.x.x label: backend 10.x.x.x label: frontend Services Internet
37. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 10.x.x.x label: analytics 10.x.x.x label: backend 10.x.x.x label: frontend Cloud Load Balancer Services Internet
38. Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 10.x.x.x label: analytics 10.x.x.x label: backend 10.x.x.x label: frontend Cloud Load Balancer stats.example.com api.example.com example.com Services Internet
39. Demo 39 https://github.com/asatrya/locust_k8s
40. 40 Locust Cluster Target App Cluster Loads traffic
41. Target URLs 41 ▷ /profile ▷ /login ▷ /article/1 ▷ /article/2 ▷ /article/3 ▷ /article/4 ▷ /article/5 ▷ /article/6 (will return 404) ▷ /article/7 (will return 404) ▷ /bottleneck (slow endpoint)
42. Integrate with CI/CD (not in demo yet) 42 ▷ Setup cluster ▷ Deploy Locust master & slaves ▷ Run test ▷ Wait ▷ Stop test ▷ Read the result
43. 43 # setup cluster # ... commands here … # start test curl -X POST -F 'locust_count=4' -F 'hatch_rate=4' http://locust-master:8089/swarm {"success": true, "message": "Swarming started"} # run test for 2 minutes sleep 2m # stop test curl http://locust-master:8089/stop {"success": true, "message": "Test stopped"} # download result curl http://locust-master:8089/stats/requests/csv # cleanup cluster gcloud container clusters delete locust-cluster