Distributed Load Testing Using Locust and Kubernetes

Slide 1

Slide 1 text

Distributed Load Testing Using Locust and Kubernetes Aditya Satrya Head of Engineering at Jabar Digital Service

Slide 2

Slide 2 text

Outline 2 ▷ Load Testing ▷ Locust ▷ Kubernetes

Slide 3

Slide 3 text

Load Testing 3

Slide 4

Slide 4 text

4 Why?

Slide 5

Slide 5 text

5 https://www.thinkwithgoogle.com/marketing-resources/data-measurement/mobile-page-speed-new-industry-benchmarks/

Slide 6

Slide 6 text

6 Latency increases as request grows

Slide 7

Slide 7 text

Common Basic Metrics 7 ▷ Latency (Response Time) ▷ Throughput (Request / second)

Slide 8

Slide 8 text

8 Percentiles

Slide 9

Slide 9 text

Lead to decisions 9 ▷ Does the server have enough resources (CPU, memory, etc.) to handle the anticipated load? ▷ Does the server respond quickly enough to provide a good user experience? ▷ Is our application running efﬁciently? ▷ Do we need to scale up our server hardware, or scale out to multiple servers? ▷ Are there any pages or API calls that are particularly resource intensive?

Slide 10

Slide 10 text

Locust 10

Slide 11

Slide 11 text

11 ▷ Open source (MIT license) ▷ Supports Python 2.7, 3.5, 3.6, 3.7, 3.8 ▷ https://locust.io/ ▷ https://github.com/locustio/locust

Slide 12

Slide 12 text

What I like about Locust 12 ▷ Deﬁne user behaviour in Python code ▷ Distributed ▷ Run in command line or from UI ▷ Exportable result to CSV

Slide 13

Slide 13 text

13 Example

Slide 14

Slide 14 text

Kubernetes 14

Slide 15

Slide 15 text

15 Why?

Slide 16

Slide 16 text

16 Generate enough traﬃc

Slide 17

Slide 17 text

What is Kubernetes? 17 ▷ Open-source system for automating: ○ deployment ○ scaling ○ management of containerized applications

Slide 18

Slide 18 text

Master & Node Architecture 18

Slide 19

Slide 19 text

19 How Kubernetes works?

Slide 20

Slide 20 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers

Slide 21

Slide 21 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers

Slide 22

Slide 22 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest

Slide 23

Slide 23 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest

Slide 24

Slide 24 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest frontend: 4/5 X

Slide 25

Slide 25 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest frontend: 4/5

Slide 26

Slide 26 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest frontend: 5/5

Slide 27

Slide 27 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest Run 3 instances of backend:latest

Slide 28

Slide 28 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest Run 3 instances of backend:latest Run 10 instances of analytics:latest analytics: 7/10 frontend: 4/5 backend: 3/3

Slide 29

Slide 29 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest Run 3 instances of backend:latest Run 10 instances of analytics:latest

Slide 30

Slide 30 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Master 1 Master 2 Master 3 Workers Run 5 instances of frontend:latest Run 3 instances of backend:latest Run 10 instances of analytics:latest

Slide 31

Slide 31 text

Slide 32

Slide 32 text

32 Networking aspect

Slide 33

Slide 33 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5

Slide 34

Slide 34 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 10.x.x.x label: analytics Services

Slide 35

Slide 35 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 10.x.x.x label: analytics 10.x.x.x label: backend 10.x.x.x label: frontend Services

Slide 36

Slide 36 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 10.x.x.x label: analytics 10.x.x.x label: backend 10.x.x.x label: frontend Services Internet

Slide 37

Slide 37 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 10.x.x.x label: analytics 10.x.x.x label: backend 10.x.x.x label: frontend Cloud Load Balancer Services Internet

Slide 38

Slide 38 text

Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 10.x.x.x label: analytics 10.x.x.x label: backend 10.x.x.x label: frontend Cloud Load Balancer stats.example.com api.example.com example.com Services Internet

Slide 39

Slide 39 text

Demo 39 https://github.com/asatrya/locust_k8s

Slide 40

Slide 40 text

40 Locust Cluster Target App Cluster Loads traffic

Slide 41

Slide 41 text

Target URLs 41 ▷ /proﬁle ▷ /login ▷ /article/1 ▷ /article/2 ▷ /article/3 ▷ /article/4 ▷ /article/5 ▷ /article/6 (will return 404) ▷ /article/7 (will return 404) ▷ /bottleneck (slow endpoint)

Slide 42

Slide 42 text

Integrate with CI/CD (not in demo yet) 42 ▷ Setup cluster ▷ Deploy Locust master & slaves ▷ Run test ▷ Wait ▷ Stop test ▷ Read the result

Slide 43

Slide 43 text

43 # setup cluster # ... commands here … # start test curl -X POST -F 'locust_count=4' -F 'hatch_rate=4' http://locust-master:8089/swarm {"success": true, "message": "Swarming started"} # run test for 2 minutes sleep 2m # stop test curl http://locust-master:8089/stop {"success": true, "message": "Test stopped"} # download result curl http://locust-master:8089/stats/requests/csv # cleanup cluster gcloud container clusters delete locust-cluster