Slide 1

Slide 1 text

Load testing UK’s biggest food publisher using AWS & Artillery

Slide 2

Slide 2 text

João Tiago (JT) Software Engineer @ YLD @johnytiago

Slide 3

Slide 3 text

The Client ● Publishing Industry ● 15K+ peak active users ● PHP + JS Stack ● Legacy codebase ● Lack of good practices

Slide 4

Slide 4 text

The Engage ● 6 months project ● Initial Engagement ● Improve DevOps processes ● Build CI/CD pipelines ● Help set up Infrastructure as Code

Slide 5

Slide 5 text

● 4 months into the project ● The client was going to release a new version of the application. ● Dozens of new services. ● No assurance it would handle production load The Spinoff

Slide 6

Slide 6 text

The Problem Release of major version of the app, running on a new infrastructure. ● Lots of features being implemented ● No considerations regarding performance ● Teams struggled with ownership

Slide 7

Slide 7 text

Load testing engine integrated with the CI pipeline Maintainability ● We won’t be at the client for long. ● It as to be easy to understand; Scalability ● Has to produce production-like load. Solution Requirements

Slide 8

Slide 8 text

The Solution

Slide 9

Slide 9 text

Tests Runner Why Artillery? ● Flexible and lightweight. ● Allows writing tests using templated YAML. ● Access to Artillery's creator.

Slide 10

Slide 10 text

Tests Runner Why Artillery? ● Per-environment configuration ● Define SLOs and make tests fail if these are not met.

Slide 11

Slide 11 text

Project Structure ● Includes all of the tests’ dependencies; ● Delimited ownership; ● Easier packaging and shipping. scenarios/example-api ├── local │ ├── user-ids.csv │ └── song-ids.csv ├── preproduction │ ├── user-ids.csv │ └── song-ids.csv ├── overwrites.env # Env var configuration ├── setup.sh # Setup script executed once before the tests ├── teardown.sh # Teardown script ├── helpers.js # Scenario helper functions ├── config.yml # Service endpoints and SLOs configuration └── spec.yml # Scenario description

Slide 12

Slide 12 text

Project Structure Single Github Repo ● Dozens services; ● Stored infra configuration; ● Github Actions: ● Build and push Docker image to ECR; ● Scenarios synced to S3.

Slide 13

Slide 13 text

13 aws s3 cp --recursive s3://$SCENARIOS_S3_BUCKET/$SERVICE/ . source overwrites.env npx artillery run --config config.yaml -e $ENVIRONMENT spec.yml

Slide 14

Slide 14 text

14 Scaling Load Tests Increased the # of Artillery containers Deployed to an ECS Cluster, through a Lambda Function. You’ll get “High CPU”, Bruh

Slide 15

Slide 15 text

CI Integration CodeBuild Step ● Has the service’s config which passes to the lambda; ● Step locking for free; ● Great place for beforeAll & afterAll hooks.

Slide 16

Slide 16 text

16

Slide 17

Slide 17 text

Inspiration Reference: https://aws.amazon.com/solutions/implementations/distributed-load-testing-on-aws/

Slide 18

Slide 18 text

18 Local Setup

Slide 19

Slide 19 text

Results

Slide 20

Slide 20 text

20 Results 10 RPS +100 RPS Results of running the load tests against a single instance container locally on 2Gb RAM 1vCPU

Slide 21

Slide 21 text

Concluding Remarks

Slide 22

Slide 22 text

What Was Good ● Extremely exciting project to work on; ● Utilised novel and refreshing tools; ● Exposed and helped fix countless critical performance issues before launch; ● Multiple teams were able to update their tests independently.

Slide 23

Slide 23 text

Opportunities For Improvement ● We started testing later then we should. ● We wrote the first set of test for all services. It was tedious . ● We're not actually testing the production environment.

Slide 24

Slide 24 text

Key Takeaways ● Start testing as soon as you can. ● Run the load tests locally. ● Make writing load tests part of your development lifecycle. ● Make load testing part of your CI pipeline. ● The best way to test production, is by testing production.

Slide 25

Slide 25 text

Thank you. Any questions? 25