Load testing UK’s biggest food publisher using AWS & Artillery

João Tiago (JT) Software Engineer @ YLD @johnytiago

The Client ● Publishing Industry ● 15K+ peak active users ● PHP + JS Stack ● Legacy codebase ● Lack of good practices

The Engage ● 6 months project ● Initial Engagement ● Improve DevOps processes ● Build CI/CD pipelines ● Help set up Infrastructure as Code

● 4 months into the project ● The client was going to release a new version of the application. ● Dozens of new services. ● No assurance it would handle production load The Spinoff

The Problem Release of major version of the app, running on a new infrastructure. ● Lots of features being implemented ● No considerations regarding performance ● Teams struggled with ownership

Load testing engine integrated with the CI pipeline Maintainability ● We won’t be at the client for long. ● It as to be easy to understand; Scalability ● Has to produce production-like load. Solution Requirements

The Solution

Tests Runner Why Artillery? ● Flexible and lightweight. ● Allows writing tests using templated YAML. ● Access to Artillery's creator.

Tests Runner Why Artillery? ● Per-environment configuration ● Define SLOs and make tests fail if these are not met.

Project Structure ● Includes all of the tests’ dependencies; ● Delimited ownership; ● Easier packaging and shipping. scenarios/example-api ├── local │ ├── user-ids.csv │ └── song-ids.csv ├── preproduction │ ├── user-ids.csv │ └── song-ids.csv ├── overwrites.env # Env var configuration ├── setup.sh # Setup script executed once before the tests ├── teardown.sh # Teardown script ├── helpers.js # Scenario helper functions ├── config.yml # Service endpoints and SLOs configuration └── spec.yml # Scenario description

Project Structure Single Github Repo ● Dozens services; ● Stored infra configuration; ● Github Actions: ● Build and push Docker image to ECR; ● Scenarios synced to S3.

13 aws s3 cp --recursive s3://$SCENARIOS_S3_BUCKET/$SERVICE/ . source overwrites.env npx artillery run --config config.yaml -e $ENVIRONMENT spec.yml

14 Scaling Load Tests Increased the # of Artillery containers Deployed to an ECS Cluster, through a Lambda Function. You’ll get “High CPU”, Bruh

CI Integration CodeBuild Step ● Has the service’s config which passes to the lambda; ● Step locking for free; ● Great place for beforeAll & afterAll hooks.

16

Inspiration Reference: https://aws.amazon.com/solutions/implementations/distributed-load-testing-on-aws/

18 Local Setup

Results

20 Results 10 RPS +100 RPS Results of running the load tests against a single instance container locally on 2Gb RAM 1vCPU

Concluding Remarks

What Was Good ● Extremely exciting project to work on; ● Utilised novel and refreshing tools; ● Exposed and helped fix countless critical performance issues before launch; ● Multiple teams were able to update their tests independently.

Opportunities For Improvement ● We started testing later then we should. ● We wrote the first set of test for all services. It was tedious . ● We're not actually testing the production environment.

Key Takeaways ● Start testing as soon as you can. ● Run the load tests locally. ● Make writing load tests part of your development lifecycle. ● Make load testing part of your CI pipeline. ● The best way to test production, is by testing production.

Thank you. Any questions? 25