Run tests at scale with on-demand Selenium Grid using AWS Fargate

My talk from DSTC 2019 edition based on a cloud-hosted container solution to run 1000s of tests in an on-demand, scalable Selenium Grid.

YouTube link of talk- https://www.youtube.com/watch?v=npSlm1YUp-Q

Blog - https://meghamehta.tech/2020/07/13/run-selenium-tests-in-containers-using-aws-ecs-fargate/

Transcript

1. RUN TESTS AT SCALE WITH ON-DEMAND SELENIUM GRID USING AWS FARGATE Presenter: Megha Mehta
2. Megha Mehta Senior Automation Engineer S&P Global Market Intelligence, Gurgaon https://www.linkedin.com/in/m-mehta/ SPEAKER mmehta.ten@gmail.com @meghamehtatech
3. PROBLEM DESCRIPTION • Slower feedback • Hardware provisioning challenges • Additional maintenance effort • Challenges with cross-browser testing
4. • UI tests take long to run • More expensive • Longer waits for feedback without parallelization SLOWER FEEDBACK
5. HARDWARE PROVISIONING CHALLENGES • Performance of under-provisioned hardware suffers under peak loads • Avoid wastage due to over-provisioning by optimizing capacity utilization Application stops responding Infrastructure slow under load Tests failed
6. • Frequent software upgrades and patches for system efficiency -Typical maintenance activities – browser upgrades, OS hotfixes, security patches, system restart, disk cleanup etc. • This takes focus away from testing ADDITIONAL MAINTENANCE EFFORT
7. CHALLENGES WITH CROSS-BROWSER TESTING CHROME 75 FIREFOX 60 FIREFOX 65 On different versions of same browser On different browsers
8. SOLUTION Auto- provisioned with minimal maintenance requirements • Auto-scalable in response to varying load • Reliable and cost-effective Run tests in hyper-parallel mode for faster feedback Dynamic and quick to change
9. OUR SOLUTION A dockerized Selenium Grid running on AWS
10. TAKEAWAYS • Designing your own high-capacity Selenium Grid using Docker and AWS • A scalable yet disposable grid • Faster feedback – reduce execution time from hours to minutes • Enable Continuous Integration • Major cost savings compared to traditional on-prem infrastructure
11. ARCHITECTURE DEEP DIVE
12. |WHY DOCKER CONTAINERS? VMs heavyweight Take up lot of memory resources Startup time in minutes Containers lighweight Require lower memory Startup time in seconds
13. SELENIUM GRID – HUB SETUP • Hub should have High Bandwidth to support large number of requests from test clients and nodes • High CPU needed for 1000’s of tests but lower memory is fine • In our case, hub is single container of 16GB RAM, 4 vCPU
14. SELENIUM GRID – NODE SETUP • Containers of 16 GB RAM, 4 vCPU • Each container can run up to 20 browsers without any performance hit
15. Docker Swarm AWS EKS (Kubernetes) AWS ECS/Fargate Do we need to provision instances in advance? Yes Yes No, automatically done by AWS Is it suitable for short workloads? Yes No, master node always up Yes How cost-effective is the solution? * Charged for provisioned EC2’s Additional charges for cluster management Charged only for duration that containers are running *Considering the container cluster only runs for a few hours HOW TO MANAGE SO MANY CONTAINERS? Deciding factors ↓ Orchestration solutions 
16. CONTAINER ORCHESTRATION USING FARGATE • No need to provision, configure, or scale clusters of machines to run containers – unlike other orchestration solutions • Focus on designing and building Selenium Grid instead of managing the infrastructure that runs it AWS Fargate
17. LOAD BALANCER • Selenium Hub is behind an AWS LB which listens to incoming requests from clients and responses from nodes on port 4444 • Domain Name System (DNS) name received by the LB can be quite complex and lengthy - Use a friendly DNS name for load balancer using AWS Route53 AWS Application Load Balancer AWS Route53
18. • ECR – AWS Elastic Container Registry to store Docker images • Official selenium-node images modified to expose container IP to hub for registration CONTAINER_IP=$(curl -s http://169.254.170.2/v2/metadata | jq -r '.Containers[1].Networks[0].IPv4Addresses[0]') ECR – REPOSITORY FOR DOCKER IMAGES AWS ECR
19. VPC Application Load Balancer Node containers - Chrome Node containers - Firefox Autoscaling Group Amazon ECR Jenkins Node Jenkins node Jenkins Node AWS Fargate Selenium Hub Selenium Grid
20. • Monitor container metrics like CPU and memory utilization during tests • Scale out – add 2 node containers whenever CPU Utilization > 80% for 2 min • Scale down to ZERO if idle > 5 mins AUTO-SCALING USING CLOUDWATCH
21. VPC Application Load Balancer Node containers - Chrome Node containers - Firefox Autoscaling Group AWS Fargate Selenium Hub Selenium Grid Amazon ECR Jenkins Node Jenkins node Jenkins Node CloudWatch logs Cloudwatch Alarm [Triggered on CPU utilization > 80%]
22. CLOUDFORMATION – BRINGING IT ALL TOGETHER • Describe and provision all the infrastructure resources using CloudFormation template • Version control/replicate/update the templates like application code AWS CloudFormation
23. *Calculations shown for US East (N. Virginia) Region. Also, costs of other services like ECR, AWS ALB etc. are not included. COST BENEFITS 41 Fargate Containers (1 Hub + 40 Nodes, each of 4vCPU and 16GB configuration) x 10 minutes to complete 800 tests x 5 times a day x 30 days in a month = 1025 hours ~ $238.59 per month* 41 VMs x ~ $80 per VM per month ~ $3280 per month
24. COST BENEFITS $39360 $2856 Annual costs Fargate containers On-prem VM That’s a saving of ~93%!
25. SUMMARY • Designing your own high-capacity Selenium Grid using Docker and AWS • A scalable yet disposable grid • Faster feedback – reduce execution time from hours to minutes • Enable Continuous Integration • Major cost savings compared to traditional on-prem infrastructure
26. • Task networking in Fargate https://aws.amazon.com/blogs/compute/task-networking-in-aws-fargate/ • Comparison between clusters https://www.trinimbus.com/blog/fresh-reinvent-2017-aws-introduces-elastic-container-service- kubernetes-eks-fargate • FARGATE pricing calculator http://fargate-pricing-calculator.site.s3-website-us-east-1.amazonaws.com/ REFERENCES
27. CLOUDFORMATION TEMPLATE https://github.com/mmehta-10/selenium-grid- fargate
28. THANK YOU!