RUN TESTS AT SCALE WITH ON-DEMAND SELENIUM GRID USING AWS FARGATE Presenter: Megha Mehta 

Megha Mehta Senior Automation Engineer S&P Global Market Intelligence, Gurgaon https://www.linkedin.com/in/m-mehta/ SPEAKER [email protected] @meghamehtatech 

PROBLEM DESCRIPTION • Slower feedback • Hardware provisioning challenges • Additional maintenance effort • Challenges with cross-browser testing 

• UI tests take long to run • More expensive • Longer waits for feedback without parallelization SLOWER FEEDBACK 

HARDWARE PROVISIONING CHALLENGES • Performance of under-provisioned hardware suffers under peak loads • Avoid wastage due to over-provisioning by optimizing capacity utilization Application stops responding Infrastructure slow under load Tests failed 

• Frequent software upgrades and patches for system efficiency -Typical maintenance activities – browser upgrades, OS hotfixes, security patches, system restart, disk cleanup etc. • This takes focus away from testing ADDITIONAL MAINTENANCE EFFORT 

CHALLENGES WITH CROSS-BROWSER TESTING CHROME 75 FIREFOX 60 FIREFOX 65 On different versions of same browser On different browsers 

SOLUTION Auto- provisioned with minimal maintenance requirements • Auto-scalable in response to varying load • Reliable and cost-effective Run tests in hyper-parallel mode for faster feedback Dynamic and quick to change 

OUR SOLUTION A dockerized Selenium Grid running on AWS 

TAKEAWAYS • Designing your own high-capacity Selenium Grid using Docker and AWS • A scalable yet disposable grid • Faster feedback – reduce execution time from hours to minutes • Enable Continuous Integration • Major cost savings compared to traditional on-prem infrastructure 

ARCHITECTURE DEEP DIVE 

|WHY DOCKER CONTAINERS? VMs heavyweight Take up lot of memory resources Startup time in minutes Containers lighweight Require lower memory Startup time in seconds 

SELENIUM GRID – HUB SETUP • Hub should have High Bandwidth to support large number of requests from test clients and nodes • High CPU needed for 1000’s of tests but lower memory is fine • In our case, hub is single container of 16GB RAM, 4 vCPU 

SELENIUM GRID – NODE SETUP • Containers of 16 GB RAM, 4 vCPU • Each container can run up to 20 browsers without any performance hit 

Docker Swarm AWS EKS (Kubernetes) AWS ECS/Fargate Do we need to provision instances in advance? Yes Yes No, automatically done by AWS Is it suitable for short workloads? Yes No, master node always up Yes How cost-effective is the solution? * Charged for provisioned EC2’s Additional charges for cluster management Charged only for duration that containers are running *Considering the container cluster only runs for a few hours HOW TO MANAGE SO MANY CONTAINERS? Deciding factors ↓ Orchestration solutions  

CONTAINER ORCHESTRATION USING FARGATE • No need to provision, configure, or scale clusters of machines to run containers – unlike other orchestration solutions • Focus on designing and building Selenium Grid instead of managing the infrastructure that runs it AWS Fargate 

LOAD BALANCER • Selenium Hub is behind an AWS LB which listens to incoming requests from clients and responses from nodes on port 4444 • Domain Name System (DNS) name received by the LB can be quite complex and lengthy - Use a friendly DNS name for load balancer using AWS Route53 AWS Application Load Balancer AWS Route53 

• ECR – AWS Elastic Container Registry to store Docker images • Official selenium-node images modified to expose container IP to hub for registration CONTAINER_IP=$(curl -s http://169.254.170.2/v2/metadata | jq -r '.Containers[1].Networks[0].IPv4Addresses[0]') ECR – REPOSITORY FOR DOCKER IMAGES AWS ECR 

VPC Application Load Balancer Node containers - Chrome Node containers - Firefox Autoscaling Group Amazon ECR Jenkins Node Jenkins node Jenkins Node AWS Fargate Selenium Hub Selenium Grid 

• Monitor container metrics like CPU and memory utilization during tests • Scale out – add 2 node containers whenever CPU Utilization > 80% for 2 min • Scale down to ZERO if idle > 5 mins AUTO-SCALING USING CLOUDWATCH 

VPC Application Load Balancer Node containers - Chrome Node containers - Firefox Autoscaling Group AWS Fargate Selenium Hub Selenium Grid Amazon ECR Jenkins Node Jenkins node Jenkins Node CloudWatch logs Cloudwatch Alarm [Triggered on CPU utilization > 80%] 

CLOUDFORMATION – BRINGING IT ALL TOGETHER • Describe and provision all the infrastructure resources using CloudFormation template • Version control/replicate/update the templates like application code AWS CloudFormation 

*Calculations shown for US East (N. Virginia) Region. Also, costs of other services like ECR, AWS ALB etc. are not included. COST BENEFITS 41 Fargate Containers (1 Hub + 40 Nodes, each of 4vCPU and 16GB configuration) x 10 minutes to complete 800 tests x 5 times a day x 30 days in a month = 1025 hours ~ $238.59 per month* 41 VMs x ~ $80 per VM per month ~ $3280 per month 

COST BENEFITS $39360 $2856 Annual costs Fargate containers On-prem VM That’s a saving of ~93%! 

SUMMARY • Designing your own high-capacity Selenium Grid using Docker and AWS • A scalable yet disposable grid • Faster feedback – reduce execution time from hours to minutes • Enable Continuous Integration • Major cost savings compared to traditional on-prem infrastructure 

• Task networking in Fargate https://aws.amazon.com/blogs/compute/task-networking-in-aws-fargate/ • Comparison between clusters https://www.trinimbus.com/blog/fresh-reinvent-2017-aws-introduces-elastic-container-service- kubernetes-eks-fargate • FARGATE pricing calculator http://fargate-pricing-calculator.site.s3-website-us-east-1.amazonaws.com/ REFERENCES 

CLOUDFORMATION TEMPLATE https://github.com/mmehta-10/selenium-grid- fargate 

THANK YOU!