Powering Test Environments with Amazon EKS using Serverless Tool

#awsugchn #AWSCommunityDay

Transcript

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CHENNAI
2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Powering Test Environments with Amazon EKS using Serverless Tool KARTHIKEYAN VENKATRAMAN | August 10, 2019
3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Subscription management platform that helps untangle recurring revenue businesses from spaghetti billing. • Hosted on AWS Cloud. 120 + Currencies 20+ Payment Gateways 8,500 + Customers 50 + Countries Employees 300+
4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BUSINESS CHALLENGE With growing development teams, our key business challenge was to improve the release cycle time. Spin up Test Environments on the fly to empower Integration Testing across various microservices
5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Chargebee Application Stack
6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Monolithic application • Multiple internal microservices • Hosted on EC2 Instances • In-house deployment tool
7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Development Time Local • Feature, Bug Fixes • Unit Testing 2-3 Days DEV • Manual and Integrations Test • Database Migrations • Feature Demos for Product Teams 2-3 Hours STAGING • Database Migrations • Sanity Testing • Patch Fix Go Live PRODUCTION • Database Migrations • Rolling Upgrades • GO Live 1-2 Hours • Functional Tests(covers 60% of use cases) • Local Database Schema Generation • Parallel Test executions • Go or No-Go for Dev QA (Automated) Deployment Stages and Timelines:
8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Bottleneck with Dev Majority of the features that land in dev took a minimum of two days to progress to Staging. LOCAL I’m waiting for the DEV :( DEV STG PROD
9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Migrations & Validations Team Collaborations & Dependencies Testing & Bug Fixes Infra Changes & Environment Updates Delay in Dev
10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Solution: Launch Dynamic PREDEV Environments LOCAL I’m in Predev and will release to Prod Shortly DEV STG PROD PRE DEV PRE DEV PRE DEV PRE DEV
11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 1. Immutable Setup 2. Parallelize Build 3. Perform Database Migrations with Production alike Schema 4. Onboard New Services quickly as they evolve 5. Low Operational Overhead 6. CI/CD 7. Security & Compliance Solution Requirements
12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Reclone Dev environments with EC2 instances. • Containers. Options
13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EC2 INSTANCES CONTAINERS We opted to go with containers using AWS EKS. Pros: • Templated setup • Identical to other environments. Cons: • Infrastructure costs. • Operational overheads. Pros: • Build for the future. • Low operational overhead. • Low infra costs. Cons: • Infrastructure is different from other environments.
14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Elastic Kubernetes Service (EKS) • Prior experience with Kubernetes • Managed control plane service • Active community support • Cloud agnostic WORKER NODES VPC EKS AZ-1 AZ-2 AZ-3
15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TOOLS FUNCTIONALITY PURPOSE AWS Cloudformation Infrastructure as a Code Provision infrastructure resources in AWS Cloud Environment. AWS Elastic Container Service for Kubernetes (EKS) Container Orchestration Engine Deploy, manage and scale Container applications. AWS Elastic Container Registry (ECR) Container Registry Store and manage Docker Container images AWS ALB Ingress Controller Ingress Controller (Maintained & Supported by AWS) To create application load balancer which routes HTTP[s] traffic across different endpoints. Tool Kit
16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TOOLS FUNCTIONALITY PURPOSE KUBE2IAM Fine Grained IAM permissions to Pods Provides IAM Credentials for Container to make calls to AWS Service based on the Pod level Roles. ExternalDNS DNS mapping for Kubernetes Services and Ingresses(Community Project) Kubernetes Services and Ingress is mapped to AWS Route53 for DNS resolution. HELM Kubernetes Application Manager(CNCF Project) Manage Kubernetes Applications using charts. Create, Version Control and Publish Kubernetes Resources. Jenkins Open Source Automation Tool Used as Build Server with various Pipeline Jobs which helps to provision Predev Environment by communicating with EKS and ECR
17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • EKS control plane for cluster management • Cloudformation for VPC and EKS Worker Nodes Set up • Docker Images in AWS ECR • ALB Ingress Controller for Load Balancing ◦ One per predev environment. • External DNS and Route53 for DNS • Kube2IAM for Policy management • Cloudwatch for Monitoring. Solution Overview WORKER NODES VPC EKS AZ-1 AZ-2 RDS (1..N) DYNAMODB ALB (1..N) ROUTE 53 ECR IAM S3 CLOUDWATCH CLOUDFORMATION
18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Leveraging ALB with Kubernetes Ingress Controller
19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. KUBE PROXY AWS-NODE-VPC PD1(APP) PD2(UI-CDNOTES) PD3(ELASTICMQ) PD1(AUDIT) PD1(ELASTICMQ) KUBE PROXY KUBE2IAM TILLER PD1(UI-INV) PD2(DB-MIG) AWS-NODE-VPC EXTERNALDNS ALB-INGRESS CONTROLLER KUBERNETES CLUSTER KUBERNETES APISERVER KUBE2IAM NODEPORT NODEPORT NODEPORT Rule Host: *.pd1.in Rule Host: *.audits.pd1.in Rule Host: *.ui.pd1.in Path: /invoices APP-Target Group(Instance Mode) AUIDIT-Target Group(Instance Mode) UI-INV-Target Group(Instance Mode) APPLICATION LOAD BALANCER Ingress Resource Creation using HELM CHARTS DAEMONSET PODS APPLICATION PODS EKS-WORKER-NODE EKS-WORKER-NODE
20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Minions - Self Service Portal for Predev Setup
21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • To ease developers to quickly provision Predev environments. • Helps : ◦ Provision/De-Provision environments ◦ CI/CD ◦ Track build and deployment status ◦ Control access Minions (Serverless Tool)
22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Minions Serverless Architecture 1. Single page application (SPA). 2. Frontend served by Cloudfront with S3 origin.(vue.js). 3. AWS Cognito with User Pool for authentication. 4. API Gateway backed by Lambda function used to perform server side functions. 5. Leveraging AWS WAF with Cloudfront and API Gateway for additional security. 6. Lambda Function used SSM RUN Command to trigger Jenkins commands. 7. Jenkins installs/updates/deletes environment in EKS using Helm charts. JENKINS SSM RUN COMMAND LAMBDA API GATEWAY S3 CLOUDFRONT DYNAMODB COGNITO DEVELOPERS SQS SNS
23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CI/CD For Predev
24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DEVELOPERS GITHUB JENKINS ECR EKS Developers commit to their feature branch Docker images pushed to ECR Jenkins polls the branch for changes EKS pulls the images from ECR to setup pods Jenkins updates the pods using helm charts CI/CD For Predev DOCKER IMAGES Jenkins builds the docker images
25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deployment Stages and Timelines (Post Predev Implementation): Development Time LOCAL • Feature ,Bug Fixes. • Unit Testing 2-3 Hours DEV • Database Migrations • Sanity Test 2-3 Hours STAGING • Database migrations • Sanity Testing • Patch Fix Go Live PRODUCTION • Database Migrations. • Rolling Upgrades • GO Live 1-2 Hours • Functional Tests • Local Database Schema Generation • Parallel Test Executions • Go or No-Go for Dev QA(Automated) Development Time PREDEV • Feature Testing ,Bug Fixes. • Database Migration • Functional and Integration Tests • User Acceptance Tests
26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank You