Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MMT Edge

MMT Edge

Deployments Redesigned!

Paritosh

April 10, 2019
Tweet

Other Decks in Programming

Transcript

  1. About Me Paritosh Anand – Lead DevOps @ MMT •

    Python Programmer • Docker Propellent • Inclined towards – CI / CD, Monitoring, AWS
  2. Agenda 1. MakeMyTrip’s journey towards achieving 100% deployment automation. 2.

    Platform agnostic approach for Deployments. 3. Blue Green deployments. 4. Importance of CANARY & Staggered deployments 5. Tracking deployments & ensuring compliance. 6. Empowering DEV teams.
  3. Features 1. Multi Eco-system support & pluggable. 2. Agentless (Python’s

    Fabric) 3. JIRA & CANARY Integrated 4. Horizontally Scalable 5. DC Redundant 6. Dockerized 7. Logs Storage @Elasticsearch 8. Fast & Graphical Reports
  4. Why CANARY? ü Safety Net ü Fail Fast ü Faster

    Feedback ü Max. Damage Control ü Secure Deployment Strategy
  5. Tech Stack & Architecture •Python • Django (Edge-UI) • Fabric

    (Remote Exec) • Celery (Task Exec) • Boto (AWS Integration) •Rabbit MQ (Task Queue) •Elastic Search (Logs/Reports) •MySQL (Config DB) •D3 (Visualization)
  6. Integration approach with Jira • Change control management • Tagging

    deployment tickets with dev tickets • Better compliance management • Scheduled deployments • Auto deployment trigger once change is approved.
  7. Staggered or Zero Downtime Deployments • Approach – Datacenters •

    Graceful traffic drain by taking servers OOS from F5 Load Balancer • Health Check validations before enabling traffic. • Canary metric comparison is based on server level comparison • Batch creation for parallel deployment execution • Approach – AWS Cloud (Blue/Green Deployments) • Ensures graceful termination of instances via ASG • Create replica green ASG from blue ASG • Canary metric comparison is based on deployment version
  8. Fabric – Remote Command Execution • Fabric – designed to

    execute shell commands remotely • Fabric uses SSH protocol • Agentless – No need to run agent service on production nodes • Benefits • Ability to execute commands in parallel i.e. on multiple servers at once • Sanitization of commands before execution • Setup and tracking is easy for SYS Admins • Extending Fabric for property deployments.
  9. Env. Agnostic Projects • Environments • Logical definitions of environments

    like AWS, DC, AWS-ECS, STORM • Health checks attributes • Projects • Canary comparison is on or off • Notifications are on or off • Project + Env level configs • Details of servers • Jira ID for tracking
  10. Action lifecycle | Plans | Env. Agnostic • Implementation approach

    for Actions • Implementation of Actions for different environments are defined exclusively. • Highly customized deployment strategy can be developed for each environment. • Support of Hard Deploy, Rollback and Restarts • Plans • Deploy (Hard Deploy) • Restart (Hard Restart) • Rollback (Hard Rollback)