Blue/Green Deployments

Real-world Automated Blue/Green Deployment
By David Goodall - Director, SysAdmin and DevOps consultant at MySysAdmin Ltd.

Abstract
Featuring AWS and Chef to provision a Blue/Green production Drupal architecture.Co-starring Bash scripts, Varnish, CloudFormation, RDS, Elasticache, Autoscaling, ELB, and HA NAT.

Transcript

1. Real World Blue/Green Deployment David Goodall MySysAdmin.co.uk
2. Who has used: • AWS • Autoscaling • Chef • Chef Server • Blue / Green deployment • Other deployment systems? Which?
3. What is Blue/Green Deployment?
4. What is “Real-World”?
5. The Brief • Create new VPC for a website • Use AWS (eu-central-1) • Make it more robust, maintainable, (quicker?) • Use Chef • Apply DevOps principles (infrastructure as code)
6. The Options • Terraform (not available for eu-central-1 due to v4 signing) • Ansible (VPC modules not production ready) • Cloud formation (JSON, 56k limit) • Chef provisioning (not production ready)
7. The Options • Terraform (not available for eu-central-1 due to v4 signing) • Ansible (VPC modules not production ready) • Cloud formation (JSON, 56k limit) • Chef provisioning (not production ready) • BASH scripts and Chef
11. | MGMT | BLUE | GREEN —————————————————————————————————— PUB a | 10.100.0.0/27 | 10.100.10.0/27 | 10.100.20.0/27 PUB b | 10.100.0.32/27 | 10.100.10.32/27 | 10.100.20.32/27 ——————————————————————————————————- PRIV a | 10.100.1.0/27 | 10.100.11.0/27 | 10.100.21.0/27 PRIV b | 10.100.1.32/27 | 10.100.11.32/27 | 10.100.21.32/27 ——————————————————————————————————- PERSIST a | 10.100.2.0/27 | 10.100.12.0/27 | 10.100.22.0/27 PERSIST b | 10.100.2.32/27 | 10.100.12.32/27 | 10.100.22.32/27 VPC: 10.100.0.0/16 18 subnets: • Room for 30 instances in each subnet, with 21 more stacks, 7 more tiers and 6 more subnets (AZ’s) in each tier/stack available for expansion. • Network ACLs used to keep traffic within stacks, ie control IPs not ports • Security groups used to control ports, not IPs • Easier to maintain
12. Inside the VPC, infrastructure is controlled by BASH scripts and Chef 2 bash scripts are used to create a stack: • create-varnish-autoscale.sh • create-drupal-autoscale.sh (also create-ha-nat-autoscale.sh to create the NAT infrastructure) • AWS CLI commands to create RDS instance, load balancers, elasticache clusters (instead of local memcache) launch configurations and autoscaling groups • Tags and userdata hook into Chef • S3 used for keys, website and DB content • Instances are assigned an IAM role to enable them to access S3 resources and for possible future stuff (instances cannot be given roles after creation) Stack Creation
13. Demo
14. Chef server: • Manually set up • Encrypted databag created • Roles, Environments defined Varnish cookbook: • Installs and configures varnish • Cron+BASH+AWS CLI to monitor backends and update config Drupal cookbook: • Master & slave recipes • Website content and config is downloaded from S3 and modified according to Chef attributes • DB user is created, DB is populated from S3 • lsyncd is used to replicate content to slave(s) • S3 used for master public key • Slave uses Cron+BASH+AWS CLI to monitor for changes • Cron+BASH+AWS CLI used to monitor for slave and varnish instances Chef roles
15. • Each instance has to be thought of as ephemeral, but they need to know about each other. • Drupal master and slave need to have the same config and content • Drupal needs to know about the Varnish server(s) • Varnish needs to know about the Drupal server(s) • Any instance can be blown away and replaced at any time • Autoscaling: there can potentially be 1,2,3,10 or 100 of any instance Cattle, not pets (but how do they commoonicate?)
16. Roles, Environments, Stacks etc Can get confusing and are used to mean specific things by different people / products • Vanish / Drupal master / Drupal slave = ROLE • Live / Staging = ENV • Blue / Green = STACK In Blue/Green, both stacks should be identical (apart from version) BUT staging has different requirements! • Locked down access • Robots.txt - do not index! Used a bolt-on recipe to make changes for staging - needs to be removed from runlist when switching to live.
17. Demo
18. Questions?
19. Thanks • Andy Gale - Bristol DevOps • Chris Hawes - Basekit
20. David Goodall Email: david@mysysadmin.co.uk Web: www.mysysadmin.co.uk LinkedIn: dgoodall Twitter: @MySysAdm1n Facebook: MySysAdmin G+: +MysysadminCoUk Experienced SysAdmin, DevOps and IT Consultancy Services