REA Shipper @ Infracoders

Some details of rea-shipper, REA's deployment tooling for Docker-based apps

Transcript

1. REA Shipper Mike Williams May 2015 @Infracoders Melbourne
2. About me ex- Kiwi ex- ThoughtWorks ex- Lonely Planet ex- Cogent ex- Unix Sysadmin ex- Ruby/Java/Perl Dev ex- Agile ex- Consultant currently Product Manager for Delivery Engineering for (one bit of) REA Group
3. some REA context then now separate “Delivery” and “Operations multi-disciplinary teams ~ 8 teams ~ 40 teams a few monoliths (and lots of scripts) micro-services! debs, RPMs debs, RPMs, AMIs, gems, tarballs, Docker! 2 data-centers lots of AWS accounts
4. problem <app/> solution!
5. deployment monitoring logging packaging <app/> problem solution!
6. deployment monitoring logging packaging <app/> problem solution! Waste!
7. <app/>
8. <app/> configuration
9. <app/> configuration requests
10. <app/> configuration logs requests
11. <app/> configuration logs requests
12. <app/> configuration logs requests provides the abstraction we need
13. 1. Codebase - One codebase tracked in revision control, many deploys 2. Dependencies - Explicitly declare and isolate dependencies 3. Config - Store config in the environment 4. Backing Services - Treat backing services as attached resources 5. Build, release, run - Strictly separate build and run stages 6. Processes - Execute the app as one or more stateless processes 7. Port binding - Export services via port binding 8. Concurrency - Scale out via the process model 9. Disposability - Maximize robustness with fast startup and graceful shutdown 10. Dev/prod parity - Keep development, staging, and production as similar as possible 11. Logs - Treat logs as event streams 12. Admin processes - Run admin/management tasks as one-off processes from http://12factor.net The Twelve Factors
14. “PaaS-shaped” apps • adopt “12 Factor App” conventions • package/distribute as Docker images Just-in-time (half-assed) PaaS • provides a subset of typical PaaS features • usable in existing REA AWS environments
15. rea-shipper.yml CloudFormation template CloudFormation stack rea-shipper AWS CloudFormation
16. ASG ELB
17. <app/> $CONFIG nginx log collector requests Splunk New Relic generic AMI supporting services your application docker-registry
18. Any web application any web-app framework any programming language any Linux variant Splunk support captures stdout/stderr no app support required New Relic support application monitoring system monitoring deployment notification CloudWatch support alerts you when service is down via email or web-hook Auto-scaling multiple servers load-balancing support for scaling schedules Zero-downtime deployments safe upgrades safe config changes
19. Immutable ASG deployment ASG ELB
20. Immutable ASG deployment ASG ELB ASG
21. Immutable ASG deployment ASG ELB ASG
22. Immutable ASG deployment ASG ELB ASG
23. Immutable ASG deployment ELB ASG
24. DEMO?
25. /usr/local/bin/cfn-signal -e $ERROR_STATUS \ --stack {{AWS::StackName}} --region {{AWS::Region}} \ --resource #{auto_scaling_group_name}" while !is_instance_healthy?(elb, instance_id) sleep 5 end Instance asks ELB: “How good am I?” … then tells CloudFormation to proceed
26. “Friends don’t let friends program in XML JSON” — me ( ADVERTISEMENT )
28. REA Shipper — “just enough half-assed PaaS on demand” It’s only a stepping stone towards PaaS. What’s next, for REA? • AWS Elastic Container Service • Apache Mesos • Kubernetes • CloudFoundry
29. Thank Questions? We’re hiring! Good operators “Delivery engineers” ALL the developers