DevOpsTO - AWS and Ansible with theScore

Transcript

1. AWS and Ansible with theScore #DevOpsTO Luke Reeves @lukereeves
2. What is theScore? Several applications - Sports, eSports and Fantasy Support across iOS, Android and web platforms Over ten million unique users and installations Billions of push notifications sent
3. All-in with AWS EC2 Autoscaling via AMIs ELB, sometimes RedShift, sometimes CloudFront RDS Elasticache S3 Cloudwatch Route 53 IAM CloudTrail
4. Ansible We’ve tried other CM tools such as Chef and Salt Procedural instruction order much easier to work with than functional Playbook format is very straightforward Having no master server by default gives us a lot of flexibility All infrastructure can be described in a single repo and atomic changes are possible
5. Multiple Uses Provisioning hosts Installing application dependencies on hosts Deploying and updating applications Running commands across groups
6. Provisioning Role We have a “common” role that applies to all of our server images Other roles can always depend on the common role being run first on any hosts Repeatability is a must - the common role should be able to run at any time to add new base packages, user SSH keys, etc.
7. Application Roles Applications are broken into environments, groups and roles The roles are fine-grained and usually represent small to large clusters Example roles are “cms_loadbalancer”, “esports_api”
8. Autoscaling Overview Sports is primarily event based, so autoscaling each application cluster is a good fit News can also have surges and is not as easy to predict With AWS you pay for exactly what resources you use, so tuning auto-scaling is crucial
9. Provisioning, first try Create Ansible playbooks for various server roles and components (e.g. Rails server, memcached host) Use a server with scaling SNS hooks that applies the playbook to any new hosts that spin up. Post-creation the same server reconfigures any load balancers to add new hosts.
10. Provisioning, second try Use the same Ansible playbooks for each role. After any deployments (changes) to running instances create an AMI snapshot of the host. Update the scaling group for the changed role to point to the AMI. Use the EC2 APIs to configure load balancers on a schedule (every minute works fine).
11. Provisioning Using the snapshot method allows us to bring up new EBS-backed hosts in an average of 30 seconds as opposed to 10+ minutes. Leveraging tags in the EC2 hosts and scaling groups lets us create load balancer configurations with minimal lag between host creation and registration for requests. Make sure you freeze scaling groups during the deployment and snapshot, and unfreeze them afterwards. The faster you can scale up and down, the more cost effective AWS is. This also applies to tightening scaling thresholds (CPU usage, etc).
12. Gaps Examples of wiring up load balancers… don’t work in practice Some fights with the template system (Jinja 2) - it’s very simple and you can’t just outright embed Python code When we started two years ago we had to build out AWS description plugins We curse “bundle AMI” but don’t have a good replacement :-)
13. Next Steps Using consul and consul-template for config data propagation CircleCI running all deployments after the tests pass - why waste our time running the command? Putting all application-relevant configuration (Ruby version for example) in the app’s own GitHub repos.
14. Conclusion Ansible is a great tool; there are definitely functions we would use other tools for but it still is a great base Integrating Ansible and AWS has reduced our work (and drama) hugely Questions? Luke Reeves @lukereeves