TaskRabbit's Ansible Tips & Tricks

Evan Tahler, Director of Technology at TaskRabbit will discuss Ansible tips and tricks, and how/why TaskRabbit transitioned to Ansible.

Presented for "Pittsburgh Code & Supply" http://www.meetup.com/Pittsburgh-Code-Supply/events/223535992/

Links:
- GIST for testing Vagrant SSH stuff with RSpec https://gist.github.com/evantahler/9fa167a8be811fa710e2
- Relevant Blog Posts: http://tech.taskrabbit.com/posts.html#ansible

Transcript

1. TaskRabbit’s Ansible Tips & Tricks or: “sometimes you need to do more than converge” July 13th, 2015 PGH Code & Supply
3. {{ BioPage }} @evantahler www.evantahler.com • Director of Technology @ TaskRabbit • actionhero.js, elasticdump, node-resque, etc • Yinzer: CMU + ModCloth + Failed Startups. We are Hiring! Talk to me later!
6. TaskRabbit Tech History Hosts: - 2008 Amazon - 2010 RackSpace - 2012 Joyent - 2014 Co-Lo OS: - 2008 Windows - 2009 Ubuntu 10 - 2013 SmartOS - 2014 Ubuntu 12 - 2015 Ubuntu 14 Philosophy - 2008 Basic Rails - 2010 Monolithic App - 2012 SOA - 2014 Rails Engines+ Rails: - 2008 Rails 2 - 2012 Rails 3 - 2014 Rails 4 - 2014 Node.js 0.6 - 2015 Node.js 0.8 Databases: - 2008 mySQL - 2012 Riak - 2012 Postgres + GIS - 2012 Redis - 2013 Percona MySQL - 2013 Couchbase - 2013 Percona Cluster - 2014 Elasticsearch - 2015 Redshift As a company seeks to find product-market fit, you should expect to grow & change your infrastructure rapidly Optimization is the game of a profitable (market-fit) company. Stay over- provisioned and quick to change until you get there.
7. Up until 2012, a bash script was all we needed • 4 types of servers ◦ Database (MySQL) ◦ App: Web ◦ App: Resque (background worker) ◦ Memcache • Easy enough for one person to manage Provisioning: bash << (curl -s http://gist.github.com/make_server.sh) • and it worked great
8. TaskRabbit, ~ 2012 • Roles, IPs, etc were stored in Capistrano (deployment tool) • We rarely changed servers (even though were were “in the cloud”) • Then we got religion: ◦ Service Oriented Architecture
9. From an internal presentation in 2013
10. Things become painful, but slowly • Creating/Deploying configuration files was our biggest problem ◦ We kept the same OS, same general layout of all hosts, etc. ◦ Created a special DSL in yml to auto-build the collection of config file and rsync them • Then we had more and more specialized servers: ◦ Load Balancers, Log aggregators, specialized databases (for each service!), etc
11. The Chef Days: 2013 - 2014 • ~ 50 servers, 10 DBs, 7 Deployable Apps • A. We need to provision servers automatically, as we were now scaling up and down with demand ◦ We usually had warning, speed wasn’t an issue ◦ Deployment was still via other tools • B. We needed to keep the “special” (read: data) servers (dbs) up to date as well as the application servers ◦ Application servers and Special servers referenced each other
12. The Chef Days: 2013 - 2014 • What is cool about chef (compared to bash scripts): ◦ Recoverable server creation ◦ Faster changes to configs ◦ Segmentable (and combinable) roles for each application ▪ Want to change a server role? OK! ▪ Want to spin up a new data center? Possible! ◦ Local AND System-wide updates via nesting ▪ Want to change the ruby version? ▪ Want to install a custom package (Like R-lang or Couchbase) ◦ Incidentally, this is why we don't use Heroku, and manage our own stuff. That, and we save a lot of $ From an internal presentation in 2013
13. Notes from the Chef days • We only “pushed” deployments (ran `chef- client` on demand) • We hosted our own chef-server • We were a ruby shop, which let us do some fancy (dangerous) things directly ◦ Incidentally, this was why we picked Chef over Puppet
14. Chef had one big problem It was only for provisioning. • We used other tools for deployment, which lead to duplication of data (or very slow API calls to the chef server) • We were writing custom knife plugins to do what should be simple
15. Chef had one big problem ~170 lines to run `kill -s USR2 `cat /path/to/unicorn.pid` ` on all `web` servers:
16. Chef had one big problem In essence, Chef and I had a philosophical disagreement: In the real world, we sometimes wanted to use our tools to do things other than “converge our servers to a known state”. - Sometimes I *wanted* an old version of the code on a specific server. - Sometimes I *didn’t want* to do a full chef-client run to update a deployment. - Sometimes I *wanted* to directly control a service or process
17. So, we changed tools.
18. Is Ansible a Better Chef? • No need for a remote server; it all runs locally over SSH • A TON of great built-in packages (redis, mysql, http, etc) • Smaller and more Declarative ◦ YMLs and nothing else.
19. Is Ansible a Better Chef? (cont) • It runs locally, so you can arrange your project how you want. ◦ Vault? ◦ git-submodules ▪ Separate permissions for secrets • Secrets per environment • It’s really easy to test ◦ vagrant integration
20. Is Ansible a Better Chef? (cont) • But most importantly, Ansible allows for playbooks that both converge state and act
21. Tip & Trick #1: Make Many small and Clear Playbooks provision.yml (like chef-client) deploy.yml (like capistrano) restart.yml (like we tried to make with the knife extensions)
22. Tip & Trick #2: Learn the variable hierarchy If you can link variable definitions to host groups. You can reference host groups in your playbooks.
23. Tip & Trick #2: Learn the variable hierarchy https://docs.ansible.com/playbooks_variables.html
24. Tip & Trick #2: Learn the variable hierarchy If done correctly, you can do awesome things. I like to think of this as the “DSL” of our ansible implementation. Most folks shouldn’t have to write a playbook.
25. Tip & Trick #3: State Files There’s no good way to tell if this is a “new” server or not. Lay down files!
26. Tip & Trick #4: Shell Out and Hack There are times I just couldn’t get Ansbile’s build-in commands to work. Don’t be afraid to shell out. The shell module is actually one of the best I’ve ever seen, loading ENV properly, allowing sudo, etc. Many of Ansible’s built-in commands can be used for other purposes
27. Tip & Trick #4: Shell Out How to update SSH Authorized Keys (including deauth)
28. Tip & Trick #5: Test Ansible can build an inventory file from Vagrant!
29. Tip & Trick #5: Test More details in a blog post... coming soon! The dream... ..but messy to get working
30. Tip & Trick #5: Test More details in a blog post... coming soon! https://gist.github.com/evantahler/9fa167a8be811fa710e2
31. What do I hate about Ansible? The logger.
32. What do I hate about Ansible? • There is no if/else • Loop Syntax ◦ I miss ruby blocks • Debugging • Variable Scoping & play re-use
33. Ansible Tower • inventory shenanigans
34. Ansible Tower
35. Thanks! We blog about this stuff @ tech.taskrabbit.com evan@taskrabbit.com