Upgrade to Pro — share decks privately, control downloads, hide ads and more …

TaskRabbit's Ansible Tips & Tricks

TaskRabbit's Ansible Tips & Tricks

Evan Tahler, Director of Technology at TaskRabbit will discuss Ansible tips and tricks, and how/why TaskRabbit transitioned to Ansible.

Presented for "Pittsburgh Code & Supply" http://www.meetup.com/Pittsburgh-Code-Supply/events/223535992/

- GIST for testing Vagrant SSH stuff with RSpec https://gist.github.com/evantahler/9fa167a8be811fa710e2
- Relevant Blog Posts: http://tech.taskrabbit.com/posts.html#ansible

Evan Tahler

July 13, 2015

More Decks by Evan Tahler

Other Decks in Technology


  1. TaskRabbit’s Ansible Tips & Tricks or: “sometimes you need to

    do more than converge” July 13th, 2015 PGH Code & Supply
  2. {{ BioPage }} @evantahler www.evantahler.com • Director of Technology @

    TaskRabbit • actionhero.js, elasticdump, node-resque, etc • Yinzer: CMU + ModCloth + Failed Startups. We are Hiring! Talk to me later!
  3. TaskRabbit Tech History Hosts: - 2008 Amazon - 2010 RackSpace

    - 2012 Joyent - 2014 Co-Lo OS: - 2008 Windows - 2009 Ubuntu 10 - 2013 SmartOS - 2014 Ubuntu 12 - 2015 Ubuntu 14 Philosophy - 2008 Basic Rails - 2010 Monolithic App - 2012 SOA - 2014 Rails Engines+ Rails: - 2008 Rails 2 - 2012 Rails 3 - 2014 Rails 4 - 2014 Node.js 0.6 - 2015 Node.js 0.8 Databases: - 2008 mySQL - 2012 Riak - 2012 Postgres + GIS - 2012 Redis - 2013 Percona MySQL - 2013 Couchbase - 2013 Percona Cluster - 2014 Elasticsearch - 2015 Redshift As a company seeks to find product-market fit, you should expect to grow & change your infrastructure rapidly Optimization is the game of a profitable (market-fit) company. Stay over- provisioned and quick to change until you get there.
  4. Up until 2012, a bash script was all we needed

    • 4 types of servers ◦ Database (MySQL) ◦ App: Web ◦ App: Resque (background worker) ◦ Memcache • Easy enough for one person to manage Provisioning: bash << (curl -s http://gist.github.com/make_server.sh) • and it worked great
  5. TaskRabbit, ~ 2012 • Roles, IPs, etc were stored in

    Capistrano (deployment tool) • We rarely changed servers (even though were were “in the cloud”) • Then we got religion: ◦ Service Oriented Architecture
  6. Things become painful, but slowly • Creating/Deploying configuration files was

    our biggest problem ◦ We kept the same OS, same general layout of all hosts, etc. ◦ Created a special DSL in yml to auto-build the collection of config file and rsync them • Then we had more and more specialized servers: ◦ Load Balancers, Log aggregators, specialized databases (for each service!), etc
  7. The Chef Days: 2013 - 2014 • ~ 50 servers,

    10 DBs, 7 Deployable Apps • A. We need to provision servers automatically, as we were now scaling up and down with demand ◦ We usually had warning, speed wasn’t an issue ◦ Deployment was still via other tools • B. We needed to keep the “special” (read: data) servers (dbs) up to date as well as the application servers ◦ Application servers and Special servers referenced each other
  8. The Chef Days: 2013 - 2014 • What is cool

    about chef (compared to bash scripts): ◦ Recoverable server creation ◦ Faster changes to configs ◦ Segmentable (and combinable) roles for each application ▪ Want to change a server role? OK! ▪ Want to spin up a new data center? Possible! ◦ Local AND System-wide updates via nesting ▪ Want to change the ruby version? ▪ Want to install a custom package (Like R-lang or Couchbase) ◦ Incidentally, this is why we don't use Heroku, and manage our own stuff. That, and we save a lot of $ From an internal presentation in 2013
  9. Notes from the Chef days • We only “pushed” deployments

    (ran `chef- client` on demand) • We hosted our own chef-server • We were a ruby shop, which let us do some fancy (dangerous) things directly ◦ Incidentally, this was why we picked Chef over Puppet
  10. Chef had one big problem It was only for provisioning.

    • We used other tools for deployment, which lead to duplication of data (or very slow API calls to the chef server) • We were writing custom knife plugins to do what should be simple
  11. Chef had one big problem ~170 lines to run `kill

    -s USR2 `cat /path/to/unicorn.pid` ` on all `web` servers:
  12. Chef had one big problem In essence, Chef and I

    had a philosophical disagreement: In the real world, we sometimes wanted to use our tools to do things other than “converge our servers to a known state”. - Sometimes I *wanted* an old version of the code on a specific server. - Sometimes I *didn’t want* to do a full chef-client run to update a deployment. - Sometimes I *wanted* to directly control a service or process
  13. Is Ansible a Better Chef? • No need for a

    remote server; it all runs locally over SSH • A TON of great built-in packages (redis, mysql, http, etc) • Smaller and more Declarative ◦ YMLs and nothing else.
  14. Is Ansible a Better Chef? (cont) • It runs locally,

    so you can arrange your project how you want. ◦ Vault? ◦ git-submodules ▪ Separate permissions for secrets • Secrets per environment • It’s really easy to test ◦ vagrant integration
  15. Is Ansible a Better Chef? (cont) • But most importantly,

    Ansible allows for playbooks that both converge state and act
  16. Tip & Trick #1: Make Many small and Clear Playbooks

    provision.yml (like chef-client) deploy.yml (like capistrano) restart.yml (like we tried to make with the knife extensions)
  17. Tip & Trick #2: Learn the variable hierarchy If you

    can link variable definitions to host groups. You can reference host groups in your playbooks.
  18. Tip & Trick #2: Learn the variable hierarchy If done

    correctly, you can do awesome things. I like to think of this as the “DSL” of our ansible implementation. Most folks shouldn’t have to write a playbook.
  19. Tip & Trick #3: State Files There’s no good way

    to tell if this is a “new” server or not. Lay down files!
  20. Tip & Trick #4: Shell Out and Hack There are

    times I just couldn’t get Ansbile’s build-in commands to work. Don’t be afraid to shell out. The shell module is actually one of the best I’ve ever seen, loading ENV properly, allowing sudo, etc. Many of Ansible’s built-in commands can be used for other purposes
  21. Tip & Trick #4: Shell Out How to update SSH

    Authorized Keys (including deauth)
  22. Tip & Trick #5: Test More details in a blog

    post... coming soon! The dream... ..but messy to get working
  23. Tip & Trick #5: Test More details in a blog

    post... coming soon! https://gist.github.com/evantahler/9fa167a8be811fa710e2
  24. What do I hate about Ansible? • There is no

    if/else • Loop Syntax ◦ I miss ruby blocks • Debugging • Variable Scoping & play re-use