TaskRabbit's Ansible Tips & Tricks

Evan Tahler, Director of Technology at TaskRabbit will discuss Ansible tips and tricks, and how/why TaskRabbit transitioned to Ansible.

Presented for "Pittsburgh Code & Supply" http://www.meetup.com/Pittsburgh-Code-Supply/events/223535992/

Links:
- GIST for testing Vagrant SSH stuff with RSpec https://gist.github.com/evantahler/9fa167a8be811fa710e2
- Relevant Blog Posts: http://tech.taskrabbit.com/posts.html#ansible

Transcript

1. TaskRabbit’s Ansible
   Tips & Tricks
   or: “sometimes you need to do more than converge”
   July 13th, 2015 PGH Code & Supply
   

   View Slide

2. View Slide

3. {{ BioPage }}
   @evantahler
   www.evantahler.com
   ● Director of Technology @ TaskRabbit
   ● actionhero.js, elasticdump, node-resque, etc
   ● Yinzer: CMU + ModCloth + Failed Startups.
   We are Hiring! Talk to me later!
   

   View Slide

4. View Slide

5. View Slide

6. TaskRabbit Tech History
   Hosts:
   - 2008 Amazon
   - 2010 RackSpace
   - 2012 Joyent
   - 2014 Co-Lo
   OS:
   - 2008 Windows
   - 2009 Ubuntu 10
   - 2013 SmartOS
   - 2014 Ubuntu 12
   - 2015 Ubuntu 14
   Philosophy
   - 2008 Basic Rails
   - 2010 Monolithic App
   - 2012 SOA
   - 2014 Rails Engines+
   Rails:
   - 2008 Rails 2
   - 2012 Rails 3
   - 2014 Rails 4
   - 2014 Node.js 0.6
   - 2015 Node.js 0.8
   Databases:
   - 2008 mySQL
   - 2012 Riak
   - 2012 Postgres + GIS
   - 2012 Redis
   - 2013 Percona MySQL
   - 2013 Couchbase
   - 2013 Percona Cluster
   - 2014 Elasticsearch
   - 2015 Redshift
   As a company seeks to
   find product-market fit,
   you should expect to
   grow & change your
   infrastructure rapidly
   Optimization is the game
   of a profitable (market-fit)
   company. Stay over-
   provisioned and quick to
   change until you get
   there.
   

   View Slide

7. Up until 2012, a bash script was all we needed
   ● 4 types of servers
   ○ Database (MySQL)
   ○ App: Web
   ○ App: Resque (background worker)
   ○ Memcache
   ● Easy enough for one person to manage
   Provisioning: bash << (curl -s http://gist.github.com/make_server.sh)
   ● and it worked great
   

   View Slide

8. TaskRabbit, ~ 2012
   ● Roles, IPs, etc were stored in Capistrano
   (deployment tool)
   ● We rarely changed servers (even though
   were were “in the cloud”)
   ● Then we got religion:
   ○ Service Oriented Architecture
   

   View Slide

9. From an internal presentation in 2013
   

   View Slide

10. Things become painful, but slowly
    ● Creating/Deploying configuration files was our biggest
    problem
    ○ We kept the same OS, same general layout of all hosts, etc.
    ○ Created a special DSL in yml to auto-build the collection of
    config file and rsync them
    ● Then we had more and more specialized servers:
    ○ Load Balancers, Log aggregators, specialized databases (for
    each service!), etc
    

    View Slide

11. The Chef Days: 2013 - 2014
    ● ~ 50 servers, 10 DBs, 7 Deployable Apps
    ● A. We need to provision servers
    automatically, as we were now scaling up
    and down with demand
    ○ We usually had warning, speed wasn’t
    an issue
    ○ Deployment was still via other tools
    ● B. We needed to keep the “special” (read:
    data) servers (dbs) up to date as well as
    the application servers
    ○ Application servers and Special
    servers referenced each other
    

    View Slide

12. The Chef Days: 2013 - 2014
    ● What is cool about chef (compared to bash scripts):
    ○ Recoverable server creation
    ○ Faster changes to configs
    ○ Segmentable (and combinable) roles for each application
    ■ Want to change a server role? OK!
    ■ Want to spin up a new data center? Possible!
    ○ Local AND System-wide updates via nesting
    ■ Want to change the ruby version?
    ■ Want to install a custom package (Like R-lang or Couchbase)
    ○ Incidentally, this is why we don't use Heroku, and manage our own
    stuff. That, and we save a lot of $
    From an internal presentation in 2013
    

    View Slide

13. Notes from the Chef days
    ● We only “pushed” deployments (ran `chef-
    client` on demand)
    ● We hosted our own chef-server
    ● We were a ruby shop, which let us do some
    fancy (dangerous) things directly
    ○ Incidentally, this was why we picked Chef over
    Puppet
    

    View Slide

14. Chef had one big problem
    It was only for provisioning.
    ● We used other tools for deployment, which
    lead to duplication of data (or very slow API
    calls to the chef server)
    ● We were writing custom knife plugins to do
    what should be simple
    

    View Slide

15. Chef had one big problem
    ~170 lines to run `kill -s USR2 `cat /path/to/unicorn.pid`
    ` on all `web` servers:
    

    View Slide

16. Chef had one big problem
    In essence, Chef and I had a philosophical disagreement:
    In the real world, we sometimes wanted to
    use our tools to do things other than
    “converge our servers to a known state”.
    - Sometimes I *wanted* an old version of the code on a specific server.
    - Sometimes I *didn’t want* to do a full chef-client run to update a deployment.
    - Sometimes I *wanted* to directly control a service or process
    

    View Slide

17. So, we changed tools.
    

    View Slide

18. Is Ansible a Better Chef?
    ● No need for a remote server; it all runs
    locally over SSH
    ● A TON of great built-in packages (redis,
    mysql, http, etc)
    ● Smaller and more Declarative
    ○ YMLs and nothing else.
    

    View Slide

19. Is Ansible a Better Chef? (cont)
    ● It runs locally, so you can arrange your
    project how you want.
    ○ Vault?
    ○ git-submodules
    ■ Separate permissions for secrets
    ● Secrets per environment
    ● It’s really easy to test
    ○ vagrant integration
    

    View Slide

20. Is Ansible a Better Chef? (cont)
    ● But most importantly, Ansible allows for
    playbooks that both converge state and act
    

    View Slide

21. Tip & Trick #1: Make Many small and Clear Playbooks
    provision.yml (like chef-client)
    deploy.yml (like capistrano)
    restart.yml (like we tried to
    make with the knife
    extensions)
    

    View Slide

22. Tip & Trick #2: Learn the variable hierarchy
    If you can link
    variable definitions
    to host groups.
    You can reference
    host groups in your
    playbooks.
    

    View Slide

23. Tip & Trick #2: Learn the variable hierarchy
    https://docs.ansible.com/playbooks_variables.html
    

    View Slide

24. Tip & Trick #2: Learn the variable hierarchy
    If done correctly, you can
    do awesome things.
    I like to think of this as
    the “DSL” of our ansible
    implementation.
    Most folks shouldn’t have
    to write a playbook.
    

    View Slide

25. Tip & Trick #3: State Files
    There’s no good way to tell if this is a “new”
    server or not. Lay down files!
    

    View Slide

26. Tip & Trick #4: Shell Out and Hack
    There are times I just couldn’t get Ansbile’s build-in
    commands to work. Don’t be afraid to shell out.
    The shell module is actually one of the best I’ve ever
    seen, loading ENV properly, allowing sudo, etc.
    Many of Ansible’s built-in commands can be used for
    other purposes
    

    View Slide

27. Tip & Trick #4: Shell Out
    How to update SSH Authorized Keys (including deauth)
    

    View Slide

28. Tip & Trick #5: Test
    Ansible can build an inventory file from Vagrant!
    

    View Slide

29. Tip & Trick #5: Test
    More details in a blog post... coming soon!
    The dream...
    ..but messy to get
    working
    

    View Slide

30. Tip & Trick #5: Test
    More details in a blog post... coming soon!
    https://gist.github.com/evantahler/9fa167a8be811fa710e2
    

    View Slide

31. What do I hate about Ansible?
    The logger.
    

    View Slide

32. What do I hate about Ansible?
    ● There is no if/else
    ● Loop Syntax
    ○ I miss ruby blocks
    ● Debugging
    ● Variable Scoping &
    play re-use
    

    View Slide

33. Ansible Tower
    ● inventory
    shenanigans
    

    View Slide

34. Ansible Tower
    

    View Slide

35. Thanks!
    We blog about this stuff @ tech.taskrabbit.com
    evan@taskrabbit.com
    

    View Slide