Upgrade to Pro — share decks privately, control downloads, hide ads and more …

TaskRabbit's Ansible Tips & Tricks

TaskRabbit's Ansible Tips & Tricks

Evan Tahler, Director of Technology at TaskRabbit will discuss Ansible tips and tricks, and how/why TaskRabbit transitioned to Ansible.

Presented for "Pittsburgh Code & Supply" http://www.meetup.com/Pittsburgh-Code-Supply/events/223535992/

Links:
- GIST for testing Vagrant SSH stuff with RSpec https://gist.github.com/evantahler/9fa167a8be811fa710e2
- Relevant Blog Posts: http://tech.taskrabbit.com/posts.html#ansible

Evan Tahler

July 13, 2015
Tweet

More Decks by Evan Tahler

Other Decks in Technology

Transcript

  1. TaskRabbit’s Ansible
    Tips & Tricks
    or: “sometimes you need to do more than converge”
    July 13th, 2015 PGH Code & Supply

    View Slide

  2. View Slide

  3. {{ BioPage }}
    @evantahler
    www.evantahler.com
    ● Director of Technology @ TaskRabbit
    ● actionhero.js, elasticdump, node-resque, etc
    ● Yinzer: CMU + ModCloth + Failed Startups.
    We are Hiring! Talk to me later!

    View Slide

  4. View Slide

  5. View Slide

  6. TaskRabbit Tech History
    Hosts:
    - 2008 Amazon
    - 2010 RackSpace
    - 2012 Joyent
    - 2014 Co-Lo
    OS:
    - 2008 Windows
    - 2009 Ubuntu 10
    - 2013 SmartOS
    - 2014 Ubuntu 12
    - 2015 Ubuntu 14
    Philosophy
    - 2008 Basic Rails
    - 2010 Monolithic App
    - 2012 SOA
    - 2014 Rails Engines+
    Rails:
    - 2008 Rails 2
    - 2012 Rails 3
    - 2014 Rails 4
    - 2014 Node.js 0.6
    - 2015 Node.js 0.8
    Databases:
    - 2008 mySQL
    - 2012 Riak
    - 2012 Postgres + GIS
    - 2012 Redis
    - 2013 Percona MySQL
    - 2013 Couchbase
    - 2013 Percona Cluster
    - 2014 Elasticsearch
    - 2015 Redshift
    As a company seeks to
    find product-market fit,
    you should expect to
    grow & change your
    infrastructure rapidly
    Optimization is the game
    of a profitable (market-fit)
    company. Stay over-
    provisioned and quick to
    change until you get
    there.

    View Slide

  7. Up until 2012, a bash script was all we needed
    ● 4 types of servers
    ○ Database (MySQL)
    ○ App: Web
    ○ App: Resque (background worker)
    ○ Memcache
    ● Easy enough for one person to manage
    Provisioning: bash << (curl -s http://gist.github.com/make_server.sh)
    ● and it worked great

    View Slide

  8. TaskRabbit, ~ 2012
    ● Roles, IPs, etc were stored in Capistrano
    (deployment tool)
    ● We rarely changed servers (even though
    were were “in the cloud”)
    ● Then we got religion:
    ○ Service Oriented Architecture

    View Slide

  9. From an internal presentation in 2013

    View Slide

  10. Things become painful, but slowly
    ● Creating/Deploying configuration files was our biggest
    problem
    ○ We kept the same OS, same general layout of all hosts, etc.
    ○ Created a special DSL in yml to auto-build the collection of
    config file and rsync them
    ● Then we had more and more specialized servers:
    ○ Load Balancers, Log aggregators, specialized databases (for
    each service!), etc

    View Slide

  11. The Chef Days: 2013 - 2014
    ● ~ 50 servers, 10 DBs, 7 Deployable Apps
    ● A. We need to provision servers
    automatically, as we were now scaling up
    and down with demand
    ○ We usually had warning, speed wasn’t
    an issue
    ○ Deployment was still via other tools
    ● B. We needed to keep the “special” (read:
    data) servers (dbs) up to date as well as
    the application servers
    ○ Application servers and Special
    servers referenced each other

    View Slide

  12. The Chef Days: 2013 - 2014
    ● What is cool about chef (compared to bash scripts):
    ○ Recoverable server creation
    ○ Faster changes to configs
    ○ Segmentable (and combinable) roles for each application
    ■ Want to change a server role? OK!
    ■ Want to spin up a new data center? Possible!
    ○ Local AND System-wide updates via nesting
    ■ Want to change the ruby version?
    ■ Want to install a custom package (Like R-lang or Couchbase)
    ○ Incidentally, this is why we don't use Heroku, and manage our own
    stuff. That, and we save a lot of $
    From an internal presentation in 2013

    View Slide

  13. Notes from the Chef days
    ● We only “pushed” deployments (ran `chef-
    client` on demand)
    ● We hosted our own chef-server
    ● We were a ruby shop, which let us do some
    fancy (dangerous) things directly
    ○ Incidentally, this was why we picked Chef over
    Puppet

    View Slide

  14. Chef had one big problem
    It was only for provisioning.
    ● We used other tools for deployment, which
    lead to duplication of data (or very slow API
    calls to the chef server)
    ● We were writing custom knife plugins to do
    what should be simple

    View Slide

  15. Chef had one big problem
    ~170 lines to run `kill -s USR2 `cat /path/to/unicorn.pid`
    ` on all `web` servers:

    View Slide

  16. Chef had one big problem
    In essence, Chef and I had a philosophical disagreement:
    In the real world, we sometimes wanted to
    use our tools to do things other than
    “converge our servers to a known state”.
    - Sometimes I *wanted* an old version of the code on a specific server.
    - Sometimes I *didn’t want* to do a full chef-client run to update a deployment.
    - Sometimes I *wanted* to directly control a service or process

    View Slide

  17. So, we changed tools.

    View Slide

  18. Is Ansible a Better Chef?
    ● No need for a remote server; it all runs
    locally over SSH
    ● A TON of great built-in packages (redis,
    mysql, http, etc)
    ● Smaller and more Declarative
    ○ YMLs and nothing else.

    View Slide

  19. Is Ansible a Better Chef? (cont)
    ● It runs locally, so you can arrange your
    project how you want.
    ○ Vault?
    ○ git-submodules
    ■ Separate permissions for secrets
    ● Secrets per environment
    ● It’s really easy to test
    ○ vagrant integration

    View Slide

  20. Is Ansible a Better Chef? (cont)
    ● But most importantly, Ansible allows for
    playbooks that both converge state and act

    View Slide

  21. Tip & Trick #1: Make Many small and Clear Playbooks
    provision.yml (like chef-client)
    deploy.yml (like capistrano)
    restart.yml (like we tried to
    make with the knife
    extensions)

    View Slide

  22. Tip & Trick #2: Learn the variable hierarchy
    If you can link
    variable definitions
    to host groups.
    You can reference
    host groups in your
    playbooks.

    View Slide

  23. Tip & Trick #2: Learn the variable hierarchy
    https://docs.ansible.com/playbooks_variables.html

    View Slide

  24. Tip & Trick #2: Learn the variable hierarchy
    If done correctly, you can
    do awesome things.
    I like to think of this as
    the “DSL” of our ansible
    implementation.
    Most folks shouldn’t have
    to write a playbook.

    View Slide

  25. Tip & Trick #3: State Files
    There’s no good way to tell if this is a “new”
    server or not. Lay down files!

    View Slide

  26. Tip & Trick #4: Shell Out and Hack
    There are times I just couldn’t get Ansbile’s build-in
    commands to work. Don’t be afraid to shell out.
    The shell module is actually one of the best I’ve ever
    seen, loading ENV properly, allowing sudo, etc.
    Many of Ansible’s built-in commands can be used for
    other purposes

    View Slide

  27. Tip & Trick #4: Shell Out
    How to update SSH Authorized Keys (including deauth)

    View Slide

  28. Tip & Trick #5: Test
    Ansible can build an inventory file from Vagrant!

    View Slide

  29. Tip & Trick #5: Test
    More details in a blog post... coming soon!
    The dream...
    ..but messy to get
    working

    View Slide

  30. Tip & Trick #5: Test
    More details in a blog post... coming soon!
    https://gist.github.com/evantahler/9fa167a8be811fa710e2

    View Slide

  31. What do I hate about Ansible?
    The logger.

    View Slide

  32. What do I hate about Ansible?
    ● There is no if/else
    ● Loop Syntax
    ○ I miss ruby blocks
    ● Debugging
    ● Variable Scoping &
    play re-use

    View Slide

  33. Ansible Tower
    ● inventory
    shenanigans

    View Slide

  34. Ansible Tower

    View Slide

  35. Thanks!
    We blog about this stuff @ tech.taskrabbit.com
    [email protected]

    View Slide