$30 off During Our Annual Pro Sale. View Details »

Automate or Die

Aaron Quint
January 18, 2013

Automate or Die

Explaining Paperless Post's deploy methodology and how we got there. When I Ship, You Ship, We Ship.

Aaron Quint

January 18, 2013
Tweet

More Decks by Aaron Quint

Other Decks in Programming

Transcript

  1. Automate or die
    Aaron Quint / DevOpsDays NYC / 1/2013
    Friday, January 18, 13

    View Slide

  2. It’s me, @AQ
    HeYO
    Friday, January 18, 13

    View Slide

  3. Friday, January 18, 13

    View Slide

  4. • Food Obsessed (CodeFoodBar)
    Friday, January 18, 13

    View Slide

  5. • Food Obsessed (CodeFoodBar)
    • Open Source Ruby & JS (Sammy.js, et al)
    Friday, January 18, 13

    View Slide

  6. • Food Obsessed (CodeFoodBar)
    • Open Source Ruby & JS (Sammy.js, et al)
    • BROOKLYNNNNNNNNN (get hype)
    Friday, January 18, 13

    View Slide

  7. • Food Obsessed (CodeFoodBar)
    • Open Source Ruby & JS (Sammy.js, et al)
    • BROOKLYNNNNNNNNN (get hype)
    • twitter: @aq
    Friday, January 18, 13

    View Slide

  8. • Food Obsessed (CodeFoodBar)
    • Open Source Ruby & JS (Sammy.js, et al)
    • BROOKLYNNNNNNNNN (get hype)
    • twitter: @aq
    • github: quirkey
    Friday, January 18, 13

    View Slide

  9. CTO of
    Friday, January 18, 13

    View Slide

  10. Chief Taco Officer
    CTO of
    Friday, January 18, 13

    View Slide

  11. www.paperlesspost.com
    Friday, January 18, 13

    View Slide

  12. Friday, January 18, 13

    View Slide

  13. • Most individual user-generated cards sent in a single week:
    ~2 million
    • Registered user growth over December 2011: 211% (3.1x)
    • Revenue growth over December 2011: 135% (2.3x)
    • Number of commits to main app git repo: 44,545
    • Number of commits to chef git repo: 4,447
    • Peak traffic served: ~300 Mbps
    • Number of production deploys across applications: 1,631
    • Number of staging deploys across applications: 10,600
    • Unique animated gifs posted in Campfire: 3,208
    Friday, January 18, 13

    View Slide

  14. DA DIP
    Friday, January 18, 13

    View Slide

  15. DA SHIP
    Friday, January 18, 13

    View Slide

  16. WHEN I SHIP
    YOU SHIP
    WE SHIP
    Friday, January 18, 13

    View Slide

  17. It’s about being efficient
    Automation is not
    about being lazy
    Friday, January 18, 13

    View Slide

  18. Developers aren’t
    slow, they’re ...
    Friday, January 18, 13

    View Slide

  19. Distracted
    Friday, January 18, 13

    View Slide

  20. blocked
    Friday, January 18, 13

    View Slide

  21. churning
    Friday, January 18, 13

    View Slide

  22. Complexity = Not just large, but distributed
    applications get more
    complex over time
    Friday, January 18, 13

    View Slide

  23. how to allow for
    growing complexity
    while
    not slowing down
    Friday, January 18, 13

    View Slide

  24. - A DevOps Oath
    I, [YOU], Do solemnly swear to help
    developers solve their problems,
    quickly and without [too many] yak-
    shaves and make their lives easier, so
    they can ship code, so that the
    company can improve, so that we all
    can be successful. I will do what it
    takes to avoid repeated tasks, help
    unblock the blocked, and generally
    make things work better.
    Friday, January 18, 13

    View Slide

  25. You want
    Friday, January 18, 13

    View Slide

  26. Friday, January 18, 13

    View Slide

  27. • Notifications
    Friday, January 18, 13

    View Slide

  28. • Notifications
    • One click deploys
    Friday, January 18, 13

    View Slide

  29. • Notifications
    • One click deploys
    • Continuous integration
    Friday, January 18, 13

    View Slide

  30. • Notifications
    • One click deploys
    • Continuous integration
    • Distributed/fast deploys
    Friday, January 18, 13

    View Slide

  31. • Notifications
    • One click deploys
    • Continuous integration
    • Distributed/fast deploys
    • Zero config Dev environment
    Friday, January 18, 13

    View Slide

  32. • Notifications
    • One click deploys
    • Continuous integration
    • Distributed/fast deploys
    • Zero config Dev environment
    • Developer Metrics
    Friday, January 18, 13

    View Slide

  33. • Notifications
    • One click deploys
    • Continuous integration
    • Distributed/fast deploys
    • Zero config Dev environment
    • Developer Metrics
    • Deploy History/Tracking
    Friday, January 18, 13

    View Slide

  34. • Notifications
    • One click deploys
    • Continuous integration
    • Distributed/fast deploys
    • Zero config Dev environment
    • Developer Metrics
    • Deploy History/Tracking
    • Integrated Troubleshooting
    Friday, January 18, 13

    View Slide

  35. You HAVE
    Friday, January 18, 13

    View Slide

  36. A: Start Small
    Q: But how to start?
    Friday, January 18, 13

    View Slide

  37. It’s daunting, and incremental
    improvements reap rewards
    DON’t DO IT ALL AT ONCE
    Friday, January 18, 13

    View Slide

  38. Distill into a perfect
    complainy-developer cocktail
    Start with the whining
    Friday, January 18, 13

    View Slide

  39. Deploys are too complicated,
    so I end up not doing them
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  40. $ git checkout staging
    $ git pull root staging
    $ git merge my_branch
    $ git push root staging
    $ cap staging deploy:long
    Friday, January 18, 13

    View Slide

  41. WE COULD
    • Build a web ui to handle deploys
    • Make a simple script that wraps the
    common commands
    • Ignore them, haters gonna hate
    Friday, January 18, 13

    View Slide

  42. Friday, January 18, 13

    View Slide

  43. ಠ_ಠ
    Friday, January 18, 13

    View Slide

  44. I keep losing my changes or
    forget to commit
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  45. I keep losing my changes or
    forget to commit
    ಠ_ಠ
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  46. I keep losing my changes or
    forget to commit
    ಠ_ಠ
    I want notifications when
    someone deploys
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  47. I keep losing my changes or
    forget to commit
    ಠ_ಠ
    I want notifications when
    someone deploys
    ಠ_ಠ
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  48. I keep losing my changes or
    forget to commit
    ಠ_ಠ
    I want notifications when
    someone deploys
    I want to deploy multiple
    branches at once
    ಠ_ಠ
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  49. For developers it’s a familiar interface
    Lots of things can
    be solved with
    command line tools
    Friday, January 18, 13

    View Slide

  50. Devtools was born
    Friday, January 18, 13

    View Slide

  51. Design an aPI around
    your common tasks
    Friday, January 18, 13

    View Slide

  52. $ pp rails deploy earth quirkey/branch1 mrb/f2
    # pp deploy [branches, ...]
    Friday, January 18, 13

    View Slide

  53. • Does the git flow (checkout, reset,
    merge, push)
    • Handles conflicts and stashing
    changes
    • Runs the deploy
    • Sends notifications to email list/
    campfire rooms
    Friday, January 18, 13

    View Slide

  54. $ pp boot renderer restart
    # pp boot [stop|start|restart|status]
    Friday, January 18, 13

    View Slide

  55. • Handles booting local services
    • Has all powerful boot/restart
    commands
    • Monitors service status
    Friday, January 18, 13

    View Slide

  56. • Ruby gem (on internal gemserver)
    • Self-updating (pp devtools update)
    • Built on top of Thor
    • Made to be extensible (easy to add
    apps, tasks, recipes)
    DEvTOOLS
    Friday, January 18, 13

    View Slide

  57. WHEN I SHIP
    YOU SHIP
    WE SHIP
    Friday, January 18, 13

    View Slide

  58. Friday, January 18, 13

    View Slide

  59. ಠ_ಠ
    Friday, January 18, 13

    View Slide

  60. I need to deploy but I don’t have
    ssh credentials
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  61. I need to deploy but I don’t have
    ssh credentials
    ಠ_ಠ
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  62. I need to deploy but I don’t have
    ssh credentials
    ಠ_ಠ
    It takes too long to run the test
    builds
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  63. I need to deploy but I don’t have
    ssh credentials
    ಠ_ಠ
    It takes too long to run the test
    builds
    ಠ_ಠ
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  64. I need to deploy but I don’t have
    ssh credentials
    ಠ_ಠ
    It takes too long to run the test
    builds
    Are we deploying? What step are
    we at?
    ಠ_ಠ
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  65. ; http://jenkins-ci.org
    Jenkins!
    Friday, January 18, 13

    View Slide

  66. It just needs to be told what to deploy
    (It has an HTTP API)
    Jenkins can just
    run devtools
    Friday, January 18, 13

    View Slide

  67. Text
    # On your machine
    $ pp rails deploy earth quirkey/branch1 mrb/f2
    # Does git merging, sends request to jenkins
    # On Jenkins, in a `deploy` job
    $ pp $APP deploy $ENV $BRANCHES --local
    # Skips the git stuff, just does the actual
    # deploy and sends the notifications
    Friday, January 18, 13

    View Slide

  68. Jenkins
    • Global visibility (changed the game)
    • No fear of losing network
    connectivity during a deploy
    • Single point of orchestration
    • Being able to see the deploys and
    builds made the team want to make
    them faster and better
    Friday, January 18, 13

    View Slide

  69. WHEN I SHIP
    YOU SHIP
    WE SHIP
    Friday, January 18, 13

    View Slide

  70. Friday, January 18, 13

    View Slide

  71. ಠ_ಠ
    Friday, January 18, 13

    View Slide

  72. What branches are
    deployed to staging?
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  73. What branches are
    deployed to staging?
    ಠ_ಠ
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  74. What branches are
    deployed to staging?
    ಠ_ಠ
    What just got deployed to
    production?
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  75. What branches are
    deployed to staging?
    ಠ_ಠ
    What just got deployed to
    production?
    ಠ_ಠ
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  76. What branches are
    deployed to staging?
    ಠ_ಠ
    What just got deployed to
    production?
    Johnny just said he pushed
    something, but I can’t find it
    ಠ_ಠ
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  77. Now we just need to collect the data
    We unified the
    entry points
    Friday, January 18, 13

    View Slide

  78. A web UI and database for devtools
    Enter: PUSH PARTY
    Friday, January 18, 13

    View Slide

  79. Text
    Friday, January 18, 13

    View Slide

  80. And separated them from the CLI
    We extracted the
    “tools” of Devtools
    Friday, January 18, 13

    View Slide

  81. > require 'paperless-devtools'
    > app = PaperlessDevtools.app :rails
    > app.tools(:git).current_branch
    #=> "master"
    > app.tools(:jenkins).start_build("master")
    #=> send build to jenkins
    Friday, January 18, 13

    View Slide

  82. Rails, Postgres, Devtools, Compass
    Then we built a
    WebApp around it
    Friday, January 18, 13

    View Slide

  83. Friday, January 18, 13

    View Slide

  84. devtools makes api calls to pushparty
    Friday, January 18, 13

    View Slide

  85. devtools makes api calls to pushparty
    which records deploys in postgres
    Friday, January 18, 13

    View Slide

  86. devtools makes api calls to pushparty
    which records deploys in postgres
    then uses devtools to send them to jenkins
    Friday, January 18, 13

    View Slide

  87. devtools makes api calls to pushparty
    which records deploys in postgres
    then uses devtools to send them to jenkins
    which uses devtools to send progress and
    status to pushparty
    Friday, January 18, 13

    View Slide

  88. devtools makes api calls to pushparty
    which records deploys in postgres
    then uses devtools to send them to jenkins
    which uses devtools to send progress and
    status to pushparty
    which uses devtools to announce the
    completed deploy in campfire
    Friday, January 18, 13

    View Slide

  89. devtools makes api calls to pushparty
    which records deploys in postgres
    then uses devtools to send them to jenkins
    which uses devtools to send progress and
    status to pushparty
    which uses devtools to announce the
    completed deploy in campfire
    Friday, January 18, 13

    View Slide

  90. Extremely easy to see what was merged/
    deployed, in what state and at what time
    Now we have DATA
    Friday, January 18, 13

    View Slide

  91. BRANCHES
    FLOW
    Friday, January 18, 13

    View Slide

  92. WHEN I SHIP
    YOU SHIP
    WE SHIP
    Friday, January 18, 13

    View Slide

  93. Working with chef [with
    multiple team members working
    on multiple projects in multiple
    staging environments] is a pain
    in the butt.
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  94. $ pp chef deploy earth johnnydtan/graphite_cluster
    # git merging
    # cookbook testing
    # cookbook uploading
    # environment munging
    Friday, January 18, 13

    View Slide

  95. http://www.meetup.com/Chef-NYC/
    Come to #CHEF NYC
    Friday, January 18, 13

    View Slide

  96. WHEN I SHIP
    YOU SHIP
    WE SHIP
    Friday, January 18, 13

    View Slide

  97. We have all this data in graphite,
    chef, sensu, pushparty, but when
    theres an issue it’s hard to
    coalate the problems
    ಠ_ಠ
    Friday, January 18, 13

    View Slide

  98. Devtools wrappers for different tools,
    pulled into a single web ui
    HOTPOT
    Friday, January 18, 13

    View Slide

  99. Friday, January 18, 13

    View Slide

  100. Awesome things I don’t have
    time to talk about
    Friday, January 18, 13

    View Slide

  101. Awesome things I don’t have
    time to talk about
    • pp chef create_node
    Friday, January 18, 13

    View Slide

  102. Awesome things I don’t have
    time to talk about
    • pp chef create_node
    • pp vagrant setup
    Friday, January 18, 13

    View Slide

  103. Awesome things I don’t have
    time to talk about
    • pp chef create_node
    • pp vagrant setup
    • pushparty bashrc’s
    Friday, January 18, 13

    View Slide

  104. Awesome things I don’t have
    time to talk about
    • pp chef create_node
    • pp vagrant setup
    • pushparty bashrc’s
    • graphite integration
    Friday, January 18, 13

    View Slide

  105. Awesome things I don’t have
    time to talk about
    • pp chef create_node
    • pp vagrant setup
    • pushparty bashrc’s
    • graphite integration
    • deploy build time graphs
    Friday, January 18, 13

    View Slide

  106. Awesome things I don’t have
    time to talk about
    • pp chef create_node
    • pp vagrant setup
    • pushparty bashrc’s
    • graphite integration
    • deploy build time graphs
    • more every day
    Friday, January 18, 13

    View Slide

  107. Things to remember
    Friday, January 18, 13

    View Slide

  108. Build for your team for right now, not for
    every team at any size
    Don’t expect to build
    the perfect system
    Friday, January 18, 13

    View Slide

  109. Not the other way around
    Start with a workflow
    then tool around it
    Friday, January 18, 13

    View Slide

  110. Then add the cool shit that
    they didn’t know they needed
    Build what your
    team needs first
    Friday, January 18, 13

    View Slide

  111. WHEN I SHIP
    YOU SHIP
    WE SHIP
    Friday, January 18, 13

    View Slide

  112. github.com/quirkey
    twitter.com/aq
    quirkey.com
    paperlesspost.com
    THANKS!
    Friday, January 18, 13

    View Slide