Tweet
Tweet

Slide 1

Slide 1 text

Automate or die Aaron Quint / DevOpsDays NYC / 1/2013 Friday, January 18, 13

Slide 2

Slide 2 text

It’s me, @AQ HeYO Friday, January 18, 13

Slide 3

Slide 3 text

Friday, January 18, 13

Slide 4

Slide 4 text

• Food Obsessed (CodeFoodBar) Friday, January 18, 13

Slide 5

Slide 5 text

• Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) Friday, January 18, 13

Slide 6

Slide 6 text

• Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • BROOKLYNNNNNNNNN (get hype) Friday, January 18, 13

Slide 7

Slide 7 text

• Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • BROOKLYNNNNNNNNN (get hype) • twitter: @aq Friday, January 18, 13

Slide 8

Slide 8 text

• Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • BROOKLYNNNNNNNNN (get hype) • twitter: @aq • github: quirkey Friday, January 18, 13

Slide 9

Slide 9 text

CTO of Friday, January 18, 13

Slide 10

Slide 10 text

Chief Taco Officer CTO of Friday, January 18, 13

Slide 11

Slide 11 text

www.paperlesspost.com Friday, January 18, 13

Slide 12

Slide 12 text

Friday, January 18, 13

Slide 13

Slide 13 text

• Most individual user-generated cards sent in a single week: ~2 million • Registered user growth over December 2011: 211% (3.1x) • Revenue growth over December 2011: 135% (2.3x) • Number of commits to main app git repo: 44,545 • Number of commits to chef git repo: 4,447 • Peak traffic served: ~300 Mbps • Number of production deploys across applications: 1,631 • Number of staging deploys across applications: 10,600 • Unique animated gifs posted in Campfire: 3,208 Friday, January 18, 13

Slide 14

Slide 14 text

DA DIP Friday, January 18, 13

Slide 15

Slide 15 text

DA SHIP Friday, January 18, 13

Slide 16

Slide 16 text

WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13

Slide 17

Slide 17 text

It’s about being efficient Automation is not about being lazy Friday, January 18, 13

Slide 18

Slide 18 text

Developers aren’t slow, they’re ... Friday, January 18, 13

Slide 19

Slide 19 text

Distracted Friday, January 18, 13

Slide 20

Slide 20 text

blocked Friday, January 18, 13

Slide 21

Slide 21 text

churning Friday, January 18, 13

Slide 22

Slide 22 text

Complexity = Not just large, but distributed applications get more complex over time Friday, January 18, 13

Slide 23

Slide 23 text

how to allow for growing complexity while not slowing down Friday, January 18, 13

Slide 24

Slide 24 text

- A DevOps Oath I, [YOU], Do solemnly swear to help developers solve their problems, quickly and without [too many] yak- shaves and make their lives easier, so they can ship code, so that the company can improve, so that we all can be successful. I will do what it takes to avoid repeated tasks, help unblock the blocked, and generally make things work better. Friday, January 18, 13

Slide 25

Slide 25 text

You want Friday, January 18, 13

Slide 26

Slide 26 text

Friday, January 18, 13

Slide 27

Slide 27 text

• Notifications Friday, January 18, 13

Slide 28

Slide 28 text

• Notifications • One click deploys Friday, January 18, 13

Slide 29

Slide 29 text

• Notifications • One click deploys • Continuous integration Friday, January 18, 13

Slide 30

Slide 30 text

• Notifications • One click deploys • Continuous integration • Distributed/fast deploys Friday, January 18, 13

Slide 31

Slide 31 text

• Notifications • One click deploys • Continuous integration • Distributed/fast deploys • Zero config Dev environment Friday, January 18, 13

Slide 32

Slide 32 text

• Notifications • One click deploys • Continuous integration • Distributed/fast deploys • Zero config Dev environment • Developer Metrics Friday, January 18, 13

Slide 33

Slide 33 text

• Notifications • One click deploys • Continuous integration • Distributed/fast deploys • Zero config Dev environment • Developer Metrics • Deploy History/Tracking Friday, January 18, 13

Slide 34

Slide 34 text

• Notifications • One click deploys • Continuous integration • Distributed/fast deploys • Zero config Dev environment • Developer Metrics • Deploy History/Tracking • Integrated Troubleshooting Friday, January 18, 13

Slide 35

Slide 35 text

You HAVE Friday, January 18, 13

Slide 36

Slide 36 text

A: Start Small Q: But how to start? Friday, January 18, 13

Slide 37

Slide 37 text

It’s daunting, and incremental improvements reap rewards DON’t DO IT ALL AT ONCE Friday, January 18, 13

Slide 38

Slide 38 text

Distill into a perfect complainy-developer cocktail Start with the whining Friday, January 18, 13

Slide 39

Slide 39 text

Deploys are too complicated, so I end up not doing them ಠ_ಠ Friday, January 18, 13

Slide 40

Slide 40 text

$ git checkout staging $ git pull root staging $ git merge my_branch $ git push root staging $ cap staging deploy:long Friday, January 18, 13

Slide 41

Slide 41 text

WE COULD • Build a web ui to handle deploys • Make a simple script that wraps the common commands • Ignore them, haters gonna hate Friday, January 18, 13

Slide 42

Slide 42 text

Friday, January 18, 13

Slide 43

Slide 43 text

ಠ_ಠ Friday, January 18, 13

Slide 44

Slide 44 text

I keep losing my changes or forget to commit ಠ_ಠ Friday, January 18, 13

Slide 45

Slide 45 text

I keep losing my changes or forget to commit ಠ_ಠ ಠ_ಠ Friday, January 18, 13

Slide 46

Slide 46 text

I keep losing my changes or forget to commit ಠ_ಠ I want notifications when someone deploys ಠ_ಠ Friday, January 18, 13

Slide 47

Slide 47 text

I keep losing my changes or forget to commit ಠ_ಠ I want notifications when someone deploys ಠ_ಠ ಠ_ಠ Friday, January 18, 13

Slide 48

Slide 48 text

I keep losing my changes or forget to commit ಠ_ಠ I want notifications when someone deploys I want to deploy multiple branches at once ಠ_ಠ ಠ_ಠ Friday, January 18, 13

Slide 49

Slide 49 text

For developers it’s a familiar interface Lots of things can be solved with command line tools Friday, January 18, 13

Slide 50

Slide 50 text

Devtools was born Friday, January 18, 13

Slide 51

Slide 51 text

Design an aPI around your common tasks Friday, January 18, 13

Slide 52

Slide 52 text

$ pp rails deploy earth quirkey/branch1 mrb/f2 # pp deploy [branches, ...] Friday, January 18, 13

Slide 53

Slide 53 text

• Does the git flow (checkout, reset, merge, push) • Handles conflicts and stashing changes • Runs the deploy • Sends notifications to email list/ campfire rooms Friday, January 18, 13

Slide 54

Slide 54 text

$ pp boot renderer restart # pp boot [stop|start|restart|status] Friday, January 18, 13

Slide 55

Slide 55 text

• Handles booting local services • Has all powerful boot/restart commands • Monitors service status Friday, January 18, 13

Slide 56

Slide 56 text

• Ruby gem (on internal gemserver) • Self-updating (pp devtools update) • Built on top of Thor • Made to be extensible (easy to add apps, tasks, recipes) DEvTOOLS Friday, January 18, 13

Slide 57

Slide 57 text

WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13

Slide 58

Slide 58 text

Friday, January 18, 13

Slide 59

Slide 59 text

ಠ_ಠ Friday, January 18, 13

Slide 60

Slide 60 text

I need to deploy but I don’t have ssh credentials ಠ_ಠ Friday, January 18, 13

Slide 61

Slide 61 text

I need to deploy but I don’t have ssh credentials ಠ_ಠ ಠ_ಠ Friday, January 18, 13

Slide 62

Slide 62 text

I need to deploy but I don’t have ssh credentials ಠ_ಠ It takes too long to run the test builds ಠ_ಠ Friday, January 18, 13

Slide 63

Slide 63 text

I need to deploy but I don’t have ssh credentials ಠ_ಠ It takes too long to run the test builds ಠ_ಠ ಠ_ಠ Friday, January 18, 13

Slide 64

Slide 64 text

I need to deploy but I don’t have ssh credentials ಠ_ಠ It takes too long to run the test builds Are we deploying? What step are we at? ಠ_ಠ ಠ_ಠ Friday, January 18, 13

Slide 65

Slide 65 text

; http://jenkins-ci.org Jenkins! Friday, January 18, 13

Slide 66

Slide 66 text

It just needs to be told what to deploy (It has an HTTP API) Jenkins can just run devtools Friday, January 18, 13

Slide 67

Slide 67 text

Text # On your machine $ pp rails deploy earth quirkey/branch1 mrb/f2 # Does git merging, sends request to jenkins # On Jenkins, in a `deploy` job $ pp $APP deploy $ENV $BRANCHES --local # Skips the git stuff, just does the actual # deploy and sends the notifications Friday, January 18, 13

Slide 68

Slide 68 text

Jenkins • Global visibility (changed the game) • No fear of losing network connectivity during a deploy • Single point of orchestration • Being able to see the deploys and builds made the team want to make them faster and better Friday, January 18, 13

Slide 69

Slide 69 text

WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13

Slide 70

Slide 70 text

Friday, January 18, 13

Slide 71

Slide 71 text

ಠ_ಠ Friday, January 18, 13

Slide 72

Slide 72 text

What branches are deployed to staging? ಠ_ಠ Friday, January 18, 13

Slide 73

Slide 73 text

What branches are deployed to staging? ಠ_ಠ ಠ_ಠ Friday, January 18, 13

Slide 74

Slide 74 text

What branches are deployed to staging? ಠ_ಠ What just got deployed to production? ಠ_ಠ Friday, January 18, 13

Slide 75

Slide 75 text

What branches are deployed to staging? ಠ_ಠ What just got deployed to production? ಠ_ಠ ಠ_ಠ Friday, January 18, 13

Slide 76

Slide 76 text

What branches are deployed to staging? ಠ_ಠ What just got deployed to production? Johnny just said he pushed something, but I can’t find it ಠ_ಠ ಠ_ಠ Friday, January 18, 13

Slide 77

Slide 77 text

Now we just need to collect the data We unified the entry points Friday, January 18, 13

Slide 78

Slide 78 text

A web UI and database for devtools Enter: PUSH PARTY Friday, January 18, 13

Slide 79

Slide 79 text

Text Friday, January 18, 13

Slide 80

Slide 80 text

And separated them from the CLI We extracted the “tools” of Devtools Friday, January 18, 13

Slide 81

Slide 81 text

> require 'paperless-devtools' > app = PaperlessDevtools.app :rails > app.tools(:git).current_branch #=> "master" > app.tools(:jenkins).start_build("master") #=> send build to jenkins Friday, January 18, 13

Slide 82

Slide 82 text

Rails, Postgres, Devtools, Compass Then we built a WebApp around it Friday, January 18, 13

Slide 83

Slide 83 text

Friday, January 18, 13

Slide 84

Slide 84 text

devtools makes api calls to pushparty Friday, January 18, 13

Slide 85

Slide 85 text

devtools makes api calls to pushparty which records deploys in postgres Friday, January 18, 13

Slide 86

Slide 86 text

devtools makes api calls to pushparty which records deploys in postgres then uses devtools to send them to jenkins Friday, January 18, 13

Slide 87

Slide 87 text

devtools makes api calls to pushparty which records deploys in postgres then uses devtools to send them to jenkins which uses devtools to send progress and status to pushparty Friday, January 18, 13

Slide 88

Slide 88 text

devtools makes api calls to pushparty which records deploys in postgres then uses devtools to send them to jenkins which uses devtools to send progress and status to pushparty which uses devtools to announce the completed deploy in campfire Friday, January 18, 13

Slide 89

Slide 89 text

devtools makes api calls to pushparty which records deploys in postgres then uses devtools to send them to jenkins which uses devtools to send progress and status to pushparty which uses devtools to announce the completed deploy in campfire Friday, January 18, 13

Slide 90

Slide 90 text

Extremely easy to see what was merged/ deployed, in what state and at what time Now we have DATA Friday, January 18, 13

Slide 91

Slide 91 text

BRANCHES FLOW Friday, January 18, 13

Slide 92

Slide 92 text

WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13

Slide 93

Slide 93 text

Working with chef [with multiple team members working on multiple projects in multiple staging environments] is a pain in the butt. ಠ_ಠ Friday, January 18, 13

Slide 94

Slide 94 text

$ pp chef deploy earth johnnydtan/graphite_cluster # git merging # cookbook testing # cookbook uploading # environment munging Friday, January 18, 13

Slide 95

Slide 95 text

http://www.meetup.com/Chef-NYC/ Come to #CHEF NYC Friday, January 18, 13

Slide 96

Slide 96 text

WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13

Slide 97

Slide 97 text

We have all this data in graphite, chef, sensu, pushparty, but when theres an issue it’s hard to coalate the problems ಠ_ಠ Friday, January 18, 13

Slide 98

Slide 98 text

Devtools wrappers for different tools, pulled into a single web ui HOTPOT Friday, January 18, 13

Slide 99

Slide 99 text

Friday, January 18, 13

Slide 100

Slide 100 text

Awesome things I don’t have time to talk about Friday, January 18, 13

Slide 101

Slide 101 text

Awesome things I don’t have time to talk about • pp chef create_node Friday, January 18, 13

Slide 102

Slide 102 text

Awesome things I don’t have time to talk about • pp chef create_node • pp vagrant setup Friday, January 18, 13

Slide 103

Slide 103 text

Awesome things I don’t have time to talk about • pp chef create_node • pp vagrant setup • pushparty bashrc’s Friday, January 18, 13

Slide 104

Slide 104 text

Awesome things I don’t have time to talk about • pp chef create_node • pp vagrant setup • pushparty bashrc’s • graphite integration Friday, January 18, 13

Slide 105

Slide 105 text

Awesome things I don’t have time to talk about • pp chef create_node • pp vagrant setup • pushparty bashrc’s • graphite integration • deploy build time graphs Friday, January 18, 13

Slide 106

Slide 106 text

Awesome things I don’t have time to talk about • pp chef create_node • pp vagrant setup • pushparty bashrc’s • graphite integration • deploy build time graphs • more every day Friday, January 18, 13

Slide 107

Slide 107 text

Things to remember Friday, January 18, 13

Slide 108

Slide 108 text

Build for your team for right now, not for every team at any size Don’t expect to build the perfect system Friday, January 18, 13

Slide 109

Slide 109 text

Not the other way around Start with a workflow then tool around it Friday, January 18, 13

Slide 110

Slide 110 text

Then add the cool shit that they didn’t know they needed Build what your team needs first Friday, January 18, 13

Slide 111

Slide 111 text

WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13

Slide 112

Slide 112 text

github.com/quirkey twitter.com/aq quirkey.com paperlesspost.com THANKS! Friday, January 18, 13