ChatOps at GitHub

Hubot, GitHub's open source chat bot, is completely revolutionizing how we do ops at GitHub. Automating deployment, graphing, monitoring, provisioning, tweeting, and many other things with Hubot has enabled and enhanced our culture of remote work. We're standardizing how we interact with the applications and servers that run GitHub by continuously expanding our library of Hubot commands that any GitHubber can run in multiple Campfire chat rooms. Interacting with Hubot in the middle other of conversations has increased our awareness of what other GitHubbers are working on and the speed at which new team members learn common practices. In this talk, I'll give examples of how we deploy and interact with github.com via Hubot, describe benefits we've seen from this approach, and describe how you can bring ChatOps into your daily workflow.

Transcript

1. at
   ChatOps
   GitHub
   

   View Slide

2. Jesse Newland
   jnewland
   

   View Slide

3. at
   ChatOps
   GitHub
   

   View Slide

4. *
   

   View Slide

5. View Slide

6. View Slide

7. git checkout -b git-gh13
   # bump version number
   git commit -am ‘bump git version’
   git push origin git-gh13
   # create pull request
   github/puppet
   

   View Slide

8. Build #108816
   (5fe75932f26ea62cb5fc5e3d0cb302cc2461d11e)
   of puppet/git-gh13 was successful(421s) github/
   puppet@567ea48...5fe7593
   

   View Slide

9. hubot deploy puppet to production
   puppet noop git-gh13 fs1
   puppet force git-gh13 fs1
   graph me -1h @collectd.load(fs1)
   # merge pull request
   # ci runs
   # auto-deploy
   hubot puppet force production fs
   graph me -1h @collectd.load(fs*)
   log me hooks github/github
   

   View Slide

10. ** [out :: REDACTED ] Bootstrapping...
    ** [out :: REDACTED ] Gem environment up-to-date.
    ** [out :: REDACTED ] Running librarian-puppet...
    ** [out :: REDACTED ] Generating puppet environments...
    ** [out :: REDACTED ] Cleaning up deleted branches...
    ** [out :: REDACTED ] Done!
    ** [out :: REDACTED ] Sending 'restart' command
    ** [out :: REDACTED ] The following watches were affected:
    ** [out :: REDACTED ] puppetmaster_unicorn
    ...
    

    View Slide

11. hubot deploy puppet to production
    puppet noop git-gh13 fs1
    puppet force git-gh13 fs1
    graph me -1h @collectd.load(fs1)
    # merge pull request
    # ci runs
    # auto-deploy
    hubot puppet force production fs
    graph me -1h @collectd.load(fs*)
    log me hooks github/github
    

    View Slide

12. ...
    ** [out :: fs1a.rs.github.com] info: Applying
    configuration version
    '8fb1a2716d5f950b836e511471a2bdac3ed27090'
    ** [out :: fs1a.rs.github.com] notice: /Stage[main]/
    Github::Common_packages/Package[git]/ensure: would have
    changed from '1:1.7.10-1+github12' to
    '1:1.7.10-1+github13'
    ...
    

    View Slide

13. hubot deploy puppet to production
    puppet noop git-gh13 fs1
    puppet force git-gh13 fs1
    graph me -1h @collectd.load(fs1)
    # merge pull request
    # ci runs
    # auto-deploy
    hubot puppet force production fs
    graph me -1h @collectd.load(fs*)
    log me hooks github/github
    

    View Slide

14. ** [out :: fs1a.rs.github.com] info: Applying
    configuration version
    ** [out :: fs1a.rs.github.com] notice: /Stage[main]/
    Github::Common_packages/Package[git]/ensure: ensure
    changed '1:1.7.10-1+github12' to '1:1.7.10-1+github13'
    ...
    

    View Slide

15. hubot deploy puppet to production
    puppet noop git-gh13 fs1
    puppet force git-gh13 fs1
    graph me -1h @collectd.load(fs1)
    # merge pull request
    # ci runs
    # auto-deploy
    hubot puppet force production fs
    graph me -1h @collectd.load(fs*)
    log me hooks github/github
    

    View Slide

16. View Slide

17. hubot deploy puppet to production
    puppet noop git-gh13 fs1
    puppet force git-gh13 fs1
    graph me -1h @collectd.load(fs1)
    # merge pull request
    # ci runs
    # auto-deploy
    hubot puppet force production fs
    graph me -1h @collectd.load(fs*)
    log me hooks github/github
    

    View Slide

18. [puppet/master] Merge pull request #8406
    from github/git-gh13 - Jesse Newland
    ( github/puppet@567ea48 )
    

    View Slide

19. hubot deploy puppet to production
    puppet noop git-gh13 fs1
    puppet force git-gh13 fs1
    graph me -1h @collectd.load(fs1)
    # merge pull request
    # ci runs
    # auto-deploy
    hubot puppet force production fs
    graph me -1h @collectd.load(fs*)
    log me hooks github/github
    

    View Slide

20. Build #108817
    (5fe75932f26ea62cb5fc5e3d0cb302cc2461d11e)
    of puppet/master was successful(399s)
    

    View Slide

21. hubot deploy puppet to production
    puppet noop git-gh13 fs1
    puppet force git-gh13 fs1
    graph me -1h @collectd.load(fs1)
    # merge pull request
    # ci runs
    # auto-deploy
    hubot puppet force production fs
    graph me -1h @collectd.load(fs*)
    log me hooks github/github
    

    View Slide

22. jnewland is deploying puppet/master
    (863f7ff..567ea48) to production [#37789]
    

    View Slide

23. ** [out :: REDACTED ] Bootstrapping...
    ** [out :: REDACTED ] Gem environment up-to-date.
    ** [out :: REDACTED ] Running librarian-puppet...
    ** [out :: REDACTED ] Generating puppet environments...
    ** [out :: REDACTED ] Cleaning up deleted branches...
    ** [out :: REDACTED ] Done!
    ** [out :: REDACTED ] Sending 'restart' command
    ** [out :: REDACTED ] The following watches were affected:
    ** [out :: REDACTED ] puppetmaster_unicorn
    ...
    

    View Slide

24. hubot deploy puppet to production
    puppet noop git-gh13 fs1
    puppet force git-gh13 fs1
    graph me -1h @collectd.load(fs*)
    # merge pull request
    # ci runs
    # auto-deploy
    hubot puppet force production fs
    graph me -1h @collectd.load(fs*)
    log me hooks github/github
    

    View Slide

25. ** [out :: fs2a.rs.github.com] info: Applying
    configuration version
    '8fb1a2716d5f950b836e511471a2bdac3ed27090'
    ** [out :: fs7b.rs.github.com] info: Applying
    configuration version
    '8fb1a2716d5f950b836e511471a2bdac3ed27090'
    ** [out :: fs2a.rs.github.com] notice: /Stage[main]/
    Github::Common_packages/Package[git]/ensure: ensure
    changed '1:1.7.10-1+github12' to '1:1.7.10-1+github13'
    ** [out :: fs7b.rs.github.com] notice: /Stage[main]/
    Github::Common_packages/Package[git]/ensure: ensure
    changed '1:1.7.10-1+github12' to '1:1.7.10-1+github13'
    ...
    

    View Slide

26. hubot deploy puppet to production
    puppet noop git-gh13 fs1
    puppet force git-gh13 fs1
    graph me -1h @collectd.load(fs1)
    # merge pull request
    # ci runs
    # auto-deploy
    hubot puppet force production fs
    graph me -1h @collectd.load(fs*)
    log me hooks github/github
    

    View Slide

27. View Slide

28. hubot deploy puppet to production
    puppet noop git-gh13 fs1
    puppet force git-gh13 fs1
    graph me -1h @collectd.load(fs1)
    # merge pull request
    # ci runs
    # auto-deploy
    hubot puppet force production fs
    graph me -1h @collectd.load(fs*)
    log me hooks github/github
    

    View Slide

29. View Slide

30. Boom
    

    View Slide

31. ChatOps
    

    View Slide

32. 
    Hubot
    

    View Slide

33. heaven
    janky
    shell
    graphme
    Hubot
    

    View Slide

34. hubot
    heaven
    janky
    shell
    graphme
    ALL OF
    THE APIS
    

    View Slide

35. Why is this stupid
    chat bot so
    important to Ops?
    

    View Slide

36. hubot deploy puppet to production
    puppet noop git-gh13 fs1
    puppet force git-gh13 fs1
    graph me -1h @collectd.load(fs1)
    # merge pull request
    # ci runs
    # auto-deploy
    hubot puppet force production fs
    graph me -1h @collectd.load(fs*)
    log me hooks github/github
    

    View Slide

37. Everyone sees all
    of that happen
    

    View Slide

38. Everyone sees all
    of that happen
    on their first day
    

    View Slide

39. View Slide

40. hubot ci status github/smoke-perf
    

    View Slide

41. hubot deploy github/smoke-perf to prod/fe1
    

    View Slide

42. hubot graph me -10min @app-perf
    

    View Slide

43. hubot procs unicorn
    

    View Slide

44. hubot resque critical
    

    View Slide

45. hubot graph me -10min @collectd.load(fe*)
    

    View Slide

46. hubot conns fe1
    

    View Slide

47. hubot log me smoke fe1
    

    View Slide

48. hubot lbctl disable fe1
    

    View Slide

49. hubot status yellow Bad deploy. Reverting now.
    

    View Slide

50. hubot who’s on call
    

    View Slide

51. hubot pingdom checks
    

    View Slide

52. hubot upset me
    

    View Slide

53. hubot deploy github to prod/fe1
    

    View Slide

54. hubot log me smoke fe1
    

    View Slide

55. hubot air drum me
    

    View Slide

56. hubot lbctl enable fe1
    

    View Slide

57. hubot status green All systems go.
    

    View Slide

58. hubot whois 4.9.23.22
    

    View Slide

59. hubot khanify spammers
    

    View Slide

60. hubot play in the air tonight
    

    View Slide

61. ChatOps
    

    View Slide

62. By placing tools
    directly in the
    middle of the
    conversation
    

    View Slide

63. Everyone
    is pairing
    all of the time
    

    View Slide

64. Teaching
    by
    doing
    

    View Slide

65. This was always my main
    motivation with hubot - teaching
    by doing by making things
    visible. It's an extremely
    powerful teaching
    technique - @rtomayko
    

    View Slide

66. View Slide

67. Communicate
    by
    doing
    

    View Slide

68. THINGS I
    HAVEN’T ASKED
    RECENTLY
    

    View Slide

69. THINGS I
    HAVEN’T ASKED
    RECENTLY
    how’s that deploy going?
    

    View Slide

70. THINGS I
    HAVEN’T ASKED
    RECENTLY
    how’s that deploy going?
    are you deploying that or should i?
    

    View Slide

71. THINGS I
    HAVEN’T ASKED
    RECENTLY
    how’s that deploy going?
    are you deploying that or should i?
    is anyone responding to that nagios alert?
    

    View Slide

72. THINGS I
    HAVEN’T ASKED
    RECENTLY
    how’s that deploy going?
    are you deploying that or should i?
    is anyone responding to that nagios alert?
    is that branch green?
    

    View Slide

73. THINGS I
    HAVEN’T ASKED
    RECENTLY
    is that branch green?
    how’s that deploy going?
    are you deploying that or should i?
    is anyone responding to that nagios alert?
    how does load look?
    

    View Slide

74. THINGS I
    HAVEN’T ASKED
    RECENTLY
    is that branch green?
    how’s that deploy going?
    are you deploying that or should i?
    is anyone responding to that nagios alert?
    how does load look?
    did anyone update the status page?
    

    View Slide

75. THINGS I
    HAVEN’T ASKED
    RECENTLY
    how’s that deploy going?
    are you deploying that or should i?
    is anyone responding to that nagios alert?
    is that branch green?
    how does load look?
    did that deploy finish?
    did anyone update the status page?
    

    View Slide

76. View Slide

77. View Slide

78. http://www.flickr.com/photos/7997249@N06/6061305639/
    

    View Slide

79. Hide
    the
    ugly
    

    View Slide

80. View Slide

81. [nines] hubot opened issue #4263: Nagios
    (229906) - fs3b/syslog - Tue Sept 25 23:40:18
    PDT 2012. github/nines#4263
    

    View Slide

82. hubot nagios ack fs3b/syslog
    # fix stuff
    nagios check fs3b/syslog
    nagios status fs3b/syslog
    hubot nagios downtime fs3b/syslog 90
    nagios mute fs3b/syslog
    nagios unmute fs3b/syslog
    

    View Slide

83. View Slide

84. View Slide

85. Mobile
    FTW
    

    View Slide

86. View Slide

87. ChatOps
    

    View Slide

88. View Slide

89. View Slide

90. Work at GitHub
    jesse@github.com
    

    View Slide