ChatOps: Infrastructure As Conversation

Those tasked with building and maintaining resilient services and systems at scale are aware of the concepts behind “Infrastructure as Code”. Configuration management tools and services such as Puppet, Ansible, Chef, and others have become a staple in the provisioning, management, and more of critical components of systems worldwide.

With the growing popularity of ChatOps (the concept of moving actions and context related to Operations into a persistent group-chat tool) more and more teams are beginning to find even more efficiencies and benefits. What does it look like when teams have begun to manage much of their infrastructure from the interface of group-chat? What concerns and pitfalls must be considered? Where should teams begin their efforts towards a ChatOps approach? What’s on the horizon and beyond as more and more shift their actions away from the command-line-interface and web browsers and begin to leverage API’s, bots, and group-chat tools in a secure and consistent way?

Transcript

1. CHATOPS:
   INFRASTRUCTURE AS CONVERSATION
   

   View Slide

2. JASON HAND
   jasonhand
   #ChatOps
   

   View Slide

3. @jasonhand ~ #ChatOps
   https://www.cheatography.com/uploads/davidpol_1439395467_Agile%20Manifesto.png
   

   View Slide

4. Delivery Pipeline
   Plan Code Build Test Release Deploy Operate
   Agile Infrastructure (DevOps)
   Continuous Delivery
   Continuous Integration
   Agile Development
   @jasonhand ~ #ChatOps
   

   View Slide

5. @jasonhand ~ #ChatOps
   http://2.bp.blogspot.com/-3Vv68jBEYdo/VfGlfe3aB0I/AAAAAAAAAMo/vrJl7PBd8m0/s1600/contact-center-acd-bottleneck.jpg
   

   View Slide

6. USER & CUSTOMER EXPECTATIONS
   Users expect working software,
   high availability, innovation, and
   a sense of empathy
   

   View Slide

7. AVAILABILITY & RELIABILITY
   High availability is the result of
   high resiliency & reliability
   

   View Slide

8. AVAILABILITY & RELIABILITY
   High resiliency & reliability is
   the result of improved MTTR
   Fail Fast == Respond & Recover Fast
   

   View Slide

9. Continuous Delivery
   @jasonhand ~ #ChatOps
   

   View Slide

10. Continuous Improvement
    @jasonhand ~ #ChatOps
    

    View Slide

11. A
    CONTINUOUS IMPROVEMENT
    It is not enough to do your best;
    you must know what to do, and
    then do your best.
    - W. Edwards Deming
    Act
    Check
    Do
    @jasonhand ~ #ChatOps
    Plan
    

    View Slide

12. @jasonhand ~ #ChatOps
    

    View Slide

13. DevOps Buzzword Bingo
    Continuous Integration
    Continuous Testing
    Continuous Delivery
    Continuous Monitoring
    Continuous Deployment ChatOps
    Containerization
    Microservices
    High Availability
    Devs On-call
    Delivery Pipeline
    Configuration Management
    Infrastructure As Code
    Virtualization
    Immutable Infrastructure
    Feedback Loops
    Experimentation
    Culture
    Release Automation
    Lean
    Kanban
    @jasonhand ~ #ChatOps
    Measurement Sharing
    

    View Slide

14. DEVOPS PRINCIPLES
    CULTURE
    AUTOMATION
    MEASUREMENT
    SHARING
    LEAN
    @jasonhand ~ #ChatOps
    

    View Slide

15. What is ChatOps?
    ChatOps is an approach to communication that allows teams to collaborate and manage many
    aspects of their infrastructure, code, and data from the comfort and safety of a chat room.
    Through the use of chatbots and scripts, teams can execute commands, query information, and
    distribute knowledge across the entire organization and all from the space where conversations are
    already taking place, a chat room.
    “We're taking the things we're already doing and putting them
    in line with the conversations we're already having”
    - James Fryman (StackStorm)
    @jasonhand ~ #ChatOps
    

    View Slide

16. 1
    1 1
    2 1
    3 1
    4
    Origins
    IRC XMPP/
    Jabber
    Campfire HipChat/
    Slack/Flowdock
    api
    @jasonhand ~ #ChatOps
    “Conversation-driven development”
    

    View Slide

17. A
    HUBOT: THE EARLY YEARS
    Placing tools right in the middle of
    the conversation
    - Jesse Newland (GitHub) - 2012
    @jasonhand ~ #ChatOps
    

    View Slide

18. Problems
    Devs & Ops
    Teams
    Have
    Problems
    Our
    Bots
    Solve
    Maturation of ChatOps : Phase 1
    adopted from @davidjbland
    @jasonhand ~ #ChatOps
    

    View Slide

19. Problems
    Devs & Ops
    Teams
    Have
    Problems
    Our
    Bots
    Solve
    Maturation of ChatOps : Phase 2
    adopted from @davidjbland
    @jasonhand ~ #ChatOps
    

    View Slide

20. GitHub
    Pull Request Notifier
    Merge code
    Show current status
    Display “commit” link
    Commit Notifier
    Display “issue” link
    Display repo commuters
    Show Open Pull Requests
    Show Repo Activity
    Show open issues
    Search for string in code
    @jasonhand ~ #ChatOps
    

    View Slide

21. 1
    1 1
    2 1
    3 1
    4
    THE NEW INTERFACE
    Notifications/
    Information
    Alerting/
    Paging
    Command
    Execution
    Collaboration/
    Communication
    @jasonhand ~ #ChatOps
    Feedback Loops
    Lower MTTR
    Higher availability & happier users
    

    View Slide

22. A
    THE NEW INTERFACE
    A tool to help the way our people
    systems work that in turn helps the
    way our technical systems work.
    - Mark Imbriaco (operable.io)
    https://youtu.be/pCVvYCjvoZI
    @jasonhand ~ #ChatOps
    

    View Slide

23. SHIELDS UP
    @jasonhand ~ #ChatOps
    laserfirecreations.com/Large%20Images/ShldsUpNew.jpg
    https://stardate.org/sites/default/files/images/gallery/10%2012%2018.jpg
    DDOS
    Attack
    Shields Up!
    

    View Slide

24. Problems
    Devs &
    Ops
    Teams
    Have
    Problems
    Our
    Bots
    Solve
    Maturation of ChatOps : Phase 3
    adopted from @davidjbland
    @jasonhand ~ #ChatOps
    

    View Slide

25. Timelines
    @jasonhand ~ #ChatOps
    

    View Slide

26. SITUATIONAL AWARENESS
    @jasonhand ~ #ChatOps
    blog.vmware.com
    

    View Slide

27. A
    SITUATIONAL AWARENESS
    What makes a great team —
    everyone on the field is aware of
    everything, all the time. The team
    with the most situational
    awareness will win.
    - Mary Poppendieck (Lean
    Software Development)
    DevOpsDays
    Minneapolis - 2015
    @jasonhand ~ #ChatOps
    

    View Slide

28. Examples
    @jasonhand ~ #ChatOps
    

    View Slide

29. Deploy Code
    git push origin master
    Counting objects: 7, done.
    Compressing objects: 100% (4/4), done.
    Writing objects: 100% (4/4), 407 bytes, done.
    Total 4 (delta 2), reused 0 (delta 0)
    To git@github.com:qrush/gitready.git
    361303d..f2cd831 master -> master
    jhand$ git commit -am "Adding ChatOps link"
    Created commit f2cd831: Adding ChatOps link
    1 files changed, 1 insertions(+), 0 deletions(-)
    @jasonhand ~ #ChatOps
    jhand$
    

    View Slide

30. Deploy Code
    Created commit f2cd831: Adding ChatOps link
    1 files changed, 1 insertions(+), 0 deletions(-)
    foobot git commit -am "Adding ChatOps link"
    @jasonhand ~ #ChatOps
    Counting objects: 7, done.
    Compressing objects: 100% (4/4), done.
    Writing objects: 100% (4/4), 407 bytes, done.
    Total 4 (delta 2), reused 0 (delta 0)
    To git@github.com:qrush/gitready.git
    361303d..f2cd831 master -> master
    foobot git push origin master
    

    View Slide

31. Build Infrastructure
    Platform install commit-1248994 (#789).
    Started by user: jhand (09.04.2015.11:03:24)
    Platform install commit-1248994 complete
    jhand$ build commit-1248994
    @jasonhand ~ #ChatOps
    

    View Slide

32. Build Infrastructure
    building commit-1248994
    Platform install commit-1248994 (#789).
    Started by user: jhand (09.04.2015.11:03:24)
    Platform install commit-1248994 complete
    foobot build commit-1248994
    @jasonhand ~ #ChatOps
    

    View Slide

33. Manage Incidents
    Incident 778467 has been
    acknowledged by (jhand)
    Stopped paging jhand -
    (Backend Engineering Team) @
    2:51 p.m. MT
    jhand$ ack incident 778467
    @jasonhand ~ #ChatOps
    

    View Slide

34. Manage Incidents
    Incident 778467 has been
    acknowledged by (jhand)
    Stopped paging jhand -
    (Backend Engineering Team) @
    2:51 p.m. MT
    foobot ack me incident 778467
    @jasonhand ~ #ChatOps
    

    View Slide

35. Manage Incidents
    Runbook says to look at the graph first
    Please review runbook for prod2.cass24.victorops.com
    @jasonhand ~ #ChatOps
    

    View Slide

36. Pull Graphs
    foobot graph me prod2.cass24.victorops.com - cpu -15
    @jasonhand ~ #ChatOps
    

    View Slide

37. Kick Services
    foobot restart apache 192.168.0.345
    @jasonhand ~ #ChatOps
    apache on host 192.168.0.345 has been restarted
    one moment .. restarting apache on requested host
    

    View Slide

38. Moving terminal and API to Chat
    @jasonhand ~ #ChatOps
    

    View Slide

39. 1
    1 1
    2 1
    3 1
    4
    Benefits of ChatOps
    Sharing Speed Learning Security
    @jasonhand ~ #ChatOps
    

    View Slide

40. Security continued
    @jasonhand ~ #ChatOps
    Security, compliance,
    and regulatory
    requirements https://youtu.be/18PNEwt18P0
    

    View Slide

41. Bots
    github.com/litaio/lita
    @jasonhand ~ #ChatOps
    github.com/github/hubot
    github.com/errbotio/errbot
    Lita
    

    View Slide

42. More Bots
    github.com/devth/yetibot
    @jasonhand ~ #ChatOps
    Hal-9001
    github.com/Netflix/hal-9001
    github.com/elastic/jarvis
    (Previously Scorebot)
    github.com/micro/micro/tree/master/bot
    

    View Slide

43. Cog : ChatOps Platform
    operable.io
    @jasonhand ~ #ChatOps
    github.com/operable/cog
    w/ Access Control Mechanism
    

    View Slide

44. Stackstorm
    @jasonhand ~ #ChatOps
    stackstorm.com
    Event-driven
    automation
    

    View Slide

45. http://eng.localytics.com/serverless-slackbots-powered-by-aws/
    

    View Slide

46. OSS (Persistent) Group Chat
    @jasonhand ~ #ChatOps
    Zulip Jappix Mattermost
    RocketChat
    

    View Slide

47. It’s getting even easier
    @jasonhand ~ #ChatOps
    

    View Slide

48. @jasonhand ~ #ChatOps
    3rd-party App Directories
    

    View Slide

49. Problems
    Devs & Ops
    Teams
    Have
    Problems
    Our
    Bots
    Solve
    Maturation of ChatOps : Phase 4 & Beyond
    adopted from @davidjbland
    @jasonhand ~ #ChatOps
    

    View Slide

50. CONCERNS
    ALERT FATIGUE
    CONTEXT SWITCHING
    LARGE TEAMS
    AUTHENTICATION
    @jasonhand ~ #ChatOps
    

    View Slide

51. ADDITIONAL RESOURCES
    reddit.com/r/chatops
    @jasonhand
    talk-chatops.slack.com
    twitter.com/hashtag/chatops
    medium.com/search?q=chatops
    jasonhand.com/tag/chatops
    

    View Slide

52. ADDITIONAL RESOURCES
    https://youtu.be/9C2aZIhVk3U
    https://youtu.be/mBbyO4UsWiE
    

    View Slide

53. ChatOps For Dummies
    jhand.co/ChatOps4Dummies
    

    View Slide

54. ChatOps on O’Reilly
    jhand.co/ChatOpsSurvey
    

    View Slide

55. THANK YOU!
    Slides:
    jhand.co/1siEWHd
    

    View Slide

56. !56
    More
    https://medium.com/@cloudoapp/we-asked-hipchat-what-their-next-big-move-is-will-hipchat-join-the-bot-hype-heres-their-answer-90bae12e0cce#.pvzw4tzf7
    https://www.usenix.org/conference/srecon16/program/presentation/tobey
    https://getcarina.com/blog/moving-chatops-into-carina/
    http://dev.haufe.com/irc-chatops/
    https://blog.micro.mu/2016/04/25/the-micro-bot.html
    http://www.infoworld.com/article/3062703/devops/how-chatops-can-revolutionize-your-business.html
    https://aws.amazon.com/blogs/aws/new-slack-integration-blueprints-for-aws-lambda/
    https://victorops.com/chatops/
    http://www.jasonhand.com/tag/chatops/
    

    View Slide

57. !57
    Abstract
    Those tasked with building and maintaining resilient services and systems at scale are
    aware of the concepts behind “Infrastructure as Code”. Configuration management
    tools and services such as Puppet, Ansible, Chef, and others have become a staple in
    the provisioning, management, and more of critical components of systems worldwide.
    With the growing popularity of ChatOps (the concept of moving actions and context
    related to Operations into a persistent group-chat tool) more and more teams are
    beginning to find even more efficiencies and benefits. What does it look like when
    teams have begun to manage much of their infrastructure from the interface of group-
    chat? What concerns and pitfalls must be considered? Where should teams begin their
    efforts towards a ChatOps approach? What’s on the horizon and beyond as more and
    more shift their actions away from the command-line-interface and web browsers and
    begin to leverage API’s, bots, and group-chat tools in a secure and consistent way?
    

    View Slide