Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ChatOps: Infrastructure As Conversation

ChatOps: Infrastructure As Conversation

Those tasked with building and maintaining resilient services and systems at scale are aware of the concepts behind “Infrastructure as Code”. Configuration management tools and services such as Puppet, Ansible, Chef, and others have become a staple in the provisioning, management, and more of critical components of systems worldwide.

With the growing popularity of ChatOps (the concept of moving actions and context related to Operations into a persistent group-chat tool) more and more teams are beginning to find even more efficiencies and benefits. What does it look like when teams have begun to manage much of their infrastructure from the interface of group-chat? What concerns and pitfalls must be considered? Where should teams begin their efforts towards a ChatOps approach? What’s on the horizon and beyond as more and more shift their actions away from the command-line-interface and web browsers and begin to leverage API’s, bots, and group-chat tools in a secure and consistent way?

j.hand

May 17, 2016
Tweet

More Decks by j.hand

Other Decks in Technology

Transcript

  1. CHATOPS:
    INFRASTRUCTURE AS CONVERSATION

    View Slide

  2. JASON HAND
    jasonhand
    #ChatOps

    View Slide

  3. @jasonhand ~ #ChatOps
    https://www.cheatography.com/uploads/davidpol_1439395467_Agile%20Manifesto.png

    View Slide

  4. Delivery Pipeline
    Plan Code Build Test Release Deploy Operate
    Agile Infrastructure (DevOps)
    Continuous Delivery
    Continuous Integration
    Agile Development
    @jasonhand ~ #ChatOps

    View Slide

  5. @jasonhand ~ #ChatOps
    http://2.bp.blogspot.com/-3Vv68jBEYdo/VfGlfe3aB0I/AAAAAAAAAMo/vrJl7PBd8m0/s1600/contact-center-acd-bottleneck.jpg

    View Slide

  6. USER & CUSTOMER EXPECTATIONS
    Users expect working software,
    high availability, innovation, and
    a sense of empathy

    View Slide

  7. AVAILABILITY & RELIABILITY
    High availability is the result of
    high resiliency & reliability

    View Slide

  8. AVAILABILITY & RELIABILITY
    High resiliency & reliability is
    the result of improved MTTR
    Fail Fast == Respond & Recover Fast

    View Slide

  9. Continuous Delivery
    @jasonhand ~ #ChatOps

    View Slide

  10. Continuous Improvement
    @jasonhand ~ #ChatOps

    View Slide

  11. A
    CONTINUOUS IMPROVEMENT
    It is not enough to do your best;
    you must know what to do, and
    then do your best.
    - W. Edwards Deming
    Act
    Check
    Do
    @jasonhand ~ #ChatOps
    Plan

    View Slide

  12. @jasonhand ~ #ChatOps

    View Slide

  13. DevOps Buzzword Bingo
    Continuous Integration
    Continuous Testing
    Continuous Delivery
    Continuous Monitoring
    Continuous Deployment ChatOps
    Containerization
    Microservices
    High Availability
    Devs On-call
    Delivery Pipeline
    Configuration Management
    Infrastructure As Code
    Virtualization
    Immutable Infrastructure
    Feedback Loops
    Experimentation
    Culture
    Release Automation
    Lean
    Kanban
    @jasonhand ~ #ChatOps
    Measurement Sharing

    View Slide

  14. DEVOPS PRINCIPLES
    CULTURE
    AUTOMATION
    MEASUREMENT
    SHARING
    LEAN
    @jasonhand ~ #ChatOps

    View Slide

  15. What is ChatOps?
    ChatOps is an approach to communication that allows teams to collaborate and manage many
    aspects of their infrastructure, code, and data from the comfort and safety of a chat room.
    Through the use of chatbots and scripts, teams can execute commands, query information, and
    distribute knowledge across the entire organization and all from the space where conversations are
    already taking place, a chat room.
    “We're taking the things we're already doing and putting them
    in line with the conversations we're already having”
    - James Fryman (StackStorm)
    @jasonhand ~ #ChatOps

    View Slide

  16. 1
    1 1
    2 1
    3 1
    4
    Origins
    IRC XMPP/
    Jabber
    Campfire HipChat/
    Slack/Flowdock
    api
    @jasonhand ~ #ChatOps
    “Conversation-driven development”

    View Slide

  17. A
    HUBOT: THE EARLY YEARS
    Placing tools right in the middle of
    the conversation
    - Jesse Newland (GitHub) - 2012
    @jasonhand ~ #ChatOps

    View Slide

  18. Problems
    Devs & Ops
    Teams
    Have
    Problems
    Our
    Bots
    Solve
    Maturation of ChatOps : Phase 1
    adopted from @davidjbland
    @jasonhand ~ #ChatOps

    View Slide

  19. Problems
    Devs & Ops
    Teams
    Have
    Problems
    Our
    Bots
    Solve
    Maturation of ChatOps : Phase 2
    adopted from @davidjbland
    @jasonhand ~ #ChatOps

    View Slide

  20. GitHub
    Pull Request Notifier
    Merge code
    Show current status
    Display “commit” link
    Commit Notifier
    Display “issue” link
    Display repo commuters
    Show Open Pull Requests
    Show Repo Activity
    Show open issues
    Search for string in code
    @jasonhand ~ #ChatOps

    View Slide

  21. 1
    1 1
    2 1
    3 1
    4
    THE NEW INTERFACE
    Notifications/
    Information
    Alerting/
    Paging
    Command
    Execution
    Collaboration/
    Communication
    @jasonhand ~ #ChatOps
    Feedback Loops
    Lower MTTR
    Higher availability & happier users

    View Slide

  22. A
    THE NEW INTERFACE
    A tool to help the way our people
    systems work that in turn helps the
    way our technical systems work.
    - Mark Imbriaco (operable.io)
    https://youtu.be/pCVvYCjvoZI
    @jasonhand ~ #ChatOps

    View Slide

  23. SHIELDS UP
    @jasonhand ~ #ChatOps
    laserfirecreations.com/Large%20Images/ShldsUpNew.jpg
    https://stardate.org/sites/default/files/images/gallery/10%2012%2018.jpg
    DDOS
    Attack
    Shields Up!

    View Slide

  24. Problems
    Devs &
    Ops
    Teams
    Have
    Problems
    Our
    Bots
    Solve
    Maturation of ChatOps : Phase 3
    adopted from @davidjbland
    @jasonhand ~ #ChatOps

    View Slide

  25. Timelines
    @jasonhand ~ #ChatOps

    View Slide

  26. SITUATIONAL AWARENESS
    @jasonhand ~ #ChatOps
    blog.vmware.com

    View Slide

  27. A
    SITUATIONAL AWARENESS
    What makes a great team —
    everyone on the field is aware of
    everything, all the time. The team
    with the most situational
    awareness will win.
    - Mary Poppendieck (Lean
    Software Development)
    DevOpsDays
    Minneapolis - 2015
    @jasonhand ~ #ChatOps

    View Slide

  28. Examples
    @jasonhand ~ #ChatOps

    View Slide

  29. Deploy Code
    git push origin master
    Counting objects: 7, done.
    Compressing objects: 100% (4/4), done.
    Writing objects: 100% (4/4), 407 bytes, done.
    Total 4 (delta 2), reused 0 (delta 0)
    To [email protected]:qrush/gitready.git
    361303d..f2cd831 master -> master
    jhand$ git commit -am "Adding ChatOps link"
    Created commit f2cd831: Adding ChatOps link
    1 files changed, 1 insertions(+), 0 deletions(-)
    @jasonhand ~ #ChatOps
    jhand$

    View Slide

  30. Deploy Code
    Created commit f2cd831: Adding ChatOps link
    1 files changed, 1 insertions(+), 0 deletions(-)
    foobot git commit -am "Adding ChatOps link"
    @jasonhand ~ #ChatOps
    Counting objects: 7, done.
    Compressing objects: 100% (4/4), done.
    Writing objects: 100% (4/4), 407 bytes, done.
    Total 4 (delta 2), reused 0 (delta 0)
    To [email protected]:qrush/gitready.git
    361303d..f2cd831 master -> master
    foobot git push origin master

    View Slide

  31. Build Infrastructure
    Platform install commit-1248994 (#789).
    Started by user: jhand (09.04.2015.11:03:24)
    Platform install commit-1248994 complete
    jhand$ build commit-1248994
    @jasonhand ~ #ChatOps

    View Slide

  32. Build Infrastructure
    building commit-1248994
    Platform install commit-1248994 (#789).
    Started by user: jhand (09.04.2015.11:03:24)
    Platform install commit-1248994 complete
    foobot build commit-1248994
    @jasonhand ~ #ChatOps

    View Slide

  33. Manage Incidents
    Incident 778467 has been
    acknowledged by (jhand)
    Stopped paging jhand -
    (Backend Engineering Team) @
    2:51 p.m. MT
    jhand$ ack incident 778467
    @jasonhand ~ #ChatOps

    View Slide

  34. Manage Incidents
    Incident 778467 has been
    acknowledged by (jhand)
    Stopped paging jhand -
    (Backend Engineering Team) @
    2:51 p.m. MT
    foobot ack me incident 778467
    @jasonhand ~ #ChatOps

    View Slide

  35. Manage Incidents
    Runbook says to look at the graph first
    Please review runbook for prod2.cass24.victorops.com
    @jasonhand ~ #ChatOps

    View Slide

  36. Pull Graphs
    foobot graph me prod2.cass24.victorops.com - cpu -15
    @jasonhand ~ #ChatOps

    View Slide

  37. Kick Services
    foobot restart apache 192.168.0.345
    @jasonhand ~ #ChatOps
    apache on host 192.168.0.345 has been restarted
    one moment .. restarting apache on requested host

    View Slide

  38. Moving terminal and API to Chat
    @jasonhand ~ #ChatOps

    View Slide

  39. 1
    1 1
    2 1
    3 1
    4
    Benefits of ChatOps
    Sharing Speed Learning Security
    @jasonhand ~ #ChatOps

    View Slide

  40. Security continued
    @jasonhand ~ #ChatOps
    Security, compliance,
    and regulatory
    requirements https://youtu.be/18PNEwt18P0

    View Slide

  41. Bots
    github.com/litaio/lita
    @jasonhand ~ #ChatOps
    github.com/github/hubot
    github.com/errbotio/errbot
    Lita

    View Slide

  42. More Bots
    github.com/devth/yetibot
    @jasonhand ~ #ChatOps
    Hal-9001
    github.com/Netflix/hal-9001
    github.com/elastic/jarvis
    (Previously Scorebot)
    github.com/micro/micro/tree/master/bot

    View Slide

  43. Cog : ChatOps Platform
    operable.io
    @jasonhand ~ #ChatOps
    github.com/operable/cog
    w/ Access Control Mechanism

    View Slide

  44. Stackstorm
    @jasonhand ~ #ChatOps
    stackstorm.com
    Event-driven
    automation

    View Slide

  45. http://eng.localytics.com/serverless-slackbots-powered-by-aws/

    View Slide

  46. OSS (Persistent) Group Chat
    @jasonhand ~ #ChatOps
    Zulip Jappix Mattermost
    RocketChat

    View Slide

  47. It’s getting even easier
    @jasonhand ~ #ChatOps

    View Slide

  48. @jasonhand ~ #ChatOps
    3rd-party App Directories

    View Slide

  49. Problems
    Devs & Ops
    Teams
    Have
    Problems
    Our
    Bots
    Solve
    Maturation of ChatOps : Phase 4 & Beyond
    adopted from @davidjbland
    @jasonhand ~ #ChatOps

    View Slide

  50. CONCERNS
    ALERT FATIGUE
    CONTEXT SWITCHING
    LARGE TEAMS
    AUTHENTICATION
    @jasonhand ~ #ChatOps

    View Slide

  51. ADDITIONAL RESOURCES
    reddit.com/r/chatops
    @jasonhand
    talk-chatops.slack.com
    twitter.com/hashtag/chatops
    medium.com/search?q=chatops
    jasonhand.com/tag/chatops

    View Slide

  52. ADDITIONAL RESOURCES
    https://youtu.be/9C2aZIhVk3U
    https://youtu.be/mBbyO4UsWiE

    View Slide

  53. ChatOps For Dummies
    jhand.co/ChatOps4Dummies

    View Slide

  54. ChatOps on O’Reilly
    jhand.co/ChatOpsSurvey

    View Slide

  55. THANK YOU!
    Slides:
    jhand.co/1siEWHd

    View Slide

  56. !56
    More
    https://medium.com/@cloudoapp/we-asked-hipchat-what-their-next-big-move-is-will-hipchat-join-the-bot-hype-heres-their-answer-90bae12e0cce#.pvzw4tzf7
    https://www.usenix.org/conference/srecon16/program/presentation/tobey
    https://getcarina.com/blog/moving-chatops-into-carina/
    http://dev.haufe.com/irc-chatops/
    https://blog.micro.mu/2016/04/25/the-micro-bot.html
    http://www.infoworld.com/article/3062703/devops/how-chatops-can-revolutionize-your-business.html
    https://aws.amazon.com/blogs/aws/new-slack-integration-blueprints-for-aws-lambda/
    https://victorops.com/chatops/
    http://www.jasonhand.com/tag/chatops/

    View Slide

  57. !57
    Abstract
    Those tasked with building and maintaining resilient services and systems at scale are
    aware of the concepts behind “Infrastructure as Code”. Configuration management
    tools and services such as Puppet, Ansible, Chef, and others have become a staple in
    the provisioning, management, and more of critical components of systems worldwide.
    With the growing popularity of ChatOps (the concept of moving actions and context
    related to Operations into a persistent group-chat tool) more and more teams are
    beginning to find even more efficiencies and benefits. What does it look like when
    teams have begun to manage much of their infrastructure from the interface of group-
    chat? What concerns and pitfalls must be considered? Where should teams begin their
    efforts towards a ChatOps approach? What’s on the horizon and beyond as more and
    more shift their actions away from the command-line-interface and web browsers and
    begin to leverage API’s, bots, and group-chat tools in a secure and consistent way?

    View Slide