$30 off During Our Annual Pro Sale. View Details »

ChatOps for Outages at Lean Agile KC

Aaron Blythe
November 09, 2016

ChatOps for Outages at Lean Agile KC

Walk through the process of incident response with the use of Slack, PagerDuty, Amazon Alexa Services, AWS Lambda and the Amazon Echo.

Aaron Blythe

November 09, 2016
Tweet

More Decks by Aaron Blythe

Other Decks in Technology

Transcript

  1. Lean for Outages
    Aaron Blythe

    View Slide

  2. Voice-Controlled
    ChatOps
    For the Remote Worker
    2

    View Slide

  3. 3
    http://samples.jbpub.com/9780763782610/82610_CH02_PASS02.pdf

    View Slide

  4. “You build it, you run it.”
    - Dr. Werner Vogels CTO Amazon
    4
    http://queue.acm.org/detail.cfm?id=1142065 published 2006

    View Slide

  5. 5
    https://www.3cs.ch/lean-incident-management/

    View Slide

  6. Cost of Outages - per minute
    • $5,617 in 2010
    • $7,908 in 2013
    • $8,851 in 2016
    http://www.emersonnetworkpower.com/en-US/About/NewsRoom/NewsReleases/Pages/Emerson-Network-Power-Study-Says-Unplanned-Data-Center-
    Outages-Cost-Companies-Nearly-9000-Per-Minute-.aspx

    View Slide

  7. Twitter For Outages?
    • Companies that invest in customer service on Twitter have found
    that resolving issues via Twitter:
    • costs one-sixth as much as call centers
    • can lead to over 95% of issues being resolved in channel
    • can achieve a customer satisfaction rate of over 90%
    https://blog.twitter.com/2015/how-twitter-data-can-play-a-role-in-customer-service

    View Slide

  8. Cross Functional Teams
    • Decrease Cycle Time to Prod (DevOps)
    • Increase visibility (everyone)
    • Minimize Unplanned Work or Interruptions (Project Planners)

    View Slide

  9. Voice-Controlled
    ChatOps
    For the Remote Worker
    9

    View Slide

  10. 10
    Aaron Blythe
    @ablythe
    Kansas City
    DevOps
    http://www.meetup.com/DevOps-Kansas-City/
    @DevOpsKC

    View Slide

  11. View Slide

  12. View Slide

  13. https://m.signalvnoise.com/why-i-work-remotely-hint-it-has-nothing-to-do-with-
    productivity-34ace30f74fc#.q9dfcxq2u

    View Slide

  14. Persistent Video Chat
    Chat Clients
    Raspberry Pi
    Work area
    Spare Mac
    For Music, Etc.
    Standing Desk

    View Slide

  15. View Slide

  16. View Slide

  17. Voice-Controlled
    ChatOps
    For the Remote Worker
    17

    View Slide

  18. 18
    https://victorops.com/chatops-for-dummies/
    Free Download
    By: Jason Hand
    https://victorops.com/chatops/

    View Slide

  19. Chat Client + Bot
    19
    Err

    View Slide

  20. View Slide

  21. Slack Integrations

    View Slide

  22. Slack Integrations
    • Benefits
    • Quick Setup (minimal configuration)
    • Often managed by company that owns integration
    • Drawbacks
    • Often simplistic workflow

    View Slide

  23. Hubot

    View Slide

  24. Hubot Brain

    View Slide

  25. Hubot Plugins
    • hubot-pager-me
    • hubot-confluence
    • hubot-leankit
    • hubot-newrelic2
    • hubot-sumologic

    View Slide

  26. hubot-newrelic2

    View Slide

  27. View Slide

  28. http://devopsreactions.tumblr.com/post/127777547677/when-you-see-the-outage-
    starting-and-you-cant-do

    View Slide

  29. Slack
    Hubot
    PagerDuty
    Architecture
    29

    View Slide

  30. Webhooks
    30
    Pager Duty Outgoing
    Slack Incoming

    View Slide

  31. Slack
    Hubot
    PagerDuty
    Webhook
    Webhook
    Architecture
    31

    View Slide

  32. hubot-incident
    Start/Triggered
    Acknowledged
    Resolved
    Closed

    View Slide

  33. hubot-incident
    Start/Triggered
    Acknowledged
    Resolved
    Closed

    View Slide

  34. hubot-incident
    Start/Triggered
    Acknowledged
    Resolved
    Closed

    View Slide

  35. hubot-incident
    Start/Triggered
    Acknowledged
    Resolved
    Closed

    View Slide

  36. hubot-incident
    Start/Triggered
    Acknowledged
    Resolved
    Closed
    After hold Post Mortem

    View Slide

  37. Blameless Post Mortem

    View Slide

  38. View Slide

  39. http://devopsreactions.tumblr.com/post/145902399369/manager-on-a-call-during-an-
    outage

    View Slide

  40. Notes in PagerDuty
    (For Post Mortem)

    View Slide

  41. http://devopsreactions.tumblr.com/post/122408751191/alerts-when-an-outage-starts

    View Slide

  42. https://github.com/HearstAT/hubot-incident

    View Slide

  43. SLAPI Bot (Slack API)
    • Why?
    • Take advantage of Slack API (Hubot is least common denom.)
    • Language agnostic plugins
    • Docker as packaging system
    • https://github.com/ImperialLabs/slapi

    View Slide

  44. Sococo

    View Slide

  45. Small and Fun SFW Hubot Script
    • https://github.com/TheFynx/hubot-funbox

    View Slide

  46. Voice-Controlled
    ChatOps
    For the Remote Worker
    46

    View Slide

  47. Virtual Assistants
    Been around the longest. Specific to Xcode/iOS.
    I only have Windows Virtual Machines.
    Newest to the market in opening up to developing “Intents”
    Best ecosystem to write apps for voice.

    View Slide

  48. Demo
    48

    View Slide

  49. https://developer.amazon.com/
    49

    View Slide

  50. Intents, Slots and Utterances
    50
    https://developer.amazon.com/

    View Slide

  51. Intents, Slots and Utterances
    51
    https://developer.amazon.com/

    View Slide

  52. Voice Design
    52
    https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/alexa-skills-
    kit-voice-design-handbook

    View Slide

  53. Slack
    Hubot
    PagerDuty
    Webhook
    Webhook
    Architecture
    53

    View Slide

  54. Echo AWS Lambda
    Webhook
    Slack
    Hubot
    PagerDuty
    Webhook
    Webhook
    Architecture
    54

    View Slide

  55. AWS Lambda
    55
    https://aws.amazon.com/free

    1,000,000 free requests per month**

    Up to 3.2 million seconds of compute time per month** = 37 days

    ** These free tier offers do not automatically expire at the end of your 12 month AWS Free Tier term, but are available to both existing and new AWS customers indefinitely.

    View Slide

  56. Lambda Skill
    56
    https://aws.amazon.com/blogs/compute/slack-dictation-an-amazon-echo-and-aws-
    lambda-demo/

    View Slide

  57. https://github.com/aaronblythe/
    alexa_slack_skill
    57

    View Slide

  58. 58
    Echo
    $179.99
    Raspberry Pi
    $5-35
    Browser
    Free
    https://echosim.io/

    View Slide

  59. View Slide

  60. Say What?
    • https://github.com/joshnewlan/say_what

    View Slide

  61. View Slide

  62. What do you think?

    View Slide