ChatOps for Outages at Lean Agile KC

ChatOps for Outages at Lean Agile KC

Walk through the process of incident response with the use of Slack, PagerDuty, Amazon Alexa Services, AWS Lambda and the Amazon Echo.

260a95e08b7880ecd76b964203f25c87?s=128

Aaron Blythe

November 09, 2016
Tweet

Transcript

  1. Lean for Outages Aaron Blythe

  2. Voice-Controlled ChatOps For the Remote Worker 2

  3. 3 http://samples.jbpub.com/9780763782610/82610_CH02_PASS02.pdf

  4. “You build it, you run it.” - Dr. Werner Vogels

    CTO Amazon 4 http://queue.acm.org/detail.cfm?id=1142065 published 2006
  5. 5 https://www.3cs.ch/lean-incident-management/

  6. Cost of Outages - per minute • $5,617 in 2010

    • $7,908 in 2013 • $8,851 in 2016 http://www.emersonnetworkpower.com/en-US/About/NewsRoom/NewsReleases/Pages/Emerson-Network-Power-Study-Says-Unplanned-Data-Center- Outages-Cost-Companies-Nearly-9000-Per-Minute-.aspx
  7. Twitter For Outages? • Companies that invest in customer service

    on Twitter have found that resolving issues via Twitter: • costs one-sixth as much as call centers • can lead to over 95% of issues being resolved in channel • can achieve a customer satisfaction rate of over 90% https://blog.twitter.com/2015/how-twitter-data-can-play-a-role-in-customer-service
  8. Cross Functional Teams • Decrease Cycle Time to Prod (DevOps)

    • Increase visibility (everyone) • Minimize Unplanned Work or Interruptions (Project Planners)
  9. Voice-Controlled ChatOps For the Remote Worker 9

  10. 10 Aaron Blythe @ablythe Kansas City DevOps http://www.meetup.com/DevOps-Kansas-City/ @DevOpsKC

  11. None
  12. None
  13. https://m.signalvnoise.com/why-i-work-remotely-hint-it-has-nothing-to-do-with- productivity-34ace30f74fc#.q9dfcxq2u

  14. Persistent Video Chat Chat Clients Raspberry Pi Work area Spare

    Mac For Music, Etc. Standing Desk
  15. None
  16. None
  17. Voice-Controlled ChatOps For the Remote Worker 17

  18. 18 https://victorops.com/chatops-for-dummies/ Free Download By: Jason Hand https://victorops.com/chatops/

  19. Chat Client + Bot 19 Err

  20. None
  21. Slack Integrations

  22. Slack Integrations • Benefits • Quick Setup (minimal configuration) •

    Often managed by company that owns integration • Drawbacks • Often simplistic workflow
  23. Hubot

  24. Hubot Brain

  25. Hubot Plugins • hubot-pager-me • hubot-confluence • hubot-leankit • hubot-newrelic2

    • hubot-sumologic
  26. hubot-newrelic2

  27. None
  28. http://devopsreactions.tumblr.com/post/127777547677/when-you-see-the-outage- starting-and-you-cant-do

  29. Slack Hubot PagerDuty Architecture 29

  30. Webhooks 30 Pager Duty Outgoing Slack Incoming

  31. Slack Hubot PagerDuty Webhook Webhook Architecture 31

  32. hubot-incident Start/Triggered Acknowledged Resolved Closed

  33. hubot-incident Start/Triggered Acknowledged Resolved Closed

  34. hubot-incident Start/Triggered Acknowledged Resolved Closed

  35. hubot-incident Start/Triggered Acknowledged Resolved Closed

  36. hubot-incident Start/Triggered Acknowledged Resolved Closed After hold Post Mortem

  37. Blameless Post Mortem

  38. None
  39. http://devopsreactions.tumblr.com/post/145902399369/manager-on-a-call-during-an- outage

  40. Notes in PagerDuty (For Post Mortem)

  41. http://devopsreactions.tumblr.com/post/122408751191/alerts-when-an-outage-starts

  42. https://github.com/HearstAT/hubot-incident

  43. SLAPI Bot (Slack API) • Why? • Take advantage of

    Slack API (Hubot is least common denom.) • Language agnostic plugins • Docker as packaging system • https://github.com/ImperialLabs/slapi
  44. Sococo

  45. Small and Fun SFW Hubot Script • https://github.com/TheFynx/hubot-funbox

  46. Voice-Controlled ChatOps For the Remote Worker 46

  47. Virtual Assistants Been around the longest. Specific to Xcode/iOS. I

    only have Windows Virtual Machines. Newest to the market in opening up to developing “Intents” Best ecosystem to write apps for voice.
  48. Demo 48

  49. https://developer.amazon.com/ 49

  50. Intents, Slots and Utterances 50 https://developer.amazon.com/

  51. Intents, Slots and Utterances 51 https://developer.amazon.com/

  52. Voice Design 52 https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/alexa-skills- kit-voice-design-handbook

  53. Slack Hubot PagerDuty Webhook Webhook Architecture 53

  54. Echo AWS Lambda Webhook Slack Hubot PagerDuty Webhook Webhook Architecture

    54
  55. AWS Lambda 55 https://aws.amazon.com/free 1,000,000 free requests per month** Up

    to 3.2 million seconds of compute time per month** = 37 days ** These free tier offers do not automatically expire at the end of your 12 month AWS Free Tier term, but are available to both existing and new AWS customers indefinitely.
  56. Lambda Skill 56 https://aws.amazon.com/blogs/compute/slack-dictation-an-amazon-echo-and-aws- lambda-demo/

  57. https://github.com/aaronblythe/ alexa_slack_skill 57

  58. 58 Echo $179.99 Raspberry Pi $5-35 Browser Free https://echosim.io/

  59. None
  60. Say What? • https://github.com/joshnewlan/say_what

  61. None
  62. What do you think?