ChatOps for Outages at Lean Agile KC

Walk through the process of incident response with the use of Slack, PagerDuty, Amazon Alexa Services, AWS Lambda and the Amazon Echo.

Transcript

1. Lean for Outages Aaron Blythe
2. Voice-Controlled ChatOps For the Remote Worker 2
3. 3 http://samples.jbpub.com/9780763782610/82610_CH02_PASS02.pdf
4. “You build it, you run it.” - Dr. Werner Vogels CTO Amazon 4 http://queue.acm.org/detail.cfm?id=1142065 published 2006
5. 5 https://www.3cs.ch/lean-incident-management/
6. Cost of Outages - per minute • $5,617 in 2010 • $7,908 in 2013 • $8,851 in 2016 http://www.emersonnetworkpower.com/en-US/About/NewsRoom/NewsReleases/Pages/Emerson-Network-Power-Study-Says-Unplanned-Data-Center- Outages-Cost-Companies-Nearly-9000-Per-Minute-.aspx
7. Twitter For Outages? • Companies that invest in customer service on Twitter have found that resolving issues via Twitter: • costs one-sixth as much as call centers • can lead to over 95% of issues being resolved in channel • can achieve a customer satisfaction rate of over 90% https://blog.twitter.com/2015/how-twitter-data-can-play-a-role-in-customer-service
8. Cross Functional Teams • Decrease Cycle Time to Prod (DevOps) • Increase visibility (everyone) • Minimize Unplanned Work or Interruptions (Project Planners)
9. Voice-Controlled ChatOps For the Remote Worker 9
10. 10 Aaron Blythe @ablythe Kansas City DevOps http://www.meetup.com/DevOps-Kansas-City/ @DevOpsKC
13. https://m.signalvnoise.com/why-i-work-remotely-hint-it-has-nothing-to-do-with- productivity-34ace30f74fc#.q9dfcxq2u
14. Persistent Video Chat Chat Clients Raspberry Pi Work area Spare Mac For Music, Etc. Standing Desk
17. Voice-Controlled ChatOps For the Remote Worker 17
18. 18 https://victorops.com/chatops-for-dummies/ Free Download By: Jason Hand https://victorops.com/chatops/
19. Chat Client + Bot 19 Err
21. Slack Integrations
22. Slack Integrations • Benefits • Quick Setup (minimal configuration) • Often managed by company that owns integration • Drawbacks • Often simplistic workflow
23. Hubot
24. Hubot Brain
25. Hubot Plugins • hubot-pager-me • hubot-confluence • hubot-leankit • hubot-newrelic2 • hubot-sumologic
26. hubot-newrelic2
28. http://devopsreactions.tumblr.com/post/127777547677/when-you-see-the-outage- starting-and-you-cant-do
29. Slack Hubot PagerDuty Architecture 29
30. Webhooks 30 Pager Duty Outgoing Slack Incoming
31. Slack Hubot PagerDuty Webhook Webhook Architecture 31
32. hubot-incident Start/Triggered Acknowledged Resolved Closed
33. hubot-incident Start/Triggered Acknowledged Resolved Closed
34. hubot-incident Start/Triggered Acknowledged Resolved Closed
35. hubot-incident Start/Triggered Acknowledged Resolved Closed
36. hubot-incident Start/Triggered Acknowledged Resolved Closed After hold Post Mortem
37. Blameless Post Mortem
39. http://devopsreactions.tumblr.com/post/145902399369/manager-on-a-call-during-an- outage
40. Notes in PagerDuty (For Post Mortem)
41. http://devopsreactions.tumblr.com/post/122408751191/alerts-when-an-outage-starts
42. https://github.com/HearstAT/hubot-incident
43. SLAPI Bot (Slack API) • Why? • Take advantage of Slack API (Hubot is least common denom.) • Language agnostic plugins • Docker as packaging system • https://github.com/ImperialLabs/slapi
44. Sococo
45. Small and Fun SFW Hubot Script • https://github.com/TheFynx/hubot-funbox
46. Voice-Controlled ChatOps For the Remote Worker 46
47. Virtual Assistants Been around the longest. Specific to Xcode/iOS. I only have Windows Virtual Machines. Newest to the market in opening up to developing “Intents” Best ecosystem to write apps for voice.
48. Demo 48
49. https://developer.amazon.com/ 49
50. Intents, Slots and Utterances 50 https://developer.amazon.com/
51. Intents, Slots and Utterances 51 https://developer.amazon.com/
52. Voice Design 52 https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/alexa-skills- kit-voice-design-handbook
53. Slack Hubot PagerDuty Webhook Webhook Architecture 53
54. Echo AWS Lambda Webhook Slack Hubot PagerDuty Webhook Webhook Architecture 54
55. AWS Lambda 55 https://aws.amazon.com/free 1,000,000 free requests per month** Up to 3.2 million seconds of compute time per month** = 37 days ** These free tier offers do not automatically expire at the end of your 12 month AWS Free Tier term, but are available to both existing and new AWS customers indefinitely.
56. Lambda Skill 56 https://aws.amazon.com/blogs/compute/slack-dictation-an-amazon-echo-and-aws- lambda-demo/
57. https://github.com/aaronblythe/ alexa_slack_skill 57
58. 58 Echo $179.99 Raspberry Pi $5-35 Browser Free https://echosim.io/
60. Say What? • https://github.com/joshnewlan/say_what
62. What do you think?