ChatOps: Infrastructure As Conversation

Those tasked with building and maintaining resilient services and systems at scale are aware of the concepts behind “Infrastructure as Code”. Configuration management tools and services such as Puppet, Ansible, Chef, and others have become a staple in the provisioning, management, and more of critical components of systems worldwide.

With the growing popularity of ChatOps (the concept of moving actions and context related to Operations into a persistent group-chat tool) more and more teams are beginning to find even more efficiencies and benefits. What does it look like when teams have begun to manage much of their infrastructure from the interface of group-chat? What concerns and pitfalls must be considered? Where should teams begin their efforts towards a ChatOps approach? What’s on the horizon and beyond as more and more shift their actions away from the command-line-interface and web browsers and begin to leverage API’s, bots, and group-chat tools in a secure and consistent way?

Transcript

1. CHATOPS: INFRASTRUCTURE AS CONVERSATION
2. JASON HAND jasonhand #ChatOps
3. @jasonhand ~ #ChatOps https://www.cheatography.com/uploads/davidpol_1439395467_Agile%20Manifesto.png
4. Delivery Pipeline Plan Code Build Test Release Deploy Operate Agile Infrastructure (DevOps) Continuous Delivery Continuous Integration Agile Development @jasonhand ~ #ChatOps
5. @jasonhand ~ #ChatOps http://2.bp.blogspot.com/-3Vv68jBEYdo/VfGlfe3aB0I/AAAAAAAAAMo/vrJl7PBd8m0/s1600/contact-center-acd-bottleneck.jpg
6. USER & CUSTOMER EXPECTATIONS Users expect working software, high availability, innovation, and a sense of empathy
7. AVAILABILITY & RELIABILITY High availability is the result of high resiliency & reliability
8. AVAILABILITY & RELIABILITY High resiliency & reliability is the result of improved MTTR Fail Fast == Respond & Recover Fast
9. Continuous Delivery @jasonhand ~ #ChatOps
10. Continuous Improvement @jasonhand ~ #ChatOps
11. A CONTINUOUS IMPROVEMENT It is not enough to do your best; you must know what to do, and then do your best. - W. Edwards Deming Act Check Do @jasonhand ~ #ChatOps Plan
12. @jasonhand ~ #ChatOps
13. DevOps Buzzword Bingo Continuous Integration Continuous Testing Continuous Delivery Continuous Monitoring Continuous Deployment ChatOps Containerization Microservices High Availability Devs On-call Delivery Pipeline Configuration Management Infrastructure As Code Virtualization Immutable Infrastructure Feedback Loops Experimentation Culture Release Automation Lean Kanban @jasonhand ~ #ChatOps Measurement Sharing
14. DEVOPS PRINCIPLES CULTURE AUTOMATION MEASUREMENT SHARING LEAN @jasonhand ~ #ChatOps
15. What is ChatOps? ChatOps is an approach to communication that allows teams to collaborate and manage many aspects of their infrastructure, code, and data from the comfort and safety of a chat room. Through the use of chatbots and scripts, teams can execute commands, query information, and distribute knowledge across the entire organization and all from the space where conversations are already taking place, a chat room. “We're taking the things we're already doing and putting them in line with the conversations we're already having” - James Fryman (StackStorm) @jasonhand ~ #ChatOps
16. 1 1 1 2 1 3 1 4 Origins IRC XMPP/ Jabber Campfire HipChat/ Slack/Flowdock api @jasonhand ~ #ChatOps “Conversation-driven development”
17. A HUBOT: THE EARLY YEARS Placing tools right in the middle of the conversation - Jesse Newland (GitHub) - 2012 @jasonhand ~ #ChatOps
18. Problems Devs & Ops Teams Have Problems Our Bots Solve Maturation of ChatOps : Phase 1 adopted from @davidjbland @jasonhand ~ #ChatOps
19. Problems Devs & Ops Teams Have Problems Our Bots Solve Maturation of ChatOps : Phase 2 adopted from @davidjbland @jasonhand ~ #ChatOps
20. GitHub Pull Request Notifier Merge code Show current status Display “commit” link Commit Notifier Display “issue” link Display repo commuters Show Open Pull Requests Show Repo Activity Show open issues Search for string in code @jasonhand ~ #ChatOps
21. 1 1 1 2 1 3 1 4 THE NEW INTERFACE Notifications/ Information Alerting/ Paging Command Execution Collaboration/ Communication @jasonhand ~ #ChatOps Feedback Loops Lower MTTR Higher availability & happier users
22. A THE NEW INTERFACE A tool to help the way our people systems work that in turn helps the way our technical systems work. - Mark Imbriaco (operable.io) https://youtu.be/pCVvYCjvoZI @jasonhand ~ #ChatOps
23. SHIELDS UP @jasonhand ~ #ChatOps laserfirecreations.com/Large%20Images/ShldsUpNew.jpg https://stardate.org/sites/default/files/images/gallery/10%2012%2018.jpg DDOS Attack Shields Up!
24. Problems Devs & Ops Teams Have Problems Our Bots Solve Maturation of ChatOps : Phase 3 adopted from @davidjbland @jasonhand ~ #ChatOps
25. Timelines @jasonhand ~ #ChatOps
26. SITUATIONAL AWARENESS @jasonhand ~ #ChatOps blog.vmware.com
27. A SITUATIONAL AWARENESS What makes a great team — everyone on the field is aware of everything, all the time. The team with the most situational awareness will win. - Mary Poppendieck (Lean Software Development) DevOpsDays Minneapolis - 2015 @jasonhand ~ #ChatOps
28. Examples @jasonhand ~ #ChatOps
29. Deploy Code git push origin master Counting objects: 7, done. Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 407 bytes, done. Total 4 (delta 2), reused 0 (delta 0) To git@github.com:qrush/gitready.git 361303d..f2cd831 master -> master jhand$ git commit -am "Adding ChatOps link" Created commit f2cd831: Adding ChatOps link 1 files changed, 1 insertions(+), 0 deletions(-) @jasonhand ~ #ChatOps jhand$
30. Deploy Code Created commit f2cd831: Adding ChatOps link 1 files changed, 1 insertions(+), 0 deletions(-) foobot git commit -am "Adding ChatOps link" @jasonhand ~ #ChatOps Counting objects: 7, done. Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 407 bytes, done. Total 4 (delta 2), reused 0 (delta 0) To git@github.com:qrush/gitready.git 361303d..f2cd831 master -> master foobot git push origin master
31. Build Infrastructure Platform install commit-1248994 (#789). Started by user: jhand (09.04.2015.11:03:24) Platform install commit-1248994 complete jhand$ build commit-1248994 @jasonhand ~ #ChatOps
32. Build Infrastructure building commit-1248994 Platform install commit-1248994 (#789). Started by user: jhand (09.04.2015.11:03:24) Platform install commit-1248994 complete foobot build commit-1248994 @jasonhand ~ #ChatOps
33. Manage Incidents Incident 778467 has been acknowledged by (jhand) Stopped paging jhand - (Backend Engineering Team) @ 2:51 p.m. MT jhand$ ack incident 778467 @jasonhand ~ #ChatOps
34. Manage Incidents Incident 778467 has been acknowledged by (jhand) Stopped paging jhand - (Backend Engineering Team) @ 2:51 p.m. MT foobot ack me incident 778467 @jasonhand ~ #ChatOps
35. Manage Incidents Runbook says to look at the graph first Please review runbook for prod2.cass24.victorops.com @jasonhand ~ #ChatOps
36. Pull Graphs foobot graph me prod2.cass24.victorops.com - cpu -15 @jasonhand ~ #ChatOps
37. Kick Services foobot restart apache 192.168.0.345 @jasonhand ~ #ChatOps apache on host 192.168.0.345 has been restarted one moment .. restarting apache on requested host
38. Moving terminal and API to Chat @jasonhand ~ #ChatOps
39. 1 1 1 2 1 3 1 4 Benefits of ChatOps Sharing Speed Learning Security @jasonhand ~ #ChatOps
40. Security continued @jasonhand ~ #ChatOps Security, compliance, and regulatory requirements https://youtu.be/18PNEwt18P0
41. Bots github.com/litaio/lita @jasonhand ~ #ChatOps github.com/github/hubot github.com/errbotio/errbot Lita
42. More Bots github.com/devth/yetibot @jasonhand ~ #ChatOps Hal-9001 github.com/Netflix/hal-9001 github.com/elastic/jarvis (Previously Scorebot) github.com/micro/micro/tree/master/bot
43. Cog : ChatOps Platform operable.io @jasonhand ~ #ChatOps github.com/operable/cog w/ Access Control Mechanism
44. Stackstorm @jasonhand ~ #ChatOps stackstorm.com Event-driven automation
45. http://eng.localytics.com/serverless-slackbots-powered-by-aws/
46. OSS (Persistent) Group Chat @jasonhand ~ #ChatOps Zulip Jappix Mattermost RocketChat
47. It’s getting even easier @jasonhand ~ #ChatOps
48. @jasonhand ~ #ChatOps 3rd-party App Directories
49. Problems Devs & Ops Teams Have Problems Our Bots Solve Maturation of ChatOps : Phase 4 & Beyond adopted from @davidjbland @jasonhand ~ #ChatOps
50. CONCERNS ALERT FATIGUE CONTEXT SWITCHING LARGE TEAMS AUTHENTICATION @jasonhand ~ #ChatOps
51. ADDITIONAL RESOURCES reddit.com/r/chatops @jasonhand talk-chatops.slack.com twitter.com/hashtag/chatops medium.com/search?q=chatops jasonhand.com/tag/chatops
52. ADDITIONAL RESOURCES https://youtu.be/9C2aZIhVk3U https://youtu.be/mBbyO4UsWiE
53. ChatOps For Dummies jhand.co/ChatOps4Dummies
54. ChatOps on O’Reilly jhand.co/ChatOpsSurvey
55. THANK YOU! Slides: jhand.co/1siEWHd
56. !56 More https://medium.com/@cloudoapp/we-asked-hipchat-what-their-next-big-move-is-will-hipchat-join-the-bot-hype-heres-their-answer-90bae12e0cce#.pvzw4tzf7 https://www.usenix.org/conference/srecon16/program/presentation/tobey https://getcarina.com/blog/moving-chatops-into-carina/ http://dev.haufe.com/irc-chatops/ https://blog.micro.mu/2016/04/25/the-micro-bot.html http://www.infoworld.com/article/3062703/devops/how-chatops-can-revolutionize-your-business.html https://aws.amazon.com/blogs/aws/new-slack-integration-blueprints-for-aws-lambda/ https://victorops.com/chatops/ http://www.jasonhand.com/tag/chatops/
57. !57 Abstract Those tasked with building and maintaining resilient services and systems at scale are aware of the concepts behind “Infrastructure as Code”. Configuration management tools and services such as Puppet, Ansible, Chef, and others have become a staple in the provisioning, management, and more of critical components of systems worldwide. With the growing popularity of ChatOps (the concept of moving actions and context related to Operations into a persistent group-chat tool) more and more teams are beginning to find even more efficiencies and benefits. What does it look like when teams have begun to manage much of their infrastructure from the interface of group- chat? What concerns and pitfalls must be considered? Where should teams begin their efforts towards a ChatOps approach? What’s on the horizon and beyond as more and more shift their actions away from the command-line-interface and web browsers and begin to leverage API’s, bots, and group-chat tools in a secure and consistent way?