ChatOps at Shopify: Inviting Bots in Our Day-to-day Operations

ChatOps has already been identified as instrumental for DevOps success. In this talk, I will describe how we use chatbots to accelerate developer onboarding, increase developer productivity and manage service disruption incidents. ChatOps is about bringing tools into your conversations and using them to interact with the infrastructure. It traditionally combines a chatbot, key plugins and scripts. I will describe how we integrate these to perform actions related to the infrastructure such as rebalancing traffic, querying the infrastructure state, and other various actions.

Transcript

1. ChatOps at Shopify
3. Inviting Bots in our Day-to-Day Operations Your
4. Production Engineering
5. 1200+ $40b Rails 40+ 500k+ 10k
6. ChatOps
7. Conversation Driven Development
8. Conversation Driven Development How do you choose a Chat Bot?
9. ChatOps at Shopify
10. # defining a new command command :find_answer, 'answer', help: 'the answer to life, universe, and everything' def find_answer reply(42) end $ spy find_answer 42 $ spy help find_answer the answer to life, universe, and everything Adding Commands
11. Infrastructure
12. Region Host Web Server Load Balancers Host Job Server Host Web Server Hosts Web Servers Host Job Server Hosts Job Servers Host DB Standby The Internet Host DB Reader Load Balancers Host DB Writer Edge Router Edge Router Region Host Web Server Load Balancers Host Job Server Host Web Server Hosts Web Servers Host Job Server Hosts Job Servers Host DB Standby Host DB Reader Load Balancers Host DB Writer Edge Router Edge Router A Global Scale Resilient Web App CDN
13. • spy cdn show traffic • spy cdn backend [region]
14. • spy nginx status • spy profile nginx lua cpu
15. • spy revisions • spy shops • spy profile shopify
16. • spy resque [dc=x] • spy job working [dc=x]
17. • spy shards • spy shard load
18. Failovers • spy failover shopify pod :pod to :location
19. A pod is an isolated set of shops.
20. Region Host Web Server Load Balancers Host Web Server Shared Workers Pod 2 Pods Load Balancers Pod N Pod 5 Pod 9 Redis Pod N Shared Workers Dedicated Workers Dedicated Workers Memcache MySQL Dedicated Workers Active Passive
21. Pods • spy pods
22. Failovers • spy failover shopify pod :ids to :location
23. Failovers: User Authentication
26. And More... • spy chef environment :environment :server • spy newrelic :app • spy datadog :metric
27. Incident Management
29. The Incident Manager On Call (IMOC)’s role is to lead the incident response.
30. ➔ Shit breaks ➔ Detection ➔ Start Incident ➔ Communicate ➔ Fix ➔ Stop Incident ➔ Document (Service Disruption) ➔ Investigation ➔ Root Cause Analysis (RCA) ➔ Action Items ➔ Resolution Incident Response
31. • spy page • spy incident • spy status
32. Shit Breaks ➔ spy page imoc “order notifications not going out”
33. Start Incident ➔ spy incident start me order fraud analysis outage
34. Communicate ➔ spy incident tldr
35. Communicate with other teams ➔ spy incident tell :team message ➔ spy page datastores
36. Actions
37. Third Party Services ➔ spy status ➔ spy status :provider :status for :feature ➔ spy pager imoc res 123
38. Reminders when: [30, stop] command: :check_status_page - when: 120 command: :notify_support_atc message: 'Spy has notified the Support Response Manager (SRM) on your behalf.' - when: 120 command: :srm_fill_out_doc - when: 300 message: 'You should coordinate external comms with the support incident responder.’ - when: 600 command: :srm_checking_in - when: [3600] command: :notify_imoc_team - when: stop message: 'Please create a Service Disruptions report.
39. Stop Incident ➔ spy incident stop
40. Generating the SD report ➔ spy incident note &
41. spy helps us to reduce the impact and duration of incidents.
42. Developer Onboarding
43. Learn commands by seeing others execute them.
44. Hit The Ground Running • spy github add user :user :team • spy circle add my_new_shiny_project • spy buildkite add my_new_shiny_repo • spy shipit lock :stack *message
45. Deploying Code
46. Resiliency. What if Slack is down?
47. Benefits & Lessons Learned
48. • Increased sharing and focus • Shortened feedback loop • Eliminated manual toil • Smoother incident handling • Faster onboarding experience But, we have also learned ...
49. Summary Infrastructure Incident Management Developer Onboarding
51. Thanks! @niyodanie
52. Another Shopify Talk
53. Questions? @niyodanie