Learning from Firefighters

In this presentation, I explain what we can learn from wilderness firefighters about increasing the reliability of our systems.

This version of the talk was given at the 2024 Edition of Open Source Summit Europe as part of the "Cloud Open" track.

Transcript

1. Learning from Firefighters to improve System Reliability
2. Kerim Satirli Senior Developer Advocate, Infrastructure & Orchestration he / him @ksatirli
3. ca. 75 meters ca. 100 meters
4. ca. 825 meters ca. 300 meters
6. 10 minutes
7. 3000 acres
8. 5000 acres
9. regulations follow incidents
10. Rules are Tools
13. • logs, metrics, traces • outliers and predictions • track missing data Order #1 Keep informed on system conditions and forecasts.
14. Know what your system is doing at all times. • architecture • external factors • (actual) weather Order #2
15. Base all actions on current and expected behavior of the system. • chaos testing • game days • incident retrospectives Order #3
16. • message queues • degrade gracefully • communicate early Order #4 Identify minimum reliability levels and make them known.
17. • monitor honey tokens • guard against insiders • possibly bait attackers Order #5 Post honeypots when there is possible danger.
18. Be alert. Keep calm. Think clearly. Act decisively. • rotate roles consistently • trust your training • know your plays Order #6
19. Maintain prompt comms with your own and adjoining teams. • organization • corp comms • customers Order #7
20. • verify comprehension • document choices • buddy system Order #8 Give clear instructions and ensure they are understood.
21. • avoid tunnel vision • don't overextend • accuracy beats expediency Order #9 Maintain control of your systems at all times.
22. • restore supported services • collect forensics • keep taking notes Order #10 Protect systems aggressively, having provided for safety first.
23. Summary
24. systems reliability is a product of structured command and control
25. systems reliability is a product of structured command and control structured command and control requires situational awareness
26. systems reliability is a product of structured command and control situational awareness requires strong communication structured command and control requires situational awareness systems reliability is a product of structured command and control structured command and control requires situational awareness
27. systems reliability is a product of strong communication
28. Incident Management for Operations Schnepp Vidal, Hawley 2017 Fatal Defect Chasing Killer Computer Bugs Ivars Peterson 1995
29. Thank you speakerdeck.com/ksatirli