Machine Learning Operations (MLOps) - Active Failures and Latent Conditions

This talk will discuss risk assessment in ML Systems from the perspective of reliability, operations and especially causal aspects that can lead to outages in ML Systems.

Transcript

1. Flávio Clésio April, 2020 Machine Learning Operations Active Failures, Latent Conditions MLOps Community
2. ABOUT ME Flávio Clésio • Machine Learning Engineer @ MyHammer AG • MSc. in Production Engineering (Machine Learning in Credit Derivatives/NPL) • Specialist in Database Engineering and Business Intelligence • Blogger @ flavioclesio.com • Talked in some venues (Strata Hadoop World, Spark Summit, PAPIS.io, The Developers Conference and so on…) flavioclesio
3. CURRENT STATE OF ML SYSTEMS Machine Learning Systems play a huge role in several businesses from the banking industry, recommender systems until health domains. When we talk about high stakes Machine Learning in Production we can consider that this era of "a-data-scientist-with-a-script-in-a-single-machine" is officially over.
4. This talk will discuss risk assessment in ML Systems from the perspective of reliability, operations and especially causal aspects that can lead to outages in ML Systems. WHAT IT’S ABOUT?
5. SURVIVORSHIP BIAS Several posts, conference talks, papers most of the time they presents only what worked extremely well, how those solutions generated revenue for the company and other happy cases.
6. SURVIVORSHIP BIAS Almost no one disclosures what went wrong during the development of these solutions. This is essentially a problem given that we are only seeing the final outcome and not how that outcome was generated and the failures/errors made along the way.
7. • Not sexy • People can feel blame or silly to talk about their errors • Can turns in a "bad personal/corporate branding" • Who died in during the process cannot tell what went wrong FAILURE: A NOT SO ROMANTIC TOPIC
8. HOW IT LOOKS LIKE
9. • Amazon: The data on a load-balancer was deleted and that caused a disruption in practically an entire AWS region at the time; • Gitlab: A deletion of a production database led to an 18-hour unavailability with loss of customer data; • Knight Capital: Lack of code review culture allowed an engineer to deploy a code 8 years outdated in production. Outcome: losses of $ 172,222 per second for 45 minutes (or U$ 465 million). SOME SPECIFIC FAILURE CASES • European Space Agency: A conversion from a 16-bit to 64-bit number caused an overflow in the rocket steering system that triggered a chain of events that caused the rocket to be destroyed and a loss of more than $ 370 million • NASA: A degradation from an engineering culture to political/product culture led to a catastrophic failure that not only cost billions of dollars but also killed the crew in the Challenger space shuttle.
10. FAILURE = LEARNING = OPPORTUNITY TO IMPROVE • There is always a lesson to be learned in the face of what went wrong • A good culture it’s not about blaming or not thinking about the problems, but to analyse them, learn and improve. • For every lesson learned, all system becomes more reliable
11. The aviation industry it’s one example where the reliability are significantly increased for every incident/accident. This is one of industries that becomes more reliable even with the increase of transactions along the time; and because of that the number of fatalities has been falling year by year. RELIABILITY BENCHMARK
12. This model was created by James Reason in early 90ies as a general framework for understanding the dynamics of accident causation. The idea was to identify latent conditions and active failures to put in place countermeasures to minimize a unwanted variability in human behaviour in socio-technological systems. Sources: “Human error: models and management” and "The contribution of latent human failures to the breakdown of complex systems" SWISS CHEESE MODEL
13. DEFENSES, BARRIERS, SAFEGUARDS [...] High-tech systems have many defensive layers: some are designed (alarms, physical barriers, automatic shutdowns, etc.), others rely on people (surgeons, anesthetists, pilots, control room operators, etc.) and others rely on procedures and administrative controls. [...] Source: Human error: models and management
14. [...] In an ideal world, each defensive layer would be intact. In reality, however, they are more like slices of Swiss cheese, with many holes - although, unlike cheese, these holes are continually opening, closing and moving. The presence of holes in any "slice" does not normally cause a bad result. This can usually happen only when the holes in several layers line up momentarily to allow for an accident opportunity trajectory - bringing risks to harmful contact with victims[...] Source: Human error: models and management DEFENSES, BARRIERS, SAFEGUARDS
15. SWISS CHEESE MODEL Source: Understanding models of error and how they apply in clinical practice
16. • Local fixes are appealing because of sounds productive and look good for the other teams (e.g. see how this person solved the problem very fast?) • Cultivates several dormant problems and silent risks that can potentially cause harm, instead to eliminate them • Promotes a stagnated engineering culture instead to aim for a continuous reform (i.e. not only cosmetic enhancements but a substantial reform) • Local fixes it’s like have a problem with mosquitoes and keep swatting them every day, instead to solve the problem draining the swamps in which they breed. WHY NOT JUST FIX THE PROBLEM AND MOVE ON?
17. That is, in this case each slice of Swiss cheese would be a line of defense with projected layers (e.g., monitoring, alarms, code push locks in production, etc.) and / or the procedural layers that involve people (e.g., cultural aspects , training and qualification of commiters in the repository, rollback mechanisms, unit and integration tests, etc.). LATENT CONDITIONS AND ACTIVE FAILURES
18. LATENT CONDITIONS [...] Latent conditions are like a kind of situations intrinsically resident within the system; which are consequences of design, engineering decisions, who wrote the rules or procedures and even the highest hierarchical levels in an organization. These latent conditions can lead to two types of adverse effects, which are situations that cause error and the creation of vulnerabilities. That is, the solution has a design that increases the likelihood of high negative impact events that can be equivalent to a causal or contributing factor.[...] Source: Human error: models and management
19. ACTIVE FAILURES [...]Active failures are insecure acts or minor transgressions committed by people who are in direct contact with the system; these acts can be mistakes, lapses, distortions, omissions, errors and procedural violations.[...] Source: Human error: models and management
20. HUMAN FACTORS Source: Human Factors Analysis and Classification System (HFACS)
21. • Absence of Code Review culture (e.g. London Whale and Knight Capital) • Culture of improvised technical arrangements (e.g. workarounds) • Lack of observability • Democracy-type decisions with less informed people rather than consensus between experts and risk-takers SOME LATENT CONDITIONS IN ML
22. • Resumé-Driven Development • Unreviewed code going into production • Data Leakage in model training • Lack of reproducibility / replicability • Glue code SOME ACTIVE FAILURES IN ML
23. A SWISS CHEESE OF AN OUTAGE IN ML SYSTEM
24. • Post-Mortems • Automation • Monitoring & Alerts ◦ Observability ▪ Metrics ▪ Logs ▪ Application Performance Monitoring • Continuous reform and permanent assessment SOME STRATEGIES TO MINIMIZE RISK SURFACE
25. • Orchestration (e.g. Mesos, Airflow, Kubernetes, AWS ECS, Kubeflow) • Observability (e.g. Elasticsearch, Kibana, Prometheus, Sentry, Grafana, FluentBit, Datadog) • ML Experiment Management ( e.g. ModelChimp, Randopt, Forge, Lore, Datmo, Studio ML, Sacred, MLFlow, Polyaxon) • Data Versioning and management (e.g. DVC, Pachyderm, Snorkel) • ML SaaS (e.g. Algorithmia, Peltarion, Databricks, Seldon IO, Google AI Platform, AWS Sage Maker, Azure ML Studio, Dotscience, Daitaku DSS, Domino AI, Polyaxon, Weights & Biases, Spell, Gradient, Paperspace, H2O AI, Stack ML, Comet, Valohai, Neptune AI) SOME TOOLS
26. • There’s no silver bullet regarding risk management in ML Platforms. The hard part it’s to know how perceive and manage those risks • An outage never happens due to a single reason. Outages are several latent conditions and active failures combined and aligned that triggers the event • Human factors plays a huge role in outages • If possible, share your mistakes. When someone shares what went wrong, everyone learns and all ML systems becomes more robust. FINAL REMARKS
27. THANK YOU! @flavioclesio fclesio flavioclesio flavioclesio.com
28. • Reason, James. “The contribution of latent human failures to the breakdown of complex systems.” Philosophical Transactions of the Royal Society of London. B, Biological Sciences 327.1241 (1990): 475-484. • Reason, J. “Human error: models and management.” BMJ (Clinical research ed.) vol. 320,7237 (2000): 768-70. doi:10.1136/bmj.320.7237.768 • Morgenthaler, J. David, et al. “Searching for build debt: Experiences managing technical debt at Google.” 2012 Third International Workshop on Managing Technical Debt (MTD). IEEE, 2012. • Alahdab, Mohannad, and Gül Çalıklı. “Empirical Analysis of Hidden Technical Debt Patterns in Machine Learning Software.” International Conference on Product-Focused Software Process Improvement. Springer, Cham, 2019. REFERENCES
29. • Perneger, Thomas V. “The Swiss cheese model of safety incidents: are there holes in the metaphor?.” BMC health services research vol. 5 71. 9 Nov. 2005, doi:10.1186/1472-6963-5-71 • “Hot cheese: a processed Swiss cheese model.” JR Coll Physicians Edinb 44 (2014): 116-21. • Breck, Eric, et al. “What’s your ML Test Score? A rubric for ML production systems.” (2016). • SEC Charges Knight Capital With Violations of Market Access Rule • Machine Learning Goes Production! Engineering, Maintenance Cost, Technical Debt, Applied Data Analysis Lab Seminar REFERENCES
30. REFERENCES • Nassim Taleb – Lectures on Fat Tails, (Anti)Fragility, Precaution, and Asymmetric Exposures • Skybrary – Human Factors Analysis and Classification System (HFACS) • CEFA Aviation – Swiss Cheese Model • A List of Post-mortems • Richard Cook – How Complex Systems Fail • Airbus – Hull Losses • Number of flights performed by the global airline industry from 2004 to 2020