Vertical first, horizontal second. How MLOps facilitates efficient vertical prototyping of ML systems 2022/09/15 | Simon Stiebellehner 

Introduction Vertical first, horizontal second – MLOps & vertical prototyping 1 2 3 How MLOps enables vertical prototyping and beyond at TMNL Simon Stiebellehner Lead MLOps Engineer @ TMNL Lecturer @ University of Applied Sciences Vienna linkedin.com/in/simonstiebellehner/ [email protected] / [email protected] Healthy failing through vertical prototyping MLOps as a key enabler 

Healthy failing through vertical prototyping Failure is part of the game If you don’t fail, you are not experimenting § Failure is inherent in doing Data Science § Even more so if failure is defined as “not meeting expected business value” 

Healthy failing through vertical prototyping Impact of potential failure needs to be mitigated Risk = probability of event * impact We want to decrease the probability and impact failure. What options do we have? Managing risk of failure 

Healthy failing through vertical prototyping Impact of potential failure needs to be mitigated Managing risk of failure Increase the probability of success: § Collect the right data in good quality § Hire experienced, distinguished Data Scientists & Engineers § Use battle-proven models and techniques Risk = probability of event * impact We want to decrease the probability and impact failure. What options do we have? 

Healthy failing through vertical prototyping Impact of potential failure needs to be mitigated Managing risk of failure Increase the probability of success: § Collect the right data in good quality § Hire experienced, distinguished Data Scientists & Engineers § Use battle-proven models and techniques A classic of risk management is often forgotten: à Reduce the impact of failure Risk = probability of event * impact We want to decrease the probability and impact failure. What options do we have? 

Healthy failing through vertical prototyping Detect failure fast and early to minimize wasted effort ”Pulling the Plug” at the right time means reducing the risk of wasted effort and not killing projects with a high probability of success. But when is “the right time”? 

Healthy failing through vertical prototyping Detect failure fast and early to minimize wasted effort § Too early means discarding potentially net-positive projects ”Pulling the Plug” at the right time means reducing the risk of wasted effort and not killing projects with a high probability of success. But when is “the right time”? The “fog of war” might still the thick and you can’t yet estimate well if there’ll be net business value. 

Healthy failing through vertical prototyping Detect failure fast and early to minimize wasted effort § Too early means discarding potentially net-positive projects § Too late means wasted effort ”Pulling the Plug” at the right time means reducing the risk of wasted effort and not killing projects with a high probability of success. But when is “the right time”? The “fog of war” might still the thick and you can’t yet estimate well if there’ll be net business value. You’ve been building in thick “fog of war” for months, chasing some accuracy threshold in your 10-fold cross validation. However, in production the model suddenly fails to live up to the experimental results. 

Healthy failing through vertical prototyping Detect failure fast and early to minimize wasted effort § Too early means discarding potentially net-positive projects § Too late means wasted effort ”Pulling the Plug” at the right time means reducing the risk of wasted effort and not killing projects with a high probability of success. But when is “the right time”? The “fog of war” might still the thick and you can’t yet estimate well if there’ll be net business value. You’ve been building in thick “fog of war” for months, chasing some accuracy threshold in your 10-fold cross validation. However, in production the model suddenly fails to live up to the experimental results. The key is to eliminate uncertainty asap by collecting the right information. 

Healthy failing through vertical prototyping Detect failure fast and early to minimize wasted effort § Too early means discarding potentially net-positive projects § Too late means wasted effort ”Pulling the Plug” at the right time means reducing the risk of wasted effort and not killing projects with a high probability of success. But when is “the right time”? The “fog of war” might still the thick and you can’t yet estimate well if there’ll be net business value. You’ve been building in thick “fog of war” for months, chasing some accuracy threshold in your 10-fold cross validation. However, in production the model suddenly fails to live up to the experimental results. The key is to eliminate uncertainty asap by collecting the right information. Push through to “where business value is generated” fast and early to lift the fog of war and understand the full scope of your Machine Learning System, enabling you to make an educated decision about “when to pull the plug”. 

Healthy failing through vertical prototyping Vertical first, Horizontal second. Build Vertical Prototypes of your Machine Learning Systems to lift the fog of war Source of figure: Jones, M. C., Floyd, I. R., & Twidale, M. B. (2007). Patchwork prototyping: A rapid prototyping technique that harnesses the power of open-source software. Vertical first Build out the vertical path to the user first. à “Push through where business value is generated” Horizontal second Then iterate, optimize and measure. à Optimizing only makes sense if our target variable is as close to business value as possible. 

Healthy failing through vertical prototyping As a Data Scientist, you are taught to think “horizontal first” Horizontal focus is a result of seeing a model as the final product and success defined by an artificial metric Source of figure: Jones, M. C., Floyd, I. R., & Twidale, M. B. (2007). Patchwork prototyping: A rapid prototyping technique that harnesses the power of open-source software. Horizontal prioritization is predominant in Data Science 1. Fetch data 2. Exploratory Analysis 3. Build model 4. Evaluate model under lab-conditions (e.g. k-fold CV...) 5. Optimize model performance 

Healthy failing through vertical prototyping Where’s “business value generated”? In production. Pushing through to production and evaluating the added value of your model (or dashboard or…) under real circumstances is gold standard This is what we spend a lot of time on early in the DS lifecycle Source (image): "The End-to-End CD4ML Process” by martinfowler.com 

Healthy failing through vertical prototyping Where’s “business value generated”? In production. Pushing through to production and evaluating the added value of your model (or dashboard or…) under real circumstances is gold standard Source (image): "The End-to-End CD4ML Process” by martinfowler.com This is where we get a good idea of true business value This is what we spend a lot of time on early in the DS lifecycle 

Healthy failing through vertical prototyping Where’s “business value generated”? In production. Pushing through to production and evaluating the added value of your model (or dashboard or…) under real circumstances is gold standard Source (image): "The End-to-End CD4ML Process” by martinfowler.com ”But getting from ß here to there à is very time-consuming! We’ll waste a lot of effort if we bring models close to the user early!” 

Healthy failing through vertical prototyping Model productionization is complex and usually takes several months Executing complex processes manually is not only slow but also highly error-prone Inefficient, manual work Lots of handovers Highly error-prone Slow iterations Efficient vertical prototyping is impossible What model productionization usually is… 

MLOps as a key enabler MLOps paves and automates the Golden Path for models to production MLOps (ML + DevOps) is a set of principles and activities that aims to improve effectiveness and efficiency of model development. 

MLOps as a key enabler MLOps aims to streamline and automate large parts of how models are developed, deployed, served and monitored. What model productionization usually is… What model productionization facilitated by MLOps is… Modern factories are characterized by strealimed processes, allowing for a high degree of automation 

MLOps as a key enabler MLOps is at the heart of TMNL At TMNL, a vertical prototype of a model is deployed, served* and monitored the minute a new model project is started, without any extra effort from model developers. Vertical prototyping out of the box, solving our dilemma. *Serving does not imply having direct end-user impact without explicit approval. 

How MLOps enables vertical prototyping and beyond at TMNL Transaction Monitoring Netherlands - TMNL We’re fighting money laundering at an unprecedented scale § Joint venture of 5 Dutch banks: § Pooling pseudonymized transaction data (of businesses) at TMNL § The larger the transaction graph, the better we can detect money laundering § TMNL builds models that detect unusual patterns on the inter-bank transaction graph that might indicate money laundering § … and the modeling platform to do this effectively. 

How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? There is 1 streamlined, highly automated “Golden Path” to production, enabling efficient vertical prototyping and beyond. 

How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? There is 1 streamlined, highly automated “Golden Path” to production, enabling efficient vertical prototyping and beyond. Remote repository w/ approval rules Model Dev Teams start off from a template model repo with centrally managed CI/CD pipelines, quality checks and monitoring attached. Deployment happens from minute 1 and with every merge to release and main. Iterations on models do not happen in an isolated phase, but go all the way to UAT and prod environments (fully automated). Model repository template Tools to ease interaction with different AWS services On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry Centralized CI/CD Pipelines w/ checks On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry Centralized CI/CD Pipelines w/ checks Automated versioning & cataloging of model outputs Workflow Orchestration & scalable compute On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry Centralized CI/CD Pipelines w/ checks Automated versioning & cataloging of model outputs Workflow Orchestration & scalable compute Data (distribution) profiling & checks Integration Tests on higher envs On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry Centralized CI/CD Pipelines w/ checks Automated versioning & cataloging of model outputs Workflow Orchestration & scalable compute Data (distribution) profiling & checks Integration Tests on higher envs Contract “Production” & enablement Full auditability On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry Centralized CI/CD Pipelines w/ checks Automated versioning & cataloging of model outputs Workflow Orchestration & scalable compute Data (distribution) profiling & checks Integration Tests on higher envs Contract “Production” & enablement Full auditability Operational and model metric monitoring & alerting Automated triggering of inference pipelines On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

How MLOps enables vertical prototyping and beyond at TMNL For TMNL, MLOps is a key enabler for vertical prototyping and beyond At TMNL, we’ve been turning model development, deployment & operations into a streamlined 21st century factory Clear path to production No handovers, slim teams with E2E ownership Agreed and enforced quality standards Fast, high-quality iterations from minute 1 in model development while staying close to “where business value is generated” High degree of automation 

How MLOps enables vertical prototyping and beyond at TMNL For TMNL, MLOps is a key enabler for vertical prototyping and beyond At TMNL, we’ve been turning model development, deployment & operations into a streamlined 21st century factory Clear path to production No handovers, slim teams with E2E ownership Agreed and enforced quality standards Fast, high-quality iterations from minute 1 in model development while staying close to “where business value is generated” High degree of automation This allows us to mitigate the impact of failure inherent in doing Data Science. Clear the fog of war asap through early, no-effort vertical breakthroughs to production and iterating on models close to business value. 

Introduction Vertical first, horizontal second – MLOps & vertical prototyping 1 2 3 Healthy failing through vertical prototyping MLOps as a key enabler How MLOps enables vertical prototyping and beyond at TMNL Simon Stiebellehner Lead MLOps Engineer @ TMNL Lecturer @ University of Applied Sciences Vienna linkedin.com/in/simonstiebellehner/ [email protected] / [email protected] 

Healthy failing through vertical prototyping Let’s summarize what we know so far § There’s risk inherent in doing Data Science. Healthy failing means we want to have this risk properly managed, e.g. by reducing impact of failure § Typically, significant time is spent on early project phases using proxy metrics under lab conditions, which are often deocupled from final business value. § This makes it hard to pin down when we should “pull the plug” of a project, i.e. it’s difficult to mitigate the impact of failure. § Early vertical breakthroughs to production/the user help us optimize our models close to the user/business value, lifting the fog of war, enabling educated decisions. § However, these vertical breakthroughs are costly as taking a model to production is often fairly complex. 

Healthy failing through vertical prototyping Let’s summarize what we know so far It seems we’re in a delimma: costly lifting of the “fog of war” vs. poorly informed “pulling the plug” § There’s risk inherent in doing Data Science. Healthy failing means we want to have this risk properly managed, e.g. by reducing impact of failure § Typically, significant time is spent on early project phases using proxy metrics under lab conditions, which are often deocupled from final business value. § This makes it hard to pin down when we should “pull the plug” of a project, i.e. it’s difficult to mitigate the impact of failure. § Early vertical breakthroughs to production/the user help us optimize our models close to the user/business value, lifting the fog of war, enabling educated decisions. § However, these vertical breakthroughs are costly as taking a model to production is often fairly complex. 

Healthy failing through vertical prototyping … and it’s also complex! Source (image): "The End-to-End CD4ML Process” by martinfowler.com Reproducibility Data, Model, Code testing Deployment Strategies ML Monitoring (e.g. data, drift...) (Automated) Re-Training Consistency 

How MLOps enables vertical prototyping and beyond at TMNL It’s a loop! From banks to TMNL to banks to TMNL to … TMNL and the banks exchange data in a circular relationship, including a feedback loop 1 2 3 Pseudonymized transaction data of businesses following a data contract Feedback on alerts Pseudonymized IDs of transactions that show anomalous characteristics w/ respect to money laundering (“alerts”) 

MLOps as a key enabler Introducing MLOps principles has much in common with building a modern factory MLOps does NOT equal tooling. It’s understanding processes, streamlining and automating them. Tools are a means to do so. 1. Understand the AS-IS process how your teams are building models and how they take them to production. 2. Identify bottlenecks, inefficiencies and other pain points. 3. Map out a target/TO-BE process (“Golden Path”) and the tooling/architectural/organisational changes required to get there. 4. Start moving to the target process (3) following priorities identified in (2). 5. Continuously update and measure the process using an adequate set of metrics, e.g. time to deployment, internal user satisfaction, time spent on production incidents… 

How MLOps enables vertical prototyping and beyond at TMNL MLOps is at the heart of what we do at TMNL We treat our MLOps platform as an internal product with model development teams being our primary customers § High-over, we measure MLOps platform success by – Non-negotiable features such as full lineage and reproducibility. – How much it accelerates model development teams to “do their job”. § Understanding our users is key - we maintain rigorous process documentation of the model developer workflow: – Repeatedly measure changes and pain/effectiveness/happiness/… per activity, – … providing the MLOps platform with orientation what to focus on from a user perspective. – … what to shape from a stream-lining perspective/platform. – Handbook for new joiners and ”index” for activity-specific documentation § Over time and iterations we’ve been paving a Golden Path: A first-class, supported way how to develop, test, deploy and monitor model pipelines. 

Healthy failing through vertical prototyping As a Data Scientist, you are taught to think “horizontal first” Horizontal prioritization is a result of seeing a model/dashboard as the final product, whereas the product typically is a lot more than that. - It’s an entire system that goes far beyond the model. Source of figure: Jones, M. C., Floyd, I. R., & Twidale, M. B. (2007). Patchwork prototyping: A rapid prototyping technique that harnesses the power of open-source software. Horizontal prioritization is predominant in Data Science 1. Fetch data 2. Exploratory Analysis 3. Build model 4. Evaluate model under lab-conditions (e.g. k-fold CV...) 5. Optimize model performance 6. Profit! (business value) 

How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? There is 1 streamlined, highly automated “Golden Path” to production, enabling efficient vertical prototyping and beyond. Model Development Teams Start new modeling project Exploratory analysis Model prototype Stable model pipeline Production-grade model pipeline Proof of Concept Operations 

Model Development Teams How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Start new modeling project Exploratory analysis Model prototype Stable model pipeline Production-grade model pipeline Proof of Concept Operations Model repository template Remote repository w/ approval rules On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Tools to ease interaction with different AWS services Shared tools or infrastructure Model Dev Teams start off from a template model repo with centrally managed CI/CD pipelines, quality checks and monitoring attached. Deployment happens from minute 1 and with every merge to release and main. Iterations on models do not happen in an isolated phase, but go all the way to UAT and prod environments (fully automated). 

Model Development Teams How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Start new modeling project Exploratory analysis Model prototype Stable model pipeline Production-grade model pipeline Proof of Concept Operations Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Every element we’ve been adding as platform capability has originated from rigorous focus on the model development workflow. On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

Model Development Teams How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Start new modeling project Exploratory analysis Model prototype Stable model pipeline Production-grade model pipeline Proof of Concept Operations Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

Model Development Teams How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Start new modeling project Exploratory analysis Model prototype Stable model pipeline Production-grade model pipeline Proof of Concept Operations Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry Centralized CI/CD Pipelines w/ checks On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

Model Development Teams How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Start new modeling project Exploratory analysis Model prototype Stable model pipeline Production-grade model pipeline Proof of Concept Operations Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry Centralized CI/CD Pipelines w/ checks Automated versioning & cataloging of model outputs Workflow Orchestration & scalable compute On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

Model Development Teams How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Start new modeling project Exploratory analysis Model prototype Stable model pipeline Production-grade model pipeline Proof of Concept Operations Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry Centralized CI/CD Pipelines w/ checks Automated versioning & cataloging of model outputs Workflow Orchestration & scalable compute Data (distribution) profiling & checks Integration Tests on higher envs On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

Model Development Teams How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Start new modeling project Exploratory analysis Model prototype Stable model pipeline Production-grade model pipeline Proof of Concept Operations Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry Centralized CI/CD Pipelines w/ checks Automated versioning & cataloging of model outputs Workflow Orchestration & scalable compute Data (distribution) profiling & checks Integration Tests on higher envs Contract “Production” & enablement Full auditability On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure 

Model Development Teams How MLOps enables vertical prototyping and beyond at TMNL How does MLOps support the (simplified) Model Development Workflow? Start new modeling project Exploratory analysis Model prototype Stable model pipeline Production-grade model pipeline Proof of Concept Operations Model repository template Remote repository w/ approval rules Tools to ease interaction with different AWS services Scalable self- service notebook environment Feature Store Base docker images for kernels & jobs Experiment tracking Model Registry Centralized CI/CD Pipelines w/ checks Automated versioning & cataloging of model outputs Workflow Orchestration & scalable compute Data (distribution) profiling & checks Integration Tests on higher envs Contract “Production” & enablement Full auditability Operational and model metric monitoring & alerting Automated triggering of inference pipelines On model project initiation auto- bootstrapped of (at least partly) customized infrastructure Shared tools or infrastructure