What do we talk about when we talk about ML maturity?

In the last year, the world's buzzing about Machine Learning, with a lot of investments in this area creating new job opportunities, teams and an higher pressure to deliver. Machine Learning Engineering and Data Science Teams share a lot of challenges when compared to more "classical" software engineering counterparts but also add new ones, related to the different technological nature of their deliverables, the time horizons related to some of the projects and the different skills and roles involved in their execution.
In this talk we will go through the challenges, concepts and ideas to be applied in companies of different sizes to understand own level of maturity and try to introduce reliable frameworks to grow project after project.

What does it mean to be "mature" in shipping ML projects in the industry in the first place?

Transcript

1. What do we talk about when we talk about ML maturity? Massimo Belloni Data Science & Machine Learning Manager
2. Data Science Manager @ Bumble Inc Massimo Belloni • Machine Learning Engineer (󰏢) - lived and worked in Rotterdam (󰐗) and London (󰏅) • Currently Data Science Manager at Bumble Inc for Integrity & Safety and MLOps Who am I? @massibelloni /massibelloni massibelloni@gmail.com
3. 3 maturity (noun) the state of being mentally and emotionally well-developed, and therefore responsible; the state of being completely grown; a very advanced or developed form or state; the time when an insurance agreement or investment becomes ready to be paid. Cambridge Dictionary
4. 4 maturity (noun) the state of being mentally and emotionally well-developed, and therefore responsible; the state of being completely grown; a very advanced or developed form or state; the time when an insurance agreement or investment becomes ready to be paid. Cambridge Dictionary
5. 5 maturity (noun) the state of being mentally and emotionally well-developed, and therefore responsible; the state of being completely grown; a very advanced or developed form or state; the time when an insurance agreement or investment becomes ready to be paid. Cambridge Dictionary
6. 6 maturity (noun) the state of being mentally and emotionally well-developed, and therefore responsible; the state of being completely grown; a very advanced or developed form or state; the time when an insurance agreement or investment becomes ready to be paid. Cambridge Dictionary
7. 7 87 200
8. 8 87 percent; data science projects that never make it to production [2] 200 billions; investments in AI globally by 2025 [1] [1] Goldman Sachs, 2023 [2] VentureBeat, 2019
9. 9 Proven track record of having a quantifiable and controllable impact on relevant business metrics Ability to align data products’ output to a mature set of metrics and indicators, ease to quantify projects’ success Comfort in roadmapping and project management exercises; transparency in all iterative steps; ability to correlate steps with end results. Innovation and development of new capabilities are organic and baked in in the usual team dynamics and life-cycle Transparency & Replicability Success Criteria Business Impact Forward looking
11. 11 Having a reference for what “good” looks like from successful organisations can really help newer teams and businesses to position themselves in the industry and set achievable targets. 🧱Shared milestones It’s helpful to have a vision for the end goal, but it’s much more useful to have it broke down into subsequent iterative steps, with checkpoints along the way. It’s not uncommon to see teams settling in a middle ground that works well for the challenges they are facing. 👷 Iterative nature Enhanced visibility over shared challenges and approaches can facilitate the adoption of relevant tooling if not the creation of completely new tools (open source or not!) to address them properly. 🛠 Industry tooling ML Maturity Frameworks: Why?
12. 12 Azure (Accelerator) Source: MLOps Maturity Model Automated Training Manual deployments No MLOps DevOps but no MLOps Heavy reliance on other teams Automated Model Deployment Ownership over the full process Full MLOps Monitoring and observable downtime
13. 13 Automated Training No MLOps DevOps but no MLOps Automated Model Deployment Full MLOps ⏳ 🛠 When is the right moment to walk up? Which tools can help?
14. • Strong business need to deploy the first data product to production • Possibility to leverage internal compute / DevOps resources • Enough skills to build and own a service • Full access to data resources 14 No MLOps DevOps but no MLOps • FastAPI • Docker • Kubernetes ⏳
15. 15 Automated Training DevOps but no MLOps • Overall requirement for more transparency and faster experimentation cycles • Struggles to replicate experiments and to keep track of projects’ status • Lots of manual “boilerplate” work required to run experiments in the usual way - better to allocate quality time somewhere else • Airflow • Docker • Kubeflow Pipelines • K8s Cronjobs • ClearML ⏳
16. ⏳ 16 Automated Training Automated Model Deployment • Increased transparency and observability when deploying to different environments • Organic necessity to start supporting different frameworks • Democratise access to serving to significantly increase team’s throughput • ClearML • GitOps • Helm • KServe
17. ⏳ 17 Automated Model Deployment Full MLOps • Requirement of full end to end observability often arising from business critical systems • Solid integrations and ability to react fast when on call • Unlocking and facilitating complex debugging tasks • Full understanding of performance bottlenecks to maximise hardware utilization • Prometheus • Grafana • Vector 🛠
18. 18 Full MLOps • Moving towards end to end platform able to support more complex solutions, even before they arise • Ability to prioritise MLOps own roadmap towards industry standards and state of the art frameworks • Unlocking innovative business ideas through facilitation • Open source/industry contributions to improve state of the art tooling • Milvus • NVIDIA Triton • TensorRT • Feast ⏳ 🛠
20. Data Science Manager @ Bumble Inc Massimo Belloni Thank you! @massibelloni /massibelloni massibelloni@gmail.com