Making machine learning model deployment boring

The free lunch for machine learning is over. Organizations are quickly ramping up their abilities to automate and professionalize their machine learning processes and infrastructure. As a consequence organizational goals, processes and requirements put an increasing burden on teams to put machine learning models in production. We believe much of this burden relates to engineering issues, which with proper abstractions can be greatly reduced for product teams. In this presentation we will talk about the organizational context of ING and the design our Machine Learning Platform. In the first part we will sketch some organizational context and the requirements it brings. Next, we will picture the kind of use cases and user journey we have in mind. Finally, we will present how these considerations led the platform design we are currently deploying.

Transcript

1. Making machine learning model deployment boring Thomas Kluiters, ING Ruurtjan Pul, BigData Republic
2. ING wil softwaregeorienteerde bank worden (steeds meer producten zijn digitaal) en data driven zijn en dus ML gebruiken
3. ING wil softwaregeorienteerde bank worden (steeds meer producten zijn digitaal) en data driven zijn en dus ML gebruiken
4. Machine learning flips the equation Classical programming Rules Data Answers
5. Machine learning flips the equation Classical programming Rules Data Answers Machine learning Answers Data Rules
6. Machine learning helps many companies • Match candidates and vacancies • Recommend customer treatments to call center agents • Pick the right time for airplane maintenance
7. Machine learning helps many companies • Match candidates and vacancies • Recommend customer treatments to call center agents • Pick the right time for airplane maintenance
8. Machine learning helps many companies • Match candidates and vacancies • Recommend customer treatments to call center agents • Pick the right time for airplane maintenance
9. Machine learning requires many capabilities Statistics Algorithms Math Scripting Programming CI/CD Feature engineering Data plumbing Monitoring Stability Reliability Infrastructure Business value Compliance and Risk Integration Process Domain knowledge
10. Machine learning requires many capabilities Statistics Algorithms Math Scripting Programming CI/CD Feature engineering Data plumbing Monitoring Stability Reliability Infrastructure Business value Compliance and Risk Integration Process Domain knowledge
11. Machine learning requires many capabilities Statistics Algorithms Math Scripting Programming CI/CD Data plumbing Monitoring Stability Reliability Infrastructure Programming CI/CD Feature engineering Data plumbing Monitoring Stability Reliability Infrastructure Business value Compliance and Risk Integration Process Domain knowledge
12. Machine learning requires many capabilities Statistics Algorithms Math Scripting Programming CI/CD Feature engineering Data plumbing Monitoring Stability Reliability Infrastructure Statistics Algorithms Math Scripting Monitoring Stability Reliability Infrastructure Business value Compliance and Risk Integration Process Domain knowledge
13. Machine learning requires many capabilities Statistics Algorithms Math Scripting Programming CI/CD Feature engineering Data plumbing Monitoring Stability Reliability Infrastructure Statistics Algorithms Math Scripting Programming CI/CD Data plumbing Business value Compliance and Risk Integration Process Domain knowledge
14. Expertise oriented teams do not work Data lab Product team Data engineering Operations
15. Expertise oriented teams do not work Data lab Product team Data engineering Operations
16. Expertise oriented teams do not work Data lab Product team Data engineering Operations
17. Expertise oriented teams do not work Data lab Product team Data engineering Operations
18. Expertise oriented teams do not work Data lab Product team Data engineering Operations
19. Expertise oriented teams do not work Data lab Product team Data engineering Operations
20. Full stack teams don’t scale Feature team
21. Full stack teams don’t scale
22. Full stack teams don’t scale
23. Centralized teams should handle cross-cutting concerns Feature teams Centralized team
24. The machine learning platform
25. The machine learning platform • The platform is a service built by ING engineers, for ING engineers and data scientists • The platform is responsible for hosting machine learning models, and orchestrating them
26. The machine learning platform
27. The machine learning platform • Orchestrator as an interface between models and outside world • Kafka for streaming data • REST API for real-time and batch
28. • Allow teams to fully embrace the power and capabilities of machine learning • Only software engineers, data scientists and product owners are responsible The machine learning platform
29. A journey: the feedback use case • ING gathers user feedback • Feedback covers many categories
30. • Automatically categorize user feedback using machine learning • Establish business goals Inception of the idea Product owner
31. • Data scientists iteratively build a model • Incorporate feedback from product owner • Configure the model according to platform standards Building the model Data scientist
32. • Data scientist and software engineer agree on configuration of the model • Software engineer builds interface for consuming the platform • Data scientist deploys model on pipeline Packaging the model Software engineer & data scientist
33. • Automated pipelines deploy the model on the platform • Technologies such as Ansible, Docker and Gitlab CI • No actions are required by the platform team The packaged model Platform
34. The productionized model • Once in production, data scientists, software engineers and product owners can monitor their model • The model can be changed iteratively and quickly deployed again
35. The productionized model, monitoring
36. The productionized model, versioning • Models are tracked by their version number • Using historic data we can track the performance of the models
37. Centralized teams should handle cross-cutting concerns ING’s platform team provides machine learning deployment as a service Allowing data scientists to easily and quickly deploy models makes machine learning boring Key takeaways