MTC2018 - Mercari ML Platform

Speaker: Hirofumi Nakagawa

Mercari uses several machine learning (ML) models. After an ML model is created, it requires continuous tuning and retraining. This is why it's not possible to use multiple ML models without having a highly automated system in place. In this session, I will be talking about the fundamentals of such automation and systems, what we’re currently using, and what we're developing.

Transcript

1. Mercari ML Platform Hirofumi Nakagawa Software Engineer (SRE)
2. Software Engineer SRE Hirofumi Nakagawa
3. Mercari ML Platform
4. Kubernetes CLI Image builder Cluster Pipeline Engine Dashboard Metrics Runner Component Mercari ML Component ML Platform Architecture
5. Kubernetes CLI Image builder Cluster Pipeline Engine Dashboard Metrics Runner Component Mercari ML Component
6. Builds container image automatically. There is no need for the user to write Dockerfiles. Cluster Pipeline Engine CLI/Image builder Metrics Manages cluster resources and implements containerized data pipeline. Using Prometheus. Acquire metrics pertaining to ML model.
7. Kubernetes CLI Image builder Cluster Pipeline Engine Dashboard Metrics Runner Component Mercari ML Component
8. Componentized so that pre-processing and classifiers can be executed through container pipeline Runner ML Component Mercari ML Component Bridges the gap between the cluster and local environments using the training/serving environment Specialized components for internal use, such as data source and item categorization
9. Container-Based Pipeline
10. DataSource Image Text　 Pre-processing Image PV Picture　 Pre-processing Image PV PV Estimator Image All output is saved to PV and can be used as a cache.
11. Model Training & Serving Workflow
12. CI Training Cluster Job ・・・ Model Repository Job Job Serving Cluster REST API TF Serving Faiss ・・・ All models are turned into images and subject to version management.
13. Serving Architecture
14. Mercari API Flask SK Model SK Model SK Model TensorFlow Serving TF Model TF Model TF Model Virtual Service Basic Serving API Architecture
15. Streaming Serving API Architecture Flask SK Model SK Model SK Model TensorFlow Serving TF Model TF Model TF Model Virtual Service Proxy
16. A/B Test Architecture
17. Model Service A　 Virtual Service Model Service B　 Mercari API 90% 10% Model API Is Activated by Istio
18. Common Problems of Model Serving
19. Huge Model File vs. Container Image • Can a huge ML model file be included in the image? • If not, where should it be placed? • Trade-off between portability and load times
20. Effective Memory Usage • Normally uses several GB of memory • An environment where a model must be loaded for each process is not ideal. • Using Copy on Write (CoW) is necessary.
21. In Progress
22. Automation of model evaluation re-training Automated generation of models Automatic deployment to the production environment Simple model generation and simplification can be automated through methods like architecture search and hyper parameter optimization. Deploy the generated model to the production environment. The best model will be selected automatically. High Degree of Automation Conduct evaluation and visualization of the operation model. Automate re-training and raise its scale.
23. ML Continuous Deployment Deploy Monitoring Evaluation Hyper Parameter Optimization Re-Training Even after release, precision monitoring, hyper parameter tuning, re-training, and deployment will be conducted automatically.
24. AutoFlow Feature Extraction Components Concatenation Components Classification Components Model Builder Component Repository
25. In the Future
26. Edge Device Democratization of AI Further Automation Future Plans
27. Edge Device • An environment has been arranged that can conduct prediction for TensorFlow Lite, CoreML, etc. on the Edge site. • Predictions on Edge are considered to have great benefits for UX. • Research and investigation is underway, so stay tuned
28. Demo
29. Democratization of AI • Many models in demand • Make it possible for people other than ML engineers to make models • Necessary to prepare an environment in accordance with DataPlatform
30. Further Automation • It will be necessary to operate several thousand models in the near future. • It will be vital to conduct automation on an even greater scale.