1 MLOps in Mercari Group's Trust and Safety ML Team PyCon JP 2024 Calvin Janitra Halim 

2 Team Trust and Safety ML Position ML Engineer Work MLOps, Data Analysis, etc. Hobbies Music Production, Jamming Career 2021-04 ~ 2023-12 Rakuten 2024-01 ~ Mercari Email [email protected] [email protected] GitHub, Qiita, Spotify, Soundcloud: CJHJ LinkedIn, Medium, X: My name Calvin Janitra Halim 

3 Agenda 1. Introduction a. What’s Trust and Safety? b. ML Team’s Responsibilities 2. System Architecture a. Marketplace Service Architecture b. Fintech Service Architecture 3. Automation Steps a. MLOps Overview b. Experimentation c. Feature store d. Training e. Deployment f. Monitoring 4. Struggles and Learnings 5. Closing 

4 Introduction 

5 What’s Trust and Safety? Japan Region Fintech Marketplace 

6 What’s Trust and Safety? Trust and Safety To ensure a safe and secure marketplace Fraud detection Monitoring Prevent policy violations 

7 ML Team’s Responsibilities ● Item Moderation ● Account Moderation ● Transaction Moderation Fintech Marketplace 

8 System Architectures 

9 Marketplace Service Architecture (Realtime) 

10 Fintech Service Architecture (Batch) 

11 Service Architecture - Feature Update 

12 FT Service Architecture - Inference 

13 FT Service Architecture - Sync Results 

14 MLOps Overview Adapted from this GCP doc with modification Automation 

15 MLOps Overview Adapted from this GCP doc with modification 

16 Automation Steps 

17 Experimentation Model comparison Data loading EDA Evaluation Error analysis 

18 Package as a template notebook Experimentation Model comparison To facilitate the collection of results from models based on different configurations, we use the Vertex AI's experiment feature. Data loading We usually use Feast to make it easier to perform point-in-time joins in historical features. EDA E.g. univariate analysis helps us determine which features will contribute to the model’s performance. Evaluation PR-AUC and Feature importances are what we usually check to evaluate models. And also some other business metrics. Error analysis A deep dive into false positives and false negatives is really helpful for identifying the types of patterns the model could miss, as well as any bugs in the data we use. write from scratch 

19 Creating and Using Features 

20 Creating and Using Features Define feature tables as dbt models Table dependencies, SQL parameterized values, validation 

21 Creating and Using Features Sync between offline and online feature stores Less discrepancy 

22 Model Training Automate training through GitOps Sync config between repo and trained model 

23 Model Deployment 

24 Model Deployment Track trained models through GitOps and model registry 

25 Model Deployment Flexible Deployment Types! 

26 Deploying Pipelines Automate pipeline compl. and config through GitOps 

27 Model Monitoring 

28 Struggles and Learnings 

29 Risks and Costs Risks Automations Optimizations Reviews & Approvals Cost Monitor & Reduce 

30 Struggle with Tools Ended up with Vertex AI Feature Store dbt Dataflow BigQuery BigTable Offline features Online features Vertex AI Endpoint or Ended up with Vertex AI Model Registry GKE Pod 

31 Closing https://github.com/CJHJ