Machine Learning Infrastructure with Feature Store and Vertex AI Li Software Engineer (Machine Learning)

Li / @Li Software Engineer (Machine Learning) ● 2017/4 joined Yahoo! JAPAN ○ FrontEnd Engineer ○ 2019/1~: Machine Learning Engineer ● 2021/9 joined Merpay ○ Machine Learning Engineer ○ currently in charge of fraud prevention-related development.

Agenda 01 Background 02 Machine Learning Infrastructure 03 What we think were good after 1 year of operation 04 Future work

Agenda 01 Background 02 Machine Learning Infrastructure 03 What we think were good after 1 year of operation 04 Future work

Fraud Prevention Models ● Alert Filtering (multiple ML models) ○ ref: https://engineering.mercari.com/blog/entry/alertfiltering-ml/ ● ChargeBack Detection (ML model) ○ ref: https://engineering.mercari.com/en/blog/entry/chargeback-ml/ ● Sub Account Detection ● Suspicious Account Detection (rule-based logics) ● Suspicious Action Detection (complex network) ○ ref: https://engineering.mercari.com/blog/entry/complex-network-ml/ ● (new) Account-based Chargeback Detection ● (new) Account Takeover Detection ● (new) Continuous transaction Detection ● (new) Overdue Payment Detection

t 2020 2021 Alert Filtering Model ChargeBack Detection Sub Account Detection Dimension ~ 40 Alert Filtering Model × 4 sub model ChargeBack Detection latest version Sub Account Detection Suspicious Account Detection × ~ 4 detection Suspicious Action Detection × 2 mass fraud detection Dimension ~ 170 Features ● feature dimension increased by 4 times within 1 year ● feature used by multiple models

Agenda 01 Background 02 Machine Learning Infrastructure 03 What we think were good after 1 year of operation 04 Future work

machine learning infrastructure

Vertex AI ref: https://cloud.google.com/vertex-ai/docs/start/introduction-unified-platform?hl=ja

Feature Store(Feast) ref: https://docs.feast.dev/

Before Feature Store BigQuery GCS Model A Training Model B Training Model C Training …

After Feature Store FeatureView1 FeatureView2 FeatureView3 Feature Service1 Feature Service2 Feature Service3 FeatureView4 … … Model A Training Model B Training Model C Training … BigQuery GCS FEAST

Agenda 01 Background 02 Machine Learning Infrastructure 03 What we think were good after 1 year of operation 04 Future work

● components ○ performs one step in ML workflow ● common components ○ commonly used components such as execute_bigquery_query ● pipelines ○ ML workflow, including all of the components in the workflow Elements Definition component pipeline

● supervised learning process was similar between multiple models ● use the same training pipeline with different config files to train multiple models which reduce the cost of modeling Common Pipelines

Common Pipelines

Version Control def train(): … pipeline.py component.py com1.yaml … com1.yaml com2.yaml compile compile

Without Version Control

With Version Control

pipeline uses different versions of common components Version Control

● we use one-for-all versioning for components and pipeline files ● components and pipeline files are compiled and saved to gcs. ● we use different gcs path for different versions Version Control

Scheduled/On-demand Execution need to satisfy ● train pipeline ○ on-demand execution ● predict pipeline ○ scheduled execution ● common pipeline ○ execute with different config files

Scheduled/On-demand Execution

Continuous A/B Test

ML Monitoring with Slack Notifier

Agenda 01 Background 02 Machine Learning Infrastructure 03 What we think were good after 1 year of operation 04 Future work

Feature Online Store with Stream Ingest WIP

WIP Data Drift Detection and Re-Train

No content