#MerpayTechFest Session Title Using Feature Store and Vertex Pipelines in Fraud Prevention System Liu Songjie Software Engineer (Machine Learning)

#MerpayTechFest Software Engineer (Machine Learning) Liu Songjie I joined Merpay as a new graduate in 2019. I have been working on machine learning solutions for fraud prevention such as chargeback detecting. Recently, I've been involved in the development of Vertex AI based ML pipelines.

#MerpayTechFest Overview Background Feature Store Training-Deploying Pipeline Summary 02 03 04 01

#MerpayTechFest Background

#MerpayTechFest Fraud Prevention System ● Multiple fraud detection related solutions in the system ○ Alert Filtering (multiple ML models) ■ https://engineering.mercari.com/blog/entry/alertfiltering-ml/ ○ ChargeBack Detection (ML model) ■ https://engineering.mercari.com/en/blog/entry/chargeback-ml/ ○ Sub Account Detection ○ (New) Suspicious Account Detection (rule-based logics) ○ (New) Suspicious Action Detection (complex network) ■ https://engineering.mercari.com/blog/entry/complex-network-ml/

#MerpayTechFest Legacy Fraud Prevention System

#MerpayTechFest Features t 2020 2021 Alert Filtering Model ChargeBack Detection Sub Account Detection Dimension ~ 40 Alert Filtering Model × 4 sub model ChargeBack Detection latest version Sub Account Detection Suspicious Account Detection × ~ 4 detection Suspicious Action Detection × 2 mass fraud detection Dimension ~ 170 The total dimension of features increased by 4 times within one year

#MerpayTechFest Features ● Various data sources ○ BigQuery, Spanner, GCS,… ● Different data sources between training and predicting ○ Some of the models need to predict in real-time, so they need to use Spanner in predicting for lower latency ● The same or similar features are created multiple times for different models

#MerpayTechFest Features

#MerpayTechFest Features ● Cost of maintenance ○ Large amount of the features become difficult to maintain ● Cost of computing resource ○ The same features created multiple times for different models

#MerpayTechFest Features Features in Model B Features in Model A Features only in Model A Features only in Model B Features in both Model A & B Obviously, we can reduce costs by reusing the features of different models

#MerpayTechFest Legacy Architecture Every model has its own pipeline and works separately

#MerpayTechFest Legacy Architecture Training set BigQuery Data Sources Training AI Platform Jobs Serving AI Platform Models Outputs Predicting Input

#MerpayTechFest Legacy Architecture Training set BigQuery Data Sources Training AI Platform Jobs Serving AI Platform Models Outputs Predicting Input

#MerpayTechFest Legacy Architecture ● A common training-deploying process ○ Training sets are saved in BigQuery ○ Models are trained by AI Platform Training Jobs ○ Models are deployed on AI Platform Models ● Difference between each solution ○ Predicting inputs ■ BigQuery, Spanner, .. ○ Predicting outputs ■ Connect to different microservices or systems

#MerpayTechFest Problems & Needs ● About Features ○ Unnecessary costs on creating and maintaining features ○ Necessity of improving the data reliability ● About Legacy Architecture ○ Necessity of a shared training-deploying pipeline

#MerpayTechFest The new system architecture A feature store for all models A common pipeline for training-deploying process 02 01

#MerpayTechFest The new system architecture

#MerpayTechFest Feature Store

#MerpayTechFest Feature Store

#MerpayTechFest Feature Store (Before)

#MerpayTechFest Feature Store (After)

#MerpayTechFest What is a Feature Store ref: https://feast.dev/blog/what-is-a-feature-store/

#MerpayTechFest Feature Store Candidates Vertex Feature Store & Feast Vertex Feature Store Feast Stream Ingesting No Yes GCP Support Native Good Cost High Low We choose to use Feast for now

#MerpayTechFest Preparation ● Define FeatureView (Schema) ○ entities ○ features ○ input ○ ...

#MerpayTechFest Preparation ● Run `feast apply`

#MerpayTechFest Preparation ● Then, access the feature store by SDK

#MerpayTechFest Features in Bigquery (Offline firestore) timestamp entity feature ● Entity ● Feature ● Timestamp

#MerpayTechFest Data Stream

#MerpayTechFest Data Stream

#MerpayTechFest Training-Deploying Pipeline

#MerpayTechFest Training-Deploying Pipeline

#MerpayTechFest Vertex Pipelines

#MerpayTechFest Vertex Pipelines ● Based on Kubeflow Pipelines SDK or TensorFlow Extended ○ Kubeflow Pipelines SDK is used in our system ● Easier to get start than Kubeflow Pipelines ○ Kubeflow Pipelines must be deployed on a Kubernetes Cluster. ■ Lots of permission issues ■ Hard to start and maintain ○ Vertex Pipelines ■ Provision all resources ■ Store all the artifacts ■ Pass resources, artifacts through each steps.

#MerpayTechFest Vertex Pipelines

#MerpayTechFest Get-data ● Get data from feast offline store and save into Vertex AI Dataset. ● Run a python script in a custom docker container with feast SDK installed

#MerpayTechFest Train-model ● Train model in a Docker container by Vertex AI custom container training job ● Prepare the necessary libraries in the docker image ● Custom the training process by python script in the container

#MerpayTechFest Deploy-model ● Deploy the model to the endpoint and start serving ● Access endpoint using curl

#MerpayTechFest Pipelines

#MerpayTechFest Summary

#MerpayTechFest Summary ● The feature store and common training-predicting pipeline bring following benefits ○ ML engineers can focus more on model development and improvement ○ Increased development efficiency ○ Smoother onboarding and collaboration ● This architecture is still a tentative plan, we will keep improving it in the future

#MerpayTechFest Future Work ● A common pipeline for the prediction process ● Store prediction results / graph features in the feature store for analysis and modeling ● Continuous monitoring and quality control ● Massively producing supervised models

#MerpayTechFest Thank you for listening!