Slide 1

Slide 1 text

#MerpayTechFest Session Title Using Feature Store and Vertex Pipelines in Fraud Prevention System Liu Songjie Software Engineer (Machine Learning)

Slide 2

Slide 2 text

#MerpayTechFest Software Engineer (Machine Learning) Liu Songjie I joined Merpay as a new graduate in 2019. I have been working on machine learning solutions for fraud prevention such as chargeback detecting. Recently, I've been involved in the development of Vertex AI based ML pipelines.

Slide 3

Slide 3 text

#MerpayTechFest Overview Background Feature Store Training-Deploying Pipeline Summary 02 03 04 01

Slide 4

Slide 4 text

#MerpayTechFest Background

Slide 5

Slide 5 text

#MerpayTechFest Fraud Prevention System ● Multiple fraud detection related solutions in the system ○ Alert Filtering (multiple ML models) ■ https://engineering.mercari.com/blog/entry/alertfiltering-ml/ ○ ChargeBack Detection (ML model) ■ https://engineering.mercari.com/en/blog/entry/chargeback-ml/ ○ Sub Account Detection ○ (New) Suspicious Account Detection (rule-based logics) ○ (New) Suspicious Action Detection (complex network) ■ https://engineering.mercari.com/blog/entry/complex-network-ml/

Slide 6

Slide 6 text

#MerpayTechFest Legacy Fraud Prevention System

Slide 7

Slide 7 text

#MerpayTechFest Features t 2020 2021 Alert Filtering Model ChargeBack Detection Sub Account Detection Dimension ~ 40 Alert Filtering Model × 4 sub model ChargeBack Detection latest version Sub Account Detection Suspicious Account Detection × ~ 4 detection Suspicious Action Detection × 2 mass fraud detection Dimension ~ 170 The total dimension of features increased by 4 times within one year

Slide 8

Slide 8 text

#MerpayTechFest Features ● Various data sources ○ BigQuery, Spanner, GCS,… ● Different data sources between training and predicting ○ Some of the models need to predict in real-time, so they need to use Spanner in predicting for lower latency ● The same or similar features are created multiple times for different models

Slide 9

Slide 9 text

#MerpayTechFest Features

Slide 10

Slide 10 text

#MerpayTechFest Features ● Cost of maintenance ○ Large amount of the features become difficult to maintain ● Cost of computing resource ○ The same features created multiple times for different models

Slide 11

Slide 11 text

#MerpayTechFest Features Features in Model B Features in Model A Features only in Model A Features only in Model B Features in both Model A & B Obviously, we can reduce costs by reusing the features of different models

Slide 12

Slide 12 text

#MerpayTechFest Legacy Architecture Every model has its own pipeline and works separately

Slide 13

Slide 13 text

#MerpayTechFest Legacy Architecture Training set BigQuery Data Sources Training AI Platform Jobs Serving AI Platform Models Outputs Predicting Input

Slide 14

Slide 14 text

#MerpayTechFest Legacy Architecture Training set BigQuery Data Sources Training AI Platform Jobs Serving AI Platform Models Outputs Predicting Input

Slide 15

Slide 15 text

#MerpayTechFest Legacy Architecture ● A common training-deploying process ○ Training sets are saved in BigQuery ○ Models are trained by AI Platform Training Jobs ○ Models are deployed on AI Platform Models ● Difference between each solution ○ Predicting inputs ■ BigQuery, Spanner, .. ○ Predicting outputs ■ Connect to different microservices or systems

Slide 16

Slide 16 text

#MerpayTechFest Problems & Needs ● About Features ○ Unnecessary costs on creating and maintaining features ○ Necessity of improving the data reliability ● About Legacy Architecture ○ Necessity of a shared training-deploying pipeline

Slide 17

Slide 17 text

#MerpayTechFest The new system architecture A feature store for all models A common pipeline for training-deploying process 02 01

Slide 18

Slide 18 text

#MerpayTechFest The new system architecture

Slide 19

Slide 19 text

#MerpayTechFest Feature Store

Slide 20

Slide 20 text

#MerpayTechFest Feature Store

Slide 21

Slide 21 text

#MerpayTechFest Feature Store (Before)

Slide 22

Slide 22 text

#MerpayTechFest Feature Store (After)

Slide 23

Slide 23 text

#MerpayTechFest What is a Feature Store ref: https://feast.dev/blog/what-is-a-feature-store/

Slide 24

Slide 24 text

#MerpayTechFest Feature Store Candidates Vertex Feature Store & Feast Vertex Feature Store Feast Stream Ingesting No Yes GCP Support Native Good Cost High Low We choose to use Feast for now

Slide 25

Slide 25 text

#MerpayTechFest Preparation ● Define FeatureView (Schema) ○ entities ○ features ○ input ○ ...

Slide 26

Slide 26 text

#MerpayTechFest Preparation ● Run `feast apply`

Slide 27

Slide 27 text

#MerpayTechFest Preparation ● Then, access the feature store by SDK

Slide 28

Slide 28 text

#MerpayTechFest Features in Bigquery (Offline firestore) timestamp entity feature ● Entity ● Feature ● Timestamp

Slide 29

Slide 29 text

#MerpayTechFest Data Stream

Slide 30

Slide 30 text

#MerpayTechFest Data Stream

Slide 31

Slide 31 text

#MerpayTechFest Training-Deploying Pipeline

Slide 32

Slide 32 text

#MerpayTechFest Training-Deploying Pipeline

Slide 33

Slide 33 text

#MerpayTechFest Vertex Pipelines

Slide 34

Slide 34 text

#MerpayTechFest Vertex Pipelines ● Based on Kubeflow Pipelines SDK or TensorFlow Extended ○ Kubeflow Pipelines SDK is used in our system ● Easier to get start than Kubeflow Pipelines ○ Kubeflow Pipelines must be deployed on a Kubernetes Cluster. ■ Lots of permission issues ■ Hard to start and maintain ○ Vertex Pipelines ■ Provision all resources ■ Store all the artifacts ■ Pass resources, artifacts through each steps.

Slide 35

Slide 35 text

#MerpayTechFest Vertex Pipelines

Slide 36

Slide 36 text

#MerpayTechFest Get-data ● Get data from feast offline store and save into Vertex AI Dataset. ● Run a python script in a custom docker container with feast SDK installed

Slide 37

Slide 37 text

#MerpayTechFest Train-model ● Train model in a Docker container by Vertex AI custom container training job ● Prepare the necessary libraries in the docker image ● Custom the training process by python script in the container

Slide 38

Slide 38 text

#MerpayTechFest Deploy-model ● Deploy the model to the endpoint and start serving ● Access endpoint using curl

Slide 39

Slide 39 text

#MerpayTechFest Pipelines

Slide 40

Slide 40 text

#MerpayTechFest Summary

Slide 41

Slide 41 text

#MerpayTechFest Summary ● The feature store and common training-predicting pipeline bring following benefits ○ ML engineers can focus more on model development and improvement ○ Increased development efficiency ○ Smoother onboarding and collaboration ● This architecture is still a tentative plan, we will keep improving it in the future

Slide 42

Slide 42 text

#MerpayTechFest Future Work ● A common pipeline for the prediction process ● Store prediction results / graph features in the feature store for analysis and modeling ● Continuous monitoring and quality control ● Massively producing supervised models

Slide 43

Slide 43 text

#MerpayTechFest Thank you for listening!