Feature StoreとVertex AIを使った機械学習基盤の実現 / Machine Learning infrastructure using Feature Store and Vertex AI

Feature StoreとVertex AIを使った機械学習基盤の実現と、1年間運用してみて感じたこと、future work等を話します。
------
Merpay Tech Fest 2022は3日間のオンライン技術カンファレンスです。
IT企業で働くソフトウェアエンジニアおよびメルペイの技術スタックに興味がある方々を対象に2022年8月23日（火）から8月25日（木）までの3日間、開催します。 Merpay Tech Festは事業との関わりから技術への興味を深め、プロダクトやサービスを支えるエンジニアリングを知れるお祭りです。 セッションでは事業を支える組織・技術・課題などへの試行錯誤やアプローチを紹介予定です。お楽しみに！

■イベント関連情報
- 公式ウェブサイト：https://events.merpay.com/techfest-2022/
- 申し込みページ：https://mercari.connpass.com/event/249428/
- Twitterハッシュタグ： #MerpayTechFest
■リンク集
- メルカリ・メルペイイベント一覧：https://mercari.connpass.com/
- メルカリキャリアサイト：https://careers.mercari.com/
- メルカリエンジニアリングブログ：https://engineering.mercari.com/blog/
- メルカリエンジニア向けTwitterアカウント：https://twitter.com/mercaridevjp
- 株式会社メルペイ：https://jp.merpay.com/

Transcript

1. Machine Learning Infrastructure
   with Feature Store and Vertex AI
   Li
   Software Engineer (Machine Learning)
   

   View Slide

2. Li / @Li
   Software Engineer (Machine Learning)
   ● 2017/4 joined Yahoo! JAPAN
   ○ FrontEnd Engineer
   ○ 2019/1~: Machine Learning Engineer
   ● 2021/9 joined Merpay
   ○ Machine Learning Engineer
   ○ currently in charge of fraud
   prevention-related development.
   

   View Slide

3. Agenda
   01 Background
   02 Machine Learning Infrastructure
   03 What we think were good after 1 year of operation
   04 Future work
   

   View Slide

4. Agenda
   01 Background
   02 Machine Learning Infrastructure
   03 What we think were good after 1 year of operation
   04 Future work
   

   View Slide

5. Fraud Prevention Models
   ● Alert Filtering (multiple ML models)
   ○ ref: https://engineering.mercari.com/blog/entry/alertfiltering-ml/
   ● ChargeBack Detection (ML model)
   ○ ref: https://engineering.mercari.com/en/blog/entry/chargeback-ml/
   ● Sub Account Detection
   ● Suspicious Account Detection (rule-based logics)
   ● Suspicious Action Detection (complex network)
   ○ ref: https://engineering.mercari.com/blog/entry/complex-network-ml/
   ● (new) Account-based Chargeback Detection
   ● (new) Account Takeover Detection
   ● (new) Continuous transaction Detection
   ● (new) Overdue Payment Detection
   

   View Slide

6. t
   2020 2021
   Alert Filtering Model
   ChargeBack Detection
   Sub Account Detection
   Dimension ~ 40
   Alert Filtering Model
   × 4 sub model
   ChargeBack Detection
   latest version
   Sub Account Detection
   Suspicious Account Detection
   × ~ 4 detection
   Suspicious Action Detection
   × 2 mass fraud detection
   Dimension ~ 170
   Features
   ● feature dimension increased by 4 times within 1 year
   ● feature used by multiple models
   

   View Slide

7. Agenda
   01 Background
   02 Machine Learning Infrastructure
   03 What we think were good after 1 year of operation
   04 Future work
   

   View Slide

8. machine learning infrastructure
   

   View Slide

9. Vertex AI
   ref: https://cloud.google.com/vertex-ai/docs/start/introduction-unified-platform?hl=ja
   

   View Slide

10. Feature Store(Feast)
    ref: https://docs.feast.dev/
    

    View Slide

11. Before Feature Store
    BigQuery
    GCS
    Model A
    Training
    Model B
    Training
    Model C
    Training
    …
    

    View Slide

12. After Feature Store
    FeatureView1
    FeatureView2
    FeatureView3
    Feature
    Service1
    Feature
    Service2
    Feature
    Service3
    FeatureView4
    …
    …
    Model A
    Training
    Model B
    Training
    Model C
    Training
    …
    BigQuery
    GCS
    FEAST
    

    View Slide

13. Agenda
    01 Background
    02 Machine Learning Infrastructure
    03 What we think were good after 1 year of operation
    04 Future work
    

    View Slide

14. ● components
    ○ performs one step in ML workflow
    ● common components
    ○ commonly used components such
    as execute_bigquery_query
    ● pipelines
    ○ ML workflow, including all of the
    components in the workflow
    Elements Definition
    component
    pipeline
    

    View Slide

15. ● supervised learning process was similar between
    multiple models
    ● use the same training pipeline with different config
    files to train multiple models which reduce the cost of
    modeling
    Common Pipelines
    

    View Slide

16. Common Pipelines
    

    View Slide

17. Version Control
    def train():
    …
    pipeline.py
    component.py
    com1.yaml
    …
    com1.yaml
    com2.yaml
    compile compile
    

    View Slide

18. Without Version Control
    

    View Slide

19. With Version Control
    

    View Slide

20. pipeline uses
    different versions of
    common components
    Version Control
    

    View Slide

21. ● we use one-for-all versioning for components and
    pipeline files
    ● components and pipeline files are compiled and saved to
    gcs.
    ● we use different gcs path for different versions
    Version Control
    

    View Slide

22. Scheduled/On-demand Execution
    need to satisfy
    ● train pipeline
    ○ on-demand execution
    ● predict pipeline
    ○ scheduled execution
    ● common pipeline
    ○ execute with different config files
    

    View Slide

23. Scheduled/On-demand Execution
    

    View Slide

24. Continuous A/B Test
    

    View Slide

25. ML Monitoring with Slack Notifier
    

    View Slide

26. Agenda
    01 Background
    02 Machine Learning Infrastructure
    03 What we think were good after 1 year of operation
    04 Future work
    

    View Slide

27. Feature Online Store with Stream Ingest
    WIP
    

    View Slide

28. WIP
    Data Drift Detection and Re-Train
    

    View Slide

29. View Slide