2019/8 機械学習基盤の開発コンセプト Google Cloud Next Tokyo ’19 2020/6 パイプライン基盤(バッチ予測) Google Cloud Day Digital ’20 2020/12 リアルタイム予測 Google ML Summit ’20 2021/5 バッチ基盤拡張 Google Cloud Day Digital ’21 2022/4 Light MLOps Google Cloud Day Digital ’22
Server UI Images Container Registory KARTE System Kubernetes Engine KARTE System CI/CD Pipeline Github Actions 弊社開発者 Status Cloud Pub/Sub ML Pipeline AI Platform Pipeline Batch Train AI Platform Training Transform Train Inference KARTE Data BigQuery クライアント AI Platform Pipeline パイプライン基盤(バッチ予測)
(Experiment, Train, Deploy) All Data BigQuery Experiment AI Platform Notebook ML Pipeline AI Platform Pipelines Batch Train AI Platform Training Metadata Cloud Firestore Artifacts Cloud Storage ML Realtime System Prediction API Cloud Run Event Data Cloud Spanner API trigger Cloud Pub/Sub Data Process Cloud Dataflow Prediction Result Cloud Bigtable Event Data Cloud Pub/Sub Core Server Compute Engine User Data Cloud Bigtable Core System エンドユーザー Web App リアルタイム予測
Scientist Client Cloud Storage Cloud Run JobFlow Work ows BigQuery KARTE Data BigQuery BigQuery Ve ex Workbench Simplify - Facebook - Instagram BQ to BQ を実現するには?
Server UI Images Container Registory KARTE System Kubernetes Engine KARTE System CI/CD Pipeline Github Actions 弊社開発者 Status Cloud Pub/Sub ML Pipeline AI Platform Pipeline Batch Train AI Platform Training Transform Train Inference KARTE Data BigQuery クライアント AI Platform Pipeline バッチ予測基盤
Train, Deploy) All Data BigQuery Experiment AI Platform Notebook ML Pipeline AI Platform Pipelines Batch Train AI Platform Training Metadata Cloud Firestore Artifacts Cloud Storage ML Realtime System Prediction API Cloud Run Event Data Cloud Spanner API trigger Cloud Pub/Sub Data Process Cloud Dataflow Prediction Result Cloud Bigtable Event Data Cloud Pub/Sub Core Server Compute Engine User Data Cloud Bigtable Core System エンドユーザー Web App リアルタイム予測基盤 リアルタイム予測
event data を受け取る ② event data を DB に保存 ③ 特定 event のみを抽出して、 Prediction API に リクエスト送信 ④ DB から生データを取得し、予測 ⑤ 予測結果を DB に書き込み ML Realtime System Prediction API Cloud Run Event Data Cloud Spanner API trigger Cloud Pub/Sub Data Process Cloud Dataflow Prediction Result Cloud Bigtable Event Data Cloud Pub/Sub ① ② ③ ④ ⑤ { "event_name": "view", "sync_date": “2020-11-29 23:50:23.142 UTC", "keys": { "api_key": "xxxxxx", "user_id": "user_B", "session_id": "yyyy_662", }, "page": "https://plaid.co.jp/", ... } event data 例 リアルタイム予測基盤 リアルタイム予測