individuals can give new life to items around them—that have fallen out of use—by selling them to other customers. • Our unique challenge is around pricing - since items are in various conditions, we need to help customers ﬁnd the right price to sell their items, which is where we leverage ML. • Today, I'm going to talk about the operations side of our ML pipeline to enable features like our Price Guidance System.
Price Suggestion feature recommends a viable price range for an item during the listing process. • The Smart Pricing feature continuously updates the listing price until it hits a user-speciﬁed ﬂoor price or the item gets sold. • An ML model takes item’s title, description, category, brand, and condition to suggest the listing and ﬂoor prices in real-time. Mercari engineering | Price Guidance System leveraging Artificial Intelligence Techniques https://medium.com/mercari-engineering/price-guidance-system-74358bd96081
accelerate the iteration is the key to the success of projects. We can be able to accelerate iterations by automating manual processes with open-source MLOps and DevOps tools Organizing machine learning projects: project management guidelines. https://www.jeremyjordan.me/ml-projects-guide/
Polyaxon is an ML Ops tool to support scalable and reproducible model exploration. Continuous Training with Kubeﬂow Pipelines Kubeﬂow Pipelines is an open source for ML training pipeline management. We set up a scheduled job to build a docker image to serve a new trained model on top of it. Continuous Delivery with Spinnaker Spinnaker is an open source for Continuous Delivery. Spinnaker is able to trigger a deploy pipeline when an image is pushed to an image registry. Organizing machine learning projects: project management guidelines. https://www.jeremyjordan.me/ml-projects-guide/
an MLOps tool to support scalable and reproducible model exploration. • Polyaxon provides a yaml speciﬁcation to run hyperparameter tuning jobs on Kubernetes. The tuning jobs will run parallelly and scalably on top of a cluster autoscaler. • The yaml speciﬁcation enables other developers to reproduce the experiment easily. 1
run a parameter tuning job. Create code to train an ML model. Upload Polyaxonﬁle and the code with Polyaxon CLI. Experiments will run on Kubernetes. 1 2 3 4 My Favorite Point Polyaxon builds a docker image to run training code. A developer doesn’t have to wait for CI to build a docker image every time the developer modiﬁes the code. That prevents an interruption from happening in a development ﬂow.
• For about 2 years since Feb, 2019. How many projects/experiments we’ve run • 175 projects • About 87,000 experiments What infrastructure we’ve been using • Google Cloud Kubernetes Engine • Google Cloud Storage for logs, data, and artifacts • Regular, Preemptible x CPU, GPU node-pools • Google Filestore as NFS Persistent Volume
Learning Workﬂow Engine • Kubeﬂow Pipelines is an open source to manage end-to-end machine learning pipelines. • Kubeﬂow Pipelines has an integrated metadata store. The inputs and outputs of a stage will be automatically stored in the metadata store. • Kubeﬂow Pipelines allows a developer to implement easily a reusable component based on Python SDK. 2
for Continuous Model Deployment • KFP Web UI enables a developer to set up a scheduled job. • A pipeline submits a training job on Polyaxon and builds a docker image to serve a new trained model. • Spinnaker can trigger a deployment pipeline automatically when a new docker image is pushed to a docker registry. Mercari engineering | Continuous delivery and automation pipelines in machine learning with Polyaxon and Kubeflow Pipelines https://medium.com/mercari-engineering/continuous-delivery-and-automation-pipelines-in-machine-learning-with-polyaxon-and-kubeflow-d6a3668715de
We built a monorepo to manage pipeline versions in a git workﬂow and to share best practices. Manifests to manage projects on Polyaxon and KFP We deﬁned a manifest to prepare resources on Kubeﬂow Pipelines and Polyaxon like infrastructure as code. A KFP component to submit a Polyaxon job We developed a Kubeﬂow Pipelines component to submit a job from KFP to Polyaxon. 3
Monorepo contains KFP components and a python package to deﬁne lightweight KFP components. • CI will detect modiﬁed pipelines, and compiles and uploads them as the version: branch_name + commit_hash. • When a branch is merged into the main branch, CI will upload the updated pipelines to the production cluster. $ tree mercari-us-kubeflow-pipelines mercari-us-kubeflow-pipelines ├── components # directory for KFP components ├── docs # directory for documents ├── package │ └── merkfp # python package for lightweight KFP components ├── pipelines # directory for each project pipelines │ └── mercari-us-ml-price-suggestion │ └── train_model.py ├── projects # directory for “project” manifests │ └── mercari-us-ml-price-suggestion.yml └── scripts # directory for scripts on continuous integration
resources like Infrastructure as Code • CI will create KFP experiments and Polyaxon projects for the development and production environments to keep consistency. • CI will generate Github code owners based on “owners”. It allows each team to approve pull requests to modify project-related code. --- kind: Project name: mercari-us-ml-price-suggestion experiments: - name: “Default” - name: “Sneakers” - name: “Trading Cards” owners: - github: "@kouzoh/mercari-price-suggest-us-prod" mercari-ml-price-suggestion-us.yml
repo with a secret. The main container logs a user in to Polyaxon with a secret. The main container submits a training job through Polyaxon API. The main container trails the logs until the job ends. The component outputs Project, User, Job ID, Status for the next step. 1 2 3 4 5
delivery and automation pipelines in machine learning with Polyaxon and Kubeflow Pipelines https://medium.com/mercari-engineering/continuous-delivery-and-automation-pipelines-in-machine-learning-with-polyaxon-and-kubeflow-d6a3668715de