Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to ML Pipelines

Introduction to ML Pipelines


## Reference

### ML Pipelines for Software Engineers

GigaOm-Delivering on the Vision of MLOps - Microsoft Azure https://azure.microsoft.com/ja-jp/resources/gigaom-delivering-on-the-vision-of-mlops/
Sculley, D. and Holt, Gary and Golovin, Daniel and Davydov, Eugene and Phillips, Todd and Ebner, Dietmar and Chaudhary, Vinay and Young, Michael and Crespo, Jean-Fran\c{c}ois and Dennison, Dan (2015) Hidden Technical Debt in Machine Learning Systems, Advances in Neural Information Processing Systems 28 (NIPS 2015) https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
MLOps: 機械学習における継続的デリバリーと自動化のパイプライン - Google Cloud https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
Denis Baylor and Kevin Haas and Konstantinos Katsiapis and Sammy Leong and Rose Liu and Clemens Menwald and Hui Miao and Neoklis Polyzotis and Mitchell Trott and Martin Zinkevich, Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform, 2019 {USENIX} Conference on Operational Machine Learning (OpML 19) (2019) https://www.usenix.org/conference/opml19/presentation/baylor
Daniel Papasian and Todd Underwood, How ML Breaks: A Decade of Outages for One Large ML Pipeline, USENIX Association 2020 https://www.usenix.org/conference/opml20/presentation/papasian

### Architecture of ML Pipelines

MLOps: 機械学習における継続的デリバリーと自動化のパイプライン - Google Cloud https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
The TFX User Guide - TensorFlow https://www.tensorflow.org/tfx/guide
The ExampleGen TFX Pipeline Component - TensorFlow https://www.tensorflow.org/tfx/guide/examplegen
Akshay Naresh Modi and Chiu Yuen Koo and Chuan Yu Foo and Clemens Mewald and Denis M. Baylor and Eric Breck and Heng-Tze Cheng and Jarek Wilkiewicz and Levent Koc and Lukasz Lew and Martin A. Zinkevich and Martin Wicke and Mustafa Ispir and Neoklis Polyzotis and Noah Fiedel and Salem Elie Haykal and Steven Whang and Sudip Roy and Sukriti Ramesh and Vihan Jain and Xin Zhang and Zakaria Haque TFX: A TensorFlow-Based Production-Scale Machine Learning Platform, KDD 2017 (2017) https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
Data preprocessing for machine learning: options and recommendations - https://cloud.google.com/architecture/data-preprocessing-for-ml-with-tf-transform-pt1
Basic text classification - TensorFlow Core https://www.tensorflow.org/tutorials/keras/text_classification
The Winding Road to Better Machine Learning Infrastructure Through Tensorflow Extended and Kubeflow - Spotify Engineering https://engineering.atspotify.com/2019/12/13/the-winding-road-to-better-machine-learning-infrastructure-through-tensorflow-extended-and-kubeflow/
ML Metadata - TensorFlow https://www.tensorflow.org/tfx/guide/mlmd
Building Standard TensorFlow ModelServer - TFX https://www.tensorflow.org/tfx/serving/serving_advanced

### How to Design Your ML Pipelines

機械学習基盤 Hekatoncheir の取り組み【DeNA TechCon 2021】/techcon2021-12 https://speakerdeck.com/dena_tech/techcon2021-12

Asei Sugiyama

June 04, 2021

More Decks by Asei Sugiyama

Other Decks in Technology


  1. None
  2. None
  3. None
  4. None
  5. None
  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. None
  15. None
  16. None
  17. None
  18. None
  19. None
  20. None
  21. None
  22. None
  23. None
  24. None
  25. None
  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. None
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None
  40. None
  41. None
  42. None