Upgrade to Pro — share decks privately, control downloads, hide ads and more …

機械学習プロジェクトを頑健にする施策 ML Ops Study #2

機械学習プロジェクトを頑健にする施策 ML Ops Study #2

Takahiko Ito

May 29, 2018
Tweet

More Decks by Takahiko Ito

Other Decks in Programming

Transcript

  1. ࠓճѻ͏ൣғ ຊൃදͰѻ͏τϐοΫ ୳ࡧతͳ࣮ݧ ίʔυ੔ཧ Ϟσϧͷ σϓϩΠ 5 ϥΠϒϥϦԽ ϦϑΝΫλϦϯά ςετɺLinter

    CI όονεΫϦϓτɺ αʔϏεԽɺ CD ࣮ݧˠίʔυ੔ཧ͔ΒϓϩδΣΫτͷؤ݈ԽΛ ҙࣝ͢Δ Jupyter Notebook
  2. ໰୊ɿલͷεςʔδͰಈ͍ͯ ͍࣮ͨݧ͕ಈ࡞͠ͳ͍ 17 ࣮ݧ ςετɺlintɺϦϑΝΫλ ϦϯάɺϥΠϒϥϦԽɺ CI όονεΫϦϓτɺ CDɺαʔϏε Ϟσϧվྑ

    kubernetes ECS ίʔυ੔ཧ σϓϩΠ εςʔδ͝ͱʹಈ࡞؀ڥΛ ࡞Δίετ͕େ͖͍ɻ →ϞσϧͷվྑαΠΫϧ͕ճΒͳ͍(TдT)
  3. DockerͰ؀ڥΛԾ૝Խ 19 ࣮ݧ ςετɺlintɺϦϑΝΫλ ϦϯάɺϥΠϒϥϦԽɺ CI όονεΫϦϓτɺ CDɺαʔϏε Ϟσϧվྑ kubernetes

    ECS ίʔυ੔ཧ σϓϩΠ ࣮ݧஈ֊͔ΒҰ؏ͯ͠Dockerίϯς φ্Ͱ࡞ۀɻಈ࡞͠ͳ͍εςʔδ͕ ग़ͳ͍Α͏ʹ
  4. Docker ίϚϯυ • Docker Πϝʔδͷ࡞੒ • docker build -t ml-image

    -f ./docker/Dockerfile . • Dockerίϯςφͷ࡞੒ • docker run -it -v `pwd`:/work -p 8888:8888 — name ml-image ml-container • ͞Βʹɺ࡟আɺ࠶ੜੑ etc … 21
  5. ػೳɿCookicutter Docker Science • ΤϯδχΞϦϯάೳྗͷߴ͘ͳ͍ϝϯόͰ΋DockerΛѻ͍΍͘͢ • DockerͷίϚϯυΛ make λʔήοτͰӅṭ •

    ϙʔτϑΥϫʔυɺϑΝΠϧϚ΢ϯτઃఆɺίϯςφ࡞Γ௚͠ etc … • ࣮ݧ͔Β੔ཧɺσϓϩΠ·ͰΛҙࣝͨ͠σΟϨΫτϦߏ੒Λग़ྗ • σΟϨΫτϦߏ੒ͷڞ௨ԽʹΑΓϓϩδΣΫτͷݟ௨͠ • Cookiecutter Data Science ͷߏ੒Λࢀߟʹͨ͠ 24
  6. ϑΝΠϧɺσΟϨΫτϦߏ੒ ͷ౷Ұ 25 make init Ͱ S3͔ΒσʔλΛμ ΢ϯϩʔυ ֶशεΫ Ϧϓτ͕ओྗ͢ΔϞσ

    ϧΛอ࣋ ࣮ݧ༻ͷϊʔτϒο ΫΛอ࣋ ίʔυ੔ཧ࣌ʹ࡞ ΒΕΔϝιουɺΫϥε Λอ࣋ ϓϩδΣΫτͷϫʔ ΫϑϩʔΛ௥ه
  7. Cookiecutter Docker Science ͷ ࢖͍ํʢϓϩδΣΫτੜ੒ʣ $cookiecutter [email protected]:docker-science/cookiecutter-docker-science.git project_name [project_name]: image-classification

    project_slug [image_classification]: jupyter_host_port [8888]: description [Please Input a short description]: Classify images into several categories data_source [Please Input data source in S3]: s3://research-data/food-images 26
  8. Demo: Cookiecutter Docker Science • ϓϩδΣΫτͷੜ੒ • https://asciinema.org/a/ 6XV9dNixtzfUwWdoqLj7HG7 A2

    • Docker image / container ίϯς φ࡞੒ • https://asciinema.org/a/ 06CcXPubAj3RSiMSTy3CZDrfG • Jupyter Notebook Λ্ཱͪ͛Δ 27
  9. Cookiecutter Docker Science Λར༻ ࣮ͯ͠ݧஈ֊͔ΒԾ૝Խ؀ڥͰ࡞ۀ 28 ࣮ݧ ςετɺlintɺϦϑΝΫλ ϦϯάɺϥΠϒϥϦԽɺ CI

    όονεΫϦϓτɺ CDɺαʔϏε Ϟσϧվྑ kubernetes ECS ίʔυ੔ཧ σϓϩΠ ͢΂ͯͷεςʔδͰԾ૝؀ڥ γʔϜϨεʹεςʔδΛҠಈͰ͖Δ
  10. ͦͷ΄͔ • linter ಋೖ • logger ಋೖ • CIಋೖ •

    υΩϡϝϯτʢSphinxʣ • ࣮ݧͨ͠಺༰ͳͲΛ·ͱΊΔ • etc … 40
  11. ঢ়گ෼ੳɿ෼ۀʹΑΔ໰୊ 48 ࣮ݧ ςετɺlintɺϦϑΝΫλ ϦϯάɺϥΠϒϥϦԽɺ CI όονεΫϦϓτɺ CDɺαʔϏε Ϟσϧվྑ kubernetes

    ECS ίʔυ੔ཧ σϓϩΠ ৽͍࣮͠ݧ݁Ռ͕ͰΔͨͼʹ ϦαʔνϟˠΤϯδχΞͷόέπ ϦϨʔ͕ൃੜ
  12. ঢ়گ෼ੳɿ෼ۀʹΑΔ໰୊ 49 ࣮ݧ ςετɺlintɺϦϑΝΫλ ϦϯάɺϥΠϒϥϦԽɺ CI όονεΫϦϓτɺ CDɺαʔϏε Ϟσϧվྑ kubernetes

    ECS ίʔυ੔ཧ σϓϩΠ ίʔυ੔ཧޙͷεΫϦϓτΛϦαʔ νϟ͕ཧղͰ͖ͳ͍ 㱺վྑαΠΫϧΛճͤͳ͍ɻɻɻ