機械学習プロジェクトを頑健にする施策 ML Ops Study #2

機械学習プロジェクトを頑健にする施策 ML Ops Study #2

70c7641a3ae1099ab76137b6ba09f6f5?s=128

Takahiko Ito

May 29, 2018
Tweet

Transcript

  1. 5.

    ࠓճѻ͏ൣғ ຊൃදͰѻ͏τϐοΫ ୳ࡧతͳ࣮ݧ ίʔυ੔ཧ Ϟσϧͷ σϓϩΠ 5 ϥΠϒϥϦԽ ϦϑΝΫλϦϯά ςετɺLinter

    CI όονεΫϦϓτɺ αʔϏεԽɺ CD ࣮ݧˠίʔυ੔ཧ͔ΒϓϩδΣΫτͷؤ݈ԽΛ ҙࣝ͢Δ Jupyter Notebook
  2. 17.

    ໰୊ɿલͷεςʔδͰಈ͍ͯ ͍࣮ͨݧ͕ಈ࡞͠ͳ͍ 17 ࣮ݧ ςετɺlintɺϦϑΝΫλ ϦϯάɺϥΠϒϥϦԽɺ CI όονεΫϦϓτɺ CDɺαʔϏε Ϟσϧվྑ

    kubernetes ECS ίʔυ੔ཧ σϓϩΠ εςʔδ͝ͱʹಈ࡞؀ڥΛ ࡞Δίετ͕େ͖͍ɻ →ϞσϧͷվྑαΠΫϧ͕ճΒͳ͍(TдT)
  3. 19.

    DockerͰ؀ڥΛԾ૝Խ 19 ࣮ݧ ςετɺlintɺϦϑΝΫλ ϦϯάɺϥΠϒϥϦԽɺ CI όονεΫϦϓτɺ CDɺαʔϏε Ϟσϧվྑ kubernetes

    ECS ίʔυ੔ཧ σϓϩΠ ࣮ݧஈ֊͔ΒҰ؏ͯ͠Dockerίϯς φ্Ͱ࡞ۀɻಈ࡞͠ͳ͍εςʔδ͕ ग़ͳ͍Α͏ʹ
  4. 21.

    Docker ίϚϯυ • Docker Πϝʔδͷ࡞੒ • docker build -t ml-image

    -f ./docker/Dockerfile . • Dockerίϯςφͷ࡞੒ • docker run -it -v `pwd`:/work -p 8888:8888 — name ml-image ml-container • ͞Βʹɺ࡟আɺ࠶ੜੑ etc … 21
  5. 24.

    ػೳɿCookicutter Docker Science • ΤϯδχΞϦϯάೳྗͷߴ͘ͳ͍ϝϯόͰ΋DockerΛѻ͍΍͘͢ • DockerͷίϚϯυΛ make λʔήοτͰӅṭ •

    ϙʔτϑΥϫʔυɺϑΝΠϧϚ΢ϯτઃఆɺίϯςφ࡞Γ௚͠ etc … • ࣮ݧ͔Β੔ཧɺσϓϩΠ·ͰΛҙࣝͨ͠σΟϨΫτϦߏ੒Λग़ྗ • σΟϨΫτϦߏ੒ͷڞ௨ԽʹΑΓϓϩδΣΫτͷݟ௨͠ • Cookiecutter Data Science ͷߏ੒Λࢀߟʹͨ͠ 24
  6. 25.

    ϑΝΠϧɺσΟϨΫτϦߏ੒ ͷ౷Ұ 25 make init Ͱ S3͔ΒσʔλΛμ ΢ϯϩʔυ ֶशεΫ Ϧϓτ͕ओྗ͢ΔϞσ

    ϧΛอ࣋ ࣮ݧ༻ͷϊʔτϒο ΫΛอ࣋ ίʔυ੔ཧ࣌ʹ࡞ ΒΕΔϝιουɺΫϥε Λอ࣋ ϓϩδΣΫτͷϫʔ ΫϑϩʔΛ௥ه
  7. 26.

    Cookiecutter Docker Science ͷ ࢖͍ํʢϓϩδΣΫτੜ੒ʣ $cookiecutter git@github.com:docker-science/cookiecutter-docker-science.git project_name [project_name]: image-classification

    project_slug [image_classification]: jupyter_host_port [8888]: description [Please Input a short description]: Classify images into several categories data_source [Please Input data source in S3]: s3://research-data/food-images 26
  8. 27.

    Demo: Cookiecutter Docker Science • ϓϩδΣΫτͷੜ੒ • https://asciinema.org/a/ 6XV9dNixtzfUwWdoqLj7HG7 A2

    • Docker image / container ίϯς φ࡞੒ • https://asciinema.org/a/ 06CcXPubAj3RSiMSTy3CZDrfG • Jupyter Notebook Λ্ཱͪ͛Δ 27
  9. 28.

    Cookiecutter Docker Science Λར༻ ࣮ͯ͠ݧஈ֊͔ΒԾ૝Խ؀ڥͰ࡞ۀ 28 ࣮ݧ ςετɺlintɺϦϑΝΫλ ϦϯάɺϥΠϒϥϦԽɺ CI

    όονεΫϦϓτɺ CDɺαʔϏε Ϟσϧվྑ kubernetes ECS ίʔυ੔ཧ σϓϩΠ ͢΂ͯͷεςʔδͰԾ૝؀ڥ γʔϜϨεʹεςʔδΛҠಈͰ͖Δ
  10. 31.
  11. 40.

    ͦͷ΄͔ • linter ಋೖ • logger ಋೖ • CIಋೖ •

    υΩϡϝϯτʢSphinxʣ • ࣮ݧͨ͠಺༰ͳͲΛ·ͱΊΔ • etc … 40
  12. 48.

    ঢ়گ෼ੳɿ෼ۀʹΑΔ໰୊ 48 ࣮ݧ ςετɺlintɺϦϑΝΫλ ϦϯάɺϥΠϒϥϦԽɺ CI όονεΫϦϓτɺ CDɺαʔϏε Ϟσϧվྑ kubernetes

    ECS ίʔυ੔ཧ σϓϩΠ ৽͍࣮͠ݧ݁Ռ͕ͰΔͨͼʹ ϦαʔνϟˠΤϯδχΞͷόέπ ϦϨʔ͕ൃੜ
  13. 49.

    ঢ়گ෼ੳɿ෼ۀʹΑΔ໰୊ 49 ࣮ݧ ςετɺlintɺϦϑΝΫλ ϦϯάɺϥΠϒϥϦԽɺ CI όονεΫϦϓτɺ CDɺαʔϏε Ϟσϧվྑ kubernetes

    ECS ίʔυ੔ཧ σϓϩΠ ίʔυ੔ཧޙͷεΫϦϓτΛϦαʔ νϟ͕ཧղͰ͖ͳ͍ 㱺վྑαΠΫϧΛճͤͳ͍ɻɻɻ