Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Pythonistaに憧れた分析屋の奮闘記

kanan
June 09, 2017

 Pythonistaに憧れた分析屋の奮闘記

kanan

June 09, 2017
Tweet

More Decks by kanan

Other Decks in Programming

Transcript

  1. PythonistaʹಌΕͨ෼ੳ԰ͷฃಆه
    @kanan* 2017.06.07
    ΈΜͳͷPythonษڧձ#25

    View full-size slide

  2. ࣗݾ঺հ
    ˎKANANʢ͔ͳΜʣ
    ɹɹcompass-ID @kanan*
    ɹɹTwitter-ID @Addition_quince
    ˎ͓࢓ࣄ
    ɹɹSIerاۀͰσʔλαΠΤϯεؔ࿈ۀ຿Λ΍ͬͯ·͢
    ɹɹݴޠ͸SAS
    ˎPyLadiesʹ͸2016.ळ͔Βͪΐͪ͜ΐ͜ࢀՃ
    ˎझຯ͸ԅɻ͓ञ͕େ޷͖ɻ

    View full-size slide

  3. ࠓ೔ͷ͓͸ͳ͠
    ˎϓϩάϥϛϯάʹڵຯ͕ͳ͔ͬͨ෼ੳ԰͕
    ˎεΩϧΞοϓͨͯ͘͠PythonʹखΛग़ͦ͏ͱܾҙ͠
    ˎ৺ંΕͳ͕Βฃಆͨ͠10ϲ݄ؒͷ͓࿩Ͱ͢
    PythonistaʹಌΕͨ෼ੳ԰ͷฃಆه
    Pythonista Lv0 → Lv1.5 ※͖ͬͱੌ͍ਓ͸Lv 20͘Β͍

    View full-size slide

  4. Pythonͱͷग़ձ͍
    σʔλ෼ੳܥɹˠɹPython
    Α͋͘Δ΍ͭ

    View full-size slide

  5. ͜Ε·Ͱͷϓϩάϥϛϯάྺ
    2013 2014 2015 2016
    2012
    2011
    େֶ γεςϜΤϯδχΞ σʔλαΠΤϯςΟετ
    ˎੲ͸ϓϩάϥϛϯάͯͨͭ͠΋Γͩͬͨ
    ˎ࢓ࣄͰϚωδϝϯτ͕ଟ͘ͳΓɺ΋͸΍৮Βͳ͍ۭനͷظؒ
    ˎSAS͸ݴޠͱ͍͍ͳ͕Β΋πʔϧͬΆ͍
    ϓϩάϥϛϯάɹʹɹ࢓ࣄ

    View full-size slide

  6. ͜Ε·Ͱͷϓϩάϥϛϯάྺ
    ͪΌΜͱϓϩάϥϛϯά΍ͬͯͳ͍

    View full-size slide

  7. PythonσϏϡʔΛܾҙ
    PyLevel 0
    2016೥9݄
    ˎҟ༷ͳ΄Ͳͷ΍Δؾ
    ˎ࢒Δ͸෼ੳ԰ͷ஌ࣝ

    View full-size slide

  8. ͦͷ̍िؒޙɾɾɾ
    PyLevel 0
    2016೥9݄ ͦͷ̎
    ˎΠϯετʔϧํ๏ݕࡧ͢Δͱ
    ɹɹ΍Γํ͕஍ຯʹҧ͏
    ˎHomebrewʁpyenvʁφχιϨ
    ˎPython3ΛೖΕͨͷʹ
    ɹɹόʔδϣϯ͕Python2ͬͯͳΔ
    ˎ.bash_plofileͳ͍͠ʂ
    ˎPython͸͡ΊΔͨΊʹങͬͨ
    ɹɹMACͷ࢖͍ํ͕Θ͔Βͳ͍
    Πϯετʔϧ
    Ͱ͖ͳ͍ʂ

    View full-size slide

  9. ͦΜͳ࣌ɺPyLadiesTokyoͱग़ձ͏

    View full-size slide

  10. PyLadies TokyoͰSTEP UP
    PyLadies Tokyo ळ߹॓ 2016 [2016.10.8-10]
    PyLadies Tokyo Meet Up #16 [2016.11.27]
    PyLadies Tokyo Meet Up #17 [2016.12.11]
    PyLadies Tokyo Meet Up #20 [2016.03.25]
    *؀ڥߏங,AnacondaͱJupyterNotebookͷ࢖͍ํΛֶͿ
    *ڝٕϓϩάϥϛϯάͰPythonͷॻ͖ํΛֶͿ
    *ϚΠίϯϘʔυ(STM32 Discovery)ʹMicro PythonͰLνΧ
    *WebεΫϨΠϐϯάΛ΍ͬͯΈΔ

    View full-size slide

  11. ݸਓతʹ΋σʔλ෼ੳपลͰษڧ
    ˎσʔλ෼ੳܥ
    ɹɹɹ1) titanicੜଘऀ༧ଌ

    ɹɹɹ2) ΞϝϦΧͷՈͷച٫Ձ֨༧ଌ

    ɹɹɹ3) ίϯϏχͷച্͛༧ଌ

    ˎͦͷଞ
    ɹɹɹ1) खॻ͖਺ࣈͷೝࣝ

    ɹɹɹ2) RaspberryPi3ͰPython࢖ͬͯΈΔ
    2017.3 ʙ2017.4

    View full-size slide

  12. PyLevel 1.0
    2017೥4݄
    ˎΞϧΰϦζϜָ͍͠
    ˎԿ͔Ͱ͖Δͱخ͍͠
    ˎExcel΍SASͰ
    ɹ΍ͬͯͨࣄ͕PythonͰ΋

    ˎ΋ͬͱ৭ʑ΍ͬͯΈ͍ͨ
    ΍Δؾ෮׆ɻ΍ͬͱೖ໳ऀϨϕϧʹ

    View full-size slide

  13. νϟϨϯδ
    ࣗ෼͕ڵຯ͕͋ΔςʔϚͰ
    ΍Γ͍ͨͱࢥͬͨ΋ͷΛ΍Ζ͏

    View full-size slide

  14. ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ʲͳΜͰ΢ΟεΩʔʁʳ
    ɹˎͱΓ͋͑ͣࢲ͕޷͖
    ɹˎϫΠϯΈ͍ͨʹ໏ฑ͕Ռͯ͠ͳ͘ଟ͍Θ͚͡Όͳ͍
    ɹˎւ֎ͷϏʔϧΈ͍ͨʹຯ΍߳Γͷಛ௃͕͸͖ͬΓͯ͠Δ
    Ϙτϧͷϥϕϧ͔Β໏ฑΛ൑ผ
    ը૾ೝࣝ
    WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ޷͖ͳ໏ฑ͔ΒͨͿΜ͓ޱʹ߹͏΋ͷΛਪન
    Ϩίϝϯυ

    View full-size slide

  15. Anaconda
    Continuum Analyticsࣾఏڙͷ
    σʔλ෼ੳͰΑ͘ར༻͞ΕΔϥΠϒϥϦ܊Λ
    ҰׅΠϯετʔϧͰ͖ͪΌ͏ύοέʔδ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    Jupyter Notebook
    ϒϥ΢βͰಈ࡞͢Δର࿩ܕ࣮ߦ؀ڥ
    ίʔυهड़ͱ࣮ߦɺίϝϯτૠೖ͕Ͱ͖ɺ
    ݁Ռͷอଘ΍ڞ༗ʹ΋ศར
    AnacondaೖΕΔͱσϑΥϧτͰೖͬͯΔ
    ɹPythonҎ֎ʹ΋R, node.js, Ruby౳ෳ਺ݴޠʹରԠ
    ʲ࣮ߦ؀ڥʳ

    View full-size slide

  16. ΢ΟεΩʔͰػցֶशʹ௅ઓ
    Ϙτϧͷϥϕϧ͔Β໏ฑΛ൑ผ
    ը૾ೝࣝ
    ̍ʣҰੜݒ໋σʔλΛूΊΔ

    ̎ʣskimageΛ࢖ͬͯHOGಛ௃ྔΛऔಘ͢Δ

    ̏ʣֶशσʔλͱςετσʔλʹ෼ׂ

    ̐ʣsklearnΛ࢖ͬͯCodeBookΛ࡞੒͠BoVWʹม׵

    ̑ʣֶशͱςετ
    ■࢖༻ͨ͠ϥΠϒϥϦ

    ɹɹskimage, matplotlib, sklearn, glob, os

    View full-size slide

  17. ΢ΟεΩʔͰػցֶशʹ௅ઓ
    Ϙτϧͷϥϕϧ͔Β໏ฑΛ൑ผ
    ը૾ೝࣝ
    ̍ʣҰੜݒ໋σʔλΛूΊΔ
    899ຕ
    ࣗ෼ͷ޷͖ͳ11໏ฑ͚ͩͰ৺ંΕͯఘΊΔ

    View full-size slide

  18. ΢ΟεΩʔͰػցֶशʹ௅ઓ
    Ϙτϧͷϥϕϧ͔Β໏ฑΛ൑ผ
    ը૾ೝࣝ
    ̎ʣskimageΛ࢖ͬͯHOGಛ௃ྔΛऔಘ͢Δ
    HOG(Histograms of Oriented Gradients)
    ɹہॴྖҬ (ηϧ) ͷً౓ͷޯ഑ํ޲ΛώετάϥϜԽ
    1.ը૾Λద౰ͳαΠζʹϦαΠζ͠ɺάϨΠεέʔϧͰಡΈࠐΉ
    2.֤pixelͷً౓͔Βޯ഑ڧ౓ͱޯ഑ํ޲ΛٻΊΔ
    3.ηϧྖҬ͝ͱʹώετάϥϜΛٻΊΔʢࠓճ͸8×8ϐΫηϧʣ
    4.ϒϩοΫ͝ͱʹਖ਼نԽ͠ɺಛ௃ྔΛநग़͢Δʢࠓճ͸3×3ηϧʣ

    View full-size slide

  19. ΢ΟεΩʔͰػցֶशʹ௅ઓ
    Ϙτϧͷϥϕϧ͔Β໏ฑΛ൑ผ
    ը૾ೝࣝ
    ̎ʣskimageΛ࢖ͬͯHOGಛ௃ྔΛऔಘ͢Δ

    View full-size slide

  20. ΢ΟεΩʔͰػցֶशʹ௅ઓ
    Ϙτϧͷϥϕϧ͔Β໏ฑΛ൑ผ
    ը૾ೝࣝ
    ̐ʣsklearnΛ࢖ͬͯCodeBookΛ࡞੒͠BoVWʹม׵
    BoVW(Bag-of-Visual-Words)ܗࣜ
    ɹը૾ͷಛ௃ྔΛϕΫτϧԽ͠ώετάϥϜʹͨ͠΋ͷ
    ɾɾɾɾ
    ɾɾɾ
    ɾɾɾ
    Visual-word vectors
    Codebook

    (୅දύλʔϯͷϦετʣ
    ç
    ç
    ç
    ç

    View full-size slide

  21. ΢ΟεΩʔͰػցֶशʹ௅ઓ
    Ϙτϧͷϥϕϧ͔Β໏ฑΛ൑ผ
    ը૾ೝࣝ
    ̑ʣֶशͱςετ
    ը૾σʔλͱਖ਼ղͷϥϕϧΛֶशͤ͞Δɹ※SVMʢαϙʔτϕΫλϚγϯʣΛ࠾༻
    άϦουαʔνͰϋΠύʔύϥϝʔλνϡʔχϯά

    View full-size slide

  22. ΢ΟεΩʔͰػցֶशʹ௅ઓ
    Ϙτϧͷϥϕϧ͔Β໏ฑΛ൑ผ
    ը૾ೝࣝ
    ਖ਼౴཰
    ɹ72%
    ςετ݁Ռ

    View full-size slide

  23. ΢ΟεΩʔͰػցֶशʹ௅ઓ
    Ϙτϧͷϥϕϧ͔Β໏ฑΛ൑ผ
    ը૾ೝࣝ
    ΍ͬͯΈͨ݁Ռ
    ■ը૾ͷલॲཧ෦෼

    ɹɹɾࠓճ͸ϥϕϧΞοϓͷը૾͹͔ΓΛ࢖༻ͨ͠ͷͰɺ

    ɹɹɹҾ͖ͷࣸਅʹରԠͰ͖ͳ͍ʢ΍ͬͯΈͨΒյ໓తʣ

    ɹɹɾഎܠͱ͔΋ະߟྀͷ··

    ɹɹɾOpenCVʁͳʹͦΕ

    ■ֶश෦෼

    ɹɹɾಛ௃ྔϕʔε(BoVW)ͰͷֶशΛ࠾༻͚ͨ͠Ͳɺ

    ɹɹɹσΟʔϓϥʔχϯά࢖͏ͱ΋ͬͱਫ਼౓͕͋Δͷ͔ͳ

    View full-size slide

  24. WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ̍ʣWebαΠτ͔Β֤໏ฑͷઆ໌΍ίϝϯτΛεΫϨΠϐϯά
    ̎ʣmecabͰ୯ޠʹ෼ׂ͠ɺܗ༰ࢺͱ෭ࢺ͚ͩऔΓग़͢
    ̏ʣग़ݱͨ͠୯ޠΛ·ͱΊɺTFIDFܭࢉͰಛ௃తͳޠΛಛఆ
    ̐ʣ୯ޠΛ࢖ͬͯࣅ͍ͯΔ໏ฑΛΫϥελϦϯά

    View full-size slide

  25. WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ̍ʣWebαΠτ͔Β֤໏ฑͷઆ໌΍ίϝϯτΛεΫϨΠϐϯά
    beautiful soupΛ࢖༻
    ɹɹᶃHTMLऔಘ
    ɹɹᶄ΄͍͠৘ใͷ෦෼ͷλάΛݟ͚ͭΔ
    ɹɹᶅߏ଄ղੳͯ͠ಛఆ෦෼Λநग़
    ɹɹᶆσʔλϑϨʔϜʹม׵͢Δ

    View full-size slide

  26. WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ̍ʣWebαΠτ͔Β֤໏ฑͷઆ໌΍ίϝϯτΛεΫϨΠϐϯά
    beautiful soupΛ࢖༻
    ɹɹᶃHTMLऔಘ
    ɹɹᶄ΄͍͠৘ใͷ෦෼ͷλάΛݟ͚ͭΔ
    ɹɹᶅߏ଄ղੳͯ͠ಛఆ෦෼Λநग़
    ɹɹᶆσʔλϑϨʔϜʹม׵͢Δ
    ্ख͍͔ͣ͘

    ࣌ؒΛ࿘අ

    View full-size slide

  27. WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ̍ʣWebαΠτ͔Β֤໏ฑͷઆ໌΍ίϝϯτΛεΫϨΠϐϯά

    View full-size slide

  28. WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ̍ʣWebαΠτ͔Β֤໏ฑͷઆ໌΍ίϝϯτΛεΫϨΠϐϯά

    View full-size slide

  29. WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ̍ʣWebαΠτ͔Β֤໏ฑͷઆ໌΍ίϝϯτΛεΫϨΠϐϯά
    ໏ฑͷղઆจ
    ޱίϛจ
    ͳΜͱ͔εΫϨΠϐϯά׬ྃ
    ख࡞ۀͰCSV࡞੒

    View full-size slide

  30. WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ̎ʣmecabͰ୯ޠʹ෼ׂ͠ɺܗ༰ࢺͱ෭ࢺ͚ͩऔΓग़͢
    ߳Γ͸ڧ͍Ͱ͕͢ɺຯ͸͖ͬ͢Γ͍ͯͯ͠ɺඒຯ͍͠Ͱ͢ɻ
    ߳Γ/͸/ڧ͍/Ͱ͢/͕/ɺ/ຯ/͸/͖ͬ͢Γ/ͯ͠/͍ͯ/ɺ

    /ඒຯ͍͠/Ͱ͢/ɻ

    View full-size slide

  31. WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ̏ʣग़ݱͨ͠୯ޠΛ·ͱΊɺTFIDFܭࢉͰಛ௃తͳޠΛಛఆ
    BoW(Bag-of-Words)
    ܗࣜʹม׵
    จॻ
    ߦྻԽ
    TFIDF
    ܭࢉ

    View full-size slide

  32. WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ̏ʣग़ݱͨ͠ܗ༰ࢺΛ·ͱΊɺTFIDFܭࢉͰಛ௃తͳޠΛಛఆ
    TFIDF=TF×IDF TF→Ұ࿈ͷจॻ಺ͷͦͷ୯ޠͷग़ݱස౓
    IDF→ʢͦͷ୯ޠ͕͋Δจॻͷ਺ʗશจॻ਺ʣͷٯ਺
    ʲͦͷจॻΒ͠͞ʳ
    ɹͦͷจॻͰଟ͘Ͱͯ͘ΔͷʹɺଞͷจॻͰ͸͋·ΓͰͯ͜ͳ͍୯ޠ
    BoW(Bag-of-Words)ܗࣜʹม׵ จॻߦྻԽ TFIDFܭࢉ
    BoWܗࣜ=จॻΛ਺஋ͱͯ͠ѻ͏ʢϕΫτϧԽ͢Δʣ
    ※ࠓճ͸TFʢ୯ޠͷग़ݱස౓ʣΛ࢖༻

    View full-size slide

  33. WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ̐ʣ୯ޠΛ࢖ͬͯࣅ͍ͯΔ໏ฑΛΫϥελϦϯά
    ֤໏ฑͷಛ௃తͳ୯ޠTOP5Λ࢖ͬͯ5ͭʹΫϥελϦϯά

    …͠Α͏ͱࢥͬͨΒ
    ˎ൱ఆʮͳ͍ʯͷߟྀ

    ˎআ֎ͨ͠΄͕͍͍୯ޠ

    ɹɹɹɹɹɹɹɹɹ౳

    ߟྀෆ଍Ͱɺ

    ੾ͳ͍ΫϥελϦϯάʹ

    View full-size slide

  34. ޷͖ͳ໏ฑ͔ΒͨͿΜ͓ޱʹ߹͏΋ͷΛਪન
    Ϩίϝϯυ
    ΢ΟεΩʔͰػցֶशʹ௅ઓ
    ຊ೔·Ͱʹ౸ୡͰ͖ͣ…ɻ
    ڧௐϑΟϧλϦϯάͱ͔Ͱ΍Γ͔͚ͨͬͨͲɺ
    ·ͣσʔλΛूΊΔͱ͍͏࠷େͷน͕଴ͬͯΔɻ

    View full-size slide

  35. ΢ΟεΩʔͰػցֶशʹ௅ઓͷཱྀ͸ଓ͘…
    Ϙτϧͷϥϕϧ͔Β໏ฑΛ൑ผ
    ը૾ೝࣝ
    WEBͷ৘ใ͔Β֤໏ฑͷಛ௃Λநग़
    ςΩετղੳ
    ޷͖ͳ໏ฑ͔ΒͨͿΜ͓ޱʹ߹͏΋ͷΛਪન
    Ϩίϝϯυ
    ˎਫ਼౓޲্ʂ90%͸ਖ਼ղ͍ͨ͠ɻOpenCV࢖ͬͯΈ͍ͨ
    ˎσʔλ૿ྔͱ൱ఆޠ΍আ֎બఆɺϊΠζͷѻ͍ͷߟྀ
    ˎ͜Ε͔ΒؤுΔʂϢʔβෳ਺ධՁͰڧௐϑΟϧλϦϯά
    WebΞϓϦέʔγϣϯ

    View full-size slide

  36. ·ͱΊ
    ˎPytonistaʹಌΕͨҰհͷ෼ੳ԰͸ͨͿΜPyLv1.5͘Β͍ʹͳΕͨ
    ˎ࠷ॳͷӈ΋ࠨ΋Θ͔Βͳ͍࣌͸ɺ஥ؒͱҰॹ͕͍͍
    ˎࣗ෼ͷڵຯͱֻ͚߹ΘͤΔͷͬͯؤுΓ͕࣋ଓ͢Δ
    ˎσʔλूΊΔͷେมͬͯᷚ͸ຊ౰ͩͬͨ
    ˎ·͡ͰwebΞϓϦέʔγϣϯΘ͔Γ·ͤΜ
    ɹಘҙͳਓɺͪΐͬͱ஌ͬͯΔਓɺ͓༑ୡʹͳ͍ͬͯͩ͘͞

    View full-size slide

  37. ͋Γ͕ͱ͏͍͟͝·ͨ͠ɻ

    View full-size slide