$30 off During Our Annual Pro Sale. View Details »

Vacation Rentals of Hiroshima

hsekine
November 12, 2016

Vacation Rentals of Hiroshima

広島の民泊データを分析しよう!

hsekine

November 12, 2016
Tweet

More Decks by hsekine

Other Decks in Programming

Transcript

  1. ޿ౡͷຽധσʔλ
    Λ෼ੳ͠Α͏ʂ
    2016/11/12 ؔࠜ༟ل
    PyCon mini Hiroshima 2016

    View Slide

  2. ࣗݾ঺հ
    • ؔࠜ༟لʢ͖ͤͶ ͻΖͷΓʣ
    • גࣜձࣾSQUEEZE
    • Twitter: @checkpoint

    View Slide

  3. PythonͱͷؔΘΓʢ̍ʣ
    • PyCon JP 2014 ελοϑ
    • PyCon JP 2015 ෭࠲௕ʢϓϩάϥϜʣ
    • PyCon jp 2016 ελοϑ
    • Python΋͘΋͘ձʢओ࠵ʣ

    View Slide

  4. PythonͱͷؔΘΓʢ̎ʣ
    • LLDiver
    • PyCon JP 2014
    • Phone Symposium Tokyo 2015
    • PyCon mini Hiroshima 2015
    • PyCon mini Hiroshima 2016
    • PythonΤϯδχΞཆ੒ಡຊʢڞஶʣ

    View Slide

  5. ۀ຿ͰͷPython
    • ຽധ෺݅Λ؅ཧɺӡ༻͢ΔͨΊͷαʔϏε


    View Slide

  6. ۀ຿ͰͷPython
    • ຽധ෺݅Λ෼ੳ͢ΔͨΊͷαʔϏε



    View Slide

  7. ࢖༻ͯ͠Δٕज़

    View Slide

  8. ΞδΣϯμ
    • ޿ౡݝͷຽധʹ͍ͭͯ
    • PythonͰͷσʔλऩू
    • PythonͰͷσʔλ෼ੳ

    View Slide

  9. ຽധͱ͸
    ҰൠͷຽՈʹ॓ധ͢Δ͜ͱʢ༷ʑͳܗଶʣ

    View Slide

  10. ϓϥοτϑΥʔϜ
    COPYRIGHT (C) 2014-2016 SQUEEZE Inc.
    ALL RIGHTS RESERVED.


    View Slide

  11. ϓϥοτϑΥʔϜʢ೔ຊʣ

    View Slide

  12. ޿ౡݝͷຽധ
    • தࠃɾ࢛ࠃ஍ํͰҰ൪େ͖ͳ౎ࢢʢ޿ౡࢢʣ
    • ੈքҨ࢈ΛؚΉ๛͔ͳ؍ޫࢿݯ
    • ΦόϚถେ౷ྖͷ๚໰
    • ޿ౡΧʔϓͷηϦʔά༏উ
    • ຽധ΋੝Γ্͕͖͍ͬͯͯΔ͸ͣʂ

    View Slide

  13. ຽധσʔλͷ෼ੳ
    • σʔλͷऩू
    • σʔλͷ෼ੳ
    • σʔλͷදࣔ

    View Slide

  14. σʔλͷऩू
    • ΫϩʔϦϯά
    • εΫϨΠϐϯά
    • ౷ܭσʔλ
    • ૯຿ল౷ܭہ
    • σʔλΧλϩάαΠτ

    View Slide

  15. ΫϩʔϦϯά
    • ӳޠͷҙຯ͸ɺ[͸͏ɺΏͬ͘ΓਐΉ]
    • WebϖʔδͷϦϯΫͷ಺༰ΛͨͲΔ
    • Webϖʔδͷ಺༰Λμ΢ϯϩʔυͯ͠ऩू
    • Web APIͷσʔλΛऔΔ৔߹΋͋Δ

    View Slide

  16. εΫϨΠϐϯά
    • ӳޠͷҙຯ͸ɺ[ ࡟Δ͜ͱ ]
    • ϖʔδͷ಺༰͔Βඞཁͳ৘ใΛநग़

    View Slide

  17. όοςϦʔ෇ଐݴޠ
    ʴ
    ڧྗͳαʔυύʔςΟϥΠϒϥϦ

    View Slide

  18. ศརͳϥΠϒϥϦ
    • ඪ४ϥΠϒϥϦ
    • requests
    • BeautifulSoup
    • Scrapy
    • Selenium

    View Slide

  19. ඪ४ϥΠϒϥϦ
    • Pythonͷඪ४ϥΠϒϥϦ͸ͱͯ΋ॆ࣮
    • ωοτϫʔΫɺਖ਼نදݱɺetc
    • Pythonͷॲཧܥ͚ͩ͋Ε͹ྑ͍
    • ؆୯ͳεΫϨΠϐϯάͰ͋Ε͹े෼࣮༻త

    View Slide

  20. αϯϓϧ

    View Slide

  21. Requests
    • Python੡ͷHTTP Client
    • ਓؒʹ༏͍͠ΠϯλʔϑΣʔε
    • ͱʹ͔͘Θ͔Γ΍͍͢
    • γϯϓϧ͔ͭڧྗ

    View Slide

  22. ެࣜαΠταϯϓϧ

    View Slide

  23. αϯϓϧ(requests൛ʣ

    View Slide

  24. Beautiful Soup
    • 2004೥Ґ͔Βଘࡏ͢ΔϥΠϒϥϦ
    • HTML΍XML͔ΒσʔλΛநग़ͯ͠औಘ
    • ࠷৽όʔγϣϯ͸Beautiful Soup 4ܥ
    • Python 2.7ɺPython 3.2ʹରԠ

    View Slide

  25. αϯϓϧ

    View Slide

  26. Scrapy
    Scarpy͸଎ͯ͘ɺϋΠϨϕϧͳεΫϨΠϐϯά
    ΫϩʔϥʔͷϑϨʔϜϫʔΫɻWebαΠτͷΫ
    ϩʔϧͱɺߏ଄Խ͞ΕͨσʔλΛऔΓग़͢ͷʹ
    ࢖༻͢Δɻ෯޿͍໨తʹ࢖༻Ͱ͖ΔɻσʔλϚ
    Πχϯά͔ΒɺϞχλϦϯάɺࣗಈςετͳͲ

    View Slide

  27. Scrapyͷಛ௃
    • ΫϩʔϦϯάɺεΫϨΠϐϯάϑϨʔϜϫʔΫ
    • DjangoʹӨڹ͞Ε͍ͯΔʢMiddlewareͳͲʣ
    • εΫϨΠϐϯάʹඞཁͳػೳ͕ͦΖ͍ͬͯΔ
    • υΩϡϝϯτ͕ॆ࣮͍ͯ͠Δ

    View Slide

  28. Scrapyͷओͳػೳ
    • μ΢ϯϩʔυɺநग़ɺอଘ
    • μ΢ϯϩʔυͨ͠υΩϡϝϯτͷΩϟογϡ
    • ڧྗͳίϚϯυϥΠϯγΣϧ
    • Robots.txtͷύʔε
    • ඇಉظɺฒߦμ΢ϯϩʔυʢTwistedΛ࢖༻ʣ
    • υϝΠϯɺIPΞυϨε୯ҐͷΫϩʔϧִؒௐ੔
    • Τϥʔ࣌ͷϦτϥΠ
    • ϩάग़ྗ

    View Slide

  29. ։ൃखॱ
    • ScrapyϓϩδΣΫτͷ࡞੒
    • SpiderΛ࡞੒ʢϦϯΫநग़ɺμ΢ϯϩʔυʣ
    • ItemύΠϓϥΠϯͰσʔλΛอଘ

    View Slide

  30. ϓϩδΣΫτͷ࡞੒
    $ scrapy startproject scrapy_sample

    View Slide

  31. αϯϓϧ

    View Slide

  32. Spider࡞੒ʢެࣜαΠτΑΓʣ

    View Slide

  33. ࣮ߦ
    $ scrapy crawl dmoz_spider -o scraped_data.json

    View Slide

  34. ৄࡉ
    • Scrapyೖ໳ʢ̍ʣ
    • Scrapyೖ໳ʢ̎ʣ

    View Slide

  35. αϯϓϧʢ̍ʣ

    View Slide

  36. αϯϓϧʢ̎ʣ

    View Slide

  37. ࣮ࡍͷࣄྫͷ঺հ
    • ޿ౡݝͷຽധσʔλΛ෼ੳ
    • ෺݅৘ใ
    • Ձ֨৘ใ

    View Slide

  38. ෺݅৘ใʢ๭αΠτʣ

    View Slide

  39. ։ൃͷྲྀΕ
    • ෺݅ɺՁ֨৘ใऔಘ༻ͷεύΠμʔΛ࡞੒
    • εύΠμʔ͕Ұ࣌σʔλΛอଘʢJSONʣ
    • όονॲཧʹͯ෺݅ɺՁ֨ΛอଘʢΫϨϯδϯάʣ
    • ूܭόονʹͯσʔλΛ෼ੳͯ͠DBʹอଘ
    • ूܭσʔλΛදࣔ

    View Slide

  40. σϞ

    View Slide

  41. ෺݅਺
    0
    50
    100
    150
    200
    250
    300
    350
    400
    450
    500
    2016/2/15
    2016/2/22
    2016/2/29
    2016/3/7
    2016/3/14
    2016/3/21
    2016/3/28
    2016/4/4
    2016/4/11
    2016/4/18
    2016/4/25
    2016/5/2
    2016/5/9
    2016/5/16
    2016/5/23
    2016/5/30
    2016/6/6
    2016/6/13
    2016/6/20
    2016/6/27
    2016/7/4
    2016/7/11
    2016/7/18
    2016/7/25
    2016/8/1
    2016/8/8
    2016/8/15
    2016/8/22
    2016/8/29
    2016/9/5
    2016/9/12
    2016/9/19
    2016/9/26
    2016/10/3
    2016/10…
    2016/10…
    2016/10…
    2016/10…
    2016/11/7
    2016/11…
    2016/11…
    2016/11…
    2016/12/5
    2016/12…
    2016/12…
    2016/12…

    View Slide

  42. ෺݅਺
    • ෺݅਺ 461݅
    • 1೥Ͱ໿2ഒʢ240݅ => 461݅)
    • શࠃͰ10൪໨ʹଟ͍
    • ౦ژ౎, େࡕ෎, ژ౎෎, ԭೄ݅, ๺ւಓ, ෱Ԭ݅ɺ
    ਆಸ઒݅, ௕໺݅, Ѫ஌݅, ઍ༿ݝ, ޿ౡݝ

    View Slide

  43. ฏۉՁ֨
    0
    2000
    4000
    6000
    8000
    10000
    12000

    View Slide

  44. ฏۉՁ֨
    • ౙ৔͸ฏۉՁ͕֨௿͍ʢ5000ԁ୆ʣ
    • 8݄, 10݄, 11݄ͷि຤͕ߴ͍ʢ8000୆ʣ
    • ೥຤ɺ͓ਖ਼݄͕ϐʔΫʢ10000ԁ୆ʣ

    View Slide

  45. Քಇ཰
    0
    10
    20
    30
    40
    50
    60
    70
    80
    90
    100

    View Slide

  46. Քಇ཰
    • Նͷγʔζϯ͕ϐʔΫʢ80%ऑʣ
    • 10݄, 11݄ͷि຤͸ߴ͍ʢ70%Ҏ্ʣ
    • ౙͷγʔζϯ͸௿͍ʢ40%ҎԼʣ
    • 10/15ʢ౔), 10/29ʢ౔ʣ͕ߴ͔ͬͨ

    View Slide

  47. ·ͱΊ
    • PythonͰεΫϨΠϐϯάΛߦ͏৔߹ɺ৭ʑͳ
    Ξϓϩʔν͕͋Δɻ
    • Scrapy͸໘౗ͳॲཧΛߦͬͯ͘ΕΔͷͰΦε
    εϝ
    • ޿ౡͷຽധ͸͜Ε͔Β΋੝Γ্͕Δ͸ͣʂ

    View Slide

  48. ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View Slide