Vacation Rentals of Hiroshima

広島の民泊データを分析しよう！

Transcript

1. ޿ౡͷຽധσʔλ
   Λ෼ੳ͠Α͏ʂ
   2016/11/12 ؔࠜ༟ل
   PyCon mini Hiroshima 2016
   

   View Slide

2. ࣗݾ঺հ
   • ؔࠜ༟لʢ͖ͤͶ ͻΖͷΓʣ
   • גࣜձࣾSQUEEZE
   • Twitter: @checkpoint
   

   View Slide

3. PythonͱͷؔΘΓʢ̍ʣ
   • PyCon JP 2014 ελοϑ
   • PyCon JP 2015 ෭࠲௕ʢϓϩάϥϜʣ
   • PyCon jp 2016 ελοϑ
   • Python΋͘΋͘ձʢओ࠵ʣ
   

   View Slide

4. PythonͱͷؔΘΓʢ̎ʣ
   • LLDiver
   • PyCon JP 2014
   • Phone Symposium Tokyo 2015
   • PyCon mini Hiroshima 2015
   • PyCon mini Hiroshima 2016
   • PythonΤϯδχΞཆ੒ಡຊʢڞஶʣ
   

   View Slide

5. ۀ຿ͰͷPython
   • ຽധ෺݅Λ؅ཧɺӡ༻͢ΔͨΊͷαʔϏε
   
   
   

   View Slide

6. ۀ຿ͰͷPython
   • ຽധ෺݅Λ෼ੳ͢ΔͨΊͷαʔϏε
   
   
   
   

   View Slide

7. ࢖༻ͯ͠Δٕज़
   

   View Slide

8. ΞδΣϯμ
   • ޿ౡݝͷຽധʹ͍ͭͯ
   • PythonͰͷσʔλऩू
   • PythonͰͷσʔλ෼ੳ
   

   View Slide

9. ຽധͱ͸
   ҰൠͷຽՈʹ॓ധ͢Δ͜ͱʢ༷ʑͳܗଶʣ
   

   View Slide

10. ϓϥοτϑΥʔϜ
    COPYRIGHT (C) 2014-2016 SQUEEZE Inc.
    ALL RIGHTS RESERVED.
    
    
    

    View Slide

11. ϓϥοτϑΥʔϜʢ೔ຊʣ
    

    View Slide

12. ޿ౡݝͷຽധ
    • தࠃɾ࢛ࠃ஍ํͰҰ൪େ͖ͳ౎ࢢʢ޿ౡࢢʣ
    • ੈքҨ࢈ΛؚΉ๛͔ͳ؍ޫࢿݯ
    • ΦόϚถେ౷ྖͷ๚໰
    • ޿ౡΧʔϓͷηϦʔά༏উ
    • ຽധ΋੝Γ্͕͖͍ͬͯͯΔ͸ͣʂ
    

    View Slide

13. ຽധσʔλͷ෼ੳ
    • σʔλͷऩू
    • σʔλͷ෼ੳ
    • σʔλͷදࣔ
    

    View Slide

14. σʔλͷऩू
    • ΫϩʔϦϯά
    • εΫϨΠϐϯά
    • ౷ܭσʔλ
    • ૯຿ল౷ܭہ
    • σʔλΧλϩάαΠτ
    

    View Slide

15. ΫϩʔϦϯά
    • ӳޠͷҙຯ͸ɺ[͸͏ɺΏͬ͘ΓਐΉ]
    • WebϖʔδͷϦϯΫͷ಺༰ΛͨͲΔ
    • Webϖʔδͷ಺༰Λμ΢ϯϩʔυͯ͠ऩू
    • Web APIͷσʔλΛऔΔ৔߹΋͋Δ
    

    View Slide

16. εΫϨΠϐϯά
    • ӳޠͷҙຯ͸ɺ[ ࡟Δ͜ͱ ]
    • ϖʔδͷ಺༰͔Βඞཁͳ৘ใΛநग़
    

    View Slide

17. όοςϦʔ෇ଐݴޠ
    ʴ
    ڧྗͳαʔυύʔςΟϥΠϒϥϦ
    

    View Slide

18. ศརͳϥΠϒϥϦ
    • ඪ४ϥΠϒϥϦ
    • requests
    • BeautifulSoup
    • Scrapy
    • Selenium
    

    View Slide

19. ඪ४ϥΠϒϥϦ
    • Pythonͷඪ४ϥΠϒϥϦ͸ͱͯ΋ॆ࣮
    • ωοτϫʔΫɺਖ਼نදݱɺetc
    • Pythonͷॲཧܥ͚ͩ͋Ε͹ྑ͍
    • ؆୯ͳεΫϨΠϐϯάͰ͋Ε͹े෼࣮༻త
    

    View Slide

20. αϯϓϧ
    

    View Slide

21. Requests
    • Python੡ͷHTTP Client
    • ਓؒʹ༏͍͠ΠϯλʔϑΣʔε
    • ͱʹ͔͘Θ͔Γ΍͍͢
    • γϯϓϧ͔ͭڧྗ
    

    View Slide

22. ެࣜαΠταϯϓϧ
    

    View Slide

23. αϯϓϧ(requests൛ʣ
    

    View Slide

24. Beautiful Soup
    • 2004೥Ґ͔Βଘࡏ͢ΔϥΠϒϥϦ
    • HTML΍XML͔ΒσʔλΛநग़ͯ͠औಘ
    • ࠷৽όʔγϣϯ͸Beautiful Soup 4ܥ
    • Python 2.7ɺPython 3.2ʹରԠ
    

    View Slide

25. αϯϓϧ
    

    View Slide

26. Scrapy
    Scarpy͸଎ͯ͘ɺϋΠϨϕϧͳεΫϨΠϐϯά
    ΫϩʔϥʔͷϑϨʔϜϫʔΫɻWebαΠτͷΫ
    ϩʔϧͱɺߏ଄Խ͞ΕͨσʔλΛऔΓग़͢ͷʹ
    ࢖༻͢Δɻ෯޿͍໨తʹ࢖༻Ͱ͖ΔɻσʔλϚ
    Πχϯά͔ΒɺϞχλϦϯάɺࣗಈςετͳͲ
    

    View Slide

27. Scrapyͷಛ௃
    • ΫϩʔϦϯάɺεΫϨΠϐϯάϑϨʔϜϫʔΫ
    • DjangoʹӨڹ͞Ε͍ͯΔʢMiddlewareͳͲʣ
    • εΫϨΠϐϯάʹඞཁͳػೳ͕ͦΖ͍ͬͯΔ
    • υΩϡϝϯτ͕ॆ࣮͍ͯ͠Δ
    

    View Slide

28. Scrapyͷओͳػೳ
    • μ΢ϯϩʔυɺநग़ɺอଘ
    • μ΢ϯϩʔυͨ͠υΩϡϝϯτͷΩϟογϡ
    • ڧྗͳίϚϯυϥΠϯγΣϧ
    • Robots.txtͷύʔε
    • ඇಉظɺฒߦμ΢ϯϩʔυʢTwistedΛ࢖༻ʣ
    • υϝΠϯɺIPΞυϨε୯ҐͷΫϩʔϧִؒௐ੔
    • Τϥʔ࣌ͷϦτϥΠ
    • ϩάग़ྗ
    

    View Slide

29. ։ൃखॱ
    • ScrapyϓϩδΣΫτͷ࡞੒
    • SpiderΛ࡞੒ʢϦϯΫநग़ɺμ΢ϯϩʔυʣ
    • ItemύΠϓϥΠϯͰσʔλΛอଘ
    

    View Slide

30. ϓϩδΣΫτͷ࡞੒
    $ scrapy startproject scrapy_sample
    

    View Slide

31. αϯϓϧ
    

    View Slide

32. Spider࡞੒ʢެࣜαΠτΑΓʣ
    

    View Slide

33. ࣮ߦ
    $ scrapy crawl dmoz_spider -o scraped_data.json
    

    View Slide

34. ৄࡉ
    • Scrapyೖ໳ʢ̍ʣ
    • Scrapyೖ໳ʢ̎ʣ
    

    View Slide

35. αϯϓϧʢ̍ʣ
    

    View Slide

36. αϯϓϧʢ̎ʣ
    

    View Slide

37. ࣮ࡍͷࣄྫͷ঺հ
    • ޿ౡݝͷຽധσʔλΛ෼ੳ
    • ෺݅৘ใ
    • Ձ֨৘ใ
    

    View Slide

38. ෺݅৘ใʢ๭αΠτʣ
    

    View Slide

39. ։ൃͷྲྀΕ
    • ෺݅ɺՁ֨৘ใऔಘ༻ͷεύΠμʔΛ࡞੒
    • εύΠμʔ͕Ұ࣌σʔλΛอଘʢJSONʣ
    • όονॲཧʹͯ෺݅ɺՁ֨ΛอଘʢΫϨϯδϯάʣ
    • ूܭόονʹͯσʔλΛ෼ੳͯ͠DBʹอଘ
    • ूܭσʔλΛදࣔ
    

    View Slide

40. σϞ
    

    View Slide

41. ෺݅਺
    0
    50
    100
    150
    200
    250
    300
    350
    400
    450
    500
    2016/2/15
    2016/2/22
    2016/2/29
    2016/3/7
    2016/3/14
    2016/3/21
    2016/3/28
    2016/4/4
    2016/4/11
    2016/4/18
    2016/4/25
    2016/5/2
    2016/5/9
    2016/5/16
    2016/5/23
    2016/5/30
    2016/6/6
    2016/6/13
    2016/6/20
    2016/6/27
    2016/7/4
    2016/7/11
    2016/7/18
    2016/7/25
    2016/8/1
    2016/8/8
    2016/8/15
    2016/8/22
    2016/8/29
    2016/9/5
    2016/9/12
    2016/9/19
    2016/9/26
    2016/10/3
    2016/10…
    2016/10…
    2016/10…
    2016/10…
    2016/11/7
    2016/11…
    2016/11…
    2016/11…
    2016/12/5
    2016/12…
    2016/12…
    2016/12…
    
    

    View Slide

42. ෺݅਺
    • ෺݅਺ 461݅
    • 1೥Ͱ໿2ഒʢ240݅ => 461݅)
    • શࠃͰ10൪໨ʹଟ͍
    • ౦ژ౎, େࡕ෎, ژ౎෎, ԭೄ݅, ๺ւಓ, ෱Ԭ݅ɺ
    ਆಸ઒݅, ௕໺݅, Ѫ஌݅, ઍ༿ݝ, ޿ౡݝ
    

    View Slide

43. ฏۉՁ֨
    0
    2000
    4000
    6000
    8000
    10000
    12000
    
    

    View Slide

44. ฏۉՁ֨
    • ౙ৔͸ฏۉՁ͕֨௿͍ʢ5000ԁ୆ʣ
    • 8݄, 10݄, 11݄ͷि຤͕ߴ͍ʢ8000୆ʣ
    • ೥຤ɺ͓ਖ਼݄͕ϐʔΫʢ10000ԁ୆ʣ
    

    View Slide

45. Քಇ཰
    0
    10
    20
    30
    40
    50
    60
    70
    80
    90
    100
    
    

    View Slide

46. Քಇ཰
    • Նͷγʔζϯ͕ϐʔΫʢ80%ऑʣ
    • 10݄, 11݄ͷि຤͸ߴ͍ʢ70%Ҏ্ʣ
    • ౙͷγʔζϯ͸௿͍ʢ40%ҎԼʣ
    • 10/15ʢ౔), 10/29ʢ౔ʣ͕ߴ͔ͬͨ
    

    View Slide

47. ·ͱΊ
    • PythonͰεΫϨΠϐϯάΛߦ͏৔߹ɺ৭ʑͳ
    Ξϓϩʔν͕͋Δɻ
    • Scrapy͸໘౗ͳॲཧΛߦͬͯ͘ΕΔͷͰΦε
    εϝ
    • ޿ౡͷຽധ͸͜Ε͔Β΋੝Γ্͕Δ͸ͣʂ
    

    View Slide

48. ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide