Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Vacation Rentals of Hiroshima
hsekine
November 12, 2016
Programming
1
660
Vacation Rentals of Hiroshima
広島の民泊データを分析しよう!
hsekine
November 12, 2016
Tweet
Share
More Decks by hsekine
See All by hsekine
hsekine
1
52
hsekine
0
190
hsekine
1
1.2k
hsekine
1
110
hsekine
0
2k
hsekine
0
140
hsekine
0
37
hsekine
0
19
hsekine
0
32
Other Decks in Programming
See All in Programming
nrslib
20
13k
momofff
0
160
e10dokup
0
440
dictoss
0
160
osyo
1
360
blendthink
0
400
anchorcable
1
120
thatjeffsmith
0
370
makicamel
1
170
pdone
0
210
kazuki19992
0
440
junmikai
0
290
Featured
See All Featured
dotmariusz
94
5.1k
kneath
294
39k
erikaheidi
13
4.2k
danielanewman
1
470
chriscoyier
499
130k
frogandcode
127
20k
caitiem20
308
17k
michaelherold
225
8.5k
sferik
609
54k
pedronauck
652
110k
philnash
8
490
cherdarchuk
71
260k
Transcript
ౡͷຽധσʔλ Λੳ͠Α͏ʂ 2016/11/12 ؔࠜ༟ل PyCon mini Hiroshima 2016
ࣗݾհ • ؔࠜ༟لʢ͖ͤͶ ͻΖͷΓʣ • גࣜձࣾSQUEEZE • Twitter: @checkpoint
PythonͱͷؔΘΓʢ̍ʣ • PyCon JP 2014 ελοϑ • PyCon JP 2015
෭࠲ʢϓϩάϥϜʣ • PyCon jp 2016 ελοϑ • Python͘͘ձʢओ࠵ʣ
PythonͱͷؔΘΓʢ̎ʣ • LLDiver • PyCon JP 2014 • Phone Symposium
Tokyo 2015 • PyCon mini Hiroshima 2015 • PyCon mini Hiroshima 2016 • PythonΤϯδχΞཆಡຊʢڞஶʣ
ۀͰͷPython • ຽധ݅Λཧɺӡ༻͢ΔͨΊͷαʔϏε
ۀͰͷPython • ຽധ݅Λੳ͢ΔͨΊͷαʔϏε
༻ͯ͠Δٕज़
ΞδΣϯμ • ౡݝͷຽധʹ͍ͭͯ • PythonͰͷσʔλऩू • PythonͰͷσʔλੳ
ຽധͱ ҰൠͷຽՈʹ॓ധ͢Δ͜ͱʢ༷ʑͳܗଶʣ
ϓϥοτϑΥʔϜ COPYRIGHT (C) 2014-2016 SQUEEZE Inc. ALL RIGHTS RESERVED.
ϓϥοτϑΥʔϜʢຊʣ
ౡݝͷຽധ • தࠃɾ࢛ࠃํͰҰ൪େ͖ͳࢢʢౡࢢʣ • ੈքҨ࢈ΛؚΉ๛͔ͳ؍ޫࢿݯ • ΦόϚถେ౷ྖͷ๚ • ౡΧʔϓͷηϦʔά༏উ •
ຽധΓ্͕͖͍ͬͯͯΔͣʂ
ຽധσʔλͷੳ • σʔλͷऩू • σʔλͷੳ • σʔλͷදࣔ
σʔλͷऩू • ΫϩʔϦϯά • εΫϨΠϐϯά • ౷ܭσʔλ • ૯ল౷ܭہ •
σʔλΧλϩάαΠτ
ΫϩʔϦϯά • ӳޠͷҙຯɺ[͏ɺΏͬ͘ΓਐΉ] • WebϖʔδͷϦϯΫͷ༰ΛͨͲΔ • Webϖʔδͷ༰Λμϯϩʔυͯ͠ऩू • Web APIͷσʔλΛऔΔ߹͋Δ
εΫϨΠϐϯά • ӳޠͷҙຯɺ[ Δ͜ͱ ] • ϖʔδͷ༰͔ΒඞཁͳใΛநग़
όοςϦʔଐݴޠ ʴ ڧྗͳαʔυύʔςΟϥΠϒϥϦ
ศརͳϥΠϒϥϦ • ඪ४ϥΠϒϥϦ • requests • BeautifulSoup • Scrapy •
Selenium
ඪ४ϥΠϒϥϦ • Pythonͷඪ४ϥΠϒϥϦͱͯॆ࣮ • ωοτϫʔΫɺਖ਼نදݱɺetc • Pythonͷॲཧܥ͚ͩ͋Εྑ͍ • ؆୯ͳεΫϨΠϐϯάͰ͋Εे࣮༻త
αϯϓϧ
Requests • PythonͷHTTP Client • ਓؒʹ༏͍͠ΠϯλʔϑΣʔε • ͱʹ͔͘Θ͔Γ͍͢ • γϯϓϧ͔ͭڧྗ
ެࣜαΠταϯϓϧ
αϯϓϧ(requests൛ʣ
Beautiful Soup • 2004Ґ͔Βଘࡏ͢ΔϥΠϒϥϦ • HTMLXML͔ΒσʔλΛநग़ͯ͠औಘ • ࠷৽όʔγϣϯBeautiful Soup 4ܥ
• Python 2.7ɺPython 3.2ʹରԠ
αϯϓϧ
Scrapy Scarpyͯ͘ɺϋΠϨϕϧͳεΫϨΠϐϯά ΫϩʔϥʔͷϑϨʔϜϫʔΫɻWebαΠτͷΫ ϩʔϧͱɺߏԽ͞ΕͨσʔλΛऔΓग़͢ͷʹ ༻͢Δɻ෯͍తʹ༻Ͱ͖ΔɻσʔλϚ Πχϯά͔ΒɺϞχλϦϯάɺࣗಈςετͳͲ
Scrapyͷಛ • ΫϩʔϦϯάɺεΫϨΠϐϯάϑϨʔϜϫʔΫ • DjangoʹӨڹ͞Ε͍ͯΔʢMiddlewareͳͲʣ • εΫϨΠϐϯάʹඞཁͳػೳ͕ͦΖ͍ͬͯΔ • υΩϡϝϯτ͕ॆ࣮͍ͯ͠Δ
Scrapyͷओͳػೳ • μϯϩʔυɺநग़ɺอଘ • μϯϩʔυͨ͠υΩϡϝϯτͷΩϟογϡ • ڧྗͳίϚϯυϥΠϯγΣϧ • Robots.txtͷύʔε •
ඇಉظɺฒߦμϯϩʔυʢTwistedΛ༻ʣ • υϝΠϯɺIPΞυϨε୯ҐͷΫϩʔϧִؒௐ • Τϥʔ࣌ͷϦτϥΠ • ϩάग़ྗ
։ൃखॱ • ScrapyϓϩδΣΫτͷ࡞ • SpiderΛ࡞ʢϦϯΫநग़ɺμϯϩʔυʣ • ItemύΠϓϥΠϯͰσʔλΛอଘ
ϓϩδΣΫτͷ࡞ $ scrapy startproject scrapy_sample
αϯϓϧ
Spider࡞ʢެࣜαΠτΑΓʣ
࣮ߦ $ scrapy crawl dmoz_spider -o scraped_data.json
ৄࡉ • Scrapyೖʢ̍ʣ • Scrapyೖʢ̎ʣ
αϯϓϧʢ̍ʣ
αϯϓϧʢ̎ʣ
࣮ࡍͷࣄྫͷհ • ౡݝͷຽധσʔλΛੳ • ݅ใ • Ձ֨ใ
݅ใʢαΠτʣ
։ൃͷྲྀΕ • ݅ɺՁ֨ใऔಘ༻ͷεύΠμʔΛ࡞ • εύΠμʔ͕Ұ࣌σʔλΛอଘʢJSONʣ • όονॲཧʹͯ݅ɺՁ֨ΛอଘʢΫϨϯδϯάʣ • ूܭόονʹͯσʔλΛੳͯ͠DBʹอଘ •
ूܭσʔλΛදࣔ
σϞ
݅ 0 50 100 150 200 250 300 350 400
450 500 2016/2/15 2016/2/22 2016/2/29 2016/3/7 2016/3/14 2016/3/21 2016/3/28 2016/4/4 2016/4/11 2016/4/18 2016/4/25 2016/5/2 2016/5/9 2016/5/16 2016/5/23 2016/5/30 2016/6/6 2016/6/13 2016/6/20 2016/6/27 2016/7/4 2016/7/11 2016/7/18 2016/7/25 2016/8/1 2016/8/8 2016/8/15 2016/8/22 2016/8/29 2016/9/5 2016/9/12 2016/9/19 2016/9/26 2016/10/3 2016/10… 2016/10… 2016/10… 2016/10… 2016/11/7 2016/11… 2016/11… 2016/11… 2016/12/5 2016/12… 2016/12… 2016/12…
݅ • ݅ 461݅ • 1Ͱ2ഒʢ240݅ => 461݅) • શࠃͰ10൪ʹଟ͍
• ౦ژ, େࡕ, ژ, ԭೄ݅, ւಓ, Ԭ݅ɺ ਆಸ݅, ݅, Ѫ݅, ઍ༿ݝ, ౡݝ
ฏۉՁ֨ 0 2000 4000 6000 8000 10000 12000
ฏۉՁ֨ • ౙฏۉՁ͕͍֨ʢ5000ԁʣ • 8݄, 10݄, 11݄ͷि͕ߴ͍ʢ8000ʣ • ɺ͓ਖ਼݄͕ϐʔΫʢ10000ԁʣ
Քಇ 0 10 20 30 40 50 60 70 80
90 100
Քಇ • Նͷγʔζϯ͕ϐʔΫʢ80%ऑʣ • 10݄, 11݄ͷिߴ͍ʢ70%Ҏ্ʣ • ౙͷγʔζϯ͍ʢ40%ҎԼʣ • 10/15ʢ),
10/29ʢʣ͕ߴ͔ͬͨ
·ͱΊ • PythonͰεΫϨΠϐϯάΛߦ͏߹ɺ৭ʑͳ Ξϓϩʔν͕͋Δɻ • Scrapy໘ͳॲཧΛߦͬͯ͘ΕΔͷͰΦε εϝ • ౡͷຽധ͜Ε͔ΒΓ্͕Δͣʂ
͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠