Slide 1
Slide 1 text
ౡͷຽധσʔλ
Λੳ͠Α͏ʂ
2016/11/12 ؔࠜ༟ل
PyCon mini Hiroshima 2016
Slide 2
Slide 2 text
ࣗݾհ
• ؔࠜ༟لʢ͖ͤͶ ͻΖͷΓʣ
• גࣜձࣾSQUEEZE
• Twitter: @checkpoint
Slide 3
Slide 3 text
PythonͱͷؔΘΓʢ̍ʣ
• PyCon JP 2014 ελοϑ
• PyCon JP 2015 ෭࠲ʢϓϩάϥϜʣ
• PyCon jp 2016 ελοϑ
• Python͘͘ձʢओ࠵ʣ
Slide 4
Slide 4 text
PythonͱͷؔΘΓʢ̎ʣ
• LLDiver
• PyCon JP 2014
• Phone Symposium Tokyo 2015
• PyCon mini Hiroshima 2015
• PyCon mini Hiroshima 2016
• PythonΤϯδχΞཆಡຊʢڞஶʣ
Slide 5
Slide 5 text
ۀͰͷPython
• ຽധ݅Λཧɺӡ༻͢ΔͨΊͷαʔϏε
Slide 6
Slide 6 text
ۀͰͷPython
• ຽധ݅Λੳ͢ΔͨΊͷαʔϏε
Slide 7
Slide 7 text
༻ͯ͠Δٕज़
Slide 8
Slide 8 text
ΞδΣϯμ
• ౡݝͷຽധʹ͍ͭͯ
• PythonͰͷσʔλऩू
• PythonͰͷσʔλੳ
Slide 9
Slide 9 text
ຽധͱ
ҰൠͷຽՈʹ॓ധ͢Δ͜ͱʢ༷ʑͳܗଶʣ
Slide 10
Slide 10 text
ϓϥοτϑΥʔϜ
COPYRIGHT (C) 2014-2016 SQUEEZE Inc.
ALL RIGHTS RESERVED.
Slide 11
Slide 11 text
ϓϥοτϑΥʔϜʢຊʣ
Slide 12
Slide 12 text
ౡݝͷຽധ
• தࠃɾ࢛ࠃํͰҰ൪େ͖ͳࢢʢౡࢢʣ
• ੈքҨ࢈ΛؚΉ๛͔ͳ؍ޫࢿݯ
• ΦόϚถେ౷ྖͷ๚
• ౡΧʔϓͷηϦʔά༏উ
• ຽധΓ্͕͖͍ͬͯͯΔͣʂ
Slide 13
Slide 13 text
ຽധσʔλͷੳ
• σʔλͷऩू
• σʔλͷੳ
• σʔλͷදࣔ
Slide 14
Slide 14 text
σʔλͷऩू
• ΫϩʔϦϯά
• εΫϨΠϐϯά
• ౷ܭσʔλ
• ૯ল౷ܭہ
• σʔλΧλϩάαΠτ
Slide 15
Slide 15 text
ΫϩʔϦϯά
• ӳޠͷҙຯɺ[͏ɺΏͬ͘ΓਐΉ]
• WebϖʔδͷϦϯΫͷ༰ΛͨͲΔ
• Webϖʔδͷ༰Λμϯϩʔυͯ͠ऩू
• Web APIͷσʔλΛऔΔ߹͋Δ
Slide 16
Slide 16 text
εΫϨΠϐϯά
• ӳޠͷҙຯɺ[ Δ͜ͱ ]
• ϖʔδͷ༰͔ΒඞཁͳใΛநग़
Slide 17
Slide 17 text
όοςϦʔଐݴޠ
ʴ
ڧྗͳαʔυύʔςΟϥΠϒϥϦ
Slide 18
Slide 18 text
ศརͳϥΠϒϥϦ
• ඪ४ϥΠϒϥϦ
• requests
• BeautifulSoup
• Scrapy
• Selenium
Slide 19
Slide 19 text
ඪ४ϥΠϒϥϦ
• Pythonͷඪ४ϥΠϒϥϦͱͯॆ࣮
• ωοτϫʔΫɺਖ਼نදݱɺetc
• Pythonͷॲཧܥ͚ͩ͋Εྑ͍
• ؆୯ͳεΫϨΠϐϯάͰ͋Εे࣮༻త
Slide 20
Slide 20 text
αϯϓϧ
Slide 21
Slide 21 text
Requests
• PythonͷHTTP Client
• ਓؒʹ༏͍͠ΠϯλʔϑΣʔε
• ͱʹ͔͘Θ͔Γ͍͢
• γϯϓϧ͔ͭڧྗ
Slide 22
Slide 22 text
ެࣜαΠταϯϓϧ
Slide 23
Slide 23 text
αϯϓϧ(requests൛ʣ
Slide 24
Slide 24 text
Beautiful Soup
• 2004Ґ͔Βଘࡏ͢ΔϥΠϒϥϦ
• HTMLXML͔ΒσʔλΛநग़ͯ͠औಘ
• ࠷৽όʔγϣϯBeautiful Soup 4ܥ
• Python 2.7ɺPython 3.2ʹରԠ
Slide 25
Slide 25 text
αϯϓϧ
Slide 26
Slide 26 text
Scrapy
Scarpyͯ͘ɺϋΠϨϕϧͳεΫϨΠϐϯά
ΫϩʔϥʔͷϑϨʔϜϫʔΫɻWebαΠτͷΫ
ϩʔϧͱɺߏԽ͞ΕͨσʔλΛऔΓग़͢ͷʹ
༻͢Δɻ෯͍తʹ༻Ͱ͖ΔɻσʔλϚ
Πχϯά͔ΒɺϞχλϦϯάɺࣗಈςετͳͲ
Slide 27
Slide 27 text
Scrapyͷಛ
• ΫϩʔϦϯάɺεΫϨΠϐϯάϑϨʔϜϫʔΫ
• DjangoʹӨڹ͞Ε͍ͯΔʢMiddlewareͳͲʣ
• εΫϨΠϐϯάʹඞཁͳػೳ͕ͦΖ͍ͬͯΔ
• υΩϡϝϯτ͕ॆ࣮͍ͯ͠Δ
Slide 28
Slide 28 text
Scrapyͷओͳػೳ
• μϯϩʔυɺநग़ɺอଘ
• μϯϩʔυͨ͠υΩϡϝϯτͷΩϟογϡ
• ڧྗͳίϚϯυϥΠϯγΣϧ
• Robots.txtͷύʔε
• ඇಉظɺฒߦμϯϩʔυʢTwistedΛ༻ʣ
• υϝΠϯɺIPΞυϨε୯ҐͷΫϩʔϧִؒௐ
• Τϥʔ࣌ͷϦτϥΠ
• ϩάग़ྗ
Slide 29
Slide 29 text
։ൃखॱ
• ScrapyϓϩδΣΫτͷ࡞
• SpiderΛ࡞ʢϦϯΫநग़ɺμϯϩʔυʣ
• ItemύΠϓϥΠϯͰσʔλΛอଘ
Slide 30
Slide 30 text
ϓϩδΣΫτͷ࡞
$ scrapy startproject scrapy_sample
Slide 31
Slide 31 text
αϯϓϧ
Slide 32
Slide 32 text
Spider࡞ʢެࣜαΠτΑΓʣ
Slide 33
Slide 33 text
࣮ߦ
$ scrapy crawl dmoz_spider -o scraped_data.json
Slide 34
Slide 34 text
ৄࡉ
• Scrapyೖʢ̍ʣ
• Scrapyೖʢ̎ʣ
Slide 35
Slide 35 text
αϯϓϧʢ̍ʣ
Slide 36
Slide 36 text
αϯϓϧʢ̎ʣ
Slide 37
Slide 37 text
࣮ࡍͷࣄྫͷհ
• ౡݝͷຽധσʔλΛੳ
• ݅ใ
• Ձ֨ใ
Slide 38
Slide 38 text
݅ใʢαΠτʣ
Slide 39
Slide 39 text
։ൃͷྲྀΕ
• ݅ɺՁ֨ใऔಘ༻ͷεύΠμʔΛ࡞
• εύΠμʔ͕Ұ࣌σʔλΛอଘʢJSONʣ
• όονॲཧʹͯ݅ɺՁ֨ΛอଘʢΫϨϯδϯάʣ
• ूܭόονʹͯσʔλΛੳͯ͠DBʹอଘ
• ूܭσʔλΛදࣔ
Slide 40
Slide 40 text
σϞ
Slide 41
Slide 41 text
݅
0
50
100
150
200
250
300
350
400
450
500
2016/2/15
2016/2/22
2016/2/29
2016/3/7
2016/3/14
2016/3/21
2016/3/28
2016/4/4
2016/4/11
2016/4/18
2016/4/25
2016/5/2
2016/5/9
2016/5/16
2016/5/23
2016/5/30
2016/6/6
2016/6/13
2016/6/20
2016/6/27
2016/7/4
2016/7/11
2016/7/18
2016/7/25
2016/8/1
2016/8/8
2016/8/15
2016/8/22
2016/8/29
2016/9/5
2016/9/12
2016/9/19
2016/9/26
2016/10/3
2016/10…
2016/10…
2016/10…
2016/10…
2016/11/7
2016/11…
2016/11…
2016/11…
2016/12/5
2016/12…
2016/12…
2016/12…
Slide 42
Slide 42 text
݅
• ݅ 461݅
• 1Ͱ2ഒʢ240݅ => 461݅)
• શࠃͰ10൪ʹଟ͍
• ౦ژ, େࡕ, ژ, ԭೄ݅, ւಓ, Ԭ݅ɺ
ਆಸ݅, ݅, Ѫ݅, ઍ༿ݝ, ౡݝ
Slide 43
Slide 43 text
ฏۉՁ֨
0
2000
4000
6000
8000
10000
12000
Slide 44
Slide 44 text
ฏۉՁ֨
• ౙฏۉՁ͕͍֨ʢ5000ԁʣ
• 8݄, 10݄, 11݄ͷि͕ߴ͍ʢ8000ʣ
• ɺ͓ਖ਼݄͕ϐʔΫʢ10000ԁʣ
Slide 45
Slide 45 text
Քಇ
0
10
20
30
40
50
60
70
80
90
100
Slide 46
Slide 46 text
Քಇ
• Նͷγʔζϯ͕ϐʔΫʢ80%ऑʣ
• 10݄, 11݄ͷिߴ͍ʢ70%Ҏ্ʣ
• ౙͷγʔζϯ͍ʢ40%ҎԼʣ
• 10/15ʢ), 10/29ʢʣ͕ߴ͔ͬͨ
Slide 47
Slide 47 text
·ͱΊ
• PythonͰεΫϨΠϐϯάΛߦ͏߹ɺ৭ʑͳ
Ξϓϩʔν͕͋Δɻ
• Scrapy໘ͳॲཧΛߦͬͯ͘ΕΔͷͰΦε
εϝ
• ౡͷຽധ͜Ε͔ΒΓ্͕Δͣʂ
Slide 48
Slide 48 text
͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠