Share
Embed
Transcript
41 slides
-
ରͱత 3 ରࢹௌऀ త ͞ͳ͍͜ͱ • WebεΫϨΠϐϯά Λ;ΜΘΓ͍ͬͯ Δ •
WebεΫϨΠϐϯά ͷن੍पΓʹෆ҆Λ ײ͍ͯ͡Δ • politeύοέʔδΛ௨ ͯ͠WebεΫϨΠϐ ϯάΤνέοτͷج ૅΛֶͿ • rvestͳͲΛ༻ͨ͠ εΫϨΠϐϯάͦͷ ͷͷૢ࡞ • औಘͨ͠ใͷ༻ ্ͷ੍ݶ -
WebεΫϨΠϐϯάͱ 6 εΫϨΠϐϯά͕ҧ๏ʹͳΔ̐έʔε ✕ɹαʔόʔʹෛՙΛ͔͚ͯ͠·ͬͨ߹ ʢܐ๏233ِܭۀࡑɺಉ234ిࢠܭࢉػଛյۀࡑʣ ✕ɹݸਓใΛಉҙͳ͘औಘɺެ։ɺചങͯ͠͠·ͬͨ߹ ʢݸਓใอޢ๏ҧʣ ✕ɹεΫϨΠϐϯά͢ΔWebαΠτͷར༻نʹҧ͢Δ߹ ✕ɹஶ࡞ݖΛແࢹͨ͠ར༻ɺෳΛߦͬͯ͠·ͬͨ߹ ʢஶ࡞ݖ๏21ͳͲʣ
PigData, ITหޢ࢜ʹฉ͘ʮاۀͱͯ͠ͷεΫϨΠϐϯάҧ๏ͳͷ͔ʁʯ, https://services.sms-datatech.co.jp/pig-data/2019/07/03/scrapinglaw/ -
WebεΫϨΠϐϯάͱ εΫϨΠϐϯά͕ҧ๏ʹͳΔ̐έʔε ✕ɹαʔόʔʹෛՙΛ͔͚ͯ͠·ͬͨ߹ ʢܐ๏233ِܭۀࡑɺಉ234ిࢠܭࢉػଛյۀࡑʣ ✕ɹݸਓใΛಉҙͳ͘औಘɺެ։ɺചങͯ͠͠·ͬͨ߹ ʢݸਓใอޢ๏ҧʣ ✕ɹεΫϨΠϐϯά͢ΔWebαΠτͷར༻نʹҧ͢Δ߹ ✕ɹஶ࡞ݖΛແࢹͨ͠ར༻ɺෳΛߦͬͯ͠·ͬͨ߹ ʢஶ࡞ݖ๏21ͳͲʣ PigData,
ITหޢ࢜ʹฉ͘ʮاۀͱͯ͠ͷεΫϨΠϐϯάҧ๏ͳͷ͔ʁʯ, https://services.sms-datatech.co.jp/pig-data/2019/07/03/scrapinglaw/ 7 Be Polite!!! -
politeύοέʔδͱ 9 ϦϦʔεɿr ։ൃऀɿDmytro Perepolkin URLɿhttps://github.com/dmi3kno/polite https://www.rdocumentation.org/packages/polite/versions/0.1.1 The goal of
polite is to promote responsible web etiquette. Be Nice on the Web -
politeύοέʔδͱ 10 https://www.rdocumentation.org/packages/polite/versions/0.1.1 The three pillars of a polite session
are seeking permission, taking slowly and never asking twice. Be Nice on the Web -
11 politeͷ࣮ફ͢ΔεΫϨΠϐϯάͷΤνέοτ ɾऔಘִؒΛۭ͚Δ ʙtaking slowlyʙ ɾऔಘڐՄͷ֬ೝ ʙseeking permissionʙ ɾ֬ೝΛ܁Γฦ͞ͳ͍ ʙnever
asking twiceʙ politeύοέʔδͱ -
12 politeͷ࣮ફ͢ΔεΫϨΠϐϯάͷΤνέοτ ɾऔಘڐՄͷ֬ೝ ʙseeking permissionʙ ɾऔಘִؒΛۭ͚Δ ʙtaking slowlyʙ ɾ֬ೝΛ܁Γฦ͞ͳ͍ ʙnever
asking twiceʙ politeύοέʔδͱ ๏తͳ՝ͳͲΛશͯճආͰ͖Δ༁Ͱͳ͍ ݸਓใ ར༻ن ஶ࡞ݖ -
politeύοέʔδͷ༻खॱ 14 1, bow ʙ Introduce yourself to the hostʙ
2, scrape ʙ Scrape the content of authorized page/APIʙ ( 3, rvest ʙ helps you scrape information from web pagesʙ ) 4, nod ʙ Agree Modification Of Session Path With The Hostʙ polite༻खॱ https://www.rdocumentation.org/packages/polite/versions/0.1.1 https://www.rdocumentation.org/packages/rvest/versions/0.3.6 -
20 robots.txtͱ ΫϩʔϥʔͷࢦࣔΛهड़͢ΔςΩετϑΝΠϧͷ͜ͱ*1 *1 https://wacul-ai.com/blog/seo/internal-seo/seo-robots-txt/ *2 https://www.rdocumentation.org/packages/robotstxt/versions/0.7.13 robots.txt ࠓճରʹ͍ͯ͠ΔWebαΠτͷrobots.txtʢhttps://www.cheese.com/robots.txtʣ User-agent:
* Sitemap: https://www.cheese.com/sitemap.xml *2 politeύοέʔδͷ༻खॱʙ bowʙ -
26 ϦϦʔεɿr ։ൃऀɿ Hadley Wickham URLɿhttps://github.com/tidyverse/rvest Easily Harvest (Scrape) Web
Pages https://www.rdocumentation.org/packages/rvest/versions/0.3.6 politeύοέʔδͷ༻खॱʙ rvestʙ -
27 scrapeͰऔಘͨ͠HTML Document͔ΒඞཁͳใΛநग़ ࢀߟจݙɿ Intro to {polite} Web Scraping of
Soccer Data with R!ʢhttps://ryo-n7.github.io/2020-05-14-webscrape-soccer-data-with-R/ʣ RʹΑΔεΫϨΠϐϯάೖʢhttps://www.amazon.co.jp/dp/486354216Xʣ RϢʔβͷͨΊͷRStudio[࣮ફ]ೖ−tidyverseʹΑΔϞμϯͳੳϑϩʔͷੈք−ʢhttps://www.amazon.co.jp/dp/4774198536ʣ politeύοέʔδͷ༻खॱʙ rvestʙ -
·ͱΊ 31 politeͷखॱ ɹɹ1, bow……robots.txtͷ֬ೝɺUserAgentͷ௨ɺdelayͷઃఆɻ ɹɹ2, scrape……ίϯςϯπͷऔಘ ɹɹ( 3, rvest……ඞཁͳใͷநग़
) ɹɹ4, nod……ରύεͷมߋ politeͷ3ͭͷओػೳ ɾऔಘִؒΛۭ͚Δ ʙtaking slowlyʙ ɹɹɾऔಘڐՄͷ֬ೝ ʙseeking permissionʙ ɹɹɾ֬ೝΛ܁Γฦ͞ͳ͍ ʙnever asking twiceʙ → bowͰઃఆɻσϑΥϧτ5ඵɻ → nodͰରԠɻ → bowͰrobots.txt͔Βऔಘɻ -
ͦͷଞҙࣄ߲ 32 ར༻ن نͰεΫϨΠϐϯά͕ېࢭ͞Ε͍ͯΔαΠτͰεΫϨΠϐϯάΛߦ͏ͱɺଛഛঈ ٻͳͲͷՄೳੑ͕͋Δɻʢͨͩ͠ɺձһొͳͲϢʔβʔ͕ಉҙͨ͠߹ͷΈɻʣ ݸਓใ ݸਓใͷऔಘ͢Δࡍʹɺར༻తΛຊਓʹ໌ࣔ͢Δඞཁ͕͋Δɻ ஶ࡞ݖ ஶ࡞Λݖརऀͷಉҙͳ͘ίϐʔอଘΛ͢Δߦҝɺஶ࡞ݖ৵ʹͨΔɻ ʢͨͩ͠ɺใղੳͷͨΊͷෳɺݖརऀͷಉҙͳ͘ߦ͏͜ͱ͕Ͱ͖Δɻʣ
PigData, ITหޢ࢜ʹฉ͘ʮاۀͱͯ͠ͷεΫϨΠϐϯάҧ๏ͳͷ͔ʁʯ, https://services.sms-datatech.co.jp/pig-data/2019/07/03/scrapinglaw/ IT๏ɾAIɾFintechͷ๏ʹৄ͍͠หޢ࢜ʛதलढ़, ʮʲεΫϨΠϐϯάͱ๏ʳεΫϨΠϐϯάͬͯ๏తʹԿ͕OKͰԿ͕OUTͳͷ͔Λหޢ͕࢜ղઆɻʯ, https://it-bengosi.com/blog/scraping/ -
ͦͷଞҙࣄ߲ ར༻ن نͰεΫϨΠϐϯά͕ېࢭ͞Ε͍ͯΔαΠτͰεΫϨΠϐϯάΛߦ͏ͱɺଛഛঈ ٻͳͲͷՄೳੑ͕͋Δɻʢͨͩ͠ɺձһొͳͲϢʔβʔ͕ಉҙͨ͠߹ͷΈɻʣ ݸਓใ ݸਓใͷऔಘ͢Δࡍʹɺར༻తΛຊਓʹ໌ࣔ͢Δඞཁ͕͋Δɻ ஶ࡞ݖ ஶ࡞Λݖརऀͷಉҙͳ͘ίϐʔอଘΛ͢Δߦҝɺஶ࡞ݖ৵ʹͨΔɻ ʢͨͩ͠ɺใղੳͷͨΊͷෳɺݖརऀͷಉҙͳ͘ߦ͏͜ͱ͕Ͱ͖Δɻʣ PigData,
ITหޢ࢜ʹฉ͘ʮاۀͱͯ͠ͷεΫϨΠϐϯάҧ๏ͳͷ͔ʁʯ, https://services.sms-datatech.co.jp/pig-data/2019/07/03/scrapinglaw/ IT๏ɾAIɾFintechͷ๏ʹৄ͍͠หޢ࢜ʛதलढ़, ʮʲεΫϨΠϐϯάͱ๏ʳεΫϨΠϐϯάͬͯ๏తʹԿ͕OKͰԿ͕OUTͳͷ͔Λหޢ͕࢜ղઆɻʯ, https://it-bengosi.com/blog/scraping/ 33 ֤ࣗɺཁ֬ೝͰ͓ئ͍͠·͢……ɻ -
ࢀߟจݙ 34 R Documentation, ʮpolite packageʯ https://www.rdocumentation.org/packages/polite/versions/0.1.1 R Documentation, ʮrvest
packageʯ https://www.rdocumentation.org/packages/rvest/versions/0.3.6 ΞΫηεղੳπʔϧʮAIΞφϦετʯϒϩά, ʮrobots.txtͱʁҙຯ͔Βઃఆํ๏·Ͱৄ͘͠ղઆʯ https://wacul-ai.com/blog/seo/internal-seo/seo-robots-txt/ Octoparse, ʮWebεΫϨΠϐϯάͱʁఆ͔ٛΒԠ༻·Ͱͷઆ໌ʯ https://www.octoparse.jp/blog/web-scraping/ PigData, ITหޢ࢜ʹฉ͘ʮاۀͱͯ͠ͷεΫϨΠϐϯάҧ๏ͳͷ͔ʁʯ ɹhttps://services.sms-datatech.co.jp/pig-data/2019/07/03/scrapinglaw/ -
ࢀߟจݙ 35 จԽி, ʮஶ࡞͕ࣗ༝ʹ͑Δ߹ʯ https://www.bunka.go.jp/seisaku/chosakuken/seidokaisetsu/gaiyo/chosakubutsu_jiyu.html Stimulator, ʮWebεΫϨΠϐϯά͢ΔࡍͷϧʔϧͱPythonʹΑΔنͷಡΈࠐΈʯ https://vaaaaaanquish.hatenablog.com/entry/2017/12/01/064227 IT๏ɾAIɾFintechͷ๏ʹৄ͍͠หޢ࢜ʛதलढ़, ʮʲεΫϨΠϐϯάͱ๏ʳεΫϨΠ
ϐϯάͬͯ๏తʹԿ͕OKͰԿ͕OUTͳͷ͔Λหޢ͕࢜ղઆɻʯ https://it-bengosi.com/blog/scraping/ PigData, ʮʲεΫϨΠϐϯάʳҧ๏ʹͳΒͳ͍αʔϏεύλʔϯ5બʯ https://services.sms-datatech.co.jp/pig-data/2020/01/15/scrapinglaw3/ R Documentation, ʮrobotstxt packageʯ https://www.rdocumentation.org/packages/robotstxt/versions/0.7.13 -
ࢀߟจݙ 36 ੴా ج, ࢢ ଠ༞, ӝੜ ਅ, ౬୩ ܒ໌,
γʔΞϯυΞʔϧݚڀॴ,ʮRʹΑΔεΫϨΠϐϯάೖʯ https://www.amazon.co.jp/dp/486354216X দଜ ༏࠸, ౬୩ ܒ໌, લా , لϊఆ อྱ, ٕज़ධࣾ, ʮRϢʔβͷͨΊͷRStudio[࣮ફ]ೖ−tidyverseʹΑΔϞμϯͳੳϑϩʔͷੈք−ʯ https://www.amazon.co.jp/dp/4774198536 ʮIntro to {polite} Web Scraping of Soccer Data with R!ʯ https://ryo-n7.github.io/2020-05-14-webscrape-soccer-data-with-R/ -
Appendix 39 Ԭ࡚ࢢཱதԝਤॻؗࣄ݅ Ԭ࡚ࢢཱதԝਤॻؗࣄ݅ʢ͓͔͖͟͠ΓͭͪΎ͏͓͏ͱ͠ΐ͔Μ͚͡Μʣɺ2010 3݄ࠒʹԬ࡚ࢢཱਤॻؗͷଂॻݕࡧγεςϜʹΞΫηεো͕ൃੜ͠ɺར༻ऀͷҰ ਓ͕ୁั͞Εͨࣄ݅Ͱ͋Δɻʢதུʣ உੑ͕࣮ࡍʹߦ͍ͬͯͨͷɺଂॻݕࡧγεςϜͷ͍উखʹຬ͠ͳ͔ͬͨͨΊࣗ Ͱ࡞ͨ͠ΫϩʔϥΛ࣮ߦ͠ɺଂॻݕࡧγεςϜ͔ΒਤॻใΛऔಘ͢Δ͜ͱͰ ͋ͬͨɻʢதུʣ 20ؒͷޯཹͱऔΓௐͷޙɺ6݄14ʹஉੑͷۀͷڧ͍ҙਤ͕ೝΊΒΕͳ
͍ͱͯ͠ىૌ༛༧ॲͱͳ͕ͬͨɺઐՈٕज़ऀ͔ΒظʹΘͨΔޯཹͷਖ਼ੑ ͓ΑͼͦΕҎલʹୁั͕ඞཁͰ͋ͬͨͷ͔͕ٙࢹ͞Ε͍ͯΔɻʢWikipediaʣ https://ja.wikipedia.org/wiki/%E5%B2%A1%E5%B4%8E%E5%B8%82%E7%AB%8B%E4%B8%AD%E5%A4%AE%E5%9B%B3%E6%9B%B8%E9%A4%A8%E4%BA%8B%E4%BB%B6 -
Appendix 40 UserAgentͱ ϢʔβʔΤʔδΣϯτ(User Agent)ͱɺΣϒαΠτΞΫηε͢Δࡍ ʹ༻͞ΕΔϓϩάϥϜɺ͋Δ͍ͦΕΒΛࣝผ͢ΔͨΊͷจࣈྻͷ ͜ͱΛࢦ͢ɻ*1 *1 https://www.irep.co.jp/knowledge/glossary/detail/id=10210/ *2
https://qiita.com/nightyknite/items/b2590a69f2e0135756dc ྫɿMozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 ɹɹ(KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36 *2