ヘッドレスChromeでクローラを作った後の話

Transcript

1. ϔουϨεChromeͰ ΫϩʔϥΛ࡞ͬͨ”ޙͷ”࿩ Roppongi.js #1 Yuji Isobe

2. Yuji Isobe @yujiosaka ϓϩδΣΫτϚωʔδϟʔ at
 
 https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu

3. ΫϩʔϥΛ࡞ͬͨ࣌ͷ࿩ https://speakerdeck.com/yujiosaka/an-gazui-chu-nihetudoresuchromedekurorazuo-tuta-shi-ninannekana

4. ϔουϨεChromeͱ͸ ✓ Chrome͕ϔουϨεϞʔυͰىಈͰ͖Δ ✓ ChromeͷىಈΦϓγϣϯʹʮ--headessʯΛՃ͑Δ͚ͩ ✓ ୅දతͳϔουϨεϒϥ΢βͱ͍͑͹PhantomJS ✓ ߴ଎Ͱ҆ఆͯ͠ಈ࡞͢Δ ✓

   ඪ४΁ͷରԠ͕ૣ͍ʢES2017΍Async-Await͕࢖͑Δʣ ✓ ओͳ༻్͸ςετࣗಈԽͱΫϩʔϥ

5. Headless Chrome Crawler ✓ ϔουϨεChromeͰΫϩʔϥ ✓ ෼ࢄ؀ڥͰಈ࡞͢Δ ✓ ਂ͞༏ઌ୳ࡧʢDFSʣͱ
 ෯༏ઌ୳ࡧʢBFSʣΛαϙʔτ

   ✓ robots.txt, sitemap.xmlʹै͏ ✓ Puppeteerʹґଘ ✓ Node.jsʢJavaScriptʣ੡ https://github.com/yujiosaka/headless-chrome-crawler

6. ࠓ೔͸࡞ͬͨ”ޙͷ”࿩

7. GitHub > Insights > Traffic https://twitter.com/yujiosaka/status/967316514322890752

8. GitHub Trending Repositories ͜ͷลΛ2-3೔ؒ ͏Ζ͍ͭͯͨ https://github.com/trending

9. Hit 2000 Stars in 7 days https://github.com/yujiosaka/headless-chrome-crawler > 2000

10. ເ͕׎ͬͨ

11. ظ଴͍ͯͨ͠ϝϦοτ ✓ ඼࣭ͷ޲্ ✓ ։ൃྗͷ޲্ ✓ ϒϥϯυ޲্ ✓ ࢓ࣄʹͭͳ͕Δ ୭͔࢓ࣄ͘ΕΖ͍ͩ͘͞

    ҙ֎ͱࣗݾΞϐʔϧஏ͔͍ͣ͠ ·ͩλΠϙमਖ਼͔͠ૹΒΕͯͳ͍ Issue΍Βϝʔϧ΍ΒରԠ๩͍͠

12. ࢓ࣄதͣͬͱχϠχϠͰ͖Δ

13. ຊ୊ ଟ͘ͷਓͷ໨ʹཹ·ͬͨϥΠϒϥϦͱ ͦ͏ͳΒͳ͔ͬͨϥΠϒϥϦͷҧ͍Λ੔ཧ ※ͨͬͨαϯϓϧ̍ͷ͜ͱͳͷͰ࿩൒෼ʹฉ͍͍ͯͩ͘͞

14. λʔήοτͷ޿͞ ✓ ӳޠͰൃ৴͢Δॏཁੑ ✓ READMEΛӳޠͰॻ͍ͨΒ͓ऴ͍Ͱ͸ͳ͍ ✓ ʰ͸ͯϒϗοτΤϯτϦʔʱΑΓ
 ʰHacker News Top

    Linksʱͷํ͕10ഒྲྀೖ͕͋ͬͨ
 
 
 


15. ར༻ͷϋʔυϧ ✓ Ұ໨ݟͯԿ͕͍ͨ͠ͷ͔ϋοΩϦ෼͔Δ ✓ Headless Chrome + Crawler = Headless

    Chrome Crawler ✓ READMEͰϝϦοτΛҰ൪࠷ॳʹΞϐʔϧ͓ͯ͘͠ ✓ ͦΕͰ΋෼͔Βͳ͍ਓ޲͚ʹɺFAQΛ༻ҙ͢Δ ✓ ͍͍ͩͨͷਓ͸࠷ॳͷ਺ߦ͔͠ಡ·ͳ͍ ✓ ը૾΍ϩΰͰ΋ʮ؆୯͞ʯ͸ΞϐʔϧͰ͖Δ

16. ίʔυϦʔσΟϯάͷϋʔυϧ ✓ examples Λॆ࣮ͤ͞Δ ✓ Ұ൪ಡ·Ε͍ͯͨίϯςϯπ͸ examples ͩͬͨ ✓ πʔϧ͸ϑϧ׆༻͢Δ

    ✓ ESLint ✓ commitlint ✓ EditorConfig ✓ TypeScript / JSDoc support

17. ৴པ ✓ όοδΛϑϧ׆༻ ✓ ࠷৽ͷϏϧυঢ়گ͕Ұ໨Ͱ෼͔ΔΑ͏ʹ͢Δ ✓ άϦʔϯ͕ฒΜͰΔͱͳΜ͔҆৺͢Δ ✓ Greenkeeper࠷ڧઆ ✓

    ࠷ޙͷίϛοτ͕൒೥લͷϓϩδΣΫτͱ͔࢖͍ͨ͘ͳ͍ ✓ ͠͹Β͘αϘͬͯͯ΋ɺৗʹίϛοτͰ͖Δ

18. Happy Niya-niya Hacking!