ヘッドレスChromeでクローラを作った後の話

Transcript

1. 1.

   ϔουϨεChromeͰ ΫϩʔϥΛ࡞ͬͨ”ޙͷ”࿩ Roppongi.js #1 Yuji Isobe

2. 2.

   Yuji Isobe @yujiosaka ϓϩδΣΫτϚωʔδϟʔ at
 
 https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu

3. 3.

   ΫϩʔϥΛ࡞ͬͨ࣌ͷ࿩ https://speakerdeck.com/yujiosaka/an-gazui-chu-nihetudoresuchromedekurorazuo-tuta-shi-ninannekana

4. 4.

   ϔουϨεChromeͱ͸ ✓ Chrome͕ϔουϨεϞʔυͰىಈͰ͖Δ ✓ ChromeͷىಈΦϓγϣϯʹʮ--headessʯΛՃ͑Δ͚ͩ ✓ ୅දతͳϔουϨεϒϥ΢βͱ͍͑͹PhantomJS ✓ ߴ଎Ͱ҆ఆͯ͠ಈ࡞͢Δ ✓

   ඪ४΁ͷରԠ͕ૣ͍ʢES2017΍Async-Await͕࢖͑Δʣ ✓ ओͳ༻్͸ςετࣗಈԽͱΫϩʔϥ
5. 5.

   Headless Chrome Crawler ✓ ϔουϨεChromeͰΫϩʔϥ ✓ ෼ࢄ؀ڥͰಈ࡞͢Δ ✓ ਂ͞༏ઌ୳ࡧʢDFSʣͱ
 ෯༏ઌ୳ࡧʢBFSʣΛαϙʔτ

   ✓ robots.txt, sitemap.xmlʹै͏ ✓ Puppeteerʹґଘ ✓ Node.jsʢJavaScriptʣ੡ https://github.com/yujiosaka/headless-chrome-crawler
6. 6.

   ࠓ೔͸࡞ͬͨ”ޙͷ”࿩

7. 7.

   GitHub > Insights > Traffic https://twitter.com/yujiosaka/status/967316514322890752

8. 8.

   GitHub Trending Repositories ͜ͷลΛ2-3೔ؒ ͏Ζ͍ͭͯͨ https://github.com/trending

9. 9.

   Hit 2000 Stars in 7 days https://github.com/yujiosaka/headless-chrome-crawler > 2000

10. 10.

    ເ͕׎ͬͨ

11. 11.

    ظ଴͍ͯͨ͠ϝϦοτ ✓ ඼࣭ͷ޲্ ✓ ։ൃྗͷ޲্ ✓ ϒϥϯυ޲্ ✓ ࢓ࣄʹͭͳ͕Δ ୭͔࢓ࣄ͘ΕΖ͍ͩ͘͞

    ҙ֎ͱࣗݾΞϐʔϧஏ͔͍ͣ͠ ·ͩλΠϙमਖ਼͔͠ૹΒΕͯͳ͍ Issue΍Βϝʔϧ΍ΒରԠ๩͍͠
12. 12.

    ࢓ࣄதͣͬͱχϠχϠͰ͖Δ

13. 13.

    ຊ୊ ଟ͘ͷਓͷ໨ʹཹ·ͬͨϥΠϒϥϦͱ ͦ͏ͳΒͳ͔ͬͨϥΠϒϥϦͷҧ͍Λ੔ཧ ※ͨͬͨαϯϓϧ̍ͷ͜ͱͳͷͰ࿩൒෼ʹฉ͍͍ͯͩ͘͞

14. 14.

    λʔήοτͷ޿͞ ✓ ӳޠͰൃ৴͢Δॏཁੑ ✓ READMEΛӳޠͰॻ͍ͨΒ͓ऴ͍Ͱ͸ͳ͍ ✓ ʰ͸ͯϒϗοτΤϯτϦʔʱΑΓ
 ʰHacker News Top

    Linksʱͷํ͕10ഒྲྀೖ͕͋ͬͨ
 
 
 

15. 15.

    ར༻ͷϋʔυϧ ✓ Ұ໨ݟͯԿ͕͍ͨ͠ͷ͔ϋοΩϦ෼͔Δ ✓ Headless Chrome + Crawler = Headless

    Chrome Crawler ✓ READMEͰϝϦοτΛҰ൪࠷ॳʹΞϐʔϧ͓ͯ͘͠ ✓ ͦΕͰ΋෼͔Βͳ͍ਓ޲͚ʹɺFAQΛ༻ҙ͢Δ ✓ ͍͍ͩͨͷਓ͸࠷ॳͷ਺ߦ͔͠ಡ·ͳ͍ ✓ ը૾΍ϩΰͰ΋ʮ؆୯͞ʯ͸ΞϐʔϧͰ͖Δ
16. 16.

    ίʔυϦʔσΟϯάͷϋʔυϧ ✓ examples Λॆ࣮ͤ͞Δ ✓ Ұ൪ಡ·Ε͍ͯͨίϯςϯπ͸ examples ͩͬͨ ✓ πʔϧ͸ϑϧ׆༻͢Δ

    ✓ ESLint ✓ commitlint ✓ EditorConfig ✓ TypeScript / JSDoc support
17. 17.

    ৴པ ✓ όοδΛϑϧ׆༻ ✓ ࠷৽ͷϏϧυঢ়گ͕Ұ໨Ͱ෼͔ΔΑ͏ʹ͢Δ ✓ άϦʔϯ͕ฒΜͰΔͱͳΜ͔҆৺͢Δ ✓ Greenkeeper࠷ڧઆ ✓

    ࠷ޙͷίϛοτ͕൒೥લͷϓϩδΣΫτͱ͔࢖͍ͨ͘ͳ͍ ✓ ͠͹Β͘αϘͬͯͯ΋ɺৗʹίϛοτͰ͖Δ
18. 18.

    Happy Niya-niya Hacking!