ヘッドレスChromeでクローラを作った後の話

Transcript

1. ϔουϨεChromeͰ
   ΫϩʔϥΛ࡞ͬͨ”ޙͷ”࿩
   Roppongi.js #1
   Yuji Isobe
   

   View Slide

2. Yuji Isobe @yujiosaka
   ϓϩδΣΫτϚωʔδϟʔ at

   

   https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu
   

   View Slide

3. ΫϩʔϥΛ࡞ͬͨ࣌ͷ࿩
   https://speakerdeck.com/yujiosaka/an-gazui-chu-nihetudoresuchromedekurorazuo-tuta-shi-ninannekana
   

   View Slide

4. ϔουϨεChromeͱ͸
   ✓ Chrome͕ϔουϨεϞʔυͰىಈͰ͖Δ
   ✓ ChromeͷىಈΦϓγϣϯʹʮ--headessʯΛՃ͑Δ͚ͩ
   ✓ ୅දతͳϔουϨεϒϥ΢βͱ͍͑͹PhantomJS
   ✓ ߴ଎Ͱ҆ఆͯ͠ಈ࡞͢Δ
   ✓ ඪ४΁ͷରԠ͕ૣ͍ʢES2017΍Async-Await͕࢖͑Δʣ
   ✓ ओͳ༻్͸ςετࣗಈԽͱΫϩʔϥ
   

   View Slide

5. Headless Chrome Crawler
   ✓ ϔουϨεChromeͰΫϩʔϥ
   ✓ ෼ࢄ؀ڥͰಈ࡞͢Δ
   ✓ ਂ͞༏ઌ୳ࡧʢDFSʣͱ

   ෯༏ઌ୳ࡧʢBFSʣΛαϙʔτ
   ✓ robots.txt, sitemap.xmlʹै͏
   ✓ Puppeteerʹґଘ
   ✓ Node.jsʢJavaScriptʣ੡
   https://github.com/yujiosaka/headless-chrome-crawler
   

   View Slide

6. ࠓ೔͸࡞ͬͨ”ޙͷ”࿩
   

   View Slide

7. GitHub > Insights > Traffic
   https://twitter.com/yujiosaka/status/967316514322890752
   

   View Slide

8. GitHub Trending Repositories
   ͜ͷลΛ2-3೔ؒ
   ͏Ζ͍ͭͯͨ
   https://github.com/trending
   

   View Slide

9. Hit 2000 Stars in 7 days
   https://github.com/yujiosaka/headless-chrome-crawler
   > 2000
   

   View Slide

10. ເ͕׎ͬͨ
    

    View Slide

11. ظ଴͍ͯͨ͠ϝϦοτ
    ✓ ඼࣭ͷ޲্
    ✓ ։ൃྗͷ޲্
    ✓ ϒϥϯυ޲্
    ✓ ࢓ࣄʹͭͳ͕Δ ୭͔࢓ࣄ͘ΕΖ͍ͩ͘͞
    ҙ֎ͱࣗݾΞϐʔϧஏ͔͍ͣ͠
    ·ͩλΠϙमਖ਼͔͠ૹΒΕͯͳ͍
    Issue΍Βϝʔϧ΍ΒରԠ๩͍͠
    

    View Slide

12. ࢓ࣄதͣͬͱχϠχϠͰ͖Δ
    

    View Slide

13. ຊ୊
    ଟ͘ͷਓͷ໨ʹཹ·ͬͨϥΠϒϥϦͱ
    ͦ͏ͳΒͳ͔ͬͨϥΠϒϥϦͷҧ͍Λ੔ཧ
    ※ͨͬͨαϯϓϧ̍ͷ͜ͱͳͷͰ࿩൒෼ʹฉ͍͍ͯͩ͘͞
    

    View Slide

14. λʔήοτͷ޿͞
    ✓ ӳޠͰൃ৴͢Δॏཁੑ
    ✓ READMEΛӳޠͰॻ͍ͨΒ͓ऴ͍Ͱ͸ͳ͍
    ✓ ʰ͸ͯϒϗοτΤϯτϦʔʱΑΓ

    ʰHacker News Top Linksʱͷํ͕10ഒྲྀೖ͕͋ͬͨ

    

    

    

    

    View Slide

15. ར༻ͷϋʔυϧ
    ✓ Ұ໨ݟͯԿ͕͍ͨ͠ͷ͔ϋοΩϦ෼͔Δ
    ✓ Headless Chrome + Crawler = Headless Chrome
    Crawler
    ✓ READMEͰϝϦοτΛҰ൪࠷ॳʹΞϐʔϧ͓ͯ͘͠
    ✓ ͦΕͰ΋෼͔Βͳ͍ਓ޲͚ʹɺFAQΛ༻ҙ͢Δ
    ✓ ͍͍ͩͨͷਓ͸࠷ॳͷ਺ߦ͔͠ಡ·ͳ͍
    ✓ ը૾΍ϩΰͰ΋ʮ؆୯͞ʯ͸ΞϐʔϧͰ͖Δ
    

    View Slide

16. ίʔυϦʔσΟϯάͷϋʔυϧ
    ✓ examples Λॆ࣮ͤ͞Δ
    ✓ Ұ൪ಡ·Ε͍ͯͨίϯςϯπ͸ examples ͩͬͨ
    ✓ πʔϧ͸ϑϧ׆༻͢Δ
    ✓ ESLint
    ✓ commitlint
    ✓ EditorConfig
    ✓ TypeScript / JSDoc support
    

    View Slide

17. ৴པ
    ✓ όοδΛϑϧ׆༻
    ✓ ࠷৽ͷϏϧυঢ়گ͕Ұ໨Ͱ෼͔ΔΑ͏ʹ͢Δ
    ✓ άϦʔϯ͕ฒΜͰΔͱͳΜ͔҆৺͢Δ
    ✓ Greenkeeper࠷ڧઆ
    ✓ ࠷ޙͷίϛοτ͕൒೥લͷϓϩδΣΫτͱ͔࢖͍ͨ͘ͳ͍
    ✓ ͠͹Β͘αϘͬͯͯ΋ɺৗʹίϛοτͰ͖Δ
    

    View Slide

18. Happy Niya-niya Hacking!
    

    View Slide