Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ヘッドレスChromeでクローラを作った後の話

 ヘッドレスChromeでクローラを作った後の話

5d247ff63b1861db5e6a56d4990e5a4f?s=128

yujiosaka

March 20, 2018
Tweet

Transcript

  1. ϔουϨεChromeͰ ΫϩʔϥΛ࡞ͬͨ”ޙͷ”࿩ Roppongi.js #1 Yuji Isobe

  2. Yuji Isobe @yujiosaka ϓϩδΣΫτϚωʔδϟʔ at
 
 https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu

  3. ΫϩʔϥΛ࡞ͬͨ࣌ͷ࿩ https://speakerdeck.com/yujiosaka/an-gazui-chu-nihetudoresuchromedekurorazuo-tuta-shi-ninannekana

  4. ϔουϨεChromeͱ͸ ✓ Chrome͕ϔουϨεϞʔυͰىಈͰ͖Δ ✓ ChromeͷىಈΦϓγϣϯʹʮ--headessʯΛՃ͑Δ͚ͩ ✓ ୅දతͳϔουϨεϒϥ΢βͱ͍͑͹PhantomJS ✓ ߴ଎Ͱ҆ఆͯ͠ಈ࡞͢Δ ✓

    ඪ४΁ͷରԠ͕ૣ͍ʢES2017΍Async-Await͕࢖͑Δʣ ✓ ओͳ༻్͸ςετࣗಈԽͱΫϩʔϥ
  5. Headless Chrome Crawler ✓ ϔουϨεChromeͰΫϩʔϥ ✓ ෼ࢄ؀ڥͰಈ࡞͢Δ ✓ ਂ͞༏ઌ୳ࡧʢDFSʣͱ
 ෯༏ઌ୳ࡧʢBFSʣΛαϙʔτ

    ✓ robots.txt, sitemap.xmlʹै͏ ✓ Puppeteerʹґଘ ✓ Node.jsʢJavaScriptʣ੡ https://github.com/yujiosaka/headless-chrome-crawler
  6. ࠓ೔͸࡞ͬͨ”ޙͷ”࿩

  7. GitHub > Insights > Traffic https://twitter.com/yujiosaka/status/967316514322890752

  8. GitHub Trending Repositories ͜ͷลΛ2-3೔ؒ ͏Ζ͍ͭͯͨ https://github.com/trending

  9. Hit 2000 Stars in 7 days https://github.com/yujiosaka/headless-chrome-crawler > 2000

  10. ເ͕׎ͬͨ

  11. ظ଴͍ͯͨ͠ϝϦοτ ✓ ඼࣭ͷ޲্ ✓ ։ൃྗͷ޲্ ✓ ϒϥϯυ޲্ ✓ ࢓ࣄʹͭͳ͕Δ ୭͔࢓ࣄ͘ΕΖ͍ͩ͘͞

    ҙ֎ͱࣗݾΞϐʔϧஏ͔͍ͣ͠ ·ͩλΠϙमਖ਼͔͠ૹΒΕͯͳ͍ Issue΍Βϝʔϧ΍ΒରԠ๩͍͠
  12. ࢓ࣄதͣͬͱχϠχϠͰ͖Δ

  13. ຊ୊ ଟ͘ͷਓͷ໨ʹཹ·ͬͨϥΠϒϥϦͱ ͦ͏ͳΒͳ͔ͬͨϥΠϒϥϦͷҧ͍Λ੔ཧ ※ͨͬͨαϯϓϧ̍ͷ͜ͱͳͷͰ࿩൒෼ʹฉ͍͍ͯͩ͘͞

  14. λʔήοτͷ޿͞ ✓ ӳޠͰൃ৴͢Δॏཁੑ ✓ READMEΛӳޠͰॻ͍ͨΒ͓ऴ͍Ͱ͸ͳ͍ ✓ ʰ͸ͯϒϗοτΤϯτϦʔʱΑΓ
 ʰHacker News Top

    Linksʱͷํ͕10ഒྲྀೖ͕͋ͬͨ
 
 
 

  15. ར༻ͷϋʔυϧ ✓ Ұ໨ݟͯԿ͕͍ͨ͠ͷ͔ϋοΩϦ෼͔Δ ✓ Headless Chrome + Crawler = Headless

    Chrome Crawler ✓ READMEͰϝϦοτΛҰ൪࠷ॳʹΞϐʔϧ͓ͯ͘͠ ✓ ͦΕͰ΋෼͔Βͳ͍ਓ޲͚ʹɺFAQΛ༻ҙ͢Δ ✓ ͍͍ͩͨͷਓ͸࠷ॳͷ਺ߦ͔͠ಡ·ͳ͍ ✓ ը૾΍ϩΰͰ΋ʮ؆୯͞ʯ͸ΞϐʔϧͰ͖Δ
  16. ίʔυϦʔσΟϯάͷϋʔυϧ ✓ examples Λॆ࣮ͤ͞Δ ✓ Ұ൪ಡ·Ε͍ͯͨίϯςϯπ͸ examples ͩͬͨ ✓ πʔϧ͸ϑϧ׆༻͢Δ

    ✓ ESLint ✓ commitlint ✓ EditorConfig ✓ TypeScript / JSDoc support
  17. ৴པ ✓ όοδΛϑϧ׆༻ ✓ ࠷৽ͷϏϧυঢ়گ͕Ұ໨Ͱ෼͔ΔΑ͏ʹ͢Δ ✓ άϦʔϯ͕ฒΜͰΔͱͳΜ͔҆৺͢Δ ✓ Greenkeeper࠷ڧઆ ✓

    ࠷ޙͷίϛοτ͕൒೥લͷϓϩδΣΫτͱ͔࢖͍ͨ͘ͳ͍ ✓ ͠͹Β͘αϘͬͯͯ΋ɺৗʹίϛοτͰ͖Δ
  18. Happy Niya-niya Hacking!