Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ヘッドレスChromeでクローラを作った後の話
Search
yujiosaka
March 20, 2018
Technology
3
670
ヘッドレスChromeでクローラを作った後の話
yujiosaka
March 20, 2018
Tweet
Share
More Decks by yujiosaka
See All by yujiosaka
I was understanding WASM all wrong! 🤯
yujiosaka
2
260
Machine Learning with JavaScript
yujiosaka
0
180
JavaScriptでも機械学習がやりたかった話
yujiosaka
2
450
俺が最初にヘッドレスChromeでクローラ作った 事になんねーかな
yujiosaka
4
1.3k
『XXX』のための管理画面
yujiosaka
1
1.3k
Enjoy Deep Learning by JavaScript
yujiosaka
1
340
ひたすら楽してディープラーニング
yujiosaka
20
13k
technology x business
yujiosaka
3
570
第二回もんご祭 パネルディスカッション
yujiosaka
0
890
Other Decks in Technology
See All in Technology
Lazy application authentication with Tailscale
bluehatbrit
0
210
AI専用のリンターを作る #yumemi_patch
bengo4com
5
4.3k
Enhancing SaaS Product Reliability and Release Velocity through Optimized Testing Approach
ropqa
1
230
生まれ変わった AWS Security Hub (Preview) を紹介 #reInforce_osaka / reInforce New Security Hub
masahirokawahara
0
480
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
3
960
OSSのSNSツール「Misskey」をさわってみよう(右下ワイプで私のOSCの20年を振り返ります) / 20250705-osc2025-do
akkiesoft
0
170
オーティファイ会社紹介資料 / Autify Company Deck
autifyhq
10
130k
Reach American Airlines®️ Instantly: 19 Calling Methods for Fast Support in the USA
flyamerican
1
170
Lufthansa ®️ USA Contact Numbers: Complete 2025 Support Guide
lufthanahelpsupport
0
200
20250705 Headlamp: 專注可擴展性的 Kubernetes 用戶界面
pichuang
0
270
Backlog ユーザー棚卸しRTA、多分これが一番早いと思います
__allllllllez__
1
150
united airlines ™®️ USA Contact Numbers: Complete 2025 Support Guide
flyunitedhelp
1
340
Featured
See All Featured
Designing for Performance
lara
610
69k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
54k
Rails Girls Zürich Keynote
gr2m
95
14k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.6k
Rebuilding a faster, lazier Slack
samanthasiow
82
9.1k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.4k
Thoughts on Productivity
jonyablonski
69
4.7k
Navigating Team Friction
lara
187
15k
KATA
mclloyd
30
14k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Transcript
ϔουϨεChromeͰ ΫϩʔϥΛ࡞ͬͨ”ޙͷ” Roppongi.js #1 Yuji Isobe
Yuji Isobe @yujiosaka ϓϩδΣΫτϚωʔδϟʔ at https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu
ΫϩʔϥΛ࡞ͬͨ࣌ͷ https://speakerdeck.com/yujiosaka/an-gazui-chu-nihetudoresuchromedekurorazuo-tuta-shi-ninannekana
ϔουϨεChromeͱ ✓ Chrome͕ϔουϨεϞʔυͰىಈͰ͖Δ ✓ ChromeͷىಈΦϓγϣϯʹʮ--headessʯΛՃ͑Δ͚ͩ ✓ දతͳϔουϨεϒϥβͱ͍͑PhantomJS ✓ ߴͰ҆ఆͯ͠ಈ࡞͢Δ ✓
ඪ४ͷରԠ͕ૣ͍ʢES2017Async-Await͕͑Δʣ ✓ ओͳ༻్ςετࣗಈԽͱΫϩʔϥ
Headless Chrome Crawler ✓ ϔουϨεChromeͰΫϩʔϥ ✓ ࢄڥͰಈ࡞͢Δ ✓ ਂ͞༏ઌ୳ࡧʢDFSʣͱ ෯༏ઌ୳ࡧʢBFSʣΛαϙʔτ
✓ robots.txt, sitemap.xmlʹै͏ ✓ Puppeteerʹґଘ ✓ Node.jsʢJavaScriptʣ https://github.com/yujiosaka/headless-chrome-crawler
ࠓ࡞ͬͨ”ޙͷ”
GitHub > Insights > Traffic https://twitter.com/yujiosaka/status/967316514322890752
GitHub Trending Repositories ͜ͷลΛ2-3ؒ ͏Ζ͍ͭͯͨ https://github.com/trending
Hit 2000 Stars in 7 days https://github.com/yujiosaka/headless-chrome-crawler > 2000
ເ͕ͬͨ
ظ͍ͯͨ͠ϝϦοτ ✓ ࣭ͷ্ ✓ ։ൃྗͷ্ ✓ ϒϥϯυ্ ✓ ࣄʹͭͳ͕Δ ୭͔ࣄ͘ΕΖ͍ͩ͘͞
ҙ֎ͱࣗݾΞϐʔϧஏ͔͍ͣ͠ ·ͩλΠϙमਖ਼͔͠ૹΒΕͯͳ͍ IssueΒϝʔϧΒରԠ͍͠
ࣄதͣͬͱχϠχϠͰ͖Δ
ຊ ଟ͘ͷਓͷʹཹ·ͬͨϥΠϒϥϦͱ ͦ͏ͳΒͳ͔ͬͨϥΠϒϥϦͷҧ͍Λཧ ※ͨͬͨαϯϓϧ̍ͷ͜ͱͳͷͰʹฉ͍͍ͯͩ͘͞
λʔήοτͷ͞ ✓ ӳޠͰൃ৴͢Δॏཁੑ ✓ READMEΛӳޠͰॻ͍ͨΒ͓ऴ͍Ͱͳ͍ ✓ ʰͯϒϗοτΤϯτϦʔʱΑΓ ʰHacker News Top
Linksʱͷํ͕10ഒྲྀೖ͕͋ͬͨ
ར༻ͷϋʔυϧ ✓ ҰݟͯԿ͕͍ͨ͠ͷ͔ϋοΩϦ͔Δ ✓ Headless Chrome + Crawler = Headless
Chrome Crawler ✓ READMEͰϝϦοτΛҰ൪࠷ॳʹΞϐʔϧ͓ͯ͘͠ ✓ ͦΕͰ͔Βͳ͍ਓ͚ʹɺFAQΛ༻ҙ͢Δ ✓ ͍͍ͩͨͷਓ࠷ॳͷߦ͔͠ಡ·ͳ͍ ✓ ը૾ϩΰͰʮ؆୯͞ʯΞϐʔϧͰ͖Δ
ίʔυϦʔσΟϯάͷϋʔυϧ ✓ examples Λॆ࣮ͤ͞Δ ✓ Ұ൪ಡ·Ε͍ͯͨίϯςϯπ examples ͩͬͨ ✓ πʔϧϑϧ׆༻͢Δ
✓ ESLint ✓ commitlint ✓ EditorConfig ✓ TypeScript / JSDoc support
৴པ ✓ όοδΛϑϧ׆༻ ✓ ࠷৽ͷϏϧυঢ়گ͕ҰͰ͔ΔΑ͏ʹ͢Δ ✓ άϦʔϯ͕ฒΜͰΔͱͳΜ͔҆৺͢Δ ✓ Greenkeeper࠷ڧઆ ✓
࠷ޙͷίϛοτ͕લͷϓϩδΣΫτͱ͔͍ͨ͘ͳ͍ ✓ ͠Β͘αϘͬͯͯɺৗʹίϛοτͰ͖Δ
Happy Niya-niya Hacking!