Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
ヘッドレスChromeでクローラを作った後の話
Search
yujiosaka
March 20, 2018
Technology
770
3
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
ヘッドレスChromeでクローラを作った後の話
yujiosaka
March 20, 2018
Transcript
18 slides
ϔουϨεChromeͰ ΫϩʔϥΛ࡞ͬͨ”ޙͷ” Roppongi.js #1 Yuji Isobe
Yuji Isobe @yujiosaka ϓϩδΣΫτϚωʔδϟʔ at https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu
ΫϩʔϥΛ࡞ͬͨ࣌ͷ https://speakerdeck.com/yujiosaka/an-gazui-chu-nihetudoresuchromedekurorazuo-tuta-shi-ninannekana
ϔουϨεChromeͱ ✓ Chrome͕ϔουϨεϞʔυͰىಈͰ͖Δ ✓ ChromeͷىಈΦϓγϣϯʹʮ--headessʯΛՃ͑Δ͚ͩ ✓ දతͳϔουϨεϒϥβͱ͍͑PhantomJS ✓ ߴͰ҆ఆͯ͠ಈ࡞͢Δ ✓
ඪ४ͷରԠ͕ૣ͍ʢES2017Async-Await͕͑Δʣ ✓ ओͳ༻్ςετࣗಈԽͱΫϩʔϥ
Headless Chrome Crawler ✓ ϔουϨεChromeͰΫϩʔϥ ✓ ࢄڥͰಈ࡞͢Δ ✓ ਂ͞༏ઌ୳ࡧʢDFSʣͱ ෯༏ઌ୳ࡧʢBFSʣΛαϙʔτ
✓ robots.txt, sitemap.xmlʹै͏ ✓ Puppeteerʹґଘ ✓ Node.jsʢJavaScriptʣ https://github.com/yujiosaka/headless-chrome-crawler
ࠓ࡞ͬͨ”ޙͷ”
GitHub > Insights > Traffic https://twitter.com/yujiosaka/status/967316514322890752
GitHub Trending Repositories ͜ͷลΛ2-3ؒ ͏Ζ͍ͭͯͨ https://github.com/trending
Hit 2000 Stars in 7 days https://github.com/yujiosaka/headless-chrome-crawler > 2000
ເ͕ͬͨ
ظ͍ͯͨ͠ϝϦοτ ✓ ࣭ͷ্ ✓ ։ൃྗͷ্ ✓ ϒϥϯυ্ ✓ ࣄʹͭͳ͕Δ ୭͔ࣄ͘ΕΖ͍ͩ͘͞
ҙ֎ͱࣗݾΞϐʔϧஏ͔͍ͣ͠ ·ͩλΠϙमਖ਼͔͠ૹΒΕͯͳ͍ IssueΒϝʔϧΒରԠ͍͠
ࣄதͣͬͱχϠχϠͰ͖Δ
ຊ ଟ͘ͷਓͷʹཹ·ͬͨϥΠϒϥϦͱ ͦ͏ͳΒͳ͔ͬͨϥΠϒϥϦͷҧ͍Λཧ ※ͨͬͨαϯϓϧ̍ͷ͜ͱͳͷͰʹฉ͍͍ͯͩ͘͞
λʔήοτͷ͞ ✓ ӳޠͰൃ৴͢Δॏཁੑ ✓ READMEΛӳޠͰॻ͍ͨΒ͓ऴ͍Ͱͳ͍ ✓ ʰͯϒϗοτΤϯτϦʔʱΑΓ ʰHacker News Top
Linksʱͷํ͕10ഒྲྀೖ͕͋ͬͨ
ར༻ͷϋʔυϧ ✓ ҰݟͯԿ͕͍ͨ͠ͷ͔ϋοΩϦ͔Δ ✓ Headless Chrome + Crawler = Headless
Chrome Crawler ✓ READMEͰϝϦοτΛҰ൪࠷ॳʹΞϐʔϧ͓ͯ͘͠ ✓ ͦΕͰ͔Βͳ͍ਓ͚ʹɺFAQΛ༻ҙ͢Δ ✓ ͍͍ͩͨͷਓ࠷ॳͷߦ͔͠ಡ·ͳ͍ ✓ ը૾ϩΰͰʮ؆୯͞ʯΞϐʔϧͰ͖Δ
ίʔυϦʔσΟϯάͷϋʔυϧ ✓ examples Λॆ࣮ͤ͞Δ ✓ Ұ൪ಡ·Ε͍ͯͨίϯςϯπ examples ͩͬͨ ✓ πʔϧϑϧ׆༻͢Δ
✓ ESLint ✓ commitlint ✓ EditorConfig ✓ TypeScript / JSDoc support
৴པ ✓ όοδΛϑϧ׆༻ ✓ ࠷৽ͷϏϧυঢ়گ͕ҰͰ͔ΔΑ͏ʹ͢Δ ✓ άϦʔϯ͕ฒΜͰΔͱͳΜ͔҆৺͢Δ ✓ Greenkeeper࠷ڧઆ ✓
࠷ޙͷίϛοτ͕લͷϓϩδΣΫτͱ͔͍ͨ͘ͳ͍ ✓ ͠Β͘αϘͬͯͯɺৗʹίϛοτͰ͖Δ
Happy Niya-niya Hacking!
More Decks by yujiosaka
See All by yujiosaka
I was understanding WASM all wrong! 🤯
yujiosaka
2
370
Machine Learning with JavaScript
yujiosaka
0
250
JavaScriptでも機械学習がやりたかった話
yujiosaka
2
510
俺が最初にヘッドレスChromeでクローラ作った 事になんねーかな
yujiosaka
4
1.5k
『XXX』のための管理画面
yujiosaka
1
1.5k
Enjoy Deep Learning by JavaScript
yujiosaka
1
420
ひたすら楽してディープラーニング
yujiosaka
20
13k
technology x business
yujiosaka
3
630
第二回もんご祭 パネルディスカッション
yujiosaka
0
960
Other Decks in Technology
See All in Technology
RAGの改善を速く回すための評価基盤
yosukeyoshida
6
3.2k
AIで実装が加速したのに、 なぜ私は仕事の大半で人と話し続けていたのか?
say_no_w
0
190
Context != Context: One window, many lifecycles (AI Native DevCon, Nov 2026)
birgitta410
0
160
おうち環境でも「なるべく止めたくない!」 ― おうち環境の機器冗長化の理想と現実
yuutosi
1
600
あの日、見つかっていなかったらどうなっていたか
kazzpapa3
2
260
「任されビリティ」で引き受けて「よしなりてぃ」に頼らず任せる / Accept tasks with "makasarebility" and hand them over without relying on "yoshinarily"
pauli
1
790
LLM 基盤と kagent v0 を全部 k8s に入れてみた / Deploy both LLM and kagent v0 in k8s
kota2and3kan
0
240
目を覚ませ!僕らの認証情報が何者かに侵略されてるぞ
tkito
2
460
共通基盤の実装に Kotlin - Quarkus を選定した理由と採用したアーキテクチャ
nealle
1
130
HTTP通信に使おうとすると実はめんどくさいfile_get_contents
maguroalternative
0
220
こんなに変わった! AWSネットワークの常識
shotashiratori
16
10k
_マネジメントする人される人ナイト_2026__良いチームを作っていきたい_お気持ち_.pdf
xkxaxkx
0
800
Featured
See All Featured
AI: The stuff that nobody shows you
jnunemaker
PRO
12
1.3k
Statistics for Hackers
jakevdp
799
230k
Everyday Curiosity
cassininazir
0
400
Building a Modern Day E-commerce SEO Strategy
aleyda
45
9.3k
Testing 201, or: Great Expectations
jmmastey
46
8.4k
Faster Mobile Websites
deanohume
310
32k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
11
1.1k
Navigating the Design Leadership Dip - Product Design Week Design Leaders+ Conference 2024
apolaine
2
530
SEO for Brand Visibility & Recognition
aleyda
0
4.9k
Designing for Timeless Needs
cassininazir
1
570
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
2k
It's Worth the Effort
3n
188
29k