Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Concurrency for distributed Web crawlers
Search
Takumi Sueda
March 01, 2017
Technology
2
3.5k
Concurrency for distributed Web crawlers
golang.tokyo #4 @ eureka, Inc.
Takumi Sueda
March 01, 2017
Tweet
Share
More Decks by Takumi Sueda
See All by Takumi Sueda
令和最新版 OpenWrt を 22 年前のルーターで動かす
puhitaku
0
1.1k
組み込みLinuxの時系列
puhitaku
4
1.8k
電子辞書にステータスバーを実装する
puhitaku
0
310
知育おもちゃの遊び方 Lv. 999
puhitaku
5
3.1k
ルーターでプレゼンする
puhitaku
1
6.5k
AST as Black Magic
puhitaku
0
150
次世代の足下を舗装するコミュニティ
puhitaku
5
1.8k
電子辞書で学ぶ Linux のサスペンド
puhitaku
0
730
USB PD で迎える AC アダプター大統一時代
puhitaku
2
2.7k
Other Decks in Technology
See All in Technology
「AI駆動PO」を考えてみる - 作る速さから価値のスループットへ:検査・適応で未来を開発 / AI-driven product owner. scrummat2025
yosuke_nagai
4
670
Goにおける 生成AIによるコード生成の ベンチマーク評価入門
daisuketakeda
2
110
[2025-09-30] Databricks Genie を利用した分析基盤とデータモデリングの IVRy の現在地
wxyzzz
0
500
about #74462 go/token#FileSet
tomtwinkle
1
450
VCC 2025 Write-up
bata_24
0
180
許しとアジャイル
jnuank
1
130
pprof vs runtime/trace (FlightRecorder)
task4233
0
170
PLaMo2シリーズのvLLM実装 / PFN LLM セミナー
pfn
PRO
2
1k
AWS 잘하는 개발자 되기 - AWS 시작하기: 클라우드 개념부터 IAM까지
kimjaewook
0
110
o11yで育てる、強い内製開発組織
_awache
3
120
extension 現場で使えるXcodeショートカット一覧
ktombow
0
220
AI駆動開発を推進するためにサービス開発チームで 取り組んでいること
noayaoshiro
0
230
Featured
See All Featured
Speed Design
sergeychernyshev
32
1.1k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
The Cost Of JavaScript in 2023
addyosmani
53
9k
BBQ
matthewcrist
89
9.8k
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.7k
Art, The Web, and Tiny UX
lynnandtonic
303
21k
Facilitating Awesome Meetings
lara
56
6.6k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.7k
The World Runs on Bad Software
bkeepers
PRO
71
11k
For a Future-Friendly Web
brad_frost
180
9.9k
The Pragmatic Product Professional
lauravandoore
36
6.9k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Transcript
$PODVSSFODZGPSEJTUSJCVUFE8FCDSBXMFST HPMBOHUPLZP!FVSFLB *OD 5BLVNJ4VFEB
)J 5BLVNJ4VFEB !QVIJUBLV 'VMMFS *OD (PMBOH1ZUIPO$ 4PMEFSJOHFNCFEEFETZTUFN 8SJUJOH(PMBOHTJODF4FQUFNCFS
5PEBZˏTUPQJD JOUFSNTPGDPODVSSFODZ (PQIFS DSFBUFECZ3FOFF'SFODI $$#: GPS8FCDSBXMFST
5PEBZˏTUPQJD 8FCDSBXMFSGPSXIBU
5PEBZˏTUPQJD
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
$IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖
7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ
$IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖
7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ $SBXMUIFNBMM8JUIJOIST&WFSZEBZ
*OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN JOGPSNBOU <せ>䞔㜠䲿⣘罏 鸐濼罏 ˌGSPN薉鳤龤POUIF8&#
*OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN *OGPSNBOU $SBXMFS ˖ *ODSFBTFEFDSFBTF TDBMF DSBXMFST ˖ 1VTIMJTUPGUBSHFUBQQTJOUP"84
424 ˖ 3FQPSUTUBUVTWJB"84$MPVE8BUDI BOE4MBDL ˖ %FQMPZFEPO"84&$$POUBJOFS 4FSWJDFDBOCFTDBMFEFBTJMZ ˖ 3FUSJFWFUBSHFUBQQTGSPN424 ˖ 1VUDSBXMFETDSBQFEEBUBPO %ZOBNP%#BOE4 $PNNBOEFS
1IBTF"4DIFEVMF "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU 1VTI 1VTI 1VTI $PNNBOEFS
$PNNBOEFSQVTIFTBQQTMJTUT JOUPRVFVF "84424
1IBTF#4DBMFDSBXMFST $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFSDPVOUTOVNCFSPG BQQTUIFOTDBMFTDSBXMFST
1IBTF$$SBXMBQQT $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4 *OWPLFEDSBXMFSTCFHJODSBXMJOH SFUSJFWJOHBQQMJTUGSPNRVFVF
5IVT &BDIEJTUSJCVUFEDSBXMFSTIBWF EJFSFOUHMPCBM*1BOEDBOCF TDBMFEBSCJUSBSJMZ $BOˏUFTUJNBUFDSBXMEVSBUJPO ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖
.BOZ L UBSHFUBQQT ˖ 7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ Cleared!
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
6TFDBTF Crawl million apps every day!
$SBXMNJMMJPOBQQT4DIFNBUJD $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4
$SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT
)551 (&5 TDSBQF 1VU%# JUFN 1SPDFTTJNBHFT )551 (&5 TDSBQF "QQ "QQ 1SPDFTTUJNFPGJNBHFT EPXOMPBE DPOWFSU VQMPBE UBLFTWBSJBCMFUJNF
$SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT
$SBXMFS $BOOPUDPOUSPMFTUJNBUFDSBXMEVSBUJPO
$SBXMNJMMJPOBQQT8JUI(PSPVUJOF $SBXMFS 5JNF $SBXMEVSBUJPOJTHVBSBOUFFECZQFSJPEJDBMMZJOWPLFE(PSPVUJOFT $SBXMFS 8BJUEFTJSFEEVSBUJPOTUSJDUMZ *OWPLFOFX(PSPVUJOFFWFSZDSBXM
$SBXMNJMMJPOBQQT$PEF CFGPSF
$SBXMNJMMJPOBQQT$PEF BGUFS
6TFDBTF Continuous conversation on Slack
4MBDLDPOWFSTBUJPO$PNNBOEFS B4MBDLCPU $PNNBOEFSQSPWJEFT$IBU0QTGFBUVSF UPXBUDIDSBXMFST NJHSBUF%#
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO "QQ4UPSF64 1SPHSFTTQFSDFOUBHFJT 4JNQMFDPOWFSTBUJPOJOSFQMZ
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT )PXBCPVUDPOUJOVPVTXBUDI
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF64 TUBUVTDSBXMFS $SBXMJOH"QQ4UPSF,PSFB _NJOMBUFS_ _NJOMBUFS_
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT 4MBDLIBTˑFEJUNFTTBHF˒GFBUVSF8IZOPUVTFJU
4MBDLDPOWFSTBUJPO *NQMFNFOUFEDPOWFSTBUJPO NFTTBHF PSJFOUFE4MBDLCPU XJUIHJUIVCDPNOMPQFTTMBDL
4MBDLDPOWFSTBUJPO%FNP
$PNNBOEHPSPVUJOF *OWPLFE $POWFSTBUJPOHPSPVUJOF 4DIFNBUJD EP" 5PLFOJ[FJOWPLFDNE 1SPHSFTT9 %FBMXJUI FWFOUBOE SFBDUJPOT
&WFOU$IBO 3FBDUJPO$IBO 3FQMZTFOEFNPKJ WJBSFBDUJPODIBO 1SPHSFTT: ̕&EJU $PNNBOEOTIFE 'JOJTIFE ̕&EJU $MPTF3FBDUJPO$IBO %FTUSPZVOTVCTDSJCF
$NEHPSPVUJOF $POOFDUCFUXFFODPOWFSTBUJPOBOEHPSPVUJOF UT 1SPHSFTT9 CPEZ 1SPHSFTT9 'JSTUSFQMZ 3FBDUJPODIBO &WFOU#SPBEDBTUFS
&WFOU$IBO UJNFTUBNQ DIBO 4FOEFNPKJ 3FBDUJPODIBO 1SPHSFTT: 4FDPOESFQMZ 3FBDUJPODIBO
4MBDLDPOWFSTBUJPO$PEF 35.FWFOUTJODPNJOHGSPN4MBDL MUFSFECZ&WFOU#SPBEDBTUFS 3FQMZ FNPKJGSPNDPNNBOE *OWPLFDPNNBOE 3FBDUJPOTGSPNDPNNBOE
4MBDLDPOWFSTBUJPO$PEF *GOP54JTSFHJTUFSFE QPTUSTUUIFOBEE54 "EE54UP&WFOU#SPBEDBTUFS &EJUNFTTBHF DPOWFSTBUJPO TJODF54SFHJTUFSFE
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
1JUGBMMBOETPMVUJPO Over-consumption of TCP connection in parallelized DB operation
1JUGBMMDPOTVNF5$1DPOOFDUJPO "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFS %ZOBNP%# ,OPXOBQQUBCMF
<ˏKQGPPˏ ˏKQCBSˏ> (".&@"3$"%& (".& <ˏKQIPHFˏ ˏKQQJZPˏ> <ˏKQNJLVˏ ˏKQEBZPˏ> (".&@"%7&/563& (".&@"$5*0/ <ˏKQSJOˏ ˏKQMVLBˏ> $PNNBOEFSHFOFSBUFTBQQTMJTUGSPNBMPUPGJUFNTJOUBCMF 1BSBMMFMJ[FEXJUIHPSPVUJOFT
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: )FBMUIZTUBUF 26&3:
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ FUD 26&3:
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: 26&3: $POTVNFTBMMBWBJMBCMFDPOOFDUJPOJOXPSTUDBTF /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ
FUD
1JUGBMMDPOTVNF5$1DPOOFDUJPO GPS@ LFZSBOHFUBSHFU,FZT\ HPGVOD \ SFT FSS2VFSZ*UFN LFZ ^
UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ 2VFSZ*UFN XJMMCFDBMMFEXJUINJOJNBMTQFFEDPOUSPM DPODVSSFOUDBMMTBSFOPUDPOTJEFSFE
4PMVUJPONBLFBMJNJUXJUIDIBOOFM MJNJU$IBONBLF DIBOJOU GPS@ LFZSBOHFUBSHFU,FZT\ MJNJU$IBO HPGVOD \
SFT FSS2VFSZ*UFN LFZ MJNJU$IBO ^ UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ MJNJU$IBOCMPDLTJGMJNJU$IBOJTGVMM