Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Concurrency for distributed Web crawlers
Search
Takumi Sueda
March 01, 2017
Technology
2
3.5k
Concurrency for distributed Web crawlers
golang.tokyo #4 @ eureka, Inc.
Takumi Sueda
March 01, 2017
Tweet
Share
More Decks by Takumi Sueda
See All by Takumi Sueda
組み込みLinuxの時系列
puhitaku
4
1.7k
電子辞書にステータスバーを実装する
puhitaku
0
280
知育おもちゃの遊び方 Lv. 999
puhitaku
5
3k
ルーターでプレゼンする
puhitaku
1
6k
AST as Black Magic
puhitaku
0
150
次世代の足下を舗装するコミュニティ
puhitaku
5
1.8k
電子辞書で学ぶ Linux のサスペンド
puhitaku
0
690
USB PD で迎える AC アダプター大統一時代
puhitaku
2
2.7k
電子辞書のアイデンティティを消す方法
puhitaku
3
4.1k
Other Decks in Technology
See All in Technology
スタートアップに選択肢を 〜生成AIを活用したセカンダリー事業への挑戦〜
nstock
0
270
【LT会登壇資料】TROCCO新コネクタ「スマレジ」を活用した直営店データの分析
kazari0425
1
140
[SRE NEXT] ARR150億円_エンジニア140名_27チーム_17プロダクトから始めるSLO.pdf
satos
3
1.7k
cdk initで生成されるあのファイル達は何なのか/cdk-init-generated-files
tomoki10
1
430
VS CodeとGitHub Copilotで爆速開発!アップデートの波に乗るおさらい会 / Rapid Development with VS Code and GitHub Copilot: Catch the Latest Wave
yamachu
2
300
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
soudai
PRO
54
22k
united airlines ™®️ USA Contact Numbers: Complete 2025 Support Guide
flyunitedhelp
1
470
CDK Toolkit Libraryにおけるテストの考え方
smt7174
1
370
How to Quickly Call American Airlines®️ U.S. Customer Care : Full Guide
flyaahelpguide
0
240
Delta airlines Customer®️ USA Contact Numbers: Complete 2025 Support Guide
deltahelp
0
1.1k
第4回Snowflake 金融ユーザー会 Snowflake summit recap
tamaoki
1
330
AI エージェントと考え直すデータ基盤
na0
17
6.8k
Featured
See All Featured
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
Side Projects
sachag
455
42k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
970
Embracing the Ebb and Flow
colly
86
4.7k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
50
5.5k
The Cult of Friendly URLs
andyhume
79
6.5k
Into the Great Unknown - MozCon
thekraken
40
1.9k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
Building Adaptive Systems
keathley
43
2.7k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
How STYLIGHT went responsive
nonsquared
100
5.6k
Transcript
$PODVSSFODZGPSEJTUSJCVUFE8FCDSBXMFST HPMBOHUPLZP!FVSFLB *OD 5BLVNJ4VFEB
)J 5BLVNJ4VFEB !QVIJUBLV 'VMMFS *OD (PMBOH1ZUIPO$ 4PMEFSJOHFNCFEEFETZTUFN 8SJUJOH(PMBOHTJODF4FQUFNCFS
5PEBZˏTUPQJD JOUFSNTPGDPODVSSFODZ (PQIFS DSFBUFECZ3FOFF'SFODI $$#: GPS8FCDSBXMFST
5PEBZˏTUPQJD 8FCDSBXMFSGPSXIBU
5PEBZˏTUPQJD
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
$IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖
7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ
$IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖
7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ $SBXMUIFNBMM8JUIJOIST&WFSZEBZ
*OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN JOGPSNBOU <せ>䞔㜠䲿⣘罏 鸐濼罏 ˌGSPN薉鳤龤POUIF8&#
*OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN *OGPSNBOU $SBXMFS ˖ *ODSFBTFEFDSFBTF TDBMF DSBXMFST ˖ 1VTIMJTUPGUBSHFUBQQTJOUP"84
424 ˖ 3FQPSUTUBUVTWJB"84$MPVE8BUDI BOE4MBDL ˖ %FQMPZFEPO"84&$$POUBJOFS 4FSWJDFDBOCFTDBMFEFBTJMZ ˖ 3FUSJFWFUBSHFUBQQTGSPN424 ˖ 1VUDSBXMFETDSBQFEEBUBPO %ZOBNP%#BOE4 $PNNBOEFS
1IBTF"4DIFEVMF "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU 1VTI 1VTI 1VTI $PNNBOEFS
$PNNBOEFSQVTIFTBQQTMJTUT JOUPRVFVF "84424
1IBTF#4DBMFDSBXMFST $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFSDPVOUTOVNCFSPG BQQTUIFOTDBMFTDSBXMFST
1IBTF$$SBXMBQQT $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4 *OWPLFEDSBXMFSTCFHJODSBXMJOH SFUSJFWJOHBQQMJTUGSPNRVFVF
5IVT &BDIEJTUSJCVUFEDSBXMFSTIBWF EJFSFOUHMPCBM*1BOEDBOCF TDBMFEBSCJUSBSJMZ $BOˏUFTUJNBUFDSBXMEVSBUJPO ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖
.BOZ L UBSHFUBQQT ˖ 7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ Cleared!
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
6TFDBTF Crawl million apps every day!
$SBXMNJMMJPOBQQT4DIFNBUJD $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4
$SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT
)551 (&5 TDSBQF 1VU%# JUFN 1SPDFTTJNBHFT )551 (&5 TDSBQF "QQ "QQ 1SPDFTTUJNFPGJNBHFT EPXOMPBE DPOWFSU VQMPBE UBLFTWBSJBCMFUJNF
$SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT
$SBXMFS $BOOPUDPOUSPMFTUJNBUFDSBXMEVSBUJPO
$SBXMNJMMJPOBQQT8JUI(PSPVUJOF $SBXMFS 5JNF $SBXMEVSBUJPOJTHVBSBOUFFECZQFSJPEJDBMMZJOWPLFE(PSPVUJOFT $SBXMFS 8BJUEFTJSFEEVSBUJPOTUSJDUMZ *OWPLFOFX(PSPVUJOFFWFSZDSBXM
$SBXMNJMMJPOBQQT$PEF CFGPSF
$SBXMNJMMJPOBQQT$PEF BGUFS
6TFDBTF Continuous conversation on Slack
4MBDLDPOWFSTBUJPO$PNNBOEFS B4MBDLCPU $PNNBOEFSQSPWJEFT$IBU0QTGFBUVSF UPXBUDIDSBXMFST NJHSBUF%#
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO "QQ4UPSF64 1SPHSFTTQFSDFOUBHFJT 4JNQMFDPOWFSTBUJPOJOSFQMZ
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT )PXBCPVUDPOUJOVPVTXBUDI
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF64 TUBUVTDSBXMFS $SBXMJOH"QQ4UPSF,PSFB _NJOMBUFS_ _NJOMBUFS_
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT 4MBDLIBTˑFEJUNFTTBHF˒GFBUVSF8IZOPUVTFJU
4MBDLDPOWFSTBUJPO *NQMFNFOUFEDPOWFSTBUJPO NFTTBHF PSJFOUFE4MBDLCPU XJUIHJUIVCDPNOMPQFTTMBDL
4MBDLDPOWFSTBUJPO%FNP
$PNNBOEHPSPVUJOF *OWPLFE $POWFSTBUJPOHPSPVUJOF 4DIFNBUJD EP" 5PLFOJ[FJOWPLFDNE 1SPHSFTT9 %FBMXJUI FWFOUBOE SFBDUJPOT
&WFOU$IBO 3FBDUJPO$IBO 3FQMZTFOEFNPKJ WJBSFBDUJPODIBO 1SPHSFTT: ̕&EJU $PNNBOEOTIFE 'JOJTIFE ̕&EJU $MPTF3FBDUJPO$IBO %FTUSPZVOTVCTDSJCF
$NEHPSPVUJOF $POOFDUCFUXFFODPOWFSTBUJPOBOEHPSPVUJOF UT 1SPHSFTT9 CPEZ 1SPHSFTT9 'JSTUSFQMZ 3FBDUJPODIBO &WFOU#SPBEDBTUFS
&WFOU$IBO UJNFTUBNQ DIBO 4FOEFNPKJ 3FBDUJPODIBO 1SPHSFTT: 4FDPOESFQMZ 3FBDUJPODIBO
4MBDLDPOWFSTBUJPO$PEF 35.FWFOUTJODPNJOHGSPN4MBDL MUFSFECZ&WFOU#SPBEDBTUFS 3FQMZ FNPKJGSPNDPNNBOE *OWPLFDPNNBOE 3FBDUJPOTGSPNDPNNBOE
4MBDLDPOWFSTBUJPO$PEF *GOP54JTSFHJTUFSFE QPTUSTUUIFOBEE54 "EE54UP&WFOU#SPBEDBTUFS &EJUNFTTBHF DPOWFSTBUJPO TJODF54SFHJTUFSFE
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
1JUGBMMBOETPMVUJPO Over-consumption of TCP connection in parallelized DB operation
1JUGBMMDPOTVNF5$1DPOOFDUJPO "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFS %ZOBNP%# ,OPXOBQQUBCMF
<ˏKQGPPˏ ˏKQCBSˏ> (".&@"3$"%& (".& <ˏKQIPHFˏ ˏKQQJZPˏ> <ˏKQNJLVˏ ˏKQEBZPˏ> (".&@"%7&/563& (".&@"$5*0/ <ˏKQSJOˏ ˏKQMVLBˏ> $PNNBOEFSHFOFSBUFTBQQTMJTUGSPNBMPUPGJUFNTJOUBCMF 1BSBMMFMJ[FEXJUIHPSPVUJOFT
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: )FBMUIZTUBUF 26&3:
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ FUD 26&3:
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: 26&3: $POTVNFTBMMBWBJMBCMFDPOOFDUJPOJOXPSTUDBTF /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ
FUD
1JUGBMMDPOTVNF5$1DPOOFDUJPO GPS@ LFZSBOHFUBSHFU,FZT\ HPGVOD \ SFT FSS2VFSZ*UFN LFZ ^
UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ 2VFSZ*UFN XJMMCFDBMMFEXJUINJOJNBMTQFFEDPOUSPM DPODVSSFOUDBMMTBSFOPUDPOTJEFSFE
4PMVUJPONBLFBMJNJUXJUIDIBOOFM MJNJU$IBONBLF DIBOJOU GPS@ LFZSBOHFUBSHFU,FZT\ MJNJU$IBO HPGVOD \
SFT FSS2VFSZ*UFN LFZ MJNJU$IBO ^ UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ MJNJU$IBOCMPDLTJGMJNJU$IBOJTGVMM