Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Concurrency for distributed Web crawlers
Search
Takumi Sueda
PRO
March 01, 2017
Technology
2
3.5k
Concurrency for distributed Web crawlers
golang.tokyo #4 @ eureka, Inc.
Takumi Sueda
PRO
March 01, 2017
Tweet
Share
More Decks by Takumi Sueda
See All by Takumi Sueda
令和最新版 OpenWrt を 22 年前のルーターで動かす
puhitaku
PRO
0
820
組み込みLinuxの時系列
puhitaku
PRO
4
1.7k
電子辞書にステータスバーを実装する
puhitaku
PRO
0
300
知育おもちゃの遊び方 Lv. 999
puhitaku
PRO
5
3.1k
ルーターでプレゼンする
puhitaku
PRO
1
6.3k
AST as Black Magic
puhitaku
PRO
0
150
次世代の足下を舗装するコミュニティ
puhitaku
PRO
5
1.8k
電子辞書で学ぶ Linux のサスペンド
puhitaku
PRO
0
710
USB PD で迎える AC アダプター大統一時代
puhitaku
PRO
2
2.7k
Other Decks in Technology
See All in Technology
会社にデータエンジニアがいることでできるようになること
10xinc
9
1.6k
あなたの知らない OneDrive
murachiakira
0
240
広島発!スタートアップ開発の裏側
tsankyo
0
250
トヨタ生産方式(TPS)入門
recruitengineers
PRO
4
440
JOAI発表資料 @ 関東kaggler会
joai_committee
1
390
自社製CMSからmicroCMSへのリプレースがプロダクトグロースを加速させた話
nextbeatdev
0
150
microCMS 最新リリース情報(microCMS Meetup 2025)
microcms
0
120
見てわかるテスト駆動開発
recruitengineers
PRO
6
740
事業価値と Engineering
recruitengineers
PRO
3
930
【 LLMエンジニアがヒューマノイド開発に挑んでみた 】 - 第104回 Machine Learning 15minutes! Hybrid
soneo1127
0
120
Goss: Faiss向けの新しい本番環境対応 Goバインディング #coefl_go_jp
bengo4com
0
1.4k
Yahoo!広告ビジネス基盤におけるバックエンド開発
lycorptech_jp
PRO
1
280
Featured
See All Featured
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.8k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
4 Signs Your Business is Dying
shpigford
184
22k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Visualization
eitanlees
147
16k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.6k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.5k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Facilitating Awesome Meetings
lara
55
6.5k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Code Review Best Practice
trishagee
70
19k
Transcript
$PODVSSFODZGPSEJTUSJCVUFE8FCDSBXMFST HPMBOHUPLZP!FVSFLB *OD 5BLVNJ4VFEB
)J 5BLVNJ4VFEB !QVIJUBLV 'VMMFS *OD (PMBOH1ZUIPO$ 4PMEFSJOHFNCFEEFETZTUFN 8SJUJOH(PMBOHTJODF4FQUFNCFS
5PEBZˏTUPQJD JOUFSNTPGDPODVSSFODZ (PQIFS DSFBUFECZ3FOFF'SFODI $$#: GPS8FCDSBXMFST
5PEBZˏTUPQJD 8FCDSBXMFSGPSXIBU
5PEBZˏTUPQJD
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
$IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖
7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ
$IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖
7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ $SBXMUIFNBMM8JUIJOIST&WFSZEBZ
*OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN JOGPSNBOU <せ>䞔㜠䲿⣘罏 鸐濼罏 ˌGSPN薉鳤龤POUIF8&#
*OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN *OGPSNBOU $SBXMFS ˖ *ODSFBTFEFDSFBTF TDBMF DSBXMFST ˖ 1VTIMJTUPGUBSHFUBQQTJOUP"84
424 ˖ 3FQPSUTUBUVTWJB"84$MPVE8BUDI BOE4MBDL ˖ %FQMPZFEPO"84&$$POUBJOFS 4FSWJDFDBOCFTDBMFEFBTJMZ ˖ 3FUSJFWFUBSHFUBQQTGSPN424 ˖ 1VUDSBXMFETDSBQFEEBUBPO %ZOBNP%#BOE4 $PNNBOEFS
1IBTF"4DIFEVMF "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU 1VTI 1VTI 1VTI $PNNBOEFS
$PNNBOEFSQVTIFTBQQTMJTUT JOUPRVFVF "84424
1IBTF#4DBMFDSBXMFST $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFSDPVOUTOVNCFSPG BQQTUIFOTDBMFTDSBXMFST
1IBTF$$SBXMBQQT $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4 *OWPLFEDSBXMFSTCFHJODSBXMJOH SFUSJFWJOHBQQMJTUGSPNRVFVF
5IVT &BDIEJTUSJCVUFEDSBXMFSTIBWF EJFSFOUHMPCBM*1BOEDBOCF TDBMFEBSCJUSBSJMZ $BOˏUFTUJNBUFDSBXMEVSBUJPO ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖
.BOZ L UBSHFUBQQT ˖ 7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ Cleared!
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
6TFDBTF Crawl million apps every day!
$SBXMNJMMJPOBQQT4DIFNBUJD $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4
$SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT
)551 (&5 TDSBQF 1VU%# JUFN 1SPDFTTJNBHFT )551 (&5 TDSBQF "QQ "QQ 1SPDFTTUJNFPGJNBHFT EPXOMPBE DPOWFSU VQMPBE UBLFTWBSJBCMFUJNF
$SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT
$SBXMFS $BOOPUDPOUSPMFTUJNBUFDSBXMEVSBUJPO
$SBXMNJMMJPOBQQT8JUI(PSPVUJOF $SBXMFS 5JNF $SBXMEVSBUJPOJTHVBSBOUFFECZQFSJPEJDBMMZJOWPLFE(PSPVUJOFT $SBXMFS 8BJUEFTJSFEEVSBUJPOTUSJDUMZ *OWPLFOFX(PSPVUJOFFWFSZDSBXM
$SBXMNJMMJPOBQQT$PEF CFGPSF
$SBXMNJMMJPOBQQT$PEF BGUFS
6TFDBTF Continuous conversation on Slack
4MBDLDPOWFSTBUJPO$PNNBOEFS B4MBDLCPU $PNNBOEFSQSPWJEFT$IBU0QTGFBUVSF UPXBUDIDSBXMFST NJHSBUF%#
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO "QQ4UPSF64 1SPHSFTTQFSDFOUBHFJT 4JNQMFDPOWFSTBUJPOJOSFQMZ
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT )PXBCPVUDPOUJOVPVTXBUDI
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF64 TUBUVTDSBXMFS $SBXMJOH"QQ4UPSF,PSFB _NJOMBUFS_ _NJOMBUFS_
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT 4MBDLIBTˑFEJUNFTTBHF˒GFBUVSF8IZOPUVTFJU
4MBDLDPOWFSTBUJPO *NQMFNFOUFEDPOWFSTBUJPO NFTTBHF PSJFOUFE4MBDLCPU XJUIHJUIVCDPNOMPQFTTMBDL
4MBDLDPOWFSTBUJPO%FNP
$PNNBOEHPSPVUJOF *OWPLFE $POWFSTBUJPOHPSPVUJOF 4DIFNBUJD EP" 5PLFOJ[FJOWPLFDNE 1SPHSFTT9 %FBMXJUI FWFOUBOE SFBDUJPOT
&WFOU$IBO 3FBDUJPO$IBO 3FQMZTFOEFNPKJ WJBSFBDUJPODIBO 1SPHSFTT: ̕&EJU $PNNBOEOTIFE 'JOJTIFE ̕&EJU $MPTF3FBDUJPO$IBO %FTUSPZVOTVCTDSJCF
$NEHPSPVUJOF $POOFDUCFUXFFODPOWFSTBUJPOBOEHPSPVUJOF UT 1SPHSFTT9 CPEZ 1SPHSFTT9 'JSTUSFQMZ 3FBDUJPODIBO &WFOU#SPBEDBTUFS
&WFOU$IBO UJNFTUBNQ DIBO 4FOEFNPKJ 3FBDUJPODIBO 1SPHSFTT: 4FDPOESFQMZ 3FBDUJPODIBO
4MBDLDPOWFSTBUJPO$PEF 35.FWFOUTJODPNJOHGSPN4MBDL MUFSFECZ&WFOU#SPBEDBTUFS 3FQMZ FNPKJGSPNDPNNBOE *OWPLFDPNNBOE 3FBDUJPOTGSPNDPNNBOE
4MBDLDPOWFSTBUJPO$PEF *GOP54JTSFHJTUFSFE QPTUSTUUIFOBEE54 "EE54UP&WFOU#SPBEDBTUFS &EJUNFTTBHF DPOWFSTBUJPO TJODF54SFHJTUFSFE
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
1JUGBMMBOETPMVUJPO Over-consumption of TCP connection in parallelized DB operation
1JUGBMMDPOTVNF5$1DPOOFDUJPO "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFS %ZOBNP%# ,OPXOBQQUBCMF
<ˏKQGPPˏ ˏKQCBSˏ> (".&@"3$"%& (".& <ˏKQIPHFˏ ˏKQQJZPˏ> <ˏKQNJLVˏ ˏKQEBZPˏ> (".&@"%7&/563& (".&@"$5*0/ <ˏKQSJOˏ ˏKQMVLBˏ> $PNNBOEFSHFOFSBUFTBQQTMJTUGSPNBMPUPGJUFNTJOUBCMF 1BSBMMFMJ[FEXJUIHPSPVUJOFT
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: )FBMUIZTUBUF 26&3:
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ FUD 26&3:
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: 26&3: $POTVNFTBMMBWBJMBCMFDPOOFDUJPOJOXPSTUDBTF /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ
FUD
1JUGBMMDPOTVNF5$1DPOOFDUJPO GPS@ LFZSBOHFUBSHFU,FZT\ HPGVOD \ SFT FSS2VFSZ*UFN LFZ ^
UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ 2VFSZ*UFN XJMMCFDBMMFEXJUINJOJNBMTQFFEDPOUSPM DPODVSSFOUDBMMTBSFOPUDPOTJEFSFE
4PMVUJPONBLFBMJNJUXJUIDIBOOFM MJNJU$IBONBLF DIBOJOU GPS@ LFZSBOHFUBSHFU,FZT\ MJNJU$IBO HPGVOD \
SFT FSS2VFSZ*UFN LFZ MJNJU$IBO ^ UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ MJNJU$IBOCMPDLTJGMJNJU$IBOJTGVMM