Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Concurrency for distributed Web crawlers
Search
Takumi Sueda
March 01, 2017
Technology
2
3.4k
Concurrency for distributed Web crawlers
golang.tokyo #4 @ eureka, Inc.
Takumi Sueda
March 01, 2017
Tweet
Share
More Decks by Takumi Sueda
See All by Takumi Sueda
組み込みLinuxの時系列
puhitaku
4
1.5k
電子辞書にステータスバーを実装する
puhitaku
0
230
知育おもちゃの遊び方 Lv. 999
puhitaku
5
2.9k
ルーターでプレゼンする
puhitaku
1
5.3k
AST as Black Magic
puhitaku
0
130
次世代の足下を舗装するコミュニティ
puhitaku
5
1.8k
電子辞書で学ぶ Linux のサスペンド
puhitaku
0
650
USB PD で迎える AC アダプター大統一時代
puhitaku
2
2.6k
電子辞書のアイデンティティを消す方法
puhitaku
3
4k
Other Decks in Technology
See All in Technology
依存パッケージの更新はコツコツが勝つコツ! / phpcon_nagoya2025
blue_goheimochi
3
180
クラウドサービス事業者におけるOSS
tagomoris
3
960
明日からできる!技術的負債の返済を加速するための実践ガイド~『ホットペッパービューティー』の事例をもとに~
recruitengineers
PRO
3
530
利用終了したドメイン名の最強終活〜観測環境を育てて、分析・供養している件〜 / The Ultimate End-of-Life Preparation for Discontinued Domain Names
nttcom
2
330
データエンジニアリング領域におけるDuckDBのユースケース
chanyou0311
2
170
エンジニアの育成を支える爆速フィードバック文化
sansantech
PRO
3
1.1k
Perlの生きのこり - エンジニアがこの先生きのこるためのカンファレンス2025
kfly8
1
230
Cloud Spanner 導入で実現した快適な開発と運用について
colopl
1
920
Share my, our lessons from the road to re:Invent
naospon
0
110
Classmethod AI Talks(CATs) #17 司会進行スライド(2025.02.19) / classmethod-ai-talks-aka-cats_moderator-slides_vol17_2025-02-19
shinyaa31
0
160
2/18/25: Java meets AI: Build LLM-Powered Apps with LangChain4j
edeandrea
PRO
0
150
Exadata Database Service on Cloud@Customer セキュリティ、ネットワーク、および管理について
oracle4engineer
PRO
1
1.5k
Featured
See All Featured
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
2.1k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
GitHub's CSS Performance
jonrohan
1030
460k
Fashionably flexible responsive web design (full day workshop)
malarkey
406
66k
Why You Should Never Use an ORM
jnunemaker
PRO
55
9.2k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Building an army of robots
kneath
303
45k
What's in a price? How to price your products and services
michaelherold
244
12k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
49k
Transcript
$PODVSSFODZGPSEJTUSJCVUFE8FCDSBXMFST HPMBOHUPLZP!FVSFLB *OD 5BLVNJ4VFEB
)J 5BLVNJ4VFEB !QVIJUBLV 'VMMFS *OD (PMBOH1ZUIPO$ 4PMEFSJOHFNCFEEFETZTUFN 8SJUJOH(PMBOHTJODF4FQUFNCFS
5PEBZˏTUPQJD JOUFSNTPGDPODVSSFODZ (PQIFS DSFBUFECZ3FOFF'SFODI $$#: GPS8FCDSBXMFST
5PEBZˏTUPQJD 8FCDSBXMFSGPSXIBU
5PEBZˏTUPQJD
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
$IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖
7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ
$IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖
7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ $SBXMUIFNBMM8JUIJOIST&WFSZEBZ
*OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN JOGPSNBOU <せ>䞔㜠䲿⣘罏 鸐濼罏 ˌGSPN薉鳤龤POUIF8&#
*OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN *OGPSNBOU $SBXMFS ˖ *ODSFBTFEFDSFBTF TDBMF DSBXMFST ˖ 1VTIMJTUPGUBSHFUBQQTJOUP"84
424 ˖ 3FQPSUTUBUVTWJB"84$MPVE8BUDI BOE4MBDL ˖ %FQMPZFEPO"84&$$POUBJOFS 4FSWJDFDBOCFTDBMFEFBTJMZ ˖ 3FUSJFWFUBSHFUBQQTGSPN424 ˖ 1VUDSBXMFETDSBQFEEBUBPO %ZOBNP%#BOE4 $PNNBOEFS
1IBTF"4DIFEVMF "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU 1VTI 1VTI 1VTI $PNNBOEFS
$PNNBOEFSQVTIFTBQQTMJTUT JOUPRVFVF "84424
1IBTF#4DBMFDSBXMFST $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFSDPVOUTOVNCFSPG BQQTUIFOTDBMFTDSBXMFST
1IBTF$$SBXMBQQT $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4 *OWPLFEDSBXMFSTCFHJODSBXMJOH SFUSJFWJOHBQQMJTUGSPNRVFVF
5IVT &BDIEJTUSJCVUFEDSBXMFSTIBWF EJFSFOUHMPCBM*1BOEDBOCF TDBMFEBSCJUSBSJMZ $BOˏUFTUJNBUFDSBXMEVSBUJPO ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖
.BOZ L UBSHFUBQQT ˖ 7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ Cleared!
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
6TFDBTF Crawl million apps every day!
$SBXMNJMMJPOBQQT4DIFNBUJD $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424
"QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4
$SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT
)551 (&5 TDSBQF 1VU%# JUFN 1SPDFTTJNBHFT )551 (&5 TDSBQF "QQ "QQ 1SPDFTTUJNFPGJNBHFT EPXOMPBE DPOWFSU VQMPBE UBLFTWBSJBCMFUJNF
$SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT
$SBXMFS $BOOPUDPOUSPMFTUJNBUFDSBXMEVSBUJPO
$SBXMNJMMJPOBQQT8JUI(PSPVUJOF $SBXMFS 5JNF $SBXMEVSBUJPOJTHVBSBOUFFECZQFSJPEJDBMMZJOWPLFE(PSPVUJOFT $SBXMFS 8BJUEFTJSFEEVSBUJPOTUSJDUMZ *OWPLFOFX(PSPVUJOFFWFSZDSBXM
$SBXMNJMMJPOBQQT$PEF CFGPSF
$SBXMNJMMJPOBQQT$PEF BGUFS
6TFDBTF Continuous conversation on Slack
4MBDLDPOWFSTBUJPO$PNNBOEFS B4MBDLCPU $PNNBOEFSQSPWJEFT$IBU0QTGFBUVSF UPXBUDIDSBXMFST NJHSBUF%#
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO "QQ4UPSF64 1SPHSFTTQFSDFOUBHFJT 4JNQMFDPOWFSTBUJPOJOSFQMZ
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT )PXBCPVUDPOUJOVPVTXBUDI
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF64 TUBUVTDSBXMFS $SBXMJOH"QQ4UPSF,PSFB _NJOMBUFS_ _NJOMBUFS_
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT
4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT 4MBDLIBTˑFEJUNFTTBHF˒GFBUVSF8IZOPUVTFJU
4MBDLDPOWFSTBUJPO *NQMFNFOUFEDPOWFSTBUJPO NFTTBHF PSJFOUFE4MBDLCPU XJUIHJUIVCDPNOMPQFTTMBDL
4MBDLDPOWFSTBUJPO%FNP
$PNNBOEHPSPVUJOF *OWPLFE $POWFSTBUJPOHPSPVUJOF 4DIFNBUJD EP" 5PLFOJ[FJOWPLFDNE 1SPHSFTT9 %FBMXJUI FWFOUBOE SFBDUJPOT
&WFOU$IBO 3FBDUJPO$IBO 3FQMZTFOEFNPKJ WJBSFBDUJPODIBO 1SPHSFTT: ̕&EJU $PNNBOEOTIFE 'JOJTIFE ̕&EJU $MPTF3FBDUJPO$IBO %FTUSPZVOTVCTDSJCF
$NEHPSPVUJOF $POOFDUCFUXFFODPOWFSTBUJPOBOEHPSPVUJOF UT 1SPHSFTT9 CPEZ 1SPHSFTT9 'JSTUSFQMZ 3FBDUJPODIBO &WFOU#SPBEDBTUFS
&WFOU$IBO UJNFTUBNQ DIBO 4FOEFNPKJ 3FBDUJPODIBO 1SPHSFTT: 4FDPOESFQMZ 3FBDUJPODIBO
4MBDLDPOWFSTBUJPO$PEF 35.FWFOUTJODPNJOHGSPN4MBDL MUFSFECZ&WFOU#SPBEDBTUFS 3FQMZ FNPKJGSPNDPNNBOE *OWPLFDPNNBOE 3FBDUJPOTGSPNDPNNBOE
4MBDLDPOWFSTBUJPO$PEF *GOP54JTSFHJTUFSFE QPTUSTUUIFOBEE54 "EE54UP&WFOU#SPBEDBTUFS &EJUNFTTBHF DPOWFSTBUJPO TJODF54SFHJTUFSFE
5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖
1JUGBMMBOETPMVUJPO
1JUGBMMBOETPMVUJPO Over-consumption of TCP connection in parallelized DB operation
1JUGBMMDPOTVNF5$1DPOOFDUJPO "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFS %ZOBNP%# ,OPXOBQQUBCMF
<ˏKQGPPˏ ˏKQCBSˏ> (".&@"3$"%& (".& <ˏKQIPHFˏ ˏKQQJZPˏ> <ˏKQNJLVˏ ˏKQEBZPˏ> (".&@"%7&/563& (".&@"$5*0/ <ˏKQSJOˏ ˏKQMVLBˏ> $PNNBOEFSHFOFSBUFTBQQTMJTUGSPNBMPUPGJUFNTJOUBCMF 1BSBMMFMJ[FEXJUIHPSPVUJOFT
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: )FBMUIZTUBUF 26&3:
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ FUD 26&3:
1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: 26&3: $POTVNFTBMMBWBJMBCMFDPOOFDUJPOJOXPSTUDBTF /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ
FUD
1JUGBMMDPOTVNF5$1DPOOFDUJPO GPS@ LFZSBOHFUBSHFU,FZT\ HPGVOD \ SFT FSS2VFSZ*UFN LFZ ^
UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ 2VFSZ*UFN XJMMCFDBMMFEXJUINJOJNBMTQFFEDPOUSPM DPODVSSFOUDBMMTBSFOPUDPOTJEFSFE
4PMVUJPONBLFBMJNJUXJUIDIBOOFM MJNJU$IBONBLF DIBOJOU GPS@ LFZSBOHFUBSHFU,FZT\ MJNJU$IBO HPGVOD \
SFT FSS2VFSZ*UFN LFZ MJNJU$IBO ^ UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ MJNJU$IBOCMPDLTJGMJNJU$IBOJTGVMM