Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Concurrency for distributed Web crawlers

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Takumi Sueda Takumi Sueda
March 01, 2017
Technology
3.6k
2

Concurrency for distributed Web crawlers

golang.tokyo #4 @ eureka, Inc.

Avatar for Takumi Sueda

Takumi Sueda

March 01, 2017

More Decks by Takumi Sueda

See All by Takumi Sueda
これまでのハード遊びまとめ ノンストップ40分スペシャル
Avatar for Takumi Sueda puhitaku
1
230
令和最新版 OpenWrt を 22 年前のルーターで動かす
Avatar for Takumi Sueda puhitaku
0
5.8k
組み込みLinuxの時系列
Avatar for Takumi Sueda puhitaku
4
1.9k
電子辞書にステータスバーを実装する
Avatar for Takumi Sueda puhitaku
0
380
知育おもちゃの遊び方 Lv. 999
Avatar for Takumi Sueda puhitaku
5
3.3k
ルーターでプレゼンする
Avatar for Takumi Sueda puhitaku
1
7.5k
AST as Black Magic
Avatar for Takumi Sueda puhitaku
0
200
次世代の足下を舗装するコミュニティ
Avatar for Takumi Sueda puhitaku
5
1.9k
電子辞書で学ぶ Linux のサスペンド
Avatar for Takumi Sueda puhitaku
0
800

Other Decks in Technology

See All in Technology
シンデレラなんかになりたくない！ガラスの靴が割れた時代にどう歩く？
Avatar for Nomizo Nomizo nomizone
0
190
論文紹介：Pixal3D (SIGGRAPH 2026)
Avatar for TakatoYoshikawa tenten0727
0
740
実践 TanStack Start ― 新規プロダクトを開発して確立した、サーバーとクライアント境界の設計パターン / Practical TanStack Start Server-Client Boundary Patterns
Avatar for 株式会社カミナシ kaminashi
2
320
checker.tsにチキンレースを仕掛けてみた：型エラー(TS2589)が発生する境界線を求めて
Avatar for Hal hal_spidernight
1
210
AI時代から振り返るTerraform drift運用の歴史 / AI Age Reflections on the History of Terraform Drift Operations
Avatar for AEON aeonpeople
0
410
Agentic AI時代における メルカリのAIガバナンスとガードレール実装
Avatar for ichihara naoichihara
15
15k
イベントで大活躍する電子ペーパー名札 〜その３〜 / ビジュアルプログラミングIoTLT vol.23
Avatar for you（@youtoy） you
PRO
0
140
Anthropic AIネイティブ・スタートアップ構築のプレイブック を理解する
Avatar for 長津孝輔 nagatsu
0
190
20260528_生成AIを専属DSに_Howの次にすべきことを考える
Avatar for NobuakiOshiro doradora09
PRO
0
200
AI とサービス・デザイン / AI and Service Design
Avatar for Kenji Saito ks91
PRO
0
170
Node.js+TypeScriptにおけるCJS/ESM相互運用の最新ポイント
Avatar for Naoki Kiryu grainrigi
2
120
TypeScript の型で副作用の実行順序を制御する
Avatar for yanaemon yanaemon
2
210

Featured

See All Featured
Test your architecture with Archunit
Avatar for Yoan thirion
1
2.2k
Ruling the World: When Life Gets Gamed
Avatar for Sebastian Deterding codingconduct
0
240
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
133
19k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
75
12k
A brief & incomplete history of 
UX Design for the World Wide Web: 1989–2019
Avatar for Jason CranfordTeague jct
2
380
16th Malabo Montpellier Forum Presentation
Avatar for AKADEMIYA2063 akademiya2063
PRO
0
130
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
6.1k
Marketing to machines
Avatar for Jono Alderson jonoalderson
1
5.3k
HDC tutorial
Avatar for Michiel Stock michielstock
2
670
What’s in a name? Adding method to the madness
Avatar for The Alliance productmarketing
PRO
24
4k
Redefining SEO in the New Era of Traffic Generation
Avatar for Szymon Słowik szymonslowik
1
310
AI Search: Implications for SEO and How to Move Forward - #ShenzhenSEOConference
Avatar for Aleyda Solis aleyda
1
1.2k

Transcript

  1. $PODVSSFODZGPSEJTUSJCVUFE8FCDSBXMFST HPMBOHUPLZP!FVSFLB *OD 5BLVNJ4VFEB

  2. )J 5BLVNJ4VFEB !QVIJUBLV 'VMMFS *OD (PMBOH1ZUIPO$  4PMEFSJOHFNCFEEFETZTUFN 8SJUJOH(PMBOHTJODF4FQUFNCFS

  3. 5PEBZˏTUPQJD JOUFSNTPGDPODVSSFODZ (PQIFS DSFBUFECZ3FOFF'SFODI $$#: GPS8FCDSBXMFST

  4. 5PEBZˏTUPQJD 8FCDSBXMFSGPSXIBU

  5. 5PEBZˏTUPQJD

  6. 5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM  $SBXMNJMMJPOBQQTFWFSZEBZ  $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖

    1JUGBMMBOETPMVUJPO

  7. 5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM  $SBXMNJMMJPOBQQTFWFSZEBZ  $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖

    1JUGBMMBOETPMVUJPO

  8. $IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖

    7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ

  9. $IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖

    7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ $SBXMUIFNBMM8JUIJOIST&WFSZEBZ

  10. *OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN JOGPSNBOU <せ>䞔㜠䲿⣘罏 鸐濼罏 ˌGSPN薉鳤龤POUIF8&#

  11. *OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN *OGPSNBOU $SBXMFS ˖ *ODSFBTFEFDSFBTF TDBMF DSBXMFST ˖ 1VTIMJTUPGUBSHFUBQQTJOUP"84

    424 ˖ 3FQPSUTUBUVTWJB"84$MPVE8BUDI BOE4MBDL ˖ %FQMPZFEPO"84&$$POUBJOFS 4FSWJDFDBOCFTDBMFEFBTJMZ ˖ 3FUSJFWFUBSHFUBQQTGSPN424 ˖ 1VUDSBXMFETDSBQFEEBUBPO %ZOBNP%#BOE4 $PNNBOEFS

  12. 1IBTF"4DIFEVMF "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU 1VTI 1VTI 1VTI $PNNBOEFS

    $PNNBOEFSQVTIFTBQQTMJTUT JOUPRVFVF "84424 

  13. 1IBTF#4DBMFDSBXMFST $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424

    "QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFSDPVOUTOVNCFSPG BQQTUIFOTDBMFTDSBXMFST

  14. 1IBTF$$SBXMBQQT $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424

    "QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4 *OWPLFEDSBXMFSTCFHJODSBXMJOH SFUSJFWJOHBQQMJTUGSPNRVFVF

  15. 5IVT &BDIEJTUSJCVUFEDSBXMFSTIBWF EJFSFOUHMPCBM*1BOEDBOCF TDBMFEBSCJUSBSJMZ $BOˏUFTUJNBUFDSBXMEVSBUJPO ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖

    .BOZ L UBSHFUBQQT ˖ 7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ Cleared!

  16. 5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM  $SBXMNJMMJPOBQQTFWFSZEBZ  $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖

    1JUGBMMBOETPMVUJPO

  17. 6TFDBTF Crawl million apps every day!

  18. $SBXMNJMMJPOBQQT4DIFNBUJD $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424

    "QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4

  19. $SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT

    )551 (&5 TDSBQF 1VU%# JUFN 1SPDFTTJNBHFT )551 (&5 TDSBQF "QQ "QQ 1SPDFTTUJNFPGJNBHFT EPXOMPBE DPOWFSU VQMPBE  UBLFTWBSJBCMFUJNF

  20. $SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT

    $SBXMFS $BOOPUDPOUSPMFTUJNBUFDSBXMEVSBUJPO

  21. $SBXMNJMMJPOBQQT8JUI(PSPVUJOF $SBXMFS 5JNF $SBXMEVSBUJPOJTHVBSBOUFFECZQFSJPEJDBMMZJOWPLFE(PSPVUJOFT $SBXMFS 8BJUEFTJSFEEVSBUJPOTUSJDUMZ *OWPLFOFX(PSPVUJOFFWFSZDSBXM

  22. $SBXMNJMMJPOBQQT$PEF CFGPSF

  23. $SBXMNJMMJPOBQQT$PEF BGUFS

  24. 6TFDBTF Continuous conversation on Slack

  25. 4MBDLDPOWFSTBUJPO$PNNBOEFS B4MBDLCPU $PNNBOEFSQSPWJEFT$IBU0QTGFBUVSF UPXBUDIDSBXMFST NJHSBUF%# 

  26. 4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS

  27. 4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO  "QQ4UPSF64  1SPHSFTTQFSDFOUBHFJT 4JNQMFDPOWFSTBUJPOJOSFQMZ

  28. 4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT )PXBCPVUDPOUJOVPVTXBUDI 

  29. 4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF64 TUBUVTDSBXMFS $SBXMJOH"QQ4UPSF,PSFB _NJOMBUFS_ _NJOMBUFS_

  30. 4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT

  31. 4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT 

  32. 4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT 4MBDLIBTˑFEJUNFTTBHF˒GFBUVSF8IZOPUVTFJU

  33. 4MBDLDPOWFSTBUJPO *NQMFNFOUFEDPOWFSTBUJPO NFTTBHF 
 PSJFOUFE4MBDLCPU XJUIHJUIVCDPNOMPQFTTMBDL

  34. 4MBDLDPOWFSTBUJPO%FNP

  35. $PNNBOEHPSPVUJOF *OWPLFE $POWFSTBUJPOHPSPVUJOF 4DIFNBUJD EP" 5PLFOJ[FJOWPLFDNE 1SPHSFTT9 %FBMXJUI FWFOUBOE SFBDUJPOT

    &WFOU$IBO 3FBDUJPO$IBO 3FQMZTFOEFNPKJ WJBSFBDUJPODIBO  1SPHSFTT: ̕&EJU $PNNBOEOTIFE 'JOJTIFE ̕&EJU $MPTF3FBDUJPO$IBO %FTUSPZVOTVCTDSJCF

  36. $NEHPSPVUJOF $POOFDUCFUXFFODPOWFSTBUJPOBOEHPSPVUJOF  UT 1SPHSFTT9 CPEZ 1SPHSFTT9 'JSTUSFQMZ 3FBDUJPODIBO &WFOU#SPBEDBTUFS

    &WFOU$IBO  UJNFTUBNQ DIBO 4FOEFNPKJ 3FBDUJPODIBO 1SPHSFTT: 4FDPOESFQMZ 3FBDUJPODIBO

  37. 4MBDLDPOWFSTBUJPO$PEF 35.FWFOUTJODPNJOHGSPN4MBDL MUFSFECZ&WFOU#SPBEDBTUFS 3FQMZ FNPKJGSPNDPNNBOE *OWPLFDPNNBOE 3FBDUJPOTGSPNDPNNBOE

  38. 4MBDLDPOWFSTBUJPO$PEF *GOP54JTSFHJTUFSFE QPTUSTUUIFOBEE54 "EE54UP&WFOU#SPBEDBTUFS &EJUNFTTBHF DPOWFSTBUJPO TJODF54SFHJTUFSFE

  39. 5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM  $SBXMNJMMJPOBQQTFWFSZEBZ  $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖

    1JUGBMMBOETPMVUJPO

  40. 1JUGBMMBOETPMVUJPO Over-consumption of TCP connection in parallelized DB operation

  41. 1JUGBMMDPOTVNF5$1DPOOFDUJPO "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFS %ZOBNP%# ,OPXOBQQUBCMF 

    <ˏKQGPPˏ ˏKQCBSˏ>  (".&@"3$"%& (".& <ˏKQIPHFˏ ˏKQQJZPˏ>  <ˏKQNJLVˏ ˏKQEBZPˏ>  (".&@"%7&/563& (".&@"$5*0/ <ˏKQSJOˏ ˏKQMVLBˏ>  $PNNBOEFSHFOFSBUFTBQQTMJTUGSPNBMPUPGJUFNTJOUBCMF 1BSBMMFMJ[FEXJUIHPSPVUJOFT

  42. 1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: )FBMUIZTUBUF 26&3:

  43. 1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ FUD 26&3:

  44. 1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: 26&3: $POTVNFTBMMBWBJMBCMFDPOOFDUJPOJOXPSTUDBTF /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ

    FUD

  45. 1JUGBMMDPOTVNF5$1DPOOFDUJPO GPS@ LFZSBOHFUBSHFU,FZT\ HPGVOD \ SFT FSS2VFSZ*UFN LFZ  ^

      UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ 2VFSZ*UFN XJMMCFDBMMFEXJUINJOJNBMTQFFEDPOUSPM DPODVSSFOUDBMMTBSFOPUDPOTJEFSFE

  46. 4PMVUJPONBLFBMJNJUXJUIDIBOOFM MJNJU$IBONBLF DIBOJOU   GPS@ LFZSBOHFUBSHFU,FZT\ MJNJU$IBO HPGVOD \

    SFT FSS2VFSZ*UFN LFZ  MJNJU$IBO ^   UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ MJNJU$IBOCMPDLTJGMJNJU$IBOJTGVMM