Concurrency for distributed Web crawlers

by Takumi Sueda

Slide 1

Slide 1 text

$PODVSSFODZGPSEJTUSJCVUFE8FCDSBXMFST HPMBOHUPLZP!FVSFLB *OD 5BLVNJ4VFEB

Slide 2

Slide 2 text

)J 5BLVNJ4VFEB !QVIJUBLV 'VMMFS *OD (PMBOH1ZUIPO$ 4PMEFSJOHFNCFEEFETZTUFN 8SJUJOH(PMBOHTJODF4FQUFNCFS

Slide 3

Slide 3 text

5PEBZˏTUPQJD JOUFSNTPGDPODVSSFODZ (PQIFS DSFBUFECZ3FOFF'SFODI $$#: GPS8FCDSBXMFST

Slide 4

Slide 4 text

5PEBZˏTUPQJD 8FCDSBXMFSGPSXIBU

Slide 5

Slide 5 text

5PEBZˏTUPQJD

Slide 6

Slide 6 text

5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖ 1JUGBMMBOETPMVUJPO

Slide 7

Slide 7 text

5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖ 1JUGBMMBOETPMVUJPO

Slide 8

Slide 8 text

$IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖ 7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ

Slide 9

Slide 9 text

$IBSBDUFSJTUJDTPGDSBXMUBSHFU ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖ 7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ $SBXMUIFNBMM8JUIJOIST&WFSZEBZ

Slide 10

Slide 10 text

*OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN JOGPSNBOU <せ>䞔㜠䲿⣘罏鸐濼罏 ˌGSPN薉鳤龤POUIF8

Slide 11

Slide 11 text

*OGPSNBOU UIFEJTUSJCVUFEDSBXMFSTZTUFN *OGPSNBOU $SBXMFS ˖ *ODSFBTFEFDSFBTF TDBMF DSBXMFST ˖ 1VTIMJTUPGUBSHFUBQQTJOUP"84 424 ˖ 3FQPSUTUBUVTWJB"84$MPVE8BUDI BOE4MBDL ˖ %FQMPZFEPO"84&$$POUBJOFS 4FSWJDFDBOCFTDBMFEFBTJMZ ˖ 3FUSJFWFUBSHFUBQQTGSPN424 ˖ 1VUDSBXMFETDSBQFEEBUBPO %ZOBNP%#BOE4 $PNNBOEFS

Slide 12

Slide 12 text

1IBTF"4DIFEVMF "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU 1VTI 1VTI 1VTI $PNNBOEFS $PNNBOEFSQVTIFTBQQTMJTUT JOUPRVFVF "84424

Slide 13

Slide 13 text

1IBTF#4DBMFDSBXMFST $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFSDPVOUTOVNCFSPG BQQTUIFOTDBMFTDSBXMFST

Slide 14

Slide 14 text

1IBTF$$SBXMBQQT $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4 *OWPLFEDSBXMFSTCFHJODSBXMJOH SFUSJFWJOHBQQMJTUGSPNRVFVF

Slide 15

Slide 15 text

5IVT &BDIEJTUSJCVUFEDSBXMFSTIBWF EJFSFOUHMPCBM*1BOEDBOCF TDBMFEBSCJUSBSJMZ $BOˏUFTUJNBUFDSBXMEVSBUJPO ˖ 4JOHMFEPNBJO /PUUPNBLF)551(&5PPEPSZPVˏMM CFCBOOFEGSPNBDDFTT ˖ .BOZ L UBSHFUBQQT ˖ 7BSJBCMFDPOUFOUTˏTJ[F *ODMVEJOHJNBHFTEPXOMPBEUJNF NBZWBSZ Cleared!

Slide 16

Slide 16 text

5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖ 1JUGBMMBOETPMVUJPO

Slide 17

Slide 17 text

6TFDBTF Crawl million apps every day!

Slide 18

Slide 18 text

$SBXMNJMMJPOBQQT4DIFNBUJD $PNNBOEFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS &$*OTUBODF $SBXMFS "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU 8FC %ZOBNP%# 4

Slide 19

Slide 19 text

$SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT )551 (&5 TDSBQF 1VU%# JUFN 1SPDFTTJNBHFT )551 (&5 TDSBQF "QQ "QQ 1SPDFTTUJNFPGJNBHFT EPXOMPBE DPOWFSU VQMPBE UBLFTWBSJBCMFUJNF

Slide 20

Slide 20 text

$SBXMNJMMJPOBQQT8JUIPVU(PSPVUJOF )551 (&5 TDSBQF $SBXMFS 5JNF 1VU%# JUFN 1SPDFTT JNBHFT $SBXMFS $BOOPUDPOUSPMFTUJNBUFDSBXMEVSBUJPO

Slide 21

Slide 21 text

$SBXMNJMMJPOBQQT8JUI(PSPVUJOF $SBXMFS 5JNF $SBXMEVSBUJPOJTHVBSBOUFFECZQFSJPEJDBMMZJOWPLFE(PSPVUJOFT $SBXMFS 8BJUEFTJSFEEVSBUJPOTUSJDUMZ *OWPLFOFX(PSPVUJOFFWFSZDSBXM

Slide 22

Slide 22 text

$SBXMNJMMJPOBQQT$PEF CFGPSF

Slide 23

Slide 23 text

$SBXMNJMMJPOBQQT$PEF BGUFS

Slide 24

Slide 24 text

6TFDBTF Continuous conversation on Slack

Slide 25

Slide 25 text

4MBDLDPOWFSTBUJPO$PNNBOEFS B4MBDLCPU $PNNBOEFSQSPWJEFT$IBU0QTGFBUVSF UPXBUDIDSBXMFST NJHSBUF%# 

Slide 26

Slide 26 text

4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS

Slide 27

Slide 27 text

4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO "QQ4UPSF64  1SPHSFTTQFSDFOUBHFJT 4JNQMFDPOWFSTBUJPOJOSFQMZ

Slide 28

Slide 28 text

4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT )PXBCPVUDPOUJOVPVTXBUDI

Slide 29

Slide 29 text

4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF+BQBO TUBUVTDSBXMFS $SBXMJOH1MBZ4UPSF64 TUBUVTDSBXMFS $SBXMJOH"QQ4UPSF,PSFB _NJOMBUFS_ _NJOMBUFS_

Slide 30

Slide 30 text

4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT

Slide 31

Slide 31 text

4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT 

Slide 32

Slide 32 text

4MBDLDPOWFSTBUJPO$IBU0QTFYBNQMFT 4MBDLIBTˑFEJUNFTTBHF˒GFBUVSF8IZOPUVTFJU

Slide 33

Slide 33 text

4MBDLDPOWFSTBUJPO *NQMFNFOUFEDPOWFSTBUJPO NFTTBHF   PSJFOUFE4MBDLCPU XJUIHJUIVCDPNOMPQFTTMBDL

Slide 34

Slide 34 text

4MBDLDPOWFSTBUJPO%FNP

Slide 35

Slide 35 text

$PNNBOEHPSPVUJOF *OWPLFE $POWFSTBUJPOHPSPVUJOF 4DIFNBUJD EP" 5PLFOJ[FJOWPLFDNE 1SPHSFTT9 %FBMXJUI FWFOUBOE SFBDUJPOT &WFOU$IBO 3FBDUJPO$IBO 3FQMZTFOEFNPKJ WJBSFBDUJPODIBO  1SPHSFTT: ̕&EJU $PNNBOEOTIFE 'JOJTIFE ̕&EJU $MPTF3FBDUJPO$IBO %FTUSPZVOTVCTDSJCF

Slide 36

Slide 36 text

$NEHPSPVUJOF $POOFDUCFUXFFODPOWFSTBUJPOBOEHPSPVUJOF UT 1SPHSFTT9 CPEZ 1SPHSFTT9 'JSTUSFQMZ 3FBDUJPODIBO &WFOU#SPBEDBTUFS &WFOU$IBO UJNFTUBNQ DIBO 4FOEFNPKJ 3FBDUJPODIBO 1SPHSFTT: 4FDPOESFQMZ 3FBDUJPODIBO

Slide 37

Slide 37 text

4MBDLDPOWFSTBUJPO$PEF 35.FWFOUTJODPNJOHGSPN4MBDL MUFSFECZ&WFOU#SPBEDBTUFS 3FQMZFNPKJGSPNDPNNBOE *OWPLFDPNNBOE 3FBDUJPOTGSPNDPNNBOE

Slide 38

Slide 38 text

4MBDLDPOWFSTBUJPO$PEF *GOP54JTSFHJTUFSFE QPTUSTUUIFOBEE54 "EE54UP&WFOU#SPBEDBTUFS &EJUNFTTBHF DPOWFSTBUJPO TJODF54SFHJTUFSFE

Slide 39

Slide 39 text

5PEBZˏTUPQJD ˖ #SJFGFYQMBOBUJPOPGUIFDSBXMFSTZTUFN ˖ )PXXFVTF(PSPVUJOFBOE$IBOOFM $SBXMNJMMJPOBQQTFWFSZEBZ $POUJOVPVTDPOWFSTBUJPOPO4MBDL ˖ 1JUGBMMBOETPMVUJPO

Slide 40

Slide 40 text

1JUGBMMBOETPMVUJPO Over-consumption of TCP connection in parallelized DB operation

Slide 41

Slide 41 text

1JUGBMMDPOTVNF5$1DPOOFDUJPO "QQTMJTU 424 "QQTMJTU "QQTMJTU "QQTMJTU $PNNBOEFS %ZOBNP%# ,OPXOBQQUBCMF <ˏKQGPPˏ ˏKQCBSˏ> (".&@"3$"%& (".& <ˏKQIPHFˏ ˏKQQJZPˏ> <ˏKQNJLVˏ ˏKQEBZPˏ> (".&@"%7&/563& (".&@"$5*0/ <ˏKQSJOˏ ˏKQMVLBˏ>  $PNNBOEFSHFOFSBUFTBQQTMJTUGSPNBMPUPGJUFNTJOUBCMF 1BSBMMFMJ[FEXJUIHPSPVUJOFT

Slide 42

Slide 42 text

1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: )FBMUIZTUBUF 26&3:

Slide 43

Slide 43 text

1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ FUD 26&3:

Slide 44

Slide 44 text

1JUGBMMDPOTVNF5$1DPOOFDUJPO $PNNBOEFS 26&3: 26&3: 26&3: 26&3: $POTVNFTBMMBWBJMBCMFDPOOFDUJPOJOXPSTUDBTF /PUIFBMUIZTUBUF TMPXJOUFSOFU TMPXRVFSZ FUD

Slide 45

Slide 45 text

1JUGBMMDPOTVNF5$1DPOOFDUJPO GPS@ LFZSBOHFUBSHFU,FZT\ HPGVOD \ SFT FSS2VFSZ*UFN LFZ ^ UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ 2VFSZ*UFN XJMMCFDBMMFEXJUINJOJNBMTQFFEDPOUSPM DPODVSSFOUDBMMTBSFOPUDPOTJEFSFE

Slide 46

Slide 46 text

4PMVUJPONBLFBMJNJUXJUIDIBOOFM MJNJU$IBONBLF DIBOJOU GPS@ LFZSBOHFUBSHFU,FZT\ MJNJU$IBO HPGVOD \ SFT FSS2VFSZ*UFN LFZ MJNJU$IBO ^ UJNF4MFFQ EVSBUJPO $BMDVMBUFECZBOPUIFSHPSPVUJOF ^ MJNJU$IBOCMPDLTJGMJNJU$IBOJTGVMM