Scalable Job Queue System Built with Docker

08d5432a5bc31e6d9edec87b94cb1db1?s=47 Takashi Kokubun
September 08, 2016
Programming
13
19k

Scalable Job Queue System Built with Docker

RubyKaigi 2016
http://rubykaigi.org/2016

08d5432a5bc31e6d9edec87b94cb1db1?s=128

Takashi Kokubun

September 08, 2016
Tweet

More Decks by Takashi Kokubun

See All by Takashi Kokubun
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
7
3k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
2
590
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
0
350
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
13
8.3k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
1
410
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
8
14k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
1
150
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
5
980
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
15
9.4k

Other Decks in Programming

See All in Programming
5c097a7b75c8802b073774faa7276503?s=48 optim
5
350
3295d80faee461db1093982361256fe4?s=48 panini
0
100
9fbbe5d0b8e2ee1cdc3a576b55a2d63d?s=48 uzimaru0000
0
240
9ae4f3862d4cb0771cfd36e43252f755?s=48 kurotanshi
0
250
770ede26fb450c104128e35ac37711fd?s=48 musaprg
0
710
Fc358d63dcdcbad7a5c29198785dec2c?s=48 naototty
1
270
6e85e396c2fca10e469e67cbede8104e?s=48 yukikimoto
0
130
1763273018a6f71ffe4162dd57b60a56?s=48 maryang
0
420
0b26590a0a8b0f1da140ed5de9b68379?s=48 usamik26
0
120
A093c82d1ffb09f8a023c757ffa7419d?s=48 yuzujoe
2
190
5a065e7ea7c56f4b04c2271771688df3?s=48 raykataoka
34
18k
A3fabd96e2ca4bee5b0333ff2a640f01?s=48 translucens
2
240

Featured

See All Featured
Fde6c3a50ca518a909ef35c22c0ee0e4?s=48 destraynor
222
46k
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
281
46k
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
228
17k
A0517b08532aec8ab2aae3f65d40ad86?s=48 hannesfritz
26
850
5f2da528927a2ec9ba4fec2069cbc958?s=48 kneath
293
38k
A9704266587836f7e784235e5073b93e?s=48 jmmastey
1
50
C0390e8b11079f8761c0cd3274ed61cb?s=48 productmarketing
3
440
C44e1f7e22c3f23cff7bc130871047ef?s=48 eileencodes
110
24k
Dad095ea7038f89f760419ce475d5d14?s=48 michaelherold
224
8.2k
E4f5d494ebdc9e7cce1aecf3ce3e8bc1?s=48 shpigford
366
42k
B3ace07412a5178ea778e7eec161fc0a?s=48 jcasabona
6
380
39488f9d172ab92fd352f2cd7b73258d?s=48 mza
80
3.9k

Transcript

  1. 4DBMBCMF+PC2VFVF4ZTUFN #VJMUXJUI%PDLFS 5BLBTIJ,PLVCVO$PPLQBE*OD 4FQUFNCFSUI  3VCZ,BJHJ

  2. 5BLBTIJ,PLVCVO!LLVCVO %FWFMPQFS1SPEVDUJWJUZ(SPVQ $PPLQBE*OD )BNMJU JUBNBF\ HP NSVCZ^ 

  3. 8FSFIJSJOH IUUQTSFDSVJUDPPLQBEDPN IUUQTJOGPDPPLQBEDPNFODBSFFST

  4. 5PEBZTUPQJD w 8IBUJTBKPCRVFVF  w *OUSPEVDUJPOUP#BSCFRVF$PPLQBETOFXKPCRVFVFTZTUFN w 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  w

    )PXJT#BSCFRVFEFTJHOFE

  5. 8IBUJTBKPCRVFVF 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  6. "KPCRVFVFJT w *OUIFDPOUFYUPGBXFCBQQMJDBUJPO JUTBTZTUFNVTFEUPQFSGPSN UBTLTBTZODISPOPVTMZ w JFQFSGPSNTMPXUBTLTMBUFSJOTUFBEPGEVSJOHSFRVFTUUPBQQ TFSWFS

  7. "KPCRVFVFJT w 8JUIPVUBKPCRVFVF VTFSTIBWFUPXBJUVOUJMBQQTFSWFSDPNQMFUFTBTMPX UBTL VTFS BQQ TMPXUBTL SFRVFTU SFTQPOTF

  8. "KPCRVFVFJT w 8JUIBKPCRVFVF VTFSTEPOUIBWFUPXBJUGPSUIFTMPXUBTLCFDBVTFJUJT QFSGPSNFEBTZODISPOPVTMZ VTFS BQQ RVFVF XPSLFS FORVFVFBUBTL

    EFRVFVFBUBTL TMPXUBTL SFRVFTU SFTQPOTF

  9. )PXUPVTFBKPCRVFVFTZTUFNJO3VCZ w %FpOFBKPCVTJOHSVCZHFNGPSBKPCRVFVFTZTUFN w SFTRVF TJEFLJR  w $BMMNFUIPEUPFORVFVFBKPC w

    .POJUPSUBTLTBOEXPSLFSTXJUIBXFCDPOTPMF

  10. %FpOFBKPC FYBNQMFPGSFTRVFHFN class TestJob @queue = :default def self.perform(user_id) #

    do something end end Resque.enqueue(TestJob, 100)

  11. .FUIPETUPFORVFVFBKPC w 6TFBNFUIPEJOUIFHFNGPSBKPCRVFVFTZTUFN w 3FTRVFFORVFVF w 4JEFLJR8PSLFSQFSGPSN@BTZOD w %FMBZFE+PCFORVFVF w

    4IPSZVLFO8PSLFSQFSGPSN@BTZOD w *O3BJMT ZPVDBOVTFBOBCTUSBDUJPOMBZFS w "DUJWF+PC&ORVFVJOHQFSGPSN@MBUFS

  12. 8FCDPOTPMF w SFTRVFXFC w TJEFLJR XFCJOUFSGBDFCVOEMFE  w EFMBZFE@KPC@XFC

  13. 8FCDPOTPMF

  14. 8IBUEPFTBKPCRVFVFTZTUFNDPOTJTUPG w %BUBTUPSFGPSBRVFVF w 8PSLFSUPQFSGPSNBKPC

  15. %BUBTUPSFGPSBRVFVF w 3FEJT w 3%#.4 w .Z42- 1PTUHSF42- w 3BCCJU.2

    w "NB[PO424 w 

  16. 8PSLFSUPQFSGPSNBKPC GPS3VCZ w 6TJOH3FEJT w SFTRVF w TJEFLJR w 6TJOH3%#.4

    w EFMBZFE@KPC w 6TJOH"NB[PO424 w TIPSZVLFO

  17. *OUSPEVDUJPOUP#BSCFRVF $PPLQBETOFXKPCRVFVFTZTUFN 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  18. #BSCFRVF w IUUQTHJUIVCDPNDPPLQBECBSCFRVF w *NQMFNFOUFEBTSVCZHFN w $PSFPGPVSOFXKPCRVFVFTZTUFN w %PDVNFOUBUJPOJTXPSLJOQSPHSFTT

  19. $PNQPOFOUTJOCBSCFRVFHFN w 8PSLFS w #VJMUXJUI4FSWFSFOHJOF w &YFDVUFBKPCVTJOH%PDLFS w 4MBDLOPUJpDBUJPO w

    2VFVFJOHJOUFSGBDF 8FC"1*  w 8FCDPOTPMF w .BOBHFSFHJTUFSFEBQQMJDBUJPOTBOEOPUJpDBUJPOT w :PVDBOTFFBMPHTUPSFEJO4

  20. 2VFVFJOH"1* w 1045WKPC@FYFDVUJPOT w &ORVFVFBNFTTBHFUP424 w 1BSBNFUFST w BQQMJDBUJPOTUSJOH w

    KPCTUSJOH w RVFVFTUSJOH w NFTTBHFBOZ PCKFDU BSSBZ TUSJOH 

  21. 2VFVFJOH"1*

  22. 2VFVFJOH"1*

  23. $MJFOUJOUFSGBDFGPS3VCZ CBSCFRVF@DMJFOUHFN BarbequeClient.configure do |config| config.application = 'blog' config.default_queue =

    'default' config.endpoint = 'http://www.example.com' end response = BarbequeClient.enqueue( job: 'NotifyAuthor', message: { article_id: 1 }, ) response.message_id #=> "edb7ef55-f1d0-46cf-bcca-036acb0ebd96" response.status #=> "pending"

  24. 3FRVFVFJOH"1* w 1045WKPC@FYFDVUJPOTNFTTBHF@JESFUSJFT w 3FRVFVFBNFTTBHFUP424 w 1BSBNFUFST w EFMBZ@TFDPOETJOUFHFS

  25. 3FRVFVFJOH"1*

  26. 3FRVFVFJOH"1*

  27. 8FCDPOTPMF

  28. 8IZEJEXFOFFE BOFXKPCRVFVFTZTUFN 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  29. "TIPSUIJTUPSZPGKPCRVFVFTZTUFNTBU$PPLQBE w 4PNFMJNJUFEBQQMJDBUJPOTXFSFVTJOHBKPCRVFVF w 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF w 4JEFLJR

    3FEJT  4JEFLJR8FC

  30. 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF VOJDPSO BQQDPOTPMF BQQTFSWFS SFRVFTU HPE SFEJTTFSWFS

    SFEJT NPOJUPS SFTRVF FORVFVF EFRVFVF

  31. 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF w 4JODFSFTRVFXPSLTJONVMUJQSPDFTTFT UIFSFTOPOFFEUPUIJOL BCPVUUISFBETBGFUZ w *G3FEJTJTOPUEJTUSJCVUFE

    EVQMJDBUFNFTTBHFTBSFOUEFMJWFSFE

  32. 4JEFLJR 3FEJT  4JEFLJR8FC w 'BTUFSUIBO3FTRVF w IUUQTHJUIVCDPNNQFSIBNTJEFLJRQFSGPSNBODF w :PVNVTUJNQMFNFOUKPCTJOBUISFBETBGFXBZCFDBVTFXPSLFST

    BSFNVMUJUISFBEFE

  33. 8IZEJEOUXFVTFBKPCRVFVFTPGSFRVFOUMZ w 8FIBWFLVSPLP XIJDIJTBOVTFGVMTDIFEVMFECBUDITZTUFN w 'FBUVSFSJDIXFCDPOTPMF w 4VQQPSUKPCFYFDVUJPOXJUI%PDLFS w .BOZEFWFMPQFSTMPWFLVSPLPBOEVTFJUUPQFSGPSNBUBTL

    BTZODISPOPVTMZ w #VUVTFSTIBWFUPXBJUUIFTDIFEVMFECBUDIFYFDVUJPO w 8FTIPVMEVTFBKPCRVFVFTZTUFNGPSVTFSTDPOWFOJFODF

  34. LVSPLP *UMMCFPQFOTPVSDFEBMJUUMFMBUFS

  35. 8IBUEJEXFXBOUGPS$PPLQBETKPCRVFVFTZTUFN w $FOUSBMJ[FENBOBHFNFOUPGXPSLFST w 8FIBWFTPNBOZBQQMJDBUJPOTBOEEPOUXBOUUPNBOBHF JOGSBTUSVDUVSFGPSFBDIBQQMJDBUJPO w &BTZKPCEFQMPZNFOU w LVSPLPMJLFXFCDPOTPMF

  36. )PXUPEFQMPZBKPC w 8FSFBMSFBEZVTJOH%PDLFSUPEFQMPZNPTUPGPVSBQQMJDBUJPOT w  EFQMPZNFOUDPOpHVSBUJPOT w 8FVTF&$4UPSVO%PDLFSDPOUBJOFSTJOBDMVTUFS w &$4"NB[PO&$$POUBJOFS4FSWJDF

    w 4DBMFJOPVUDBOCFEPOFCZBEKVTUJOHEFTJSFEDBQBDJUZPG "VUP4DBMJOH(SPVQ

  37. )PXJT#BSCFRVFEFTJHOFE 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  38. %BUBTUPSFGPSBRVFVF"NB[PO424 w .BOBHFENFTTBHFRVFVF w 1SPT w 'BTU TDBMBCMFBOESFMJBCMF w &BTZUPJOUFHSBUFXJUIPUIFS"84DPNQPOFOUTMJLF"NB[PO4/4

    w $POT w .BZEFMJWFSEVQMJDBUFNFTTBHFT 2P4JTBUMFBTUPODF  w .FTTBHFEFMBZEVSBUJPOJTMJNJUFEVQUPT

  39. 2P4BUMFBTUPODF BUNPTUPODFPSFYBDUMZPODF w 2VBMJUZPG4FSWJDFJTEFDJEFECZBEBUBTUPSFGPSBRVFVFBOE JNQMFNFOUBUJPOPGBXPSLFS w 2P4 w BUNPTUPODF6OSFMJBCMFCVUOPEVQMJDBUJPO w

    BUMFBTUPODF3FMJBCMFCVUEVQMJDBUFEFMJWFSZPDDVST w FYBDUMZPODF.PTUSFMJBCMFCVUIBSEUPNBLFBTZTUFNTDBMBCMF

  40. 8IZEJEXFDIPPTF"NB[PO424 w *UTIPVMECFTDBMBCMFBOESFMJBCMFCFDBVTFJUTVTFECZBMM BQQMJDBUJPOT w %VQMJDBUFEFMJWFSZJTBMMPXFEJOPVSNPTUVTFDBTFT

  41. 8PSLFSUPQFSGPSNBKPC w #BSCFRVFXPSLFSJTJNQMFNFOUFEXJUI3VCZ w 4JODFUIFKPCQFSGPSNFEPOBKPCRVFVFTZTUFNUFOETUPCFB TMPXUBTL XFDBOJHOPSFUIFPWFSIFBEJOBXPSLFSMBZFS w 5PQFSGPSNBKPC #BSCFRVFFYFDVUFTBDPNNBOEXJUITPNF

    TQFDJBMFOWJSPONFOUWBSJBCMFTVTJOH%PDLFS w 5IFZJODMVEFBTFSJBMJ[FENFTTBHFBOEBKPCOBNF

  42. +PCFYFDVUJPOPO&$4 w #BSCFRVFXPSLFSVTFTIBLPUPQFSGPSNBKPC XIJDIJTBEFQMPZ UPPMGPS%PDLFS w IBLPIUUQTHJUIVCDPNFBHMFUNUIBLP w 8FSFBMSFBEZVTJOHUIJTUPEFQMPZNBOZBQQMJDBUJPOTUP&$4 w

    8IJMFIBLPJTPQFOTPVSDFE IBLPBEBQUFSGPS#BSCFRVFJTTUJMM DMPTFETPVSDF

  43. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU

  44. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* FORVFVF

  45. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* "NB[PO424 FORVFVF FORVFVF

  46. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF EFRVFVF

  47. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF EFRVFVF 3%4

    .Z42- TUPSFTUBUVT

  48. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* &$4CBUDIDMVTUFS BQQ CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF

    EFRVFVF 3%4 .Z42- TUPSFTUBUVT FYFDVUFBDPNNBOE

  49. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* &$4CBUDIDMVTUFS BQQ CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF

    EFRVFVF TUPSFTUEPVU  TUEFSS "NB[PO4 3%4 .Z42- TUPSFTUBUVT FYFDVUFBDPNNBOE

  50. "VUPTDBMJOH w 6TJOH"VUP4DBMJOHHSPVQUPTDBMFBO&$4DMVTUFS w 4DBMFJOPVUBSFSVOEFQFOEJOHPOSFTPVSDFVTBHFPG&$4 DMVTUFS w 4JODFBKPCJTPOFUBTLPO&$4 XFDBOFTUJNBUFBQSFDJTF SFTPVSDFUPFYFDVUFKPCT

    w 3FTPVSDFUPSVOBO&$4UBTLJTEFpOFECFGPSFIBOE w 8FDBOSFEVDFDPTUXIFOKPCTBSFOPUQFSGPSNFEGSFRVFOUMZ

  51. $PODMVTJPO w 8FEFTJHOFEBOFXKPCRVFVFTZTUFNUP w $FOUSBMJ[FNBOBHFNFOUPGKPCRVFVFTZTUFNTXJUI%PDLFS w "VUPNBUFTDBMJOHCZ&$4BOE"VUP4DBMJOH(SPVQ w 2VFVFJOH"1* XFCDPOTPMFBOEDPSFPGXPSLFSJTPQFOTPVSDFE

    w IUUQTHJUIVCDPNDPPLQBECBSCFRVF