Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scalable Job Queue System Built with Docker

08d5432a5bc31e6d9edec87b94cb1db1?s=47 Takashi Kokubun
September 08, 2016
Programming
14
20k

Scalable Job Queue System Built with Docker

RubyKaigi 2016
http://rubykaigi.org/2016

08d5432a5bc31e6d9edec87b94cb1db1?s=128

Takashi Kokubun

September 08, 2016
Tweet

More Decks by Takashi Kokubun

See All by Takashi Kokubun
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
2
1k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
19
11k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
0
270
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
8
7.5k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
2
850
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
0
540
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
13
9.6k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
1
520
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
8
16k

Other Decks in Programming

See All in Programming
9adb9a67e73d93190e1d07d077a9aae7?s=48 ykkc
4
330
A760a90c9afd981004343b9861f1e8b4?s=48 daipresents
0
230
746c5d84a717cf7f74822d248e06574e?s=48 nkmrh
3
780
Faafc04d9e69b73b9f49995fd4c94d4d?s=48 whatyouhide
0
240
1a1d418bdf51cbf8fce1317f6c80a907?s=48 satoshi0212
5
870
583e920a7e9238a1c21e923025f8f641?s=48 elainenaomi
4
170
3925ee6eaa41031bac799de0f4f528ec?s=48 to4iki
1
550
575ca492bac55e895d0e1c86f7d709fe?s=48 hschwentner
3
1k
E66449b8260b07a1cf51c5ab5eaa8180?s=48 dunglas
3
3.1k
A65c56fbce87057b9ee064122601d7b5?s=48 nirazo
4
990
8959f99d4bf2e7445e5b847d3ac18dde?s=48 pkino
0
220
351bca0f01c8cb553535791140636368?s=48 sidepelican
3
210

Featured

See All Featured
1519587b1a0febb658c36c94f6272b0d?s=48 roundedbygravity
85
8.1k
E9221b172e78e888355fa2fe4541b595?s=48 mclloyd
PRO
2
7.4k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
34
2.4k
Aebc2d07a50011e2a30d38435fde564a?s=48 jrom
119
7.4k
8ec1383b240b5ba15ffb9743fceb3c0e?s=48 philnash
20
1k
433acaea1012b25d97ae66da9b998dad?s=48 malarkey
396
61k
2f5463832ccb768ccb4a1ca3607c27ef?s=48 jacobian
260
20k
5b9fe87ec1faa67a4599782930f45ec9?s=48 sstephenson
149
12k
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
56
5.6k
7559f6cff1f5efc2d210965febd4d71c?s=48 bermonpainter
348
27k
D36d2c1821af9249b69ff7f5ed60529b?s=48 tammielis
240
23k
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
287
47k

Transcript

  1. 4DBMBCMF+PC2VFVF4ZTUFN #VJMUXJUI%PDLFS 5BLBTIJ,PLVCVO$PPLQBE*OD 4FQUFNCFSUI  3VCZ,BJHJ

  2. 5BLBTIJ,PLVCVO!LLVCVO %FWFMPQFS1SPEVDUJWJUZ(SPVQ $PPLQBE*OD )BNMJU JUBNBF\ HP NSVCZ^ 

  3. 8FSFIJSJOH IUUQTSFDSVJUDPPLQBEDPN IUUQTJOGPDPPLQBEDPNFODBSFFST

  4. 5PEBZTUPQJD w 8IBUJTBKPCRVFVF  w *OUSPEVDUJPOUP#BSCFRVF$PPLQBETOFXKPCRVFVFTZTUFN w 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  w

    )PXJT#BSCFRVFEFTJHOFE

  5. 8IBUJTBKPCRVFVF 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  6. "KPCRVFVFJT w *OUIFDPOUFYUPGBXFCBQQMJDBUJPO JUTBTZTUFNVTFEUPQFSGPSN UBTLTBTZODISPOPVTMZ w JFQFSGPSNTMPXUBTLTMBUFSJOTUFBEPGEVSJOHSFRVFTUUPBQQ TFSWFS

  7. "KPCRVFVFJT w 8JUIPVUBKPCRVFVF VTFSTIBWFUPXBJUVOUJMBQQTFSWFSDPNQMFUFTBTMPX UBTL VTFS BQQ TMPXUBTL SFRVFTU SFTQPOTF

  8. "KPCRVFVFJT w 8JUIBKPCRVFVF VTFSTEPOUIBWFUPXBJUGPSUIFTMPXUBTLCFDBVTFJUJT QFSGPSNFEBTZODISPOPVTMZ VTFS BQQ RVFVF XPSLFS FORVFVFBUBTL

    EFRVFVFBUBTL TMPXUBTL SFRVFTU SFTQPOTF

  9. )PXUPVTFBKPCRVFVFTZTUFNJO3VCZ w %FpOFBKPCVTJOHSVCZHFNGPSBKPCRVFVFTZTUFN w SFTRVF TJEFLJR  w $BMMNFUIPEUPFORVFVFBKPC w

    .POJUPSUBTLTBOEXPSLFSTXJUIBXFCDPOTPMF

  10. %FpOFBKPC FYBNQMFPGSFTRVFHFN class TestJob @queue = :default def self.perform(user_id) #

    do something end end Resque.enqueue(TestJob, 100)

  11. .FUIPETUPFORVFVFBKPC w 6TFBNFUIPEJOUIFHFNGPSBKPCRVFVFTZTUFN w 3FTRVFFORVFVF w 4JEFLJR8PSLFSQFSGPSN@BTZOD w %FMBZFE+PCFORVFVF w

    4IPSZVLFO8PSLFSQFSGPSN@BTZOD w *O3BJMT ZPVDBOVTFBOBCTUSBDUJPOMBZFS w "DUJWF+PC&ORVFVJOHQFSGPSN@MBUFS

  12. 8FCDPOTPMF w SFTRVFXFC w TJEFLJR XFCJOUFSGBDFCVOEMFE  w EFMBZFE@KPC@XFC

  13. 8FCDPOTPMF

  14. 8IBUEPFTBKPCRVFVFTZTUFNDPOTJTUPG w %BUBTUPSFGPSBRVFVF w 8PSLFSUPQFSGPSNBKPC

  15. %BUBTUPSFGPSBRVFVF w 3FEJT w 3%#.4 w .Z42- 1PTUHSF42- w 3BCCJU.2

    w "NB[PO424 w 

  16. 8PSLFSUPQFSGPSNBKPC GPS3VCZ w 6TJOH3FEJT w SFTRVF w TJEFLJR w 6TJOH3%#.4

    w EFMBZFE@KPC w 6TJOH"NB[PO424 w TIPSZVLFO

  17. *OUSPEVDUJPOUP#BSCFRVF $PPLQBETOFXKPCRVFVFTZTUFN 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  18. #BSCFRVF w IUUQTHJUIVCDPNDPPLQBECBSCFRVF w *NQMFNFOUFEBTSVCZHFN w $PSFPGPVSOFXKPCRVFVFTZTUFN w %PDVNFOUBUJPOJTXPSLJOQSPHSFTT

  19. $PNQPOFOUTJOCBSCFRVFHFN w 8PSLFS w #VJMUXJUI4FSWFSFOHJOF w &YFDVUFBKPCVTJOH%PDLFS w 4MBDLOPUJpDBUJPO w

    2VFVFJOHJOUFSGBDF 8FC"1*  w 8FCDPOTPMF w .BOBHFSFHJTUFSFEBQQMJDBUJPOTBOEOPUJpDBUJPOT w :PVDBOTFFBMPHTUPSFEJO4

  20. 2VFVFJOH"1* w 1045WKPC@FYFDVUJPOT w &ORVFVFBNFTTBHFUP424 w 1BSBNFUFST w BQQMJDBUJPOTUSJOH w

    KPCTUSJOH w RVFVFTUSJOH w NFTTBHFBOZ PCKFDU BSSBZ TUSJOH 

  21. 2VFVFJOH"1*

  22. 2VFVFJOH"1*

  23. $MJFOUJOUFSGBDFGPS3VCZ CBSCFRVF@DMJFOUHFN BarbequeClient.configure do |config| config.application = 'blog' config.default_queue =

    'default' config.endpoint = 'http://www.example.com' end response = BarbequeClient.enqueue( job: 'NotifyAuthor', message: { article_id: 1 }, ) response.message_id #=> "edb7ef55-f1d0-46cf-bcca-036acb0ebd96" response.status #=> "pending"

  24. 3FRVFVFJOH"1* w 1045WKPC@FYFDVUJPOTNFTTBHF@JESFUSJFT w 3FRVFVFBNFTTBHFUP424 w 1BSBNFUFST w EFMBZ@TFDPOETJOUFHFS

  25. 3FRVFVFJOH"1*

  26. 3FRVFVFJOH"1*

  27. 8FCDPOTPMF

  28. 8IZEJEXFOFFE BOFXKPCRVFVFTZTUFN 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  29. "TIPSUIJTUPSZPGKPCRVFVFTZTUFNTBU$PPLQBE w 4PNFMJNJUFEBQQMJDBUJPOTXFSFVTJOHBKPCRVFVF w 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF w 4JEFLJR

    3FEJT  4JEFLJR8FC

  30. 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF VOJDPSO BQQDPOTPMF BQQTFSWFS SFRVFTU HPE SFEJTTFSWFS

    SFEJT NPOJUPS SFTRVF FORVFVF EFRVFVF

  31. 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF w 4JODFSFTRVFXPSLTJONVMUJQSPDFTTFT UIFSFTOPOFFEUPUIJOL BCPVUUISFBETBGFUZ w *G3FEJTJTOPUEJTUSJCVUFE

    EVQMJDBUFNFTTBHFTBSFOUEFMJWFSFE

  32. 4JEFLJR 3FEJT  4JEFLJR8FC w 'BTUFSUIBO3FTRVF w IUUQTHJUIVCDPNNQFSIBNTJEFLJRQFSGPSNBODF w :PVNVTUJNQMFNFOUKPCTJOBUISFBETBGFXBZCFDBVTFXPSLFST

    BSFNVMUJUISFBEFE

  33. 8IZEJEOUXFVTFBKPCRVFVFTPGSFRVFOUMZ w 8FIBWFLVSPLP XIJDIJTBOVTFGVMTDIFEVMFECBUDITZTUFN w 'FBUVSFSJDIXFCDPOTPMF w 4VQQPSUKPCFYFDVUJPOXJUI%PDLFS w .BOZEFWFMPQFSTMPWFLVSPLPBOEVTFJUUPQFSGPSNBUBTL

    BTZODISPOPVTMZ w #VUVTFSTIBWFUPXBJUUIFTDIFEVMFECBUDIFYFDVUJPO w 8FTIPVMEVTFBKPCRVFVFTZTUFNGPSVTFSTDPOWFOJFODF

  34. LVSPLP *UMMCFPQFOTPVSDFEBMJUUMFMBUFS

  35. 8IBUEJEXFXBOUGPS$PPLQBETKPCRVFVFTZTUFN w $FOUSBMJ[FENBOBHFNFOUPGXPSLFST w 8FIBWFTPNBOZBQQMJDBUJPOTBOEEPOUXBOUUPNBOBHF JOGSBTUSVDUVSFGPSFBDIBQQMJDBUJPO w &BTZKPCEFQMPZNFOU w LVSPLPMJLFXFCDPOTPMF

  36. )PXUPEFQMPZBKPC w 8FSFBMSFBEZVTJOH%PDLFSUPEFQMPZNPTUPGPVSBQQMJDBUJPOT w  EFQMPZNFOUDPOpHVSBUJPOT w 8FVTF&$4UPSVO%PDLFSDPOUBJOFSTJOBDMVTUFS w &$4"NB[PO&$$POUBJOFS4FSWJDF

    w 4DBMFJOPVUDBOCFEPOFCZBEKVTUJOHEFTJSFEDBQBDJUZPG "VUP4DBMJOH(SPVQ

  37. )PXJT#BSCFRVFEFTJHOFE 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  38. %BUBTUPSFGPSBRVFVF"NB[PO424 w .BOBHFENFTTBHFRVFVF w 1SPT w 'BTU TDBMBCMFBOESFMJBCMF w &BTZUPJOUFHSBUFXJUIPUIFS"84DPNQPOFOUTMJLF"NB[PO4/4

    w $POT w .BZEFMJWFSEVQMJDBUFNFTTBHFT 2P4JTBUMFBTUPODF  w .FTTBHFEFMBZEVSBUJPOJTMJNJUFEVQUPT

  39. 2P4BUMFBTUPODF BUNPTUPODFPSFYBDUMZPODF w 2VBMJUZPG4FSWJDFJTEFDJEFECZBEBUBTUPSFGPSBRVFVFBOE JNQMFNFOUBUJPOPGBXPSLFS w 2P4 w BUNPTUPODF6OSFMJBCMFCVUOPEVQMJDBUJPO w

    BUMFBTUPODF3FMJBCMFCVUEVQMJDBUFEFMJWFSZPDDVST w FYBDUMZPODF.PTUSFMJBCMFCVUIBSEUPNBLFBTZTUFNTDBMBCMF

  40. 8IZEJEXFDIPPTF"NB[PO424 w *UTIPVMECFTDBMBCMFBOESFMJBCMFCFDBVTFJUTVTFECZBMM BQQMJDBUJPOT w %VQMJDBUFEFMJWFSZJTBMMPXFEJOPVSNPTUVTFDBTFT

  41. 8PSLFSUPQFSGPSNBKPC w #BSCFRVFXPSLFSJTJNQMFNFOUFEXJUI3VCZ w 4JODFUIFKPCQFSGPSNFEPOBKPCRVFVFTZTUFNUFOETUPCFB TMPXUBTL XFDBOJHOPSFUIFPWFSIFBEJOBXPSLFSMBZFS w 5PQFSGPSNBKPC #BSCFRVFFYFDVUFTBDPNNBOEXJUITPNF

    TQFDJBMFOWJSPONFOUWBSJBCMFTVTJOH%PDLFS w 5IFZJODMVEFBTFSJBMJ[FENFTTBHFBOEBKPCOBNF

  42. +PCFYFDVUJPOPO&$4 w #BSCFRVFXPSLFSVTFTIBLPUPQFSGPSNBKPC XIJDIJTBEFQMPZ UPPMGPS%PDLFS w IBLPIUUQTHJUIVCDPNFBHMFUNUIBLP w 8FSFBMSFBEZVTJOHUIJTUPEFQMPZNBOZBQQMJDBUJPOTUP&$4 w

    8IJMFIBLPJTPQFOTPVSDFE IBLPBEBQUFSGPS#BSCFRVFJTTUJMM DMPTFETPVSDF

  43. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU

  44. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* FORVFVF

  45. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* "NB[PO424 FORVFVF FORVFVF

  46. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF EFRVFVF

  47. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF EFRVFVF 3%4

    .Z42- TUPSFTUBUVT

  48. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* &$4CBUDIDMVTUFS BQQ CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF

    EFRVFVF 3%4 .Z42- TUPSFTUBUVT FYFDVUFBDPNNBOE

  49. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* &$4CBUDIDMVTUFS BQQ CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF

    EFRVFVF TUPSFTUEPVU  TUEFSS "NB[PO4 3%4 .Z42- TUPSFTUBUVT FYFDVUFBDPNNBOE

  50. "VUPTDBMJOH w 6TJOH"VUP4DBMJOHHSPVQUPTDBMFBO&$4DMVTUFS w 4DBMFJOPVUBSFSVOEFQFOEJOHPOSFTPVSDFVTBHFPG&$4 DMVTUFS w 4JODFBKPCJTPOFUBTLPO&$4 XFDBOFTUJNBUFBQSFDJTF SFTPVSDFUPFYFDVUFKPCT

    w 3FTPVSDFUPSVOBO&$4UBTLJTEFpOFECFGPSFIBOE w 8FDBOSFEVDFDPTUXIFOKPCTBSFOPUQFSGPSNFEGSFRVFOUMZ

  51. $PODMVTJPO w 8FEFTJHOFEBOFXKPCRVFVFTZTUFNUP w $FOUSBMJ[FNBOBHFNFOUPGKPCRVFVFTZTUFNTXJUI%PDLFS w "VUPNBUFTDBMJOHCZ&$4BOE"VUP4DBMJOH(SPVQ w 2VFVFJOH"1* XFCDPOTPMFBOEDPSFPGXPSLFSJTPQFOTPVSDFE

    w IUUQTHJUIVCDPNDPPLQBECBSCFRVF