Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scalable Job Queue System Built with Docker

08d5432a5bc31e6d9edec87b94cb1db1?s=47 Takashi Kokubun
September 08, 2016
Programming
14
20k

Scalable Job Queue System Built with Docker

RubyKaigi 2016
http://rubykaigi.org/2016

08d5432a5bc31e6d9edec87b94cb1db1?s=128

Takashi Kokubun

September 08, 2016
Tweet

More Decks by Takashi Kokubun

See All by Takashi Kokubun
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
0
160
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
8
7k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
2
780
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
0
480
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
13
9.1k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
1
500
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
8
16k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
1
200
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
5
1.2k

Other Decks in Programming

See All in Programming
438799c739f93d00abc30fa2da33c543?s=48 rukiadia
3
870
C90bac78c0fb61105cfd8239767f903d?s=48 o0h
PRO
3
1.6k
A760a90c9afd981004343b9861f1e8b4?s=48 daipresents
8
2.8k
Dc5157c1e1b11923f4cdd210ee27d4af?s=48 patorash
0
130
A8ca33aaa087bf3b664e8a0cab415455?s=48 matyo91
1
150
52cc8a838bf44a589d2572833b2dd1b9?s=48 palkan
2
310
Ebc0665c43d158c6e6450c7bd2de2fe0?s=48 ottijp
0
180
15934fa2aa7b2ce21f091e9b7cffa856?s=48 manfredsteyer
PRO
2
250
Df065863f38c6268cb70ed44b2e2f5b4?s=48 chatii
2
290
326530203cf3b827e41f68edb45e7509?s=48 bamboooo
1
130
091c8449decb2f3549c695b5f903865a?s=48 sh_akira
3
120
Acd2f6f7498ea41881e161e191aa7c02?s=48 yattom
31
11k

Featured

See All Featured
5f2da528927a2ec9ba4fec2069cbc958?s=48 kneath
294
39k
Be9caeb9d4ef9944d151af909063ed6e?s=48 samlambert
237
10k
A9179349dd2bdc67f377719f56d85656?s=48 garrettdimon
289
110k
462dad0dd24c568a32dcdb0ae895ae1b?s=48 rasmusluckow
318
19k
A9a491b0fcbe0fbce3d64063a37add99?s=48 rmw
12
870
1b75d89772b7eea1f6c89fbf362607d9?s=48 moore
125
21k
9fa239b3154a0169d99ab7bf1422dd12?s=48 marcelosomers
221
15k
79acae287842acf29084005533a7fb3b?s=48 hursman
107
9.3k
E4f5d494ebdc9e7cce1aecf3ce3e8bc1?s=48 shpigford
370
42k
812e1705ff0f8bc1bcf18587bde687d5?s=48 62gerente
586
200k
Aff5641764408271f7bc398f2097edd0?s=48 denniskardys
220
120k
C1daf20e5c49ff910745198ef9869ac2?s=48 hatefulcrawdad
257
17k

Transcript

  1. 4DBMBCMF+PC2VFVF4ZTUFN #VJMUXJUI%PDLFS 5BLBTIJ,PLVCVO$PPLQBE*OD 4FQUFNCFSUI  3VCZ,BJHJ

  2. 5BLBTIJ,PLVCVO!LLVCVO %FWFMPQFS1SPEVDUJWJUZ(SPVQ $PPLQBE*OD )BNMJU JUBNBF\ HP NSVCZ^ 

  3. 8FSFIJSJOH IUUQTSFDSVJUDPPLQBEDPN IUUQTJOGPDPPLQBEDPNFODBSFFST

  4. 5PEBZTUPQJD w 8IBUJTBKPCRVFVF  w *OUSPEVDUJPOUP#BSCFRVF$PPLQBETOFXKPCRVFVFTZTUFN w 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  w

    )PXJT#BSCFRVFEFTJHOFE

  5. 8IBUJTBKPCRVFVF 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  6. "KPCRVFVFJT w *OUIFDPOUFYUPGBXFCBQQMJDBUJPO JUTBTZTUFNVTFEUPQFSGPSN UBTLTBTZODISPOPVTMZ w JFQFSGPSNTMPXUBTLTMBUFSJOTUFBEPGEVSJOHSFRVFTUUPBQQ TFSWFS

  7. "KPCRVFVFJT w 8JUIPVUBKPCRVFVF VTFSTIBWFUPXBJUVOUJMBQQTFSWFSDPNQMFUFTBTMPX UBTL VTFS BQQ TMPXUBTL SFRVFTU SFTQPOTF

  8. "KPCRVFVFJT w 8JUIBKPCRVFVF VTFSTEPOUIBWFUPXBJUGPSUIFTMPXUBTLCFDBVTFJUJT QFSGPSNFEBTZODISPOPVTMZ VTFS BQQ RVFVF XPSLFS FORVFVFBUBTL

    EFRVFVFBUBTL TMPXUBTL SFRVFTU SFTQPOTF

  9. )PXUPVTFBKPCRVFVFTZTUFNJO3VCZ w %FpOFBKPCVTJOHSVCZHFNGPSBKPCRVFVFTZTUFN w SFTRVF TJEFLJR  w $BMMNFUIPEUPFORVFVFBKPC w

    .POJUPSUBTLTBOEXPSLFSTXJUIBXFCDPOTPMF

  10. %FpOFBKPC FYBNQMFPGSFTRVFHFN class TestJob @queue = :default def self.perform(user_id) #

    do something end end Resque.enqueue(TestJob, 100)

  11. .FUIPETUPFORVFVFBKPC w 6TFBNFUIPEJOUIFHFNGPSBKPCRVFVFTZTUFN w 3FTRVFFORVFVF w 4JEFLJR8PSLFSQFSGPSN@BTZOD w %FMBZFE+PCFORVFVF w

    4IPSZVLFO8PSLFSQFSGPSN@BTZOD w *O3BJMT ZPVDBOVTFBOBCTUSBDUJPOMBZFS w "DUJWF+PC&ORVFVJOHQFSGPSN@MBUFS

  12. 8FCDPOTPMF w SFTRVFXFC w TJEFLJR XFCJOUFSGBDFCVOEMFE  w EFMBZFE@KPC@XFC

  13. 8FCDPOTPMF

  14. 8IBUEPFTBKPCRVFVFTZTUFNDPOTJTUPG w %BUBTUPSFGPSBRVFVF w 8PSLFSUPQFSGPSNBKPC

  15. %BUBTUPSFGPSBRVFVF w 3FEJT w 3%#.4 w .Z42- 1PTUHSF42- w 3BCCJU.2

    w "NB[PO424 w 

  16. 8PSLFSUPQFSGPSNBKPC GPS3VCZ w 6TJOH3FEJT w SFTRVF w TJEFLJR w 6TJOH3%#.4

    w EFMBZFE@KPC w 6TJOH"NB[PO424 w TIPSZVLFO

  17. *OUSPEVDUJPOUP#BSCFRVF $PPLQBETOFXKPCRVFVFTZTUFN 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  18. #BSCFRVF w IUUQTHJUIVCDPNDPPLQBECBSCFRVF w *NQMFNFOUFEBTSVCZHFN w $PSFPGPVSOFXKPCRVFVFTZTUFN w %PDVNFOUBUJPOJTXPSLJOQSPHSFTT

  19. $PNQPOFOUTJOCBSCFRVFHFN w 8PSLFS w #VJMUXJUI4FSWFSFOHJOF w &YFDVUFBKPCVTJOH%PDLFS w 4MBDLOPUJpDBUJPO w

    2VFVFJOHJOUFSGBDF 8FC"1*  w 8FCDPOTPMF w .BOBHFSFHJTUFSFEBQQMJDBUJPOTBOEOPUJpDBUJPOT w :PVDBOTFFBMPHTUPSFEJO4

  20. 2VFVFJOH"1* w 1045WKPC@FYFDVUJPOT w &ORVFVFBNFTTBHFUP424 w 1BSBNFUFST w BQQMJDBUJPOTUSJOH w

    KPCTUSJOH w RVFVFTUSJOH w NFTTBHFBOZ PCKFDU BSSBZ TUSJOH 

  21. 2VFVFJOH"1*

  22. 2VFVFJOH"1*

  23. $MJFOUJOUFSGBDFGPS3VCZ CBSCFRVF@DMJFOUHFN BarbequeClient.configure do |config| config.application = 'blog' config.default_queue =

    'default' config.endpoint = 'http://www.example.com' end response = BarbequeClient.enqueue( job: 'NotifyAuthor', message: { article_id: 1 }, ) response.message_id #=> "edb7ef55-f1d0-46cf-bcca-036acb0ebd96" response.status #=> "pending"

  24. 3FRVFVFJOH"1* w 1045WKPC@FYFDVUJPOTNFTTBHF@JESFUSJFT w 3FRVFVFBNFTTBHFUP424 w 1BSBNFUFST w EFMBZ@TFDPOETJOUFHFS

  25. 3FRVFVFJOH"1*

  26. 3FRVFVFJOH"1*

  27. 8FCDPOTPMF

  28. 8IZEJEXFOFFE BOFXKPCRVFVFTZTUFN 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  29. "TIPSUIJTUPSZPGKPCRVFVFTZTUFNTBU$PPLQBE w 4PNFMJNJUFEBQQMJDBUJPOTXFSFVTJOHBKPCRVFVF w 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF w 4JEFLJR

    3FEJT  4JEFLJR8FC

  30. 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF VOJDPSO BQQDPOTPMF BQQTFSWFS SFRVFTU HPE SFEJTTFSWFS

    SFEJT NPOJUPS SFTRVF FORVFVF EFRVFVF

  31. 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF w 4JODFSFTRVFXPSLTJONVMUJQSPDFTTFT UIFSFTOPOFFEUPUIJOL BCPVUUISFBETBGFUZ w *G3FEJTJTOPUEJTUSJCVUFE

    EVQMJDBUFNFTTBHFTBSFOUEFMJWFSFE

  32. 4JEFLJR 3FEJT  4JEFLJR8FC w 'BTUFSUIBO3FTRVF w IUUQTHJUIVCDPNNQFSIBNTJEFLJRQFSGPSNBODF w :PVNVTUJNQMFNFOUKPCTJOBUISFBETBGFXBZCFDBVTFXPSLFST

    BSFNVMUJUISFBEFE

  33. 8IZEJEOUXFVTFBKPCRVFVFTPGSFRVFOUMZ w 8FIBWFLVSPLP XIJDIJTBOVTFGVMTDIFEVMFECBUDITZTUFN w 'FBUVSFSJDIXFCDPOTPMF w 4VQQPSUKPCFYFDVUJPOXJUI%PDLFS w .BOZEFWFMPQFSTMPWFLVSPLPBOEVTFJUUPQFSGPSNBUBTL

    BTZODISPOPVTMZ w #VUVTFSTIBWFUPXBJUUIFTDIFEVMFECBUDIFYFDVUJPO w 8FTIPVMEVTFBKPCRVFVFTZTUFNGPSVTFSTDPOWFOJFODF

  34. LVSPLP *UMMCFPQFOTPVSDFEBMJUUMFMBUFS

  35. 8IBUEJEXFXBOUGPS$PPLQBETKPCRVFVFTZTUFN w $FOUSBMJ[FENBOBHFNFOUPGXPSLFST w 8FIBWFTPNBOZBQQMJDBUJPOTBOEEPOUXBOUUPNBOBHF JOGSBTUSVDUVSFGPSFBDIBQQMJDBUJPO w &BTZKPCEFQMPZNFOU w LVSPLPMJLFXFCDPOTPMF

  36. )PXUPEFQMPZBKPC w 8FSFBMSFBEZVTJOH%PDLFSUPEFQMPZNPTUPGPVSBQQMJDBUJPOT w  EFQMPZNFOUDPOpHVSBUJPOT w 8FVTF&$4UPSVO%PDLFSDPOUBJOFSTJOBDMVTUFS w &$4"NB[PO&$$POUBJOFS4FSWJDF

    w 4DBMFJOPVUDBOCFEPOFCZBEKVTUJOHEFTJSFEDBQBDJUZPG "VUP4DBMJOH(SPVQ

  37. )PXJT#BSCFRVFEFTJHOFE 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  38. %BUBTUPSFGPSBRVFVF"NB[PO424 w .BOBHFENFTTBHFRVFVF w 1SPT w 'BTU TDBMBCMFBOESFMJBCMF w &BTZUPJOUFHSBUFXJUIPUIFS"84DPNQPOFOUTMJLF"NB[PO4/4

    w $POT w .BZEFMJWFSEVQMJDBUFNFTTBHFT 2P4JTBUMFBTUPODF  w .FTTBHFEFMBZEVSBUJPOJTMJNJUFEVQUPT

  39. 2P4BUMFBTUPODF BUNPTUPODFPSFYBDUMZPODF w 2VBMJUZPG4FSWJDFJTEFDJEFECZBEBUBTUPSFGPSBRVFVFBOE JNQMFNFOUBUJPOPGBXPSLFS w 2P4 w BUNPTUPODF6OSFMJBCMFCVUOPEVQMJDBUJPO w

    BUMFBTUPODF3FMJBCMFCVUEVQMJDBUFEFMJWFSZPDDVST w FYBDUMZPODF.PTUSFMJBCMFCVUIBSEUPNBLFBTZTUFNTDBMBCMF

  40. 8IZEJEXFDIPPTF"NB[PO424 w *UTIPVMECFTDBMBCMFBOESFMJBCMFCFDBVTFJUTVTFECZBMM BQQMJDBUJPOT w %VQMJDBUFEFMJWFSZJTBMMPXFEJOPVSNPTUVTFDBTFT

  41. 8PSLFSUPQFSGPSNBKPC w #BSCFRVFXPSLFSJTJNQMFNFOUFEXJUI3VCZ w 4JODFUIFKPCQFSGPSNFEPOBKPCRVFVFTZTUFNUFOETUPCFB TMPXUBTL XFDBOJHOPSFUIFPWFSIFBEJOBXPSLFSMBZFS w 5PQFSGPSNBKPC #BSCFRVFFYFDVUFTBDPNNBOEXJUITPNF

    TQFDJBMFOWJSPONFOUWBSJBCMFTVTJOH%PDLFS w 5IFZJODMVEFBTFSJBMJ[FENFTTBHFBOEBKPCOBNF

  42. +PCFYFDVUJPOPO&$4 w #BSCFRVFXPSLFSVTFTIBLPUPQFSGPSNBKPC XIJDIJTBEFQMPZ UPPMGPS%PDLFS w IBLPIUUQTHJUIVCDPNFBHMFUNUIBLP w 8FSFBMSFBEZVTJOHUIJTUPEFQMPZNBOZBQQMJDBUJPOTUP&$4 w

    8IJMFIBLPJTPQFOTPVSDFE IBLPBEBQUFSGPS#BSCFRVFJTTUJMM DMPTFETPVSDF

  43. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU

  44. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* FORVFVF

  45. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* "NB[PO424 FORVFVF FORVFVF

  46. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF EFRVFVF

  47. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF EFRVFVF 3%4

    .Z42- TUPSFTUBUVT

  48. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* &$4CBUDIDMVTUFS BQQ CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF

    EFRVFVF 3%4 .Z42- TUPSFTUBUVT FYFDVUFBDPNNBOE

  49. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* &$4CBUDIDMVTUFS BQQ CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF

    EFRVFVF TUPSFTUEPVU  TUEFSS "NB[PO4 3%4 .Z42- TUPSFTUBUVT FYFDVUFBDPNNBOE

  50. "VUPTDBMJOH w 6TJOH"VUP4DBMJOHHSPVQUPTDBMFBO&$4DMVTUFS w 4DBMFJOPVUBSFSVOEFQFOEJOHPOSFTPVSDFVTBHFPG&$4 DMVTUFS w 4JODFBKPCJTPOFUBTLPO&$4 XFDBOFTUJNBUFBQSFDJTF SFTPVSDFUPFYFDVUFKPCT

    w 3FTPVSDFUPSVOBO&$4UBTLJTEFpOFECFGPSFIBOE w 8FDBOSFEVDFDPTUXIFOKPCTBSFOPUQFSGPSNFEGSFRVFOUMZ

  51. $PODMVTJPO w 8FEFTJHOFEBOFXKPCRVFVFTZTUFNUP w $FOUSBMJ[FNBOBHFNFOUPGKPCRVFVFTZTUFNTXJUI%PDLFS w "VUPNBUFTDBMJOHCZ&$4BOE"VUP4DBMJOH(SPVQ w 2VFVFJOH"1* XFCDPOTPMFBOEDPSFPGXPSLFSJTPQFOTPVSDFE

    w IUUQTHJUIVCDPNDPPLQBECBSCFRVF