Scalable Job Queue System Built with Docker

08d5432a5bc31e6d9edec87b94cb1db1?s=47 Takashi Kokubun
September 08, 2016
Programming
14
20k

Scalable Job Queue System Built with Docker

RubyKaigi 2016
http://rubykaigi.org/2016

08d5432a5bc31e6d9edec87b94cb1db1?s=128

Takashi Kokubun

September 08, 2016
Tweet

More Decks by Takashi Kokubun

See All by Takashi Kokubun
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
0
92
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
7
3.4k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
2
670
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
0
400
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
13
8.5k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
1
450
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
8
15k
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
1
170
08d5432a5bc31e6d9edec87b94cb1db1?s=48 k0kubun
5
1.1k

Other Decks in Programming

See All in Programming
Bb49e68eb46070029845fb3022d9026f?s=48 yukilabo
1
200
264b44c0c6d60336f8b6a0fbab118984?s=48 mmazour
0
150
D79e3b0b78ca7027eb61d73028d45ad6?s=48 uchi_k
2
940
9987516c273a7874b12b742e7aef828f?s=48 smellman
0
210
7e0087a19318ded4ba2203187694740f?s=48 christianweyer
0
200
8ec2b967e38f000eb63ae345cd7c58e6?s=48 minodriven
38
5.4k
9eed44f137609e6ce3b6f1e14f80b9e1?s=48 izumin5210
6
3k
8c10fdf7a2e8ebb97041029c5efd914f?s=48 erimatsu
0
150
4a0e3afe85259f2973dbc9386b6a4bb3?s=48 ikumatadokoro
1
480
C2916036a7a60f722f34faad5ef38316?s=48 minako__ph
1
560
3931098ae486b6df619050c63e24f6cc?s=48 osyo
1
320
894a9f3c8db89d0a5cd8e1add1dbe37b?s=48 usamiuhuru
0
400

Featured

See All Featured
D83926c323d4f9289f947b4b4e76b939?s=48 jensimmons
205
10k
Be9caeb9d4ef9944d151af909063ed6e?s=48 samlambert
236
9.8k
96270e4c3e5e9806cf7245475c00b275?s=48 addyosmani
309
21k
016edace78ffe826f2348d1e0c504afd?s=48 maggiecrowley
5
310
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
497
130k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
775
240k
20bfe76b3d6105641f879fe45cfc9272?s=48 bkeepers
319
53k
5ce91fa49a613cbc3e20d5f96856473f?s=48 edds
55
9.3k
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
119
7.7k
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
29
3.2k
1177e050db6bafe62885362edf6e3537?s=48 lauravandoore
435
28k
B6a8f005f39d23ffc930508ac9da68b9?s=48 vanstee
115
4.7k

Transcript

  1. 4DBMBCMF+PC2VFVF4ZTUFN #VJMUXJUI%PDLFS 5BLBTIJ,PLVCVO$PPLQBE*OD 4FQUFNCFSUI  3VCZ,BJHJ

  2. 5BLBTIJ,PLVCVO!LLVCVO %FWFMPQFS1SPEVDUJWJUZ(SPVQ $PPLQBE*OD )BNMJU JUBNBF\ HP NSVCZ^ 

  3. 8FSFIJSJOH IUUQTSFDSVJUDPPLQBEDPN IUUQTJOGPDPPLQBEDPNFODBSFFST

  4. 5PEBZTUPQJD w 8IBUJTBKPCRVFVF  w *OUSPEVDUJPOUP#BSCFRVF$PPLQBETOFXKPCRVFVFTZTUFN w 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  w

    )PXJT#BSCFRVFEFTJHOFE

  5. 8IBUJTBKPCRVFVF 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  6. "KPCRVFVFJT w *OUIFDPOUFYUPGBXFCBQQMJDBUJPO JUTBTZTUFNVTFEUPQFSGPSN UBTLTBTZODISPOPVTMZ w JFQFSGPSNTMPXUBTLTMBUFSJOTUFBEPGEVSJOHSFRVFTUUPBQQ TFSWFS

  7. "KPCRVFVFJT w 8JUIPVUBKPCRVFVF VTFSTIBWFUPXBJUVOUJMBQQTFSWFSDPNQMFUFTBTMPX UBTL VTFS BQQ TMPXUBTL SFRVFTU SFTQPOTF

  8. "KPCRVFVFJT w 8JUIBKPCRVFVF VTFSTEPOUIBWFUPXBJUGPSUIFTMPXUBTLCFDBVTFJUJT QFSGPSNFEBTZODISPOPVTMZ VTFS BQQ RVFVF XPSLFS FORVFVFBUBTL

    EFRVFVFBUBTL TMPXUBTL SFRVFTU SFTQPOTF

  9. )PXUPVTFBKPCRVFVFTZTUFNJO3VCZ w %FpOFBKPCVTJOHSVCZHFNGPSBKPCRVFVFTZTUFN w SFTRVF TJEFLJR  w $BMMNFUIPEUPFORVFVFBKPC w

    .POJUPSUBTLTBOEXPSLFSTXJUIBXFCDPOTPMF

  10. %FpOFBKPC FYBNQMFPGSFTRVFHFN class TestJob @queue = :default def self.perform(user_id) #

    do something end end Resque.enqueue(TestJob, 100)

  11. .FUIPETUPFORVFVFBKPC w 6TFBNFUIPEJOUIFHFNGPSBKPCRVFVFTZTUFN w 3FTRVFFORVFVF w 4JEFLJR8PSLFSQFSGPSN@BTZOD w %FMBZFE+PCFORVFVF w

    4IPSZVLFO8PSLFSQFSGPSN@BTZOD w *O3BJMT ZPVDBOVTFBOBCTUSBDUJPOMBZFS w "DUJWF+PC&ORVFVJOHQFSGPSN@MBUFS

  12. 8FCDPOTPMF w SFTRVFXFC w TJEFLJR XFCJOUFSGBDFCVOEMFE  w EFMBZFE@KPC@XFC

  13. 8FCDPOTPMF

  14. 8IBUEPFTBKPCRVFVFTZTUFNDPOTJTUPG w %BUBTUPSFGPSBRVFVF w 8PSLFSUPQFSGPSNBKPC

  15. %BUBTUPSFGPSBRVFVF w 3FEJT w 3%#.4 w .Z42- 1PTUHSF42- w 3BCCJU.2

    w "NB[PO424 w 

  16. 8PSLFSUPQFSGPSNBKPC GPS3VCZ w 6TJOH3FEJT w SFTRVF w TJEFLJR w 6TJOH3%#.4

    w EFMBZFE@KPC w 6TJOH"NB[PO424 w TIPSZVLFO

  17. *OUSPEVDUJPOUP#BSCFRVF $PPLQBETOFXKPCRVFVFTZTUFN 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  18. #BSCFRVF w IUUQTHJUIVCDPNDPPLQBECBSCFRVF w *NQMFNFOUFEBTSVCZHFN w $PSFPGPVSOFXKPCRVFVFTZTUFN w %PDVNFOUBUJPOJTXPSLJOQSPHSFTT

  19. $PNQPOFOUTJOCBSCFRVFHFN w 8PSLFS w #VJMUXJUI4FSWFSFOHJOF w &YFDVUFBKPCVTJOH%PDLFS w 4MBDLOPUJpDBUJPO w

    2VFVFJOHJOUFSGBDF 8FC"1*  w 8FCDPOTPMF w .BOBHFSFHJTUFSFEBQQMJDBUJPOTBOEOPUJpDBUJPOT w :PVDBOTFFBMPHTUPSFEJO4

  20. 2VFVFJOH"1* w 1045WKPC@FYFDVUJPOT w &ORVFVFBNFTTBHFUP424 w 1BSBNFUFST w BQQMJDBUJPOTUSJOH w

    KPCTUSJOH w RVFVFTUSJOH w NFTTBHFBOZ PCKFDU BSSBZ TUSJOH 

  21. 2VFVFJOH"1*

  22. 2VFVFJOH"1*

  23. $MJFOUJOUFSGBDFGPS3VCZ CBSCFRVF@DMJFOUHFN BarbequeClient.configure do |config| config.application = 'blog' config.default_queue =

    'default' config.endpoint = 'http://www.example.com' end response = BarbequeClient.enqueue( job: 'NotifyAuthor', message: { article_id: 1 }, ) response.message_id #=> "edb7ef55-f1d0-46cf-bcca-036acb0ebd96" response.status #=> "pending"

  24. 3FRVFVFJOH"1* w 1045WKPC@FYFDVUJPOTNFTTBHF@JESFUSJFT w 3FRVFVFBNFTTBHFUP424 w 1BSBNFUFST w EFMBZ@TFDPOETJOUFHFS

  25. 3FRVFVFJOH"1*

  26. 3FRVFVFJOH"1*

  27. 8FCDPOTPMF

  28. 8IZEJEXFOFFE BOFXKPCRVFVFTZTUFN 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  29. "TIPSUIJTUPSZPGKPCRVFVFTZTUFNTBU$PPLQBE w 4PNFMJNJUFEBQQMJDBUJPOTXFSFVTJOHBKPCRVFVF w 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF w 4JEFLJR

    3FEJT  4JEFLJR8FC

  30. 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF VOJDPSO BQQDPOTPMF BQQTFSWFS SFRVFTU HPE SFEJTTFSWFS

    SFEJT NPOJUPS SFTRVF FORVFVF EFRVFVF

  31. 3FTRVF 3FEJT  JOIPVTFXFCDPOTPMF w 4JODFSFTRVFXPSLTJONVMUJQSPDFTTFT UIFSFTOPOFFEUPUIJOL BCPVUUISFBETBGFUZ w *G3FEJTJTOPUEJTUSJCVUFE

    EVQMJDBUFNFTTBHFTBSFOUEFMJWFSFE

  32. 4JEFLJR 3FEJT  4JEFLJR8FC w 'BTUFSUIBO3FTRVF w IUUQTHJUIVCDPNNQFSIBNTJEFLJRQFSGPSNBODF w :PVNVTUJNQMFNFOUKPCTJOBUISFBETBGFXBZCFDBVTFXPSLFST

    BSFNVMUJUISFBEFE

  33. 8IZEJEOUXFVTFBKPCRVFVFTPGSFRVFOUMZ w 8FIBWFLVSPLP XIJDIJTBOVTFGVMTDIFEVMFECBUDITZTUFN w 'FBUVSFSJDIXFCDPOTPMF w 4VQQPSUKPCFYFDVUJPOXJUI%PDLFS w .BOZEFWFMPQFSTMPWFLVSPLPBOEVTFJUUPQFSGPSNBUBTL

    BTZODISPOPVTMZ w #VUVTFSTIBWFUPXBJUUIFTDIFEVMFECBUDIFYFDVUJPO w 8FTIPVMEVTFBKPCRVFVFTZTUFNGPSVTFSTDPOWFOJFODF

  34. LVSPLP *UMMCFPQFOTPVSDFEBMJUUMFMBUFS

  35. 8IBUEJEXFXBOUGPS$PPLQBETKPCRVFVFTZTUFN w $FOUSBMJ[FENBOBHFNFOUPGXPSLFST w 8FIBWFTPNBOZBQQMJDBUJPOTBOEEPOUXBOUUPNBOBHF JOGSBTUSVDUVSFGPSFBDIBQQMJDBUJPO w &BTZKPCEFQMPZNFOU w LVSPLPMJLFXFCDPOTPMF

  36. )PXUPEFQMPZBKPC w 8FSFBMSFBEZVTJOH%PDLFSUPEFQMPZNPTUPGPVSBQQMJDBUJPOT w  EFQMPZNFOUDPOpHVSBUJPOT w 8FVTF&$4UPSVO%PDLFSDPOUBJOFSTJOBDMVTUFS w &$4"NB[PO&$$POUBJOFS4FSWJDF

    w 4DBMFJOPVUDBOCFEPOFCZBEKVTUJOHEFTJSFEDBQBDJUZPG "VUP4DBMJOH(SPVQ

  37. )PXJT#BSCFRVFEFTJHOFE 8IBUJTBKPCRVFVF  *OUSPEVDUJPOUP#BSCFRVF 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN  )PXJT#BSCFRVFEFTJHOFE

  38. %BUBTUPSFGPSBRVFVF"NB[PO424 w .BOBHFENFTTBHFRVFVF w 1SPT w 'BTU TDBMBCMFBOESFMJBCMF w &BTZUPJOUFHSBUFXJUIPUIFS"84DPNQPOFOUTMJLF"NB[PO4/4

    w $POT w .BZEFMJWFSEVQMJDBUFNFTTBHFT 2P4JTBUMFBTUPODF  w .FTTBHFEFMBZEVSBUJPOJTMJNJUFEVQUPT

  39. 2P4BUMFBTUPODF BUNPTUPODFPSFYBDUMZPODF w 2VBMJUZPG4FSWJDFJTEFDJEFECZBEBUBTUPSFGPSBRVFVFBOE JNQMFNFOUBUJPOPGBXPSLFS w 2P4 w BUNPTUPODF6OSFMJBCMFCVUOPEVQMJDBUJPO w

    BUMFBTUPODF3FMJBCMFCVUEVQMJDBUFEFMJWFSZPDDVST w FYBDUMZPODF.PTUSFMJBCMFCVUIBSEUPNBLFBTZTUFNTDBMBCMF

  40. 8IZEJEXFDIPPTF"NB[PO424 w *UTIPVMECFTDBMBCMFBOESFMJBCMFCFDBVTFJUTVTFECZBMM BQQMJDBUJPOT w %VQMJDBUFEFMJWFSZJTBMMPXFEJOPVSNPTUVTFDBTFT

  41. 8PSLFSUPQFSGPSNBKPC w #BSCFRVFXPSLFSJTJNQMFNFOUFEXJUI3VCZ w 4JODFUIFKPCQFSGPSNFEPOBKPCRVFVFTZTUFNUFOETUPCFB TMPXUBTL XFDBOJHOPSFUIFPWFSIFBEJOBXPSLFSMBZFS w 5PQFSGPSNBKPC #BSCFRVFFYFDVUFTBDPNNBOEXJUITPNF

    TQFDJBMFOWJSPONFOUWBSJBCMFTVTJOH%PDLFS w 5IFZJODMVEFBTFSJBMJ[FENFTTBHFBOEBKPCOBNF

  42. +PCFYFDVUJPOPO&$4 w #BSCFRVFXPSLFSVTFTIBLPUPQFSGPSNBKPC XIJDIJTBEFQMPZ UPPMGPS%PDLFS w IBLPIUUQTHJUIVCDPNFBHMFUNUIBLP w 8FSFBMSFBEZVTJOHUIJTUPEFQMPZNBOZBQQMJDBUJPOTUP&$4 w

    8IJMFIBLPJTPQFOTPVSDFE IBLPBEBQUFSGPS#BSCFRVFJTTUJMM DMPTFETPVSDF

  43. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU

  44. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* FORVFVF

  45. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* "NB[PO424 FORVFVF FORVFVF

  46. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF EFRVFVF

  47. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF EFRVFVF 3%4

    .Z42- TUPSFTUBUVT

  48. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* &$4CBUDIDMVTUFS BQQ CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF

    EFRVFVF 3%4 .Z42- TUPSFTUBUVT FYFDVUFBDPNNBOE

  49. #BSCFRVFXPSLFSXPSLqPX BQQ SFRVFTU CBSCFRVF"1* &$4CBUDIDMVTUFS BQQ CBSCFRVFXPSLFS "NB[PO424 FORVFVF FORVFVF

    EFRVFVF TUPSFTUEPVU  TUEFSS "NB[PO4 3%4 .Z42- TUPSFTUBUVT FYFDVUFBDPNNBOE

  50. "VUPTDBMJOH w 6TJOH"VUP4DBMJOHHSPVQUPTDBMFBO&$4DMVTUFS w 4DBMFJOPVUBSFSVOEFQFOEJOHPOSFTPVSDFVTBHFPG&$4 DMVTUFS w 4JODFBKPCJTPOFUBTLPO&$4 XFDBOFTUJNBUFBQSFDJTF SFTPVSDFUPFYFDVUFKPCT

    w 3FTPVSDFUPSVOBO&$4UBTLJTEFpOFECFGPSFIBOE w 8FDBOSFEVDFDPTUXIFOKPCTBSFOPUQFSGPSNFEGSFRVFOUMZ

  51. $PODMVTJPO w 8FEFTJHOFEBOFXKPCRVFVFTZTUFNUP w $FOUSBMJ[FNBOBHFNFOUPGKPCRVFVFTZTUFNTXJUI%PDLFS w "VUPNBUFTDBMJOHCZ&$4BOE"VUP4DBMJOH(SPVQ w 2VFVFJOH"1* XFCDPOTPMFBOEDPSFPGXPSLFSJTPQFOTPVSDFE

    w IUUQTHJUIVCDPNDPPLQBECBSCFRVF