Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scalable Job Queue System Built with Docker

Scalable Job Queue System Built with Docker

RubyKaigi 2016
http://rubykaigi.org/2016

Takashi Kokubun

September 08, 2016
Tweet

More Decks by Takashi Kokubun

Other Decks in Programming

Transcript

  1. 4DBMBCMF+PC2VFVF4ZTUFN
    #VJMUXJUI%PDLFS
    5BLBTIJ,PLVCVO$PPLQBE*OD
    4FQUFNCFSUI
    3VCZ,BJHJ

    View full-size slide

  2. 5BLBTIJ,PLVCVO!LLVCVO
    %FWFMPQFS1SPEVDUJWJUZ(SPVQ $PPLQBE*OD
    )BNMJU JUBNBF\ HP NSVCZ^

    View full-size slide

  3. 8FSFIJSJOH
    IUUQTSFDSVJUDPPLQBEDPN
    IUUQTJOGPDPPLQBEDPNFODBSFFST

    View full-size slide

  4. 5PEBZTUPQJD
    w 8IBUJTBKPCRVFVF
    w *OUSPEVDUJPOUP#BSCFRVF$PPLQBETOFXKPCRVFVFTZTUFN
    w 8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN
    w )PXJT#BSCFRVFEFTJHOFE

    View full-size slide

  5. 8IBUJTBKPCRVFVF
    8IBUJTBKPCRVFVF
    *OUSPEVDUJPOUP#BSCFRVF
    8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN
    )PXJT#BSCFRVFEFTJHOFE

    View full-size slide

  6. "KPCRVFVFJT
    w *OUIFDPOUFYUPGBXFCBQQMJDBUJPO JUTBTZTUFNVTFEUPQFSGPSN
    UBTLTBTZODISPOPVTMZ
    w JFQFSGPSNTMPXUBTLTMBUFSJOTUFBEPGEVSJOHSFRVFTUUPBQQ
    TFSWFS

    View full-size slide

  7. "KPCRVFVFJT
    w 8JUIPVUBKPCRVFVF VTFSTIBWFUPXBJUVOUJMBQQTFSWFSDPNQMFUFTBTMPX
    UBTL
    VTFS BQQ
    TMPXUBTL
    SFRVFTU
    SFTQPOTF

    View full-size slide

  8. "KPCRVFVFJT
    w 8JUIBKPCRVFVF VTFSTEPOUIBWFUPXBJUGPSUIFTMPXUBTLCFDBVTFJUJT
    QFSGPSNFEBTZODISPOPVTMZ
    VTFS BQQ RVFVF XPSLFS
    FORVFVFBUBTL
    EFRVFVFBUBTL
    TMPXUBTL
    SFRVFTU
    SFTQPOTF

    View full-size slide

  9. )PXUPVTFBKPCRVFVFTZTUFNJO3VCZ
    w %FpOFBKPCVTJOHSVCZHFNGPSBKPCRVFVFTZTUFN
    w SFTRVF TJEFLJR
    w $BMMNFUIPEUPFORVFVFBKPC
    w .POJUPSUBTLTBOEXPSLFSTXJUIBXFCDPOTPMF

    View full-size slide

  10. %FpOFBKPC FYBNQMFPGSFTRVFHFN

    class TestJob
    @queue = :default
    def self.perform(user_id)
    # do something
    end
    end
    Resque.enqueue(TestJob, 100)

    View full-size slide

  11. .FUIPETUPFORVFVFBKPC
    w 6TFBNFUIPEJOUIFHFNGPSBKPCRVFVFTZTUFN
    w 3FTRVFFORVFVF
    w 4JEFLJR8PSLFSQFSGPSN@BTZOD
    w %FMBZFE+PCFORVFVF
    w 4IPSZVLFO8PSLFSQFSGPSN@BTZOD
    w *O3BJMT ZPVDBOVTFBOBCTUSBDUJPOMBZFS
    w "DUJWF+PC&ORVFVJOHQFSGPSN@MBUFS

    View full-size slide

  12. 8FCDPOTPMF
    w SFTRVFXFC
    w TJEFLJR XFCJOUFSGBDFCVOEMFE

    w EFMBZFE@KPC@XFC

    View full-size slide

  13. 8IBUEPFTBKPCRVFVFTZTUFNDPOTJTUPG
    w %BUBTUPSFGPSBRVFVF
    w 8PSLFSUPQFSGPSNBKPC

    View full-size slide

  14. %BUBTUPSFGPSBRVFVF
    w 3FEJT
    w 3%#.4
    w .Z42- 1PTUHSF42-
    w 3BCCJU.2
    w "NB[PO424
    w

    View full-size slide

  15. 8PSLFSUPQFSGPSNBKPC GPS3VCZ

    w 6TJOH3FEJT
    w SFTRVF
    w TJEFLJR
    w 6TJOH3%#.4
    w EFMBZFE@KPC
    w 6TJOH"NB[PO424
    w TIPSZVLFO

    View full-size slide

  16. *OUSPEVDUJPOUP#BSCFRVF
    $PPLQBETOFXKPCRVFVFTZTUFN
    8IBUJTBKPCRVFVF
    *OUSPEVDUJPOUP#BSCFRVF
    8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN
    )PXJT#BSCFRVFEFTJHOFE

    View full-size slide

  17. #BSCFRVF
    w IUUQTHJUIVCDPNDPPLQBECBSCFRVF
    w *NQMFNFOUFEBTSVCZHFN
    w $PSFPGPVSOFXKPCRVFVFTZTUFN
    w %PDVNFOUBUJPOJTXPSLJOQSPHSFTT

    View full-size slide

  18. $PNQPOFOUTJOCBSCFRVFHFN
    w 8PSLFS
    w #VJMUXJUI4FSWFSFOHJOF
    w &YFDVUFBKPCVTJOH%PDLFS
    w 4MBDLOPUJpDBUJPO
    w 2VFVFJOHJOUFSGBDF 8FC"1*

    w 8FCDPOTPMF
    w .BOBHFSFHJTUFSFEBQQMJDBUJPOTBOEOPUJpDBUJPOT
    w :PVDBOTFFBMPHTUPSFEJO4

    View full-size slide

  19. 2VFVFJOH"1*
    w 1045WKPC@FYFDVUJPOT
    w &ORVFVFBNFTTBHFUP424
    w 1BSBNFUFST
    w BQQMJDBUJPOTUSJOH
    w KPCTUSJOH
    w RVFVFTUSJOH
    w NFTTBHFBOZ PCKFDU BSSBZ TUSJOH

    View full-size slide

  20. $MJFOUJOUFSGBDFGPS3VCZ CBSCFRVF@DMJFOUHFN

    BarbequeClient.configure do |config|
    config.application = 'blog'
    config.default_queue = 'default'
    config.endpoint = 'http://www.example.com'
    end
    response = BarbequeClient.enqueue(
    job: 'NotifyAuthor',
    message: { article_id: 1 },
    )
    response.message_id #=> "edb7ef55-f1d0-46cf-bcca-036acb0ebd96"
    response.status #=> "pending"

    View full-size slide

  21. 3FRVFVFJOH"1*
    w 1045WKPC@FYFDVUJPOTNFTTBHF@JESFUSJFT
    w 3FRVFVFBNFTTBHFUP424
    w 1BSBNFUFST
    w EFMBZ@TFDPOETJOUFHFS

    View full-size slide

  22. 3FRVFVFJOH"1*

    View full-size slide

  23. 3FRVFVFJOH"1*

    View full-size slide

  24. 8IZEJEXFOFFE
    BOFXKPCRVFVFTZTUFN
    8IBUJTBKPCRVFVF
    *OUSPEVDUJPOUP#BSCFRVF
    8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN
    )PXJT#BSCFRVFEFTJHOFE

    View full-size slide

  25. "TIPSUIJTUPSZPGKPCRVFVFTZTUFNTBU$PPLQBE
    w 4PNFMJNJUFEBQQMJDBUJPOTXFSFVTJOHBKPCRVFVF
    w 3FTRVF 3FEJT
    JOIPVTFXFCDPOTPMF
    w 4JEFLJR 3FEJT
    4JEFLJR8FC

    View full-size slide

  26. 3FTRVF 3FEJT
    JOIPVTFXFCDPOTPMF
    VOJDPSO
    BQQDPOTPMF

    BQQTFSWFS
    SFRVFTU
    HPE
    SFEJTTFSWFS
    SFEJT
    NPOJUPS
    SFTRVF
    FORVFVF
    EFRVFVF

    View full-size slide

  27. 3FTRVF 3FEJT
    JOIPVTFXFCDPOTPMF
    w 4JODFSFTRVFXPSLTJONVMUJQSPDFTTFT UIFSFTOPOFFEUPUIJOL
    BCPVUUISFBETBGFUZ
    w *G3FEJTJTOPUEJTUSJCVUFE EVQMJDBUFNFTTBHFTBSFOUEFMJWFSFE

    View full-size slide

  28. 4JEFLJR 3FEJT
    4JEFLJR8FC
    w 'BTUFSUIBO3FTRVF
    w IUUQTHJUIVCDPNNQFSIBNTJEFLJRQFSGPSNBODF
    w :PVNVTUJNQMFNFOUKPCTJOBUISFBETBGFXBZCFDBVTFXPSLFST
    BSFNVMUJUISFBEFE

    View full-size slide

  29. 8IZEJEOUXFVTFBKPCRVFVFTPGSFRVFOUMZ
    w 8FIBWFLVSPLP XIJDIJTBOVTFGVMTDIFEVMFECBUDITZTUFN
    w 'FBUVSFSJDIXFCDPOTPMF
    w 4VQQPSUKPCFYFDVUJPOXJUI%PDLFS
    w .BOZEFWFMPQFSTMPWFLVSPLPBOEVTFJUUPQFSGPSNBUBTL
    BTZODISPOPVTMZ
    w #VUVTFSTIBWFUPXBJUUIFTDIFEVMFECBUDIFYFDVUJPO
    w 8FTIPVMEVTFBKPCRVFVFTZTUFNGPSVTFSTDPOWFOJFODF

    View full-size slide

  30. LVSPLP *UMMCFPQFOTPVSDFEBMJUUMFMBUFS

    View full-size slide

  31. 8IBUEJEXFXBOUGPS$PPLQBETKPCRVFVFTZTUFN
    w $FOUSBMJ[FENBOBHFNFOUPGXPSLFST
    w 8FIBWFTPNBOZBQQMJDBUJPOTBOEEPOUXBOUUPNBOBHF
    JOGSBTUSVDUVSFGPSFBDIBQQMJDBUJPO
    w &BTZKPCEFQMPZNFOU
    w LVSPLPMJLFXFCDPOTPMF

    View full-size slide

  32. )PXUPEFQMPZBKPC
    w 8FSFBMSFBEZVTJOH%PDLFSUPEFQMPZNPTUPGPVSBQQMJDBUJPOT
    w EFQMPZNFOUDPOpHVSBUJPOT
    w 8FVTF&$4UPSVO%PDLFSDPOUBJOFSTJOBDMVTUFS
    w &$4"NB[PO&$$POUBJOFS4FSWJDF
    w 4DBMFJOPVUDBOCFEPOFCZBEKVTUJOHEFTJSFEDBQBDJUZPG
    "VUP4DBMJOH(SPVQ

    View full-size slide

  33. )PXJT#BSCFRVFEFTJHOFE
    8IBUJTBKPCRVFVF
    *OUSPEVDUJPOUP#BSCFRVF
    8IZEJEXFOFFEBOFXKPCRVFVFTZTUFN
    )PXJT#BSCFRVFEFTJHOFE

    View full-size slide

  34. %BUBTUPSFGPSBRVFVF"NB[PO424
    w .BOBHFENFTTBHFRVFVF
    w 1SPT
    w 'BTU TDBMBCMFBOESFMJBCMF
    w &BTZUPJOUFHSBUFXJUIPUIFS"84DPNQPOFOUTMJLF"NB[PO4/4
    w $POT
    w .BZEFMJWFSEVQMJDBUFNFTTBHFT 2P4JTBUMFBTUPODF

    w .FTTBHFEFMBZEVSBUJPOJTMJNJUFEVQUPT

    View full-size slide

  35. 2P4BUMFBTUPODF BUNPTUPODFPSFYBDUMZPODF
    w 2VBMJUZPG4FSWJDFJTEFDJEFECZBEBUBTUPSFGPSBRVFVFBOE
    JNQMFNFOUBUJPOPGBXPSLFS
    w 2P4
    w BUNPTUPODF6OSFMJBCMFCVUOPEVQMJDBUJPO
    w BUMFBTUPODF3FMJBCMFCVUEVQMJDBUFEFMJWFSZPDDVST
    w FYBDUMZPODF.PTUSFMJBCMFCVUIBSEUPNBLFBTZTUFNTDBMBCMF

    View full-size slide

  36. 8IZEJEXFDIPPTF"NB[PO424
    w *UTIPVMECFTDBMBCMFBOESFMJBCMFCFDBVTFJUTVTFECZBMM
    BQQMJDBUJPOT
    w %VQMJDBUFEFMJWFSZJTBMMPXFEJOPVSNPTUVTFDBTFT

    View full-size slide

  37. 8PSLFSUPQFSGPSNBKPC
    w #BSCFRVFXPSLFSJTJNQMFNFOUFEXJUI3VCZ
    w 4JODFUIFKPCQFSGPSNFEPOBKPCRVFVFTZTUFNUFOETUPCFB
    TMPXUBTL XFDBOJHOPSFUIFPWFSIFBEJOBXPSLFSMBZFS
    w 5PQFSGPSNBKPC #BSCFRVFFYFDVUFTBDPNNBOEXJUITPNF
    TQFDJBMFOWJSPONFOUWBSJBCMFTVTJOH%PDLFS
    w 5IFZJODMVEFBTFSJBMJ[FENFTTBHFBOEBKPCOBNF

    View full-size slide

  38. +PCFYFDVUJPOPO&$4
    w #BSCFRVFXPSLFSVTFTIBLPUPQFSGPSNBKPC XIJDIJTBEFQMPZ
    UPPMGPS%PDLFS
    w IBLPIUUQTHJUIVCDPNFBHMFUNUIBLP
    w 8FSFBMSFBEZVTJOHUIJTUPEFQMPZNBOZBQQMJDBUJPOTUP&$4
    w 8IJMFIBLPJTPQFOTPVSDFE IBLPBEBQUFSGPS#BSCFRVFJTTUJMM
    DMPTFETPVSDF

    View full-size slide

  39. #BSCFRVFXPSLFSXPSLqPX
    BQQ
    SFRVFTU

    View full-size slide

  40. #BSCFRVFXPSLFSXPSLqPX
    BQQ
    SFRVFTU
    CBSCFRVF"1*
    FORVFVF

    View full-size slide

  41. #BSCFRVFXPSLFSXPSLqPX
    BQQ
    SFRVFTU
    CBSCFRVF"1*
    "NB[PO424
    FORVFVF
    FORVFVF

    View full-size slide

  42. #BSCFRVFXPSLFSXPSLqPX
    BQQ
    SFRVFTU
    CBSCFRVF"1*
    CBSCFRVFXPSLFS
    "NB[PO424
    FORVFVF
    FORVFVF
    EFRVFVF

    View full-size slide

  43. #BSCFRVFXPSLFSXPSLqPX
    BQQ
    SFRVFTU
    CBSCFRVF"1*
    CBSCFRVFXPSLFS
    "NB[PO424
    FORVFVF
    FORVFVF
    EFRVFVF
    3%4 .Z42-

    TUPSFTUBUVT

    View full-size slide

  44. #BSCFRVFXPSLFSXPSLqPX
    BQQ
    SFRVFTU
    CBSCFRVF"1*
    &$4CBUDIDMVTUFS
    BQQ
    CBSCFRVFXPSLFS
    "NB[PO424
    FORVFVF
    FORVFVF
    EFRVFVF
    3%4 .Z42-

    TUPSFTUBUVT
    FYFDVUFBDPNNBOE

    View full-size slide

  45. #BSCFRVFXPSLFSXPSLqPX
    BQQ
    SFRVFTU
    CBSCFRVF"1*
    &$4CBUDIDMVTUFS
    BQQ
    CBSCFRVFXPSLFS
    "NB[PO424
    FORVFVF
    FORVFVF
    EFRVFVF
    TUPSFTUEPVU
    TUEFSS
    "NB[PO4
    3%4 .Z42-

    TUPSFTUBUVT
    FYFDVUFBDPNNBOE

    View full-size slide

  46. "VUPTDBMJOH
    w 6TJOH"VUP4DBMJOHHSPVQUPTDBMFBO&$4DMVTUFS
    w 4DBMFJOPVUBSFSVOEFQFOEJOHPOSFTPVSDFVTBHFPG&$4
    DMVTUFS
    w 4JODFBKPCJTPOFUBTLPO&$4 XFDBOFTUJNBUFBQSFDJTF
    SFTPVSDFUPFYFDVUFKPCT
    w 3FTPVSDFUPSVOBO&$4UBTLJTEFpOFECFGPSFIBOE
    w 8FDBOSFEVDFDPTUXIFOKPCTBSFOPUQFSGPSNFEGSFRVFOUMZ

    View full-size slide

  47. $PODMVTJPO
    w 8FEFTJHOFEBOFXKPCRVFVFTZTUFNUP
    w $FOUSBMJ[FNBOBHFNFOUPGKPCRVFVFTZTUFNTXJUI%PDLFS
    w "VUPNBUFTDBMJOHCZ&$4BOE"VUP4DBMJOH(SPVQ
    w 2VFVFJOH"1* XFCDPOTPMFBOEDPSFPGXPSLFSJTPQFOTPVSDFE
    w IUUQTHJUIVCDPNDPPLQBECBSCFRVF

    View full-size slide