Design & Implement Batch Stream Processing Application for Ruby

Design & Implement Batch Stream Processing Application for Ruby

沖縄Ruby会議01発表「Rubyによるバッチ業務のストリーム処理化の設計と実装」
http://regional.rubykaigi.org/okrk01/

3f1fad1dcac9f7e38db511df060a192d?s=128

Toshiaki Koshiba

March 01, 2014
Tweet

Transcript

  1. 3.
  2. 14.
  3. 17.

    l FFlluueennttdd iiss aann ooppeenn ssoouurrccee ddaattaa ccoolllleeccttoorr ddeessiiggnneedd ffoorr

    pprroocceessssiinngg hhiigghh--vvoolluummee ddaattaa ssttrreeaammss.. IUUQqVFOUEPSH
  4. 18.
  5. 21.

    IUUQ JTTVFߋ৽ JEPCBUB௨஌ .3ߋ৽ JEPCBUB௨஌ HJUQVTI JEPCBUB௨஌ HJUQVTI ճ਺ूܭ (SPXUI'PSFD

    BTUඳը Πϕϯτछ ผ൑ఆ "NB[PO424 ΤϯΩϡʔ HJUQVTI J1IPOF௨஌ Ϛείοτ ௨஌ "NB[PO424 σΩϡʔ ผͷ 'MVFOUE
  6. 24.

    TPVSDF UZQFIUUQ QPSU CJOE CPEZ@TJ[F@MJNJUN LFFQBMJWF@UJNFPVUT TPVSDF NBUDIHJUMBC  UZQFDPQZ

    TUPSF UZQFSFXSJUF BEE@QSFGJYGJMUFSFE SVMF LFZPCKFDU@LJOE QBUUFSO?   BQQFOE@UP@UBHUSVF MBTUUSVF SVMF SVMF LFZSFG QBUUFSO?SFGTaIFBETa   BQQFOE@UP@UBHUSVF SVMF TUPSF NBUDI NBUDIGJMUFSFEHJUMBC NFSHF@SFRVFTU UZQFDPQZ TUPSF UZQFJEPCBUB XFCIPPL@VSMIUUQTJEPCBUBJPXFC@IPPL@VSM NFTTBHF@UFNQMBUF<':*>5JNFOFX SFDPSE<PCKFDU@BUUSJCVUFT> <DSFBUFE@BU> TUSGUJNF NEQ ͔ΒͷʠSFDPSE<PCKFDU@BUUSJCVUFT><UJUMF>ʠͷ݅ʹ ಈ͖͕͋ΔͶ TUPSF NBUDI NBUDIGJMUFSFEHJUMBC JTTVF UZQFDPQZ TUPSF UZQFJEPCBUB XFCIPPL@VSMIUUQTJEPCBUBJPXFC@IPPL@VSM NFTTBHF@UFNQMBUF<':*>5JNFOFX SFDPSE<PCKFDU@BUUSJCVUFT> <DSFBUFE@BU> TUSGUJNF NEQ ͔ΒͷʠSFDPSE<PCKFDU@BUUSJCVUFT><UJUMF>ʡͷ݅ʹ ಈ͖͕͋ΔͶ TUPSF NBUDI NBUDIGJMUFSFEHJUMBC NBTUFS UZQFDPQZ TUPSF UZQFSFXSJUF BEE@QSFGJYSFXSJUFE TUPSF TUPSF UZQFJEPCBUB XFCIPPL@VSMIUUQTJEPCBUBJPXFC@IPPL@VSM NFTTBHF@UFNQMBUFʲϦϦʔεࠂ஌ʳSFDPSE<VTFS@OBNF>͞Μ͕ʠ SFDPSE<DPNNJUT>NBQ\]D]D<NFTTBHF>TMJDF ?  ^KPJO   ʡΛϦϦʔε͢ΔΑ TUPSF TUPSF UZQFDVSM DVSM@DPNNBOEDVSM VSM@UFNQMBUFIUUQJNLBZBDDPNBQJ@VSM PQUJPOT@UFNQMBUFEBUBVSMFODPEFNFTTBHFʲϦϦʔεࠂ஌ʳSFDPSE<VTFS@OBNF>͞Μ TUPSF TUPSF UZQFTRT RVFVF@OBNFNBLBJ[PI TRT@FOEQPJOUTRTBQOPSUIFBTUBNB[POBXTDPN TUPSF NBUDI NBUDIGJMUFSFEHJUMBC  UZQFOVMM NBUDI TPVSDF UZQFFYFD DPNNBOEFDIP LFZTVTFS@JE UBHSFXSJUFEGJMUFSFEHJUMBCNBLBJ[PINBTUFS SVO@JOUFSWBMT TPVSDF TPVSDF UZQFFYFD DPNNBOEFDIP LFZTVTFS@JE UBHSFXSJUFEGJMUFSFEHJUMBCMJHIUBETNBTUFS SVO@JOUFSWBMT TPVSDF TPVSDF UZQFFYFD DPNNBOEFDIP LFZTVTFS@JE UBHSFXSJUFEGJMUFSFEHJUMBCMJHIUBETBOBMZTZTNBTUFS SVO@JOUFSWBMT TPVSDF TPVSDF UZQFFYFD DPNNBOEFDIP LFZTVTFS@JE UBHSFXSJUFEGJMUFSFEHJUMBCQJYJWNBTUFS SVO@JOUFSWBMT TPVSDF TPVSDF UZQFFYFD DPNNBOEFDIP LFZTVTFS@JE UBHSFXSJUFEGJMUFSFEHJUMBCEJDQJYJWOFUNBTUFS SVO@JOUFSWBMT TPVSDF NBUDISFXSJUFEGJMUFSFEHJUMBC NBTUFS UZQFDPQZ TUPSF UZQFOVNFSJD@NPOJUPS VOJUNJOVUF PVUQVU@QFS@UBHZFT UBH@QSFGJYDPVOUFE BHHSFHBUFUBH NPOJUPS@LFZVTFS@JE TUPSF NBUDI NBUDIDPVOUFESFXSJUFEGJMUFSFEHJUMBC NBTUFS UZQFDPQZ TUPSF UZQFHSPXUIGPSFDBTU HGBQJ@VSMIUUQBQJ TFSWJDFHJUMBC UBH@GPSTFDUJPO OBNF@LFZTNBY SFNPWF@QSFGJYDPVOUFESFXSJUFEGJMUFSFEHJUMBC TUPSF NBUDI TPVSDF UZQFTRT TRT@FOEQPJOUTRTBQOPSUIFBTUBNB[POBXTDPN SFDFJWF@JOUFSWBM UBHNBLBJ[PITRT TPVSDF NBUDICBTI$@NBLBJ[PI  UZQFDPQZ TUPSF UZQFTTUQ TTUQ@TFSWFS TTUQ@QPSU SFRVFTU@NFUIPE/05*': SFRVFTU@WFSTJPO4451 TFOEFSΫϥ΢υి೾ TDSJQU@UFNQMBUFa͋͋οɺ৽͍͠ϦϦʔεͬͯి೾͕ඈΜͰ͖ͨχϟϯaF TUPSF TUPSF UZQFTUEPVU TUPSF NBUDI hhttttppss::////ggiisstt..ggiitthhuubb..ccoomm//bbaasshh00CC77//6655ff88118899ff77eeee7700aaff99996611ff
  7. 29.
  8. 30.

    TPVSDF UZQFUXJUUFSTFBSDI DPOTVNFS@LFZ999999999999999 DPOTVNFS@TFDSFU9999999999999999999 PBVUI@UPLFO99999999999999999999 PBVUI@UPLFO@TFDSFU9999999999999 VTFS@JEZVGV@ DPVOU SVO@JOUFSWBM SFTVMU@UZQFSFDFOU

    NFEJBZFT MBUFTU@JE@GJMF UXFFGGZ@ZVGV@@TJODF@JE@GJMFQPT UBHUXFFGGZZVGV TPVSDF NBUDIUXFFGGZ  UZQFDPQZ TUPSF UZQFOVMM TUPSF TUPSF UZQFSFXSJUF BEE@QSFGJYGJMUFSFE SVMF LFZNFEJB@VSM QBUUFSO? JHOPSFUSVF SVMF TUPSF NBUDI NBUDIGJMUFSFEUXFFGGZ  UZQFSFXSJUFJNBHF SFNPWF@UBH@QSFGJYGJMUFSFE BEE@UBH@QSFGJYSFXSJUFE JNBHF@TPVSDF@LFZNFEJB@VSM JNBHF@LFZSBX@NFEJB CBTFFODPEFGBMTF NBUDI NBUDISFXSJUFEUXFFGGZZVGV  UZQFDPQZ TUPSF UZQFJEPCBUB XFCIPPL@VSMIUUQTJEPCBUBJPVSM NFTTBHF@UFNQMBUF SFDPSE<NFEJB@VSM>SFDPSE<OBNF> ʮSFDPSE<UFYU>ʯ SFDPSE<UXFFU@VSM> TUPSF TUPSF UZQFUVNCMS DPOTVNFS@LFZ99999999999999999 DPOTVNFS@TFDSFU9999999999999999999 PBVUI@UPLFO9999999999999999 PBVUI@UPLFO@TFDSFU9999999999999999 UVNCMS@VSMUVNCMSDPN UBHT@UFNQMBUF SFDPSE<TDSFFO@OBNF> DBQUJPO@UFNQMBUFSFDPSE<UFYU> SFDPSE<UXFFU@VSM> MJOL@LFZUXFFU@VSM JNBHF@LFZSBX@NFEJB CBTFFODPEFEGBMTF QPTU@UZQFQJDUVSF TUPSF TUPSF UZQFJNBHFGJMF TBWF@EJSJNBHF GJMFOBNF@LFZJE FYUFOTJPOKQH JNBHF@LFZSBX@NFEJB CBTFFODPEFEGBMTF TUPSF NBUDI hhttttppss::////ggiisstt..ggiitthhuubb..ccoomm//bbaasshh00CC77//ff3344ddccaa00660044dddd773366990033dd99
  9. 42.

    NBUDIUXFFGGZ  UZQFSFXSJUF BEE@QSFpYpMUFSFE SVMF LFZNFEJB@VSM QBUUFSO? JHOPSFUSVF SVMF NBUDI

    NBUDIpMUFSFEUXFFGGZ  UZQFSFXSJUFJNBHF SFNPWF@QSFpYpMUFSFE BEE@QSFpYSFXSJUFE JNBHF@TPVSDF@LFZNFEJB@VSM JNBHF@LFZSBX@NFEJB CBTFFODPEFGBMTF NBUDI
  10. 45.

    NBUDISFXSJUFEUXFFGGZ  UZQFDPQZ TUPSF UZQFJEPCBUB XFCIPPL@VSMIUUQJEPCBUBXFCIPPL NFTTBHF@UFNQMBUFʮSFDPSE<GPP>ʯ TUPSF TUPSF UZQFUVNCMS

    DPOTVNFS@LFZ999999 DPOTVNFS@TFDSFU999999 PBVUI@UPLFO999999 PBVUI@UPLFO@TFDSFU99999999 UBHT@UFNQMBUFSFDPSE<TDSFFO@OBNF> DBQUJPO@UFNQMBUFSFDPSE<CBS> MJOL@LFZUXFFU@VSM JNBHF@LFZSBX@NFEJB CBTFFODPEFEGBMTF QPTU@UZQFQJDUVSF TUPSF NBUDI
  11. 48.

    TPVSDF UZQFUXJUUFSTFBSDI DPOTVNFS@LFZ999999999999999 DPOTVNFS@TFDSFU9999999999999999999 PBVUI@UPLFO99999999999999999999 PBVUI@UPLFO@TFDSFU9999999999999 VTFS@JEZVGV@ DPVOU SVO@JOUFSWBM SFTVMU@UZQFSFDFOU

    NFEJBZFT MBUFTU@JE@GJMF UXFFGGZ@ZVGV@@TJODF@JE@GJMFQPT UBHUXFFGGZZVGV TPVSDF NBUDIUXFFGGZ  UZQFDPQZ TUPSF UZQFOVMM TUPSF TUPSF UZQFSFXSJUF BEE@QSFGJYGJMUFSFE SVMF LFZNFEJB@VSM QBUUFSO? JHOPSFUSVF SVMF TUPSF NBUDI NBUDIGJMUFSFEUXFFGGZ  UZQFSFXSJUFJNBHF SFNPWF@UBH@QSFGJYGJMUFSFE BEE@UBH@QSFGJYSFXSJUFE JNBHF@TPVSDF@LFZNFEJB@VSM JNBHF@LFZSBX@NFEJB CBTFFODPEFGBMTF NBUDI NBUDISFXSJUFEUXFFGGZZVGV  UZQFDPQZ TUPSF UZQFJEPCBUB XFCIPPL@VSMIUUQTJEPCBUBJPVSM NFTTBHF@UFNQMBUF SFDPSE<NFEJB@VSM>SFDPSE<OBNF> ʮSFDPSE<UFYU>ʯ SFDPSE<UXFFU@VSM> TUPSF TUPSF UZQFUVNCMS DPOTVNFS@LFZ99999999999999999 DPOTVNFS@TFDSFU9999999999999999999 PBVUI@UPLFO9999999999999999 PBVUI@UPLFO@TFDSFU9999999999999999 UVNCMS@VSMUVNCMSDPN UBHT@UFNQMBUF SFDPSE<TDSFFO@OBNF> DBQUJPO@UFNQMBUFSFDPSE<UFYU> SFDPSE<UXFFU@VSM> MJOL@LFZUXFFU@VSM JNBHF@LFZSBX@NFEJB CBTFFODPEFEGBMTF QPTU@UZQFQJDUVSF TUPSF TUPSF UZQFJNBHFGJMF TBWF@EJSJNBHF GJMFOBNF@LFZJE FYUFOTJPOKQH JNBHF@LFZSBX@NFEJB CBTFFODPEFEGBMTF TUPSF NBUDI 変更に強い 業務処理を実装
  12. 59.