Akka streamsを活用したログ集計に優しいデータフローの構築

Transcript

1. BLLB
   TUSFBNΛ׆༻ͨ͠ϩάूܭʹ༏͍͠
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   σʔλϑϩʔͷߏஙʹ͍ͭͯ !#POpSF
   #BDLFOE
   

2. 1SPpMF ໊લ
   
   
   ໦ଜ
   ऺฏ
   4IVIFJ
   ,JNVSB
   
   
   IUUQTXXXGBDFCPPLDPNTIVIFJLJNVSB
   ॴଐ
   
   
   גࣜձࣾαΠόʔΤʔδΣϯτ
   

   
   
   ΞυςΫຊ෦
   "E5FDI4UVEJP
   %ZOBMZTU
   
   
   
   1SPEVDU
   .BOBHFS
   
   4PGUXBSF
   &OHJOFFS
   ܦྺ
   
   
   
   
   
   
   
   ೥৽ଔೖࣾ ೥໨
   5SBDLJOH 1$
   "ET 4NBSU
   1IPOF
   "ET %ZOBNJD
   3FUBSHFUJOH
   "ET (MPCBM
   &YQBOTJPO

3. %ZOBMZTUͱ͍͏޿ࠂ഑৴αʔϏεͷن໛ײ d,
   SFRTFD NT
   GPS
   FBDI 
   &$
   *OTUBODFT 'PS
   #VZJOH
   *NQSFTTJPOT 5#
   MPH
   pMFT
   
   EBZ ѻ͏ͷ͸·͊·͊େม
   ɻɻ ϚϧνϦʔδϣϯͰ
   

   ࣮ࡍ͸͜ͷഒͷྔ

4. 5PEBZ`T
   5PQJD
   d
   ϩάσʔλΛ៉ྷʹ4ʹӡͿ࿩
   d d,
   SFRTFD 5#
   QFS
   EBZ ‑‑‑‑‑

5. "LLB
   4USFBN͕ొ৔ͨ͠എܠ طଘͷ-JCSBSZ͕όά͍ͬͯͨ
   
   
   
   
   
   ,JOFTJT
   $POOFDUPS
   -JCSBSZ
   
   ͦ΋ͦ΋044ʹλμ৐Γ͍ͯ͠Δ͜ͱ͕
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   ΑΖ͘͠ͳ͍͕ɺ
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   ͍ͩͿϝϯς͞Ε͍ͯͳ͍ͷͰɺ
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   CVHpY͢Δͷ΋ɻɻ
   ͍·ͩͱ͍Ζ͍ΖͳιϦϡʔγϣϯ͕͋Δʁ
   
   
   
   
   
   ,JOFTJT
   'JSFIPTF
   ʁ
   
   
   
   ΩϨΠʹ4ʹΦϒδΣΫτΛ഑ஔ͢Δʹ͸ɺ
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   ޻෉͕ඞཁʢ-BNCEB
   'VODUJPOڬΉͷ͸ͪΐͬͱɻʣ
   
   
   
   
   

   
   
   
   
   -BNCEB
   'VODUJPO
   ʁ
   
   ,JOFTJT
   %BUB
   4USFBN͔Βͷ1PMMJOH
   *OUFSWBMΛ
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   ࢦఆͰ͖ͳ͍ͷͰ4ͷΦϒδΣΫτ਺͕๲େʹ
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   ʢ&5-Λ"84
   (MVFʹ೚ͤͯ΋Α͍͕ɻ8PSL'MPX&OHJOFొ৔ɻɻʣ 
   ,$-
   XJUI
   "LLB
   4USFBN͕γϯϓϧͰΑ͍͔΋ʁ

6. "LLB
   4USFBNTͷߏ੒ཁૉ 4PVSDF
   
   ϝοηʔδΛు͖ग़͢ίϯϙʔωϯτ
   ࢝఺
   'MPX
   
   ϝοηʔδΛड͚औͬͯϝοηʔδΛૹΔίϯϙʔωϯτ
   
   4JOL
   
   ϝοηʔδΛड͚औΔίϯϙʔωϯτ ऴ఺
   ࢀߟURL: http://chariotsolutions.com/wp-content/uploads/blog/2016/08/akkastreamstopology.png

   ͜ΕΒͷίϯϙʔωϯτΛ૊Έ߹Θͤͯ 3VOOBCMF(SBQIΛߏங
   

7. ഑৴ϩάΛ4ʹӡͿͨΊͷ(SBQIߏங Kinesis Source Extract Record Flow Upload Log Flow Sink

   ˞ຊ౰͸424
   NFTTBHJOH
   'MPX΍ɺ
   
   
   
   
   
   #SPBE$BTUͳͲར༻͍ͯ͠·ׂ͕͢Ѫ
   

8. Ͳͷ΁Μ͕ूܭʹ༏͍͠ͷ͔ʁ Kinesis Source Extract Record Flow Upload Log Flow Sink

   ‑‑‑‑‑ TCVDLFUYYYYYYYYZZZZH[
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   YYYYYY[[[[[H[
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   ʜ
   TCVDLFUYYYYZZH[
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   YY[[H[
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   ʜ 6QMPBEͨ࣌ؒ͠Ͱ͸ͳ͘ɺ
   ϩά͕ग़ྗ͞Εͨ࣌ؒϕʔεͰ
   4ͷ࣌ؒ۠੾Γͷύʔςʔγϣϯ Ͱ֨ೲ 4QBSL
   $MVTUFS͸4ͷࢦఆ࣌ؒͷύεͷϩάΛ
   ର৅ʹूܭॲཧΛ૸ΒͤΔ ूܭ͸ɺIPVSMZ
   EBJMZ
   ͕ओ
   

9. Ͳͷ΁Μ͕ूܭʹ༏͍͠ͷ͔ʁ Kinesis Source Extract Record Flow Upload Log Flow Sink

   ‑‑‑‑‑ TCVDLFUYYYYYYYYZZZZH[
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   YYYYYY[[[[[H[
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   ʜ
   TCVDLFUYYYYZZH[
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   YY[[H[
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   ʜ Before After BufferType Memory Local Disc S3 Object Size 5mbલޙ 15mbલޙ ूܭ࣌ؒ 30min 15min

10. ͓·͚ɿ&$4ͷDPNQVUF
    3FTPVSDFʹ΋༏͍͠࡞Γʹ Kinesis Source Extract Record Flow Upload Log Flow Sink

    ‑‑‑‑‑ 'MPXͷத਎
    
    
    ,JOFTJT3FDPSE͔Βɺඞཁͳ৘ใΛൈ͖ग़ͯ͠ɺΩϨΠʹ͢Δ
    
    
    ύʔςΟγϣϯ͝ͱʹ-PDBMͷετϨʔδʹϩάΛόοϑΝϦϯά
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    ϝϞϦফඅ཈੍ʢ(#
    
    (#ʣ
    
    
    
    Ұఆ༰ྔҰఆ࣌ؒܦա͢Δͱ4ʹ1VU0CKFDU
    ‑‑‑‑‑

11. ͓ฉ͖͍͖ͨͩ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ