Akka streamsを活用したログ集計に優しいデータフローの構築

A8f802dfbbdd43f6e3be9936c3cbed7f?s=47 Shuhei Kimura
December 05, 2018

Akka streamsを活用したログ集計に優しいデータフローの構築

A8f802dfbbdd43f6e3be9936c3cbed7f?s=128

Shuhei Kimura

December 05, 2018
Tweet

Transcript

  1. BLLBTUSFBNΛ׆༻ͨ͠ϩάूܭʹ༏͍͠ σʔλϑϩʔͷߏஙʹ͍ͭͯ !#POpSF#BDLFOE

  2. 1SPpMF ໊લ  ໦ଜऺฏ4IVIFJ,JNVSB   IUUQTXXXGBDFCPPLDPNTIVIFJLJNVSB ॴଐ  גࣜձࣾαΠόʔΤʔδΣϯτ

      ΞυςΫຊ෦"E5FDI4UVEJP%ZOBMZTU    1SPEVDU.BOBHFS4PGUXBSF&OHJOFFS ܦྺ ೥৽ଔೖࣾ ೥໨  5SBDLJOH 1$"ET 4NBSU 1IPOF"ET %ZOBNJD 3FUBSHFUJOH "ET (MPCBM &YQBOTJPO
  3. %ZOBMZTUͱ͍͏޿ࠂ഑৴αʔϏεͷن໛ײ d,SFRTFD NTGPSFBDI  &$*OTUBODFT 'PS#VZJOH*NQSFTTJPOT 5# MPHpMFTEBZ ѻ͏ͷ͸·͊·͊େมɻɻ ϚϧνϦʔδϣϯͰ

    ࣮ࡍ͸͜ͷഒͷྔ
  4. 5PEBZ`T5PQJDdϩάσʔλΛ៉ྷʹ4ʹӡͿ࿩d d,SFRTFD 5# QFSEBZ ‑‑‑‑‑

  5. "LLB4USFBN͕ొ৔ͨ͠എܠ طଘͷ-JCSBSZ͕όά͍ͬͯͨ ,JOFTJT$POOFDUPS-JCSBSZͦ΋ͦ΋044ʹλμ৐Γ͍ͯ͠Δ͜ͱ͕ ΑΖ͘͠ͳ͍͕ɺ ͍ͩͿϝϯς͞Ε͍ͯͳ͍ͷͰɺ CVHpY͢Δͷ΋ɻɻ ͍·ͩͱ͍Ζ͍ΖͳιϦϡʔγϣϯ͕͋Δʁ ,JOFTJT'JSFIPTFʁΩϨΠʹ4ʹΦϒδΣΫτΛ഑ஔ͢Δʹ͸ɺ ޻෉͕ඞཁʢ-BNCEB'VODUJPOڬΉͷ͸ͪΐͬͱɻʣ 

    -BNCEB'VODUJPOʁ,JOFTJT%BUB4USFBN͔Βͷ1PMMJOH*OUFSWBMΛ ࢦఆͰ͖ͳ͍ͷͰ4ͷΦϒδΣΫτ਺͕๲େʹ ʢ&5-Λ"84(MVFʹ೚ͤͯ΋Α͍͕ɻ8PSL'MPX&OHJOFొ৔ɻɻʣ ,$-XJUI"LLB4USFBN͕γϯϓϧͰΑ͍͔΋ʁ
  6. "LLB4USFBNTͷߏ੒ཁૉ 4PVSDFϝοηʔδΛు͖ग़͢ίϯϙʔωϯτ ࢝఺  'MPXϝοηʔδΛड͚औͬͯϝοηʔδΛૹΔίϯϙʔωϯτ 4JOLϝοηʔδΛड͚औΔίϯϙʔωϯτ ऴ఺  ࢀߟURL: http://chariotsolutions.com/wp-content/uploads/blog/2016/08/akkastreamstopology.png

    ͜ΕΒͷίϯϙʔωϯτΛ૊Έ߹Θͤͯ 3VOOBCMF(SBQIΛߏங
  7. ഑৴ϩάΛ4ʹӡͿͨΊͷ(SBQIߏங Kinesis Source Extract Record Flow Upload Log Flow Sink

    ˞ຊ౰͸424NFTTBHJOH'MPX΍ɺ #SPBE$BTUͳͲར༻͍ͯ͠·ׂ͕͢Ѫ
  8. Ͳͷ΁Μ͕ूܭʹ༏͍͠ͷ͔ʁ Kinesis Source Extract Record Flow Upload Log Flow Sink

    ‑‑‑‑‑ TCVDLFUYYYYYYYYZZZZH[ YYYYYY[[[[[H[ ʜ TCVDLFUYYYYZZH[ YY[[H[ ʜ 6QMPBEͨ࣌ؒ͠Ͱ͸ͳ͘ɺ ϩά͕ग़ྗ͞Εͨ࣌ؒϕʔεͰ 4ͷ࣌ؒ۠੾Γͷύʔςʔγϣϯ Ͱ֨ೲ 4QBSL$MVTUFS͸4ͷࢦఆ࣌ؒͷύεͷϩάΛ ର৅ʹूܭॲཧΛ૸ΒͤΔ ूܭ͸ɺIPVSMZ EBJMZ ͕ओ
  9. Ͳͷ΁Μ͕ूܭʹ༏͍͠ͷ͔ʁ Kinesis Source Extract Record Flow Upload Log Flow Sink

    ‑‑‑‑‑ TCVDLFUYYYYYYYYZZZZH[ YYYYYY[[[[[H[ ʜ TCVDLFUYYYYZZH[ YY[[H[ ʜ Before After BufferType Memory Local Disc S3 Object Size 5mbલޙ 15mbલޙ ूܭ࣌ؒ 30min 15min
  10. ͓·͚ɿ&$4ͷDPNQVUF3FTPVSDFʹ΋༏͍͠࡞Γʹ Kinesis Source Extract Record Flow Upload Log Flow Sink

    ‑‑‑‑‑ 'MPXͷத਎ ,JOFTJT3FDPSE͔Βɺඞཁͳ৘ใΛൈ͖ग़ͯ͠ɺΩϨΠʹ͢Δ ύʔςΟγϣϯ͝ͱʹ-PDBMͷετϨʔδʹϩάΛόοϑΝϦϯά ϝϞϦফඅ཈੍ʢ(#(#ʣ Ұఆ༰ྔҰఆ࣌ؒܦա͢Δͱ4ʹ1VU0CKFDU ‑‑‑‑‑
  11. ͓ฉ͖͍͖ͨͩ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ