Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ElasticBeatsを導入してみた話/Go Conference 2016 Spring

ElasticBeatsを導入してみた話/Go Conference 2016 Spring

4c1731ac4187db79a9b16d621aa94bdc?s=128

Daichi Hirata

April 23, 2016
Tweet

Transcript

  1. &MBTUJD#FBUΛಋೖͯ͠Έͨ࿩ (P$POGFSFODF4QSJOH %BJDIJ)*3"5" $ZCFS"HFOU *OD

  2. w גࣜձࣾαΠόʔΤʔδΣϯτ
 "EUFDI4UVEJP
 ॴଐ w %BJDIJ)*3"5"!EBJDIJME w (P 3VCZ ࣗݾ঺հ

  3. None
  4. "HFOEB w &MBTUJD#FBUTʹ͍ͭͯ w #FBUTͷ࡞Γํ w ߏ੒ͱɺ,JCBOBʹදࣔ͢Δ·Ͱ

  5. None
  6. &MBTUJD#FBUT w %BUB4IJQQFSTGPS&MBTUJDTFBSDI w &MBTUJDTFBSDI޲͚ͷ+40/Λग़ྗ w (P੡ #JOBSZ  w

    3VOUJNF΁ͷґଘੑ͕ແ͍ w ܰྔͰγϯϓϧ
  7. &MBTUJD#FBUT

  8. 0GGJDJBM#FBUT w 1BDLFUCFBU w 5PQCFBU w 'JMFCFBU w 8JOMPHCFBU

  9. 1BDLFUCFBU

  10. 1BDLFUCFBU • )551 • .Z42- • 1PTUHSF42- • 3FEJT •

    5ISJGU31$ • .POHP%# • %/4 • .FNDBDIF
  11. 5PQCFBU

  12. 'JMFCFBU

  13. 8JOMPHCFBU

  14. $PNNVOJUZ#FBUT w "QBDIFCFBU w /HJOYCFBU w %PDLFSCFBU w )551CFBU w

    /BHJPTDIFDLCFBU w FUDʜ
  15. ࣮ࡍʹ͔ͭͬͯΈͨ

  16. "QBDIF,BGLB • -JOLFE*O͔Βެ։͞Εͨ෼ࢄϝοηʔδϯά γεςϜ • #SPLFS $MVTUFS 1SPEVDFS $POTVNFSͷ ͭͷίϯϙʔωϯτͰߏ੒͞ΕΔ

  17. "QBDIF,BGLB • ϦΞϧλΠϜʹॲཧΛߦ͏ετϦʔϜܥͷॲཧͱɺ େ͖ͳεύΠΫ͕ൃੜ͢ΔॲཧͷҰ࣍ड͚ͱͯ͠ ࢖༻

  18. 1BSUJUJPO0GGTFU $POTVNFS(SPVQ

  19. 1BSUJUJPO0GGTFU -BH

  20. ϞχλϦϯάπʔϧ • ZBIPPLBGLBNBOBHFS • RVBOUJpOE,BGLB0⒎TFU.POJUPS • MJOLFEJO#VSSPX

  21. ϞχλϦϯάπʔϧ • WҎ߱ͷ৽͍͠0⒎TFU4UPSF΁ͷରԠ͕· ͪ·ͪ • 0⒎TFUͷऔಘํ๏͕ͪΐͬͱඍົ • LJCBOBΛ࢖ͬͨΞΫηεϩάͷ؂ࢹ؀ڥ͕ط ʹ͋ΔͷͰɺಉ͡৔ॴͰݟΕΔΑ͏ʹ͍ͨ͠

  22. LBGLBCFBU

  23. LBGLBCFBU • 4IPQJGZTBSBNBΛ࢖ͬͯ#JOBSZ"1*Λ͋Ε ͜ΕݺΜͰ0⒎TFUΛऔಘ • +PMPLJB +.9XJUI+40/PWFS)551 
 ܦ༝Ͱ#SPLFS಺෦ͷ৘ใΛऔಘ

  24. FMBTUJDCFBUHFOFSBUPS $ cookiecutter https://github.com/elastic/beat-generator.git

  25. FMBTUJDCFBUHFOFSBUPS ɾbeater/kafkabeat.go ɾconfig/config.go ɾetc/beat.yml ͜ͷ̏ͭΛฤू͢Δ

  26. FMBTUJDCFBUHFOFSBUPS ϥΠϒϥϦͷ؅ཧʹ͸glide

  27. Configuration Logging Daemon / Service Publisher Beat custom logic events

    channel libbeat … MJCCFBU0WFSWJFX beater
  28. #FBUFSJOUFSGBDF • #FBUFS͕΍Δ΂͖͜ͱ • 4USBUFHZύλʔϯ

  29. $POGJH.FUIPE • ઃఆϑΝΠϧͷಡΈࠐΈ • $-*ϑϥάͷϋϯυϦϯά

  30. $POGJH.FUIPE

  31. $POGJH.FUIPE

  32. 4FUVQ.FUIPE • ϝΠϯϧʔϓલͷॳظԽॲཧ • ΫϥΠΞϯτͷ઀ଓͱ͔͸͜͜Ͱߦ͏

  33. 4FUVQ.FUIPE

  34. 3VO.FUIPE • ϝΠϯϧʔϓ • σʔλͷऩू • 1VCMJTIFS΁ͷૹ৴ • γάφϧϋϯυϦϯάͳͲ͸MJCCFBUʹ͓೚ͤ

  35. 3VO.FUIPE

  36. None
  37. $MFBOVQ.FUIPE • 3VOΛൈ͚ͨ௚ޙʹݺ͹ΕΔ • ֬อͨ͠Ϧιʔεͷ։์ͳͲΛߦ͏

  38. 4UPQ.FUIPE • 3VOͰγάφϧΛड͚Δͱݺ͹ΕΔ • ϝΠϯϧʔϓΛࢭΊΔҝͷॲཧΛߦ͏ • ىಈͨ͠HPSPVUJOFͷఀࢭ

  39. EFWUPPMT • JOEFYUFNQMBUFHFOFSBUPS • EPDVNFOUHFOFSBUPS • CFBUQBDLFS • %PDLFSΛ࢖ͬͯDSPTTDPNQJMJOHͱQBDLBHJOH EFCɾSQNɾXJOɾPTY

     • ,JCBOB%BTICPBSEFYQPSUJNQPSU • FUD
  40. Kafka Consumer Log Store Kafka Broker Kafka Broker in_tail GCE

    VPN elasticsearch elasticsearch kibana fluentd ߏ੒
  41. ,JCBOB%BTICPBSE

  42. ·ͱΊ • (Pͷܰྔɾߴ଎ɾϙʔλϏϦςΟ͸#FBUͷΑ ͏ͳ"HFOUʹඇৗʹϚον͢Δ • &MBTUJDTFBSDI ,JCBOBͷ؀ڥ͕͋Ε͹͙͢ ʹͰ΋࢝ΊΒΕΔ͓खܰ͞ • ڞ௨෦෼Λ͏·͘੾Γग़͠γϯϓϧʹอͬͯ

    ͍ΔͨΊɺಠࣗͷ#FBUΛ࡞Δͷ͕؆୯
  43. ·ͱΊ • LBGLBCFBU͸044Խ༧ఆͰ͢ • ؒʹ߹͍·ͤΜͰͨ͠N @@ N

  44. (P΍($1Λ࢖ͬͨ։ൃ͕͍ͨ͠ਓʂ