ElasticBeatsを導入してみた話/Go Conference 2016 Spring

by Daichi Hirata

Slide 1

Slide 1 text

&MBTUJD#FBUΛಋೖͯ͠Έͨ࿩ (P$POGFSFODF4QSJOH %BJDIJ)*3"5" $ZCFS"HFOU *OD

Slide 2

Slide 2 text

w גࣜձࣾαΠόʔΤʔδΣϯτ  "EUFDI4UVEJP  ॴଐ w %BJDIJ)*3"5"!EBJDIJME w (P 3VCZ ࣗݾ঺հ

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

"HFOEB w &MBTUJD#FBUTʹ͍ͭͯ w #FBUTͷ࡞Γํ w ߏ੒ͱɺ,JCBOBʹදࣔ͢Δ·Ͱ

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

&MBTUJD#FBUT w %BUB4IJQQFSTGPS&MBTUJDTFBSDI w &MBTUJDTFBSDI޲͚ͷ+40/Λग़ྗ w (P੡ #JOBSZ w 3VOUJNF΁ͷґଘੑ͕ແ͍ w ܰྔͰγϯϓϧ

Slide 7

Slide 7 text

&MBTUJD#FBUT

Slide 8

Slide 8 text

0GGJDJBM#FBUT w 1BDLFUCFBU w 5PQCFBU w 'JMFCFBU w 8JOMPHCFBU

Slide 9

Slide 9 text

1BDLFUCFBU

Slide 10

Slide 10 text

1BDLFUCFBU • )551 • .Z42- • 1PTUHSF42- • 3FEJT • 5ISJGU31$ • .POHP%# • %/4 • .FNDBDIF

Slide 11

Slide 11 text

5PQCFBU

Slide 12

Slide 12 text

'JMFCFBU

Slide 13

Slide 13 text

8JOMPHCFBU

Slide 14

Slide 14 text

$PNNVOJUZ#FBUT w "QBDIFCFBU w /HJOYCFBU w %PDLFSCFBU w )551CFBU w /BHJPTDIFDLCFBU w FUDʜ

Slide 15

Slide 15 text

࣮ࡍʹ͔ͭͬͯΈͨ

Slide 16

Slide 16 text

"QBDIF,BGLB • -JOLFE*O͔Βެ։͞Εͨ෼ࢄϝοηʔδϯά γεςϜ • #SPLFS $MVTUFS 1SPEVDFS $POTVNFSͷ ͭͷίϯϙʔωϯτͰߏ੒͞ΕΔ

Slide 17

Slide 17 text

"QBDIF,BGLB • ϦΞϧλΠϜʹॲཧΛߦ͏ετϦʔϜܥͷॲཧͱɺ େ͖ͳεύΠΫ͕ൃੜ͢ΔॲཧͷҰ࣍ड͚ͱͯ͠ ࢖༻

Slide 18

Slide 18 text

1BSUJUJPO0GGTFU $POTVNFS(SPVQ

Slide 19

Slide 19 text

1BSUJUJPO0GGTFU -BH

Slide 20

Slide 20 text

ϞχλϦϯάπʔϧ • ZBIPPLBGLBNBOBHFS • RVBOUJpOE,BGLB0⒎TFU.POJUPS • MJOLFEJO#VSSPX

Slide 21

Slide 21 text

ϞχλϦϯάπʔϧ • WҎ߱ͷ৽͍͠0⒎TFU4UPSF΁ͷରԠ͕· ͪ·ͪ • 0⒎TFUͷऔಘํ๏͕ͪΐͬͱඍົ • LJCBOBΛ࢖ͬͨΞΫηεϩάͷ؂ࢹ؀ڥ͕ط ʹ͋ΔͷͰɺಉ͡৔ॴͰݟΕΔΑ͏ʹ͍ͨ͠

Slide 22

Slide 22 text

LBGLBCFBU

Slide 23

Slide 23 text

LBGLBCFBU • 4IPQJGZTBSBNBΛ࢖ͬͯ#JOBSZ"1*Λ͋Ε ͜ΕݺΜͰ0⒎TFUΛऔಘ • +PMPLJB +.9XJUI+40/PWFS)551   ܦ༝Ͱ#SPLFS಺෦ͷ৘ใΛऔಘ

Slide 24

Slide 24 text

FMBTUJDCFBUHFOFSBUPS $ cookiecutter https://github.com/elastic/beat-generator.git

Slide 25

Slide 25 text

FMBTUJDCFBUHFOFSBUPS ɾbeater/kafkabeat.go ɾconfig/config.go ɾetc/beat.yml ͜ͷ̏ͭΛฤू͢Δ

Slide 26

Slide 26 text

FMBTUJDCFBUHFOFSBUPS ϥΠϒϥϦͷ؅ཧʹ͸glide

Slide 27

Slide 27 text

Configuration Logging Daemon / Service Publisher Beat custom logic events channel libbeat … MJCCFBU0WFSWJFX beater

Slide 28

Slide 28 text

#FBUFSJOUFSGBDF • #FBUFS͕΍Δ΂͖͜ͱ • 4USBUFHZύλʔϯ

Slide 29

Slide 29 text

$POGJH.FUIPE • ઃఆϑΝΠϧͷಡΈࠐΈ • $-*ϑϥάͷϋϯυϦϯά

Slide 30

Slide 30 text

$POGJH.FUIPE

Slide 31

Slide 31 text

$POGJH.FUIPE

Slide 32

Slide 32 text

4FUVQ.FUIPE • ϝΠϯϧʔϓલͷॳظԽॲཧ • ΫϥΠΞϯτͷ઀ଓͱ͔͸͜͜Ͱߦ͏

Slide 33

Slide 33 text

4FUVQ.FUIPE

Slide 34

Slide 34 text

3VO.FUIPE • ϝΠϯϧʔϓ • σʔλͷऩू • 1VCMJTIFS΁ͷૹ৴ • γάφϧϋϯυϦϯάͳͲ͸MJCCFBUʹ͓೚ͤ

Slide 35

Slide 35 text

3VO.FUIPE

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

$MFBOVQ.FUIPE • 3VOΛൈ͚ͨ௚ޙʹݺ͹ΕΔ • ֬อͨ͠Ϧιʔεͷ։์ͳͲΛߦ͏

Slide 38

Slide 38 text

4UPQ.FUIPE • 3VOͰγάφϧΛड͚Δͱݺ͹ΕΔ • ϝΠϯϧʔϓΛࢭΊΔҝͷॲཧΛߦ͏ • ىಈͨ͠HPSPVUJOFͷఀࢭ

Slide 39

Slide 39 text

EFWUPPMT • JOEFYUFNQMBUFHFOFSBUPS • EPDVNFOUHFOFSBUPS • CFBUQBDLFS • %PDLFSΛ࢖ͬͯDSPTTDPNQJMJOHͱQBDLBHJOH EFCɾSQNɾXJOɾPTY • ,JCBOB%BTICPBSEFYQPSUJNQPSU • FUD

Slide 40

Slide 40 text

Kafka Consumer Log Store Kafka Broker Kafka Broker in_tail GCE VPN elasticsearch elasticsearch kibana ﬂuentd ߏ੒

Slide 41

Slide 41 text

,JCBOB%BTICPBSE

Slide 42

Slide 42 text

·ͱΊ • (Pͷܰྔɾߴ଎ɾϙʔλϏϦςΟ͸#FBUͷΑ ͏ͳ"HFOUʹඇৗʹϚον͢Δ • &MBTUJDTFBSDI,JCBOBͷ؀ڥ͕͋Ε͹͙͢ ʹͰ΋࢝ΊΒΕΔ͓खܰ͞ • ڞ௨෦෼Λ͏·͘੾Γग़͠γϯϓϧʹอͬͯ ͍ΔͨΊɺಠࣗͷ#FBUΛ࡞Δͷ͕؆୯

Slide 43

Slide 43 text

·ͱΊ • LBGLBCFBU͸044Խ༧ఆͰ͢ • ؒʹ߹͍·ͤΜͰͨ͠N @@ N

Slide 44

Slide 44 text

(P΍($1Λ࢖ͬͨ։ൃ͕͍ͨ͠ਓʂ