fluentdサーバをchefで構築するの辛かったけどDockerも別に救世主じゃなかった / tokyu ruby kaigi 12

by h3poteto

Slide 1

Slide 1 text

ﬂuentdαʔόΛchefͰߏங͢Δ ͷਏ͔͚ͬͨͲDocker΋ผʹٹ ੈओ͡Όͳ͔ͬͨ @h3_poteto

Slide 2

Slide 2 text

ࣗݾ঺հ • Akira Fukushimaʢ෱ౡ ໌ʣ • github: h3poteto • twitter: h3_poteto • ࢓ࣄɿterraform৬ਓ@scouty

Slide 3

Slide 3 text

એ఻

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

એ఻ऴΘΓ

Slide 7

Slide 7 text

ﬂuentdαʔόΛchefͰߏங͢Δͷਏ͔ͬ ͚ͨͲDocker΋ผʹٹੈओ͡Όͳ͔ͬͨ ָԂͰ͢ɼࠓ͙͢Dockerʹ͠·͠ΐ͏ @h3_poteto

Slide 8

Slide 8 text

ղܾࡦ͕Θ͔Βͳ͍ͬͯ࿩Λ ͠Α͏ͱࢥͬͯͨͷʹɼ ղܾࡦ͕ݟ͔ͭͬͯ͠·ͬͨ

Slide 9

Slide 9 text

Έͳ͞Μ ϩάͷू໿ʹﬂuentd ࢖ͬͯ·͢ΑͶʁ

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

͜Εͷ࿩

Slide 12

Slide 12 text

aggregatorͷﬂuentdΛ chefͰߏங͍ͯͨ͠

Slide 13

Slide 13 text

Έͳ͞Μchefͷ͜ͱ ֮͑ͯ·͔͢ʁ

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

chefͭΒ͍ • immutable infra? • ͨ·ʹࣦഊ͢Δ • ΈΜͳ΋͏cookbookͷϝϯςͯ͠ͳ͍Ͱ ͠ΐʁ

Slide 16

Slide 16 text

chef-td-agent΋͋Δ

Slide 17

Slide 17 text

td-agentͷcookbook಺Ͱ͸ yum install͍ͯ͠Δ

Slide 18

Slide 18 text

ͳͷʹamazon linux ͨ·ʹݹ͍yumϦϙδτϦΛ ফͯ͘͠Δ

Slide 19

Slide 19 text

͋Δ೔ಥવ Կ΋ͯ͠ͳ͍ͷʹ ಈ͔ͳ͘ͳΔʂ

Slide 20

Slide 20 text

ﬂuentd v0.12ܥΛ ӡ༻ͯͨ͠ࠒͷ࿩

Slide 21

Slide 21 text

td-agentͷґଘʹؒҧ͑ͯ v0.14ܥͷgem͕ࠞೖͨ͠

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

amazon linuxͷyum ฏવͱtd-agent 2.3.5ʹߋ৽ ͞ΕΔ

Slide 24

Slide 24 text

͔͠΋2.3.4͕ফ͞ΕΔ

Slide 25

Slide 25 text

͜ͷޙ1ϲ݄ۙ͘yum͸ v2.3.5ͷ·· ߋ৽͞Εͳ͔ͬͨ

Slide 26

Slide 26 text

͢΂ͯͷΠϯελϯεͰchef ͷprovision͕ࣦഊ͢Δ

Slide 27

Slide 27 text

ͭΒ͍

Slide 28

Slide 28 text

chefࣗମͷόʔδϣϯΞοϓ Λ΍ͬͨͱ͖

Slide 29

Slide 29 text

ࠓ·Ͱrhelͱ൑ఆ͞Ε͍ͯͨ amazon linuxɼ৽ͨʹ amazonͱ͍͏ࣝผࢠΛ৽ઃ ͞ΕΔ

Slide 30

Slide 30 text

platformͷ൑ఆΛ͍ͯͨ͠ cookbook͕΄ͱΜͲ࢖͍෺ ʹͳΒͳ͍

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

࢓ํͳ͍ͷͰҰͭͣͭमਖ਼ͷ Pull Request

Slide 33

Slide 33 text

͚ͩͲະͩʹϦϦʔε͞Εͯ ͍ͳ͍΋ͷ΋͋Δ

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

Ͳ͏ͤ΋͏ΈΜͳcookbook ͷϝϯςͯ͠ͳ͍Ͱ͠ΐ……

Slide 37

Slide 37 text

ͭΒ͍

Slide 38

Slide 38 text

࣮͸΋͏ΈΜͳDockerͰ ӡ༻ͯ͠Δͷʁ ͔ͩΒchefϝϯς ͯ͠ͳ͍ͷʁ

Slide 39

Slide 39 text

গͳ͘ͱ΋Ξοϓσʔτ͸ chefΑΓසൟ

Slide 40

Slide 40 text

σϓϩΠ࣌ͷBuffer Ͳ͏͢Μͷʁ ؾʹͳΔͱ͜Ζ……

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

Memory or File ?

Slide 43

Slide 43 text

Ͳͬͪ΋ίϯςφࡴͨ͠Β ফ͑Δ͡ΌΜʂ ͯࢥͬͯͨΜ͚ͩͲ……

Slide 44

Slide 44 text

ﬂush_at_shutdown

Slide 45

Slide 45 text

flush_at_shutdown • SIGTERMΛड͚औͬͨͱ͖ʹBufferΛflush • flush׬ྃ·Ͱtd-agent͕଴͔ͬͯΒऴྃ • buf_memoryͰ͸σϑΥϧτͰ༗ޮ • SIGKILLͷͱ͖͸ޮ͔ͳ͍

Slide 46

Slide 46 text

ߟ͑ͯͨ͜ͱ • k8sͷPersistentVolumesͰNFSʢ΋͘͠͸ EBSʣΛvolumeϚ΢ϯτ͢Δ • k8sͷPreStopͰbufferΛS3ʹΞοϓϩʔυ͢ Δ • ﬂuentdΫϥελ2ܥ౷࡞Δʁ

Slide 47

Slide 47 text

શ෦͍Βͳ͔ͬͨʂ ﬂush_at_shutdownͰk8sʹ ͦͷ··৐ͤΕ͹ྑ͍ʂ

Slide 48

Slide 48 text

SIGKILLΛ΋Β͏ͱ͖΋͋Δ • td-agentɼtimeoutҎ಺ʹﬂush࢓੾Βͳ͍ͱऴ ྃ͠ͳ͍ • k8sͷGracePeriodSecondsҎ಺ʹίϯςφ͕ ऴྃ͠ͳ͍ͱSIGKILL

Slide 49

Slide 49 text

͜Ε͸ྲྀྔͱͷ݉Ͷ߹͍

Slide 50

Slide 50 text

ৗʹਖ਼ৗऴྃͰ͖ͳ͍ͷͰ͋ Ε͹ɼbufferͷઃఆ΍output ͷεϨου౳Λௐઅͨ͠ํ͕ ͍͍͔΋