AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

by takipone

Slide 1

Slide 1 text

"84ωΠςΟϒͳ&$$3.γεςϜ ӡ༻ʹ͔ܽͤͳ͍ϩάج൫ߏங େ୍ོଠ

Slide 2

Slide 2 text

DNEFWJP

Slide 3

Slide 3 text

ࣗݾ঺հ /BNF େ୍ོଠ!UBLJQPOF +PC 43&!QSJTNBUJY *OUFSFTU ωοτϫʔΫσϓϩΠपΓ 'BWPSJUF "NB[PO3PVUFͱ"$.

Slide 4

Slide 4 text

ΞδΣϯμ "84ωΠςΟϒͳ&$$3.γεςϜ QSJTNBUJY ͱ͸ ϩά෼ੳج൫ͷઃܭ ͭ·͍͍ͮͨͯΔͱ͜Ζ ࠓޙͷల๬

Slide 5

Slide 5 text

1. AWSωΠςΟϒͳ  EC/CRMγεςϜ prismatixͱ͸

Slide 6

Slide 6 text

QSJTNBUJY ҎԼ1[ ͸ɺ&$ͱ$3.γεςϜΛϚΠΫϩαʔϏεͱͯ͠  ఏڙ͠ݸผʹಋೖՄೳͰ͢ɻ֤γεςϜͷػೳʹ"1*Λ׆༻͢Δ͜ͱͰ  γεςϜ։ൃͱӡ༻ͷෳࡶੑɺίετɺ࣌ؒΛ࡟ݮ͠·͢ɻ

Slide 7

Slide 7 text

ߏ੒ྫ prismatix (API) ECαʔόʔ εϚϗΞϓϦ ECαΠτ

Slide 8

Slide 8 text

ࣄྫύϧί༷

Slide 9

Slide 9 text

"84ωΠςΟϒͳΠϯϑϥߏ੒ ϩʔυόϥϯα (ෛՙ෼ࢄ/TLSऴ୺) ίϯςφΫϥελ (APαʔόʔ) σʔλϕʔε (ϚωʔδυαʔϏε)

Slide 10

Slide 10 text

ϚΠΫϩαʔϏεؒͷ࿈ܞ ඇಉظૄ݁߹ͷ͘͠Έ ঎඼αʔϏε Amazon SNS (௨஌) Amazon SQS (δϣϒΩϡʔ) ࡏݿαʔϏε (ϫʔΧʔ) ঎඼ొ࿥ Πϕϯτૹ৴ Πϕϯτ͕ Ωϡʔʹཷ·Δ ΠϕϯτΛऔಘͯ͠  ࡏݿσʔλʹ൓ө ΠϕϯτΛసૹ

Slide 11

Slide 11 text

ϚΠΫϩαʔϏεؒͷ࿈ܞ αʔϏεͷϦιʔε૿ݮ εέʔϧΠϯΞ΢τ ͠΍͍͢ ঎඼αʔϏεΛ૿ڧ ࡏݿαʔϏεΛ૿ڧ

Slide 12

Slide 12 text

2. ϩά෼ੳج൫ͷઃܭ

Slide 13

Slide 13 text

QSJTNBUJYͷϩά ओʹछྨɺ͍ͣΕ΋+40/ܗࣜ ✦ ΞϓϦϩά 4QSJOH +BWB ىಈ࣌΍Τϥʔͷϝοηʔδ ✦ .%$ϩά .BQQFE%JBHOPTUJD$POUFYUT ϚΠΫϩαʔϏεͷϦΫΤετͱϨεϙϯεͷҰ෦ ͋ͱεϩʔΫΤϦϩάͳͲ

Slide 14

Slide 14 text

ϩάͷ༻్ ✦ τϥϒϧγϡʔςΟϯά ‣ ϚΠΫϩαʔϏεͷಈ࡞֬ೝ ‣ ঎඼ݕࡧ΍஫จͱͷಥ͖߹Θͤ ✦ ϩά෼ੳ ‣ όʔήϯηʔϧ΍4/4ͷεύΠΫͷࣄલରࡦͷجૅࢿྉ औΓ͜΅ͨ͘͠ͳ͍

Slide 15

Slide 15 text

ϩάج൫ͷઃܭࢥ૝ ✦ ͦΕͳΓʹେྔ݄ؒʙ5# ✦ 410' 4JOHMF1PJOUPG'BJMVSF Λආ͚͍ͨ ✦ ͳΔ΂͘༗Γ෺Λ࢖͍͍ͨ ✦ ༻్ʹΑͬͯҟͳΔಛੑ ‣ σʔλϨΠΫϦΞϧλΠϜϩάͷ૊Έ߹Θͤ

Slide 16

Slide 16 text

ϩάج൫ͷߏ੒ਤ Amazon ECS (ίϯςφΫϥελ) Amazon S3 (σʔλϨΠΫ) CloudWatch Logs (ϦΞϧλΠϜϩά) Treasure Data (ϩά෼ੳ)

Slide 17

Slide 17 text

"NB[PO&$4ίϯςφΫϥελ ✦ %PDLFSίϯςφΛ؅ཧ͢ΔϚωʔδυαʔϏε ‣ ίϯςφΛͲͷΠϯελϯε Ծ૝Ϛγϯ Ͱ࣮ߦ͢Δ ͔ΛΫϥελ͝ͱʹ؅ཧ ‣ QSJTNBUJYͰ͸'BSHBUF͸ະ࠾༻ ✦ %PDLFSʹίϯςφϩά ඪ४ग़ྗ Λѻ͏ MPHHJOHESJWFSػೳ͕͋Δ ‣ ίϯςφ͸ϩάϑΝΠϧΛѻΘͣɺϩάΛ+40/ܗࣜͰ ඪ४ग़ྗʹग़͢Α͏ΞϓϦΛߏ੒ ECSΫϥελ

Slide 18

Slide 18 text

"NB[PO4σʔλϨΠΫ ✦ ΦϯϥΠϯετϨʔδαʔϏε ‣ ߴ͍଱ٱੑ ‣ ߴ͍εέʔϥϏϦςΟ ༰ྔແ੍ݶ ‣ ྿Ձɺબ΂ΔετϨʔδΫϥε ‣ σʔλ෼ੳαʔϏεͱͷ࿈ܞ

Slide 19

Slide 19 text

"NB[PO$MPVE8BUDI-PHTϦΞϧλΠϜϩά ✦ ϚωʔδυͷϩάอଘαʔϏε ‣ ߴ͍εέʔϥϏϦςΟ ‣ ४ϦΞϧλΠϜࢀর ‣ +40/΁ͷΫΤϦΛαϙʔτ

Slide 20

Slide 20 text

"SN5SFBTVSF%BUBϩά෼ੳ ✦ σʔλ෼ੳͷ4BB4αʔϏε ඇ"84 ‣ ඇߏ଄Խσʔλʹૉૣ͘ΞΫηεͰ͖Δ  ετϨʔδ 1MB[NB%# ‣ ฒྻΫΤϦΤϯδϯ 1SFTUPͳͲ Ͱ  ΫΤϦͰ͖Δ

Slide 21

Slide 21 text

༗Γ෺ʹ͸ݶք͋Γʢʣ Amazon ECS Amazon S3 CloudWatch Logs MPHHJOHESJWFSͷ੍໿ ⭕ $MPVE8BUDI-PHTΛαϙʔτ ❌ 4ʹ͸௚઀޲͚ΒΕͳ͍ ❌ ෳ਺ग़ྗʹະରԠ ◦ ×

Slide 22

Slide 22 text

༗Γ෺ʹ͸ݶք͋Γʢʣ Amazon S3 Treasure Data %BUB$POOFDUPSͱ͍͏5SFBTVSF %BUBͷΠϯϙʔτػೳ͕͋Δ ⭕ εέδϡʔϧػೳ͕͋Γɺ೔࣍ͷ  Πϯϙʔτ͸͜ΕͰ0, ❌ Πϯϙʔτ࣌ͷϦιʔε΍ಉ࣮࣌ߦ਺ ʹ੍ݶ͕͋ΓɺॳճΠϯϙʔτʹ͸ن ໛ײ͕߹Θͳ͍ 044ͷ&NCVMLͱ%JHEBH͕தͰ  ಈ͍͍ͯΔ

Slide 23

Slide 23 text

ෆ଍Λิ͏ͨΊʹ044Λར༻ Fluentd (ετϦʔϜॲཧ) Embulk (όονॲཧ) ͲͪΒ΋ॊೈͰ๛෋ͳϓϥάΠϯΤίγεςϜ͋Γ ͨ·ͨ·5SFBTVSF%BUB੡Ͱ5%ͱͷߴ͍਌࿨ੑ

Slide 24

Slide 24 text

%PDLFSº'MVFOUE %PDLFSMPHHJOHESJWFS͕'MVFOUE΁ͷ ૹ৴Λαϙʔτ ‣ 'MVFOUEࣗମ΋%PDLFSίϯςφͱͯ͠  &$4ͷ֤ΠϯελϯεͰ࣮ߦ ‣ ϚΠΫϩαʔϏεͷίϯςφ͔Β͸  ಉΠϯελϯεͷ'MVFOUEʹϩάΛૹ৴ Fluentd  ίϯςφ

Slide 25

Slide 25 text

'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒ 4ͱ$MPVE8BUDI-PHT  ͷ྆ํʹอଘ Amazon S3 CloudWatch Logs @type copy @type s3 : @type forward : host fluentd.example.lo port 24224

Slide 26

Slide 26 text

'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒ ✦ ϦΞϧλΠϜϩά͸"HHSFHBUPS ू໿αʔόʔ Λ  ௥Ճͨ͠ଟஈߏ੒ ‣ ͋ͱ͔ΒৼΓઌΛม͑΍͍͢Α͏ʹ ‣ $MPVE8BUDI-PHTͷ"1*ίʔϧͷ੍໿ରࡦ CloudWatch Logs Aggregator Forwarder

Slide 27

Slide 27 text

'MVFOUEͷσʔλՃ޻ ✦ 'PSXBSEFS ‣ +40/ͷύʔε ‣ ΞϓϦϩάͱ.%$ϩάͷ  ۠෼͚ ‣ 4ͷύεϓϨϑΟοΫε ✦ "HHSFHBUPS ‣ Τϥʔͷநग़ ‣ $MPVE8BUDI-PHTύϥϝʔλ @type parser format json key_name log : @type rewrite_tag_filter key marker pattern AUDIT tag ${tag}.audit key message pattern .+ tag ${tag}.app

Slide 28

Slide 28 text

&NCVML ✦ 4ˠ5%΁ͷॳظόονΠϯϙʔτ ‣ 5%ͷઃఆ͕ͱʹָ͔ͩͬͨ͘ ‣ %BUB$POOFDUPSͷίϯϑΟά͕Ұ෦࢖͍ճͤͨ

Slide 29

Slide 29 text

Embulkͷฒྻ࣮ߦ͸ Ͳ͏͢Δʁ

Slide 30

Slide 30 text

&NCVMLPO"84#BUDI ✦ "84#BUDIͷδϣϒΩϡʔ  εϙοτΠϯελϯε׆༻ ‣ δϣϒͷ಺༰͸%PDLFSίϯςφͰ ࣮ߦ͢ΔίϚϯυϥΠϯ ✦ &NCVMLͷ%PDLFSΠϝʔδΛ  ༻ҙ࣮ͯ͠ߦ ‣ IUUQTIVCEPDLFSDPNSDMBTTNFUIPE FNCVMLNFUTUE εϙοτϑϦʔτ δϣϒΩϡʔ AWS Batch

Slide 31

Slide 31 text

&NCVMLͷίϯϑΟά͸؀ڥม਺Λଟ༻ exec: max_threads: {{ env.MAX_THREADS }} in: type: s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH } path_match_patterns: \.gz$ auth_method: instance endpoint: s3-ap-northeast-1.amazonaws.com parser: type: jsonl : decoders: - { type: gzip } out: type: td endpoint: api.treasuredata.com apikey: {{ env.TDAPIKEY }} database: {{ env.TDDATABASE }} table: {{ env.TDTABLE }}

Slide 32

Slide 32 text

3. ͭ·͍ͮͨ/͍ͯΔ  ͱ͜Ζ

Slide 33

Slide 33 text

Fluentd Aggregatorͷ ৑௕Խ/εέʔϧΞ΢τ CloudWatch Logs Aggregator Forwarder

Slide 34

Slide 34 text

"HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ ✦ ෳ਺ͷ"HHSFHBUPSཱ͕ͯΒΕͳ͍ ‣ $MPVE8BUDI-PHTͰ͸ಉҰͷϩάετϦʔϜʹॻ͖ࠐΉ ͱ͖͸TFRVFODFUPLFOΛҡ࣋͠ͳ͚Ε͹ͳΒͳ͍ @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name container_id auto_create_stream true ϚΠΫϩαʔϏε ͷίϯςφ*% ϩάάϧʔϓ ϩά ετϦʔϜ Aggregator ×

Slide 35

Slide 35 text

"HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ ✦ ϩάετϦʔϜ໊ʹͩ͜ΘΒͳ͍ ‣ "HHSFHBUPS͝ͱʹϩάετϦʔϜΛׂΓ౰ͯΔ ‣ ʮΠϕϯτͷݕࡧʯ͔ΒετϦʔϜԣஅͷݕࡧΛར༻ @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name "#{Socket.gethostname}" auto_create_stream true "HHSFHBUPS ͷίϯςφ*%

Slide 36

Slide 36 text

%PDLFSͷϗετϙʔτͷ੍໿ ✦ "HHSFHBUPS΋%PDLFSίϯςφ Ͱ࣮ߦ ‣ ϗετϙʔτ͕ݻఆͰϙʔτ Λ઎༗ ‣ Πϯελϯεʹίϯςφ͔͠  ࣮ߦͰ͖ͳ͍ ✦ ϗετϙʔτΛಈతʹ͢ΔͨΊʹ ϩʔυόϥϯαΛ$-#ˠ/-#ʹ  Ҡߦ NLB Aggregator ίϯςφ

Slide 37

Slide 37 text

TD΁ͷΠϯϙʔτͷδϣϒ෼ׂ

Slide 38

Slide 38 text

Πϯϙʔτͷδϣϒ෼ׂ ✦ Πϯϙʔτର৅4ϓϨϑΟοΫεΛ޿͘औΔͱ  ϝϞϦΊͬͪΌ࢖͏ ‣ %BUB$POOFDUPSϩά͕ফ͑ͯແݶϦτϥΠ˞೥݄౰࣌ ‣ "84#BUDIϝϞϦׂ౰্ݶ·Ͱ࢖ͬͯ00.,JMMFSൃಈ $ aws s3 ls --profile cm-jp-1 s3://XXXX-infra-logbucket-XXXX/applications  /condor/app/2018/10/04/05/ 2018-10-04 14:06:24 5964 00_5fbef6f0fdec_0.gz 2018-10-04 14:11:26 5971 05_5fbef6f0fdec_0.gz 2018-10-04 14:16:24 5932 10_5fbef6f0fdec_0.gz 2018-10-04 14:21:25 5954 15_5fbef6f0fdec_0.gz

Slide 39

Slide 39 text

Πϯϙʔτͷδϣϒ෼ׂ ✦ ͍Ζ͍Ζࢼͯ͠ɺ೔୯ҐͰδϣϒΛ੾Δͷ͕  ྑͦ͞͏ͱ͍͏͜ͱʹ ‣ ϑΝΠϧαΠζɺϑΝΠϧ਺ʹґଘ͢ΔͷͰ΍ͬͯΈͳ ͍ͱΘ͔Βͳ͍෦෼ ࠓճ͸ϑΝΠϧ਺͕ଟ͔ͬͨ໛༷ in: type: s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/ {{ env.MONTH }}/{{ env.DAY }}

Slide 40

Slide 40 text

5SFBTVSF%BUBͷ'"2ʹ΋هࡌ͋Γ 2 %BUB$POOFDUPSGPS4KPCJTSVOOJOHGPSBMPOH UJNF XIBUDBO*EP " $IFDLUIFDPVOUPG4pMFTUIBUZPVSDPOOFDUPS KPCJTJOHFTUJOH*GUIFSFBSFPWFS pMFT UIF QFSGPSNBODFEFHSBEFT  5PNJUJHBUFUIJTJTTVF ZPVDBO ‣ /BSSPXQBUI@QSFpYPQUJPOBOESFEVDFUIFDPVOUPG4pMFT ‣ 4FU .# UPNJO@UBTL@TJ[FPQUJPO IUUQTTVQQPSUUSFBTVSFEBUBDPNIDFOVTBSUJDMFT%BUB$POOFDUPS GPS"NB[PO4'"2GPSUIF4%BUB$POOFDUPS

Slide 41

Slide 41 text

ϩάαΠζͷ্ݶ

Slide 42

Slide 42 text

ϩάαΠζͷ্ݶ ✦ .%$ϩά͕σΧ͍ ✦ %PDLFSMPHHJOHESJWFS͸  ,#ΑΓେ͖͍ϩάΛ෼ׂ͢Δ ‣ 'MVFOUEGPSXBSEFSͷ+40/ύʔεʹࣦഊɺࣺͯΒΕΔ ‣ DPODBUϓϥάΠϯͰ݁߹ ‣ ϝϞϦ࢖༻ྔ͕௓Ͷͯ00.,JMMFSʹࡴ͞ΕͨΠϚίί ✦ $MPVE8BUDI-PHTͷ্ݶ͸,#

Slide 43

Slide 43 text

͓ۚ ✦ $MPVE8BUDI-PHT͸ϩάσʔλྔͷैྔ՝ۚ ‣ ྔ͕ଟ͍ͷͰֹ݄අ༻͕͔͞Ή ‣ &$ͳͲଞͷαʔϏεඅ༻ͱτϯτϯͱݴΘΕΔͱͭΒ ͍ ✦ ΞϓϦ͔Βग़͢ϩάͷ෼ྔΛݮΒ͢ ‣ .%$ϩάͭΒ͍

Slide 44

Slide 44 text

4. ࠓޙͷల๬

Slide 45

Slide 45 text

࣍ͷҰख ✦ ϩά؂ࢹ ✦ ϩάू໿ͱͯ͠&MBTUJDTFBSDI,JCBOBͷར༻ ✦ 5SFBTVSF%BUBΛ΋ͬͱ׆༻͍ͨ͠

Slide 46

Slide 46 text

·ͱΊ ✦ εέʔϧ͢Δϩά؅ཧج൫͸Ϋϥ΢υαʔϏεͷ  ૊Έ߹Θ͕ͤΦεεϝ ‣ σʔλϨΠΫͱϦΞϧλΠϜϩάͰͷαʔϏεͷ࢖͍෼͚ ✦ ૊Έ߹Θ͚ͤͩͰ΋ಈ͘΋ͷ͸Ͱ͖Δ͚Ͳ ‣ ཁ݅ʹରͯ͠଍Γͳ͍ͱ͜Ζͷิ͕ؒඞཁ ‣ σʔλྔɺαΠζͷධՁ΍νϡʔχϯά͕ඞཁ