2018/10/05(金)@UDX秋葉原 Developers.IO 2018のセッション資料
"84ωΠςΟϒͳ&$$3.γεςϜӡ༻ʹ͔ܽͤͳ͍ϩάج൫ߏஙେ୍ོଠ
View Slide
DNEFWJP
ࣗݾհ /BNF େ୍ོଠ!UBLJQPOF+PC 43&!QSJTNBUJY*OUFSFTU ωοτϫʔΫσϓϩΠपΓ'BWPSJUF "NB[PO3PVUFͱ"$.
ΞδΣϯμ "84ωΠςΟϒͳ&$$3.γεςϜ QSJTNBUJYͱ ϩάੳج൫ͷઃܭ ͭ·͍͍ͮͨͯΔͱ͜Ζ ࠓޙͷల
1. AWSωΠςΟϒͳ EC/CRMγεςϜprismatixͱ
QSJTNBUJY ҎԼ1[ɺ&$ͱ$3.γεςϜΛϚΠΫϩαʔϏεͱͯ͠ ఏڙ͠ݸผʹಋೖՄೳͰ͢ɻ֤γεςϜͷػೳʹ"1*Λ׆༻͢Δ͜ͱͰ γεςϜ։ൃͱӡ༻ͷෳࡶੑɺίετɺ࣌ؒΛݮ͠·͢ɻ
ߏྫ prismatix(API)ECαʔόʔεϚϗΞϓϦECαΠτ
ࣄྫύϧί༷
"84ωΠςΟϒͳΠϯϑϥߏ ϩʔυόϥϯα(ෛՙࢄ/TLSऴ)ίϯςφΫϥελ(APαʔόʔ)σʔλϕʔε(ϚωʔδυαʔϏε)
ϚΠΫϩαʔϏεؒͷ࿈ܞ ඇಉظૄ݁߹ͷ͘͠ΈαʔϏε Amazon SNS(௨)Amazon SQS(δϣϒΩϡʔ)ࡏݿαʔϏε(ϫʔΧʔ)ొΠϕϯτૹ৴Πϕϯτ͕Ωϡʔʹཷ·Δ ΠϕϯτΛऔಘͯ͠ ࡏݿσʔλʹөΠϕϯτΛసૹ
ϚΠΫϩαʔϏεؒͷ࿈ܞ αʔϏεͷϦιʔε૿ݮεέʔϧΠϯΞτ͍͢͠αʔϏεΛ૿ڧࡏݿαʔϏεΛ૿ڧ
2. ϩάੳج൫ͷઃܭ
QSJTNBUJYͷϩά ओʹछྨɺ͍ͣΕ+40/ܗࣜ✦ ΞϓϦϩά 4QSJOH +BWBىಈ࣌Τϥʔͷϝοηʔδ✦ .%$ϩά .BQQFE%JBHOPTUJD$POUFYUTϚΠΫϩαʔϏεͷϦΫΤετͱϨεϙϯεͷҰ෦͋ͱεϩʔΫΤϦϩάͳͲ
ϩάͷ༻్ ✦ τϥϒϧγϡʔςΟϯά‣ ϚΠΫϩαʔϏεͷಈ࡞֬ೝ‣ ݕࡧจͱͷಥ͖߹Θͤ✦ ϩάੳ‣ όʔήϯηʔϧ4/4ͷεύΠΫͷࣄલରࡦͷجૅࢿྉऔΓ͜΅ͨ͘͠ͳ͍
ϩάج൫ͷઃܭࢥ ✦ ͦΕͳΓʹେྔ݄ؒʙ5#✦ 410' 4JOHMF1PJOUPG'BJMVSFΛආ͚͍ͨ✦ ͳΔ͘༗ΓΛ͍͍ͨ✦ ༻్ʹΑͬͯҟͳΔಛੑ‣ σʔλϨΠΫϦΞϧλΠϜϩάͷΈ߹Θͤ
ϩάج൫ͷߏਤ Amazon ECS(ίϯςφΫϥελ)Amazon S3(σʔλϨΠΫ)CloudWatch Logs(ϦΞϧλΠϜϩά)Treasure Data(ϩάੳ)
"NB[PO&$4ίϯςφΫϥελ ✦ %PDLFSίϯςφΛཧ͢ΔϚωʔδυαʔϏε‣ ίϯςφΛͲͷΠϯελϯε ԾϚγϯͰ࣮ߦ͢Δ͔ΛΫϥελ͝ͱʹཧ‣ QSJTNBUJYͰ'BSHBUFະ࠾༻✦ %PDLFSʹίϯςφϩά ඪ४ग़ྗΛѻ͏MPHHJOHESJWFSػೳ͕͋Δ‣ ίϯςφϩάϑΝΠϧΛѻΘͣɺϩάΛ+40/ܗࣜͰඪ४ग़ྗʹग़͢Α͏ΞϓϦΛߏECSΫϥελ
"NB[PO4σʔλϨΠΫ ✦ ΦϯϥΠϯετϨʔδαʔϏε‣ ߴ͍ٱੑ‣ ߴ͍εέʔϥϏϦςΟ ༰ྔແ੍ݶ‣ ྿ՁɺબΔετϨʔδΫϥε‣ σʔλੳαʔϏεͱͷ࿈ܞ
"NB[PO$MPVE8BUDI-PHTϦΞϧλΠϜϩά ✦ ϚωʔδυͷϩάอଘαʔϏε‣ ߴ͍εέʔϥϏϦςΟ‣ ४ϦΞϧλΠϜࢀর‣ +40/ͷΫΤϦΛαϙʔτ
"SN5SFBTVSF%BUBϩάੳ ✦ σʔλੳͷ4BB4αʔϏε ඇ"84‣ ඇߏԽσʔλʹૉૣ͘ΞΫηεͰ͖Δ ετϨʔδ 1MB[NB%#‣ ฒྻΫΤϦΤϯδϯ 1SFTUPͳͲͰ ΫΤϦͰ͖Δ
༗Γʹݶք͋Γʢʣ AmazonECSAmazonS3CloudWatchLogsMPHHJOHESJWFSͷ੍⭕ $MPVE8BUDI-PHTΛαϙʔτ❌ 4ʹ͚ΒΕͳ͍❌ ෳग़ྗʹະରԠ◦×
༗Γʹݶք͋Γʢʣ AmazonS3TreasureData%BUB$POOFDUPSͱ͍͏5SFBTVSF%BUBͷΠϯϙʔτػೳ͕͋Δ⭕ εέδϡʔϧػೳ͕͋Γɺ࣍ͷ Πϯϙʔτ͜ΕͰ0,❌ Πϯϙʔτ࣌ͷϦιʔεಉ࣮࣌ߦʹ੍ݶ͕͋ΓɺॳճΠϯϙʔτʹنײ͕߹Θͳ͍044ͷ&NCVMLͱ%JHEBH͕தͰ ಈ͍͍ͯΔ
ෆΛิ͏ͨΊʹ044Λར༻ Fluentd(ετϦʔϜॲཧ)Embulk(όονॲཧ)ͲͪΒॊೈͰ๛ͳϓϥάΠϯΤίγεςϜ͋Γͨ·ͨ·5SFBTVSF%BUBͰ5%ͱͷߴ͍ੑ
%PDLFSº'MVFOUE %PDLFSMPHHJOHESJWFS͕'MVFOUEͷૹ৴Λαϙʔτ‣ 'MVFOUEࣗମ%PDLFSίϯςφͱͯ͠ &$4ͷ֤ΠϯελϯεͰ࣮ߦ‣ ϚΠΫϩαʔϏεͷίϯςφ͔Β ಉΠϯελϯεͷ'MVFOUEʹϩάΛૹ৴Fluentd ίϯςφ
'MVFOUEͷෳ0VUQVUͱଟஈߏ 4ͱ$MPVE8BUDI-PHT ͷ྆ํʹอଘAmazonS3CloudWatchLogs@type copy@type s3:@type forward:host fluentd.example.loport 24224
'MVFOUEͷෳ0VUQVUͱଟஈߏ ✦ ϦΞϧλΠϜϩά"HHSFHBUPS ूαʔόʔΛ Ճͨ͠ଟஈߏ‣ ͋ͱ͔ΒৼΓઌΛม͍͑͢Α͏ʹ‣ $MPVE8BUDI-PHTͷ"1*ίʔϧͷ੍ରࡦCloudWatchLogsAggregatorForwarder
'MVFOUEͷσʔλՃ ✦ 'PSXBSEFS‣ +40/ͷύʔε‣ ΞϓϦϩάͱ.%$ϩάͷ ͚۠‣ 4ͷύεϓϨϑΟοΫε✦ "HHSFHBUPS‣ Τϥʔͷநग़‣ $MPVE8BUDI-PHTύϥϝʔλ@type parserformat jsonkey_name log:@type rewrite_tag_filterkey markerpattern AUDITtag ${tag}.auditkey messagepattern .+tag ${tag}.app
&NCVML ✦ 4ˠ5%ͷॳظόονΠϯϙʔτ‣ 5%ͷઃఆ͕ͱʹָ͔ͩͬͨ͘‣ %BUB$POOFDUPSͷίϯϑΟά͕Ұ෦͍ճͤͨ
Embulkͷฒྻ࣮ߦͲ͏͢Δʁ
&NCVMLPO"84#BUDI ✦ "84#BUDIͷδϣϒΩϡʔ εϙοτΠϯελϯε׆༻‣ δϣϒͷ༰%PDLFSίϯςφͰ࣮ߦ͢ΔίϚϯυϥΠϯ✦ &NCVMLͷ%PDLFSΠϝʔδΛ ༻ҙ࣮ͯ͠ߦ‣ IUUQTIVCEPDLFSDPNSDMBTTNFUIPEFNCVMLNFUTUEεϙοτϑϦʔτδϣϒΩϡʔAWSBatch
&NCVMLͷίϯϑΟάڥมΛଟ༻ exec:max_threads: {{ env.MAX_THREADS }}in:type: s3bucket: {{ env.S3BUCKET }}path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH }path_match_patterns: \.gz$auth_method: instanceendpoint: s3-ap-northeast-1.amazonaws.comparser:type: jsonl:decoders:- { type: gzip }out:type: tdendpoint: api.treasuredata.comapikey: {{ env.TDAPIKEY }}database: {{ env.TDDATABASE }}table: {{ env.TDTABLE }}
3. ͭ·͍ͮͨ/͍ͯΔ ͱ͜Ζ
Fluentd AggregatorͷԽ/εέʔϧΞτCloudWatchLogsAggregatorForwarder
"HHSFHBUPSͷԽεέʔϧΞτ ✦ ෳͷ"HHSFHBUPSཱ͕ͯΒΕͳ͍‣ $MPVE8BUDI-PHTͰಉҰͷϩάετϦʔϜʹॻ͖ࠐΉͱ͖TFRVFODFUPLFOΛҡ࣋͠ͳ͚ΕͳΒͳ͍@type cloudwatch_logsregion "#{ENV['AWS_REGION']}"log_group_name_key servicelog_stream_name container_idauto_create_stream trueϚΠΫϩαʔϏεͷίϯςφ*%ϩάάϧʔϓϩάετϦʔϜAggregator×
"HHSFHBUPSͷԽεέʔϧΞτ ✦ ϩάετϦʔϜ໊ʹͩ͜ΘΒͳ͍‣ "HHSFHBUPS͝ͱʹϩάετϦʔϜΛׂΓͯΔ‣ ʮΠϕϯτͷݕࡧʯ͔ΒετϦʔϜԣஅͷݕࡧΛར༻@type cloudwatch_logsregion "#{ENV['AWS_REGION']}"log_group_name_key servicelog_stream_name "#{Socket.gethostname}"auto_create_stream true"HHSFHBUPSͷίϯςφ*%
%PDLFSͷϗετϙʔτͷ੍ ✦ "HHSFHBUPS%PDLFSίϯςφͰ࣮ߦ‣ ϗετϙʔτ͕ݻఆͰϙʔτΛ༗‣ Πϯελϯεʹίϯςφ͔͠ ࣮ߦͰ͖ͳ͍✦ ϗετϙʔτΛಈతʹ͢ΔͨΊʹϩʔυόϥϯαΛ$-#ˠ/-#ʹ Ҡߦ NLBAggregatorίϯςφ
TDͷΠϯϙʔτͷδϣϒׂ
Πϯϙʔτͷδϣϒׂ ✦ Πϯϙʔτର4ϓϨϑΟοΫεΛ͘औΔͱ ϝϞϦΊͬͪΌ͏‣ %BUB$POOFDUPSϩά͕ফ͑ͯແݶϦτϥΠ˞݄࣌‣ "84#BUDIϝϞϦ্ׂݶ·Ͱͬͯ00.,JMMFSൃಈ$ aws s3 ls --profile cm-jp-1 s3://XXXX-infra-logbucket-XXXX/applications /condor/app/2018/10/04/05/2018-10-04 14:06:24 5964 00_5fbef6f0fdec_0.gz2018-10-04 14:11:26 5971 05_5fbef6f0fdec_0.gz2018-10-04 14:16:24 5932 10_5fbef6f0fdec_0.gz2018-10-04 14:21:25 5954 15_5fbef6f0fdec_0.gz
Πϯϙʔτͷδϣϒׂ ✦ ͍Ζ͍Ζࢼͯ͠ɺ୯ҐͰδϣϒΛΔͷ͕ ྑͦ͞͏ͱ͍͏͜ͱʹ‣ ϑΝΠϧαΠζɺϑΝΠϧʹґଘ͢ΔͷͰͬͯΈͳ͍ͱΘ͔Βͳ͍෦ ࠓճϑΝΠϧ͕ଟ͔༷ͬͨin:type: s3bucket: {{ env.S3BUCKET }}path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH }}/{{ env.DAY }}
5SFBTVSF%BUBͷ'"2ʹهࡌ͋Γ 2%BUB$POOFDUPSGPS4KPCJTSVOOJOHGPSBMPOHUJNF XIBUDBO*EP " $IFDLUIFDPVOUPG4pMFTUIBUZPVSDPOOFDUPSKPCJTJOHFTUJOH*GUIFSFBSFPWFS pMFT UIFQFSGPSNBODFEFHSBEFT 5PNJUJHBUFUIJTJTTVF ZPVDBO‣ /[email protected]‣ 4FU .#[email protected]@TJ[FPQUJPOIUUQTTVQQPSUUSFBTVSFEBUBDPNIDFOVTBSUJDMFT%BUB$POOFDUPSGPS"NB[PO4'"2GPSUIF4%BUB$POOFDUPS
ϩάαΠζͷ্ݶ
ϩάαΠζͷ্ݶ ✦ .%$ϩά͕σΧ͍✦ %PDLFSMPHHJOHESJWFS ,#ΑΓେ͖͍ϩάΛׂ͢Δ‣ 'MVFOUEGPSXBSEFSͷ+40/ύʔεʹࣦഊɺࣺͯΒΕΔ‣ DPODBUϓϥάΠϯͰ݁߹‣ ϝϞϦ༻ྔ͕Ͷͯ00.,JMMFSʹࡴ͞ΕͨΠϚίί✦ $MPVE8BUDI-PHTͷ্ݶ,#
͓ۚ ✦ $MPVE8BUDI-PHTϩάσʔλྔͷैྔ՝ۚ‣ ྔ͕ଟ͍ͷͰֹ݄අ༻͕͔͞Ή‣ &$ͳͲଞͷαʔϏεඅ༻ͱτϯτϯͱݴΘΕΔͱͭΒ͍✦ ΞϓϦ͔Βग़͢ϩάͷྔΛݮΒ͢‣ .%$ϩάͭΒ͍
4. ࠓޙͷల
࣍ͷҰख ✦ ϩάࢹ✦ ϩάूͱͯ͠&MBTUJDTFBSDI,JCBOBͷར༻✦ 5SFBTVSF%BUBΛͬͱ׆༻͍ͨ͠
·ͱΊ ✦ εέʔϧ͢Δϩάཧج൫ΫϥυαʔϏεͷ Έ߹Θ͕ͤΦεεϝ‣ σʔλϨΠΫͱϦΞϧλΠϜϩάͰͷαʔϏεͷ͍͚✦ Έ߹Θ͚ͤͩͰಈ͘ͷͰ͖Δ͚Ͳ‣ ཁ݅ʹରͯ͠Γͳ͍ͱ͜Ζͷิ͕ؒඞཁ‣ σʔλྔɺαΠζͷධՁνϡʔχϯά͕ඞཁ
takipone
hiroshinakagawa
nnstt1
mixi_engineers
whywaita
nagauta
newrelic2023
oracle4engineer
shimap_sampo
hayaok3
isaoshimizu
yoyakoba
ojun9
philnash
geeforr
aki_iinuma
revolveconf
brad_frost
geoffreycrofte
mclloyd
rmw
iamctodd
chrislema
aleyda
eileencodes