Mackerelによるクラウド監視でサービスを加速させる / Essential Mackerel

!1 MackerelʹΑΔΫϥ΢υ؂ࢹͰαʔϏεΛՃ଎ͤ͞Δ 2018/08/23 mackerel.io

!2 ࣗݾ঺հ • দ໦ խ޾ (id:Songmu @songmu) • גࣜձࣾ͸ͯͳ αʔϏεγεςϜ։ൃຊ෦
• νʔϑΤϯδχΞ݉Mackerelαϒϓϩσϡʔαʔ • 3 Times ISUCON Winner using Perl • ʮΈΜͳͷGoݴޠʯڞஶऀ • ʮMackerelαʔό؂ࢹ࣮ફೖ໳ʯ୅දஶऀ

!3 .BDLFSFM https://mackerel.io

!4 .BDLFSFMͱ͸ ɾ·͢·͢ಈతͰศརʹͳΔΠϯϑϥετϥΫνϟʔ ɾޮ཰తʹෳࡶੑͱ޲͖߹͏͜ͱ͕ٻΊΒΕΔ

!6 .BDLFSFMͱ͸ ɾΫϥ΢υ࣌୅ͷ࣍ੈ୅؂ࢹαʔϏε ɾ%FW0QTͷத֩αʔϏε

!7 Ϣʔβʔ༷اۀ

!8 .BDLFSFMͷ։ൃɾϏδωεମ੍

!9 גࣜձࣾ͸ͯͳ ɾ೥૑ۀ ɾژ౎ຊࣾɾ౦ژຊళ ɾ౦ূϚβʔζ্৔ ɾ࿝ฮͷ8FCϕϯνϟʔ ɾΤϯδχΞͷࢧ͕࣋ߴ͍αʔϏε܈ ɾ͸ͯͳϒϩάɾ͸ͯͳϒοΫϚʔΫ

!10 ͸ͯͳʹ͓͚ΔαʔϏεӡ༻ ɾࣾ಺ج൫Ͱ਺ઍ୆ن໛ͷαʔόʔӡ༻ ɾ໊ऑͷΠϯϑϥΤϯδχΞ ɾݹ͔͘Βαʔόʔ؅ཧɾ؂ࢹαʔϏεΛ಺੡ ɾ4BB4Խ υοάϑʔσΟϯάͷதͰMackerel͸ੜ·Εͨ

!11 .BEFJO+BQBO ɾ.BDLFSFM͸͸ͯͳࣾ಺Ͱ಺੡͍ͯ͠·͢ ɾϏδωεɾ։ൃɾαϙʔτશͯࠃ಺ ɾ೔ຊޠɾӳޠ྆ରԠ ɾւ֎Ϣʔβʔ͸͍·͕͢ڌ఺͸·ͩ

!12 Ϋϥ΢υ࣌୅ͷ࣍ੈ୅؂ࢹαʔϏε

!13 ͳͥΫϥ΢υ͔ ɾΞδϦςΟ ɾΞϕΠϥϏϦςΟ ɾεέʔϥϏϦςΟ ɾηΩϡϦςΟ

!14 ΞδϦςΟ ढ़හੑ ɾૉૣ͍ελʔτΞοϓ ɾมԽʹڧ͍ΞʔΩςΫνϟ

!15 ΞϕΠϥϏϦςΟ Մ༻ੑ ɾো֐଱ੑΛ࣮ݱ͢Δ৑௕ߏ੒͕༰қʹ࣮ݱՄೳ ɾҰ෦ͷγεςϜ͕མͪͯ΋αʔϏεܧଓՄೳʹ ɾαʔϏεͷμ΢ϯλΠϜΛ࠷খݶʹ

!16 εέʔϥϏϦςΟ ֦ுੑ ɾٸͳαʔϏε֦େ΁ͷରԠ ɾௐୡͷૣ͞ ɾػձଛࣦΛ๷͙

!17 ηΩϡϦςΟ ɾύϒϦοΫΫϥ΢υͱಉ౳ͷηΩϡϦςΟϨϕ ϧΛࣗࣾͰ࿫͏ͷ͸࣮͸ࠔ೉ ɾ͓ۚΛࣗࣾͷۚݿʹ͔࣋ͭɺۜߦʹ༬͚Δ͔

!18 Ϋϥ΢υͷϝϦοτ Ϗδωεͷຊ࣭ʹ஫ྗ͢ΔͨΊͷΫϥ΢υ׆༻

!19 Ϋϥ΢υ͸ۜͷ஄ؙʁ

!20 ΤϯδχΞΛऔΓר͘؀ڥ ɾ·͢·͢ಈతͰศརʹͳΔΠϯϑϥετϥΫνϟʔ ɾޮ཰తʹෳࡶੑͱ޲͖߹͏͜ͱ͕ٻΊΒΕΔ ɾطଘͷ؂ࢹख๏͕Ϋϥ΢υͰ͸௨༻͠ͳ͍ ɾΫϥ΢υർΕ

!21 ϞχλϦϯάͷॏཁੑͷ޲্ • γεςϜ΁ͷཁٻͷ޲্ • Ϗδωε੒௕ͷͨΊʹ౷ҰతͳϞχλϦϯά͸ඞਢ • ػձଛࣦΛ๷͙ͨΊʹ • ޮ཰తͳαʔϏεӡ༻ͷͨΊʹ
ʮ͢΂ͯͷαʔϏεͰ݈શੑͱҰൠతͳ؂ࢹؔ࿈ͷ ϝτϦοΫΛಉ͡Α͏ʹग़ྗ͢Δ͜ͱΛ͓͢͢Ί͠ ·͢ɻ(தུ) ͲΕΛબΜͰ΋ɺඪ४Խ͢ΔΑ͏ʹ͠ ͍ͯͩ͘͞ɻʯ

!23 *OGSBTUSVDUVSFBT$PEF ɾಈతͳΠϯϑϥΛ؅ཧ͢ΔͨΊʹίʔυԽ͢Δ ɾಉ͡༻్ͷαʔόʔΛҰ੪ʹ૿Ճ ɾ৽όʔδϣϯͷΞϓϦέʔγϣϯΛҰ੪ʹ౤ೖ ɾαʔόʔߏங͚ͩͰ͸ͳ͘ӡ༻΋ίʔυԽ͢Δ

!24 Ϋϥ΢υ࣌୅ͷΠϯϑϥӡ༻ ɾγεςϜ͸࡞͓ͬͯ͠·͍Ͱ͸ͳ͍ ɾϏδωεͷͨΊʹӡ༻ʹίϛοτ͢Δ ɾ؂ࢹॏཁ

!25 ςετۦಈ։ൃʹֶͿ ɾςετ͸ςελʔ͕΍Δ͚ͩͷ΋ͷͰ͸ͳ͍ ɾ։ൃऀ͕ςετίʔυΛॻ͖ɺ$*Λ࣮ࢪ͢Δ

!26 ؂ࢹͱ%FW0QT ɾαʔϏε։ൃͱӡ༻͸྆ྠ ɾ؂ࢹΛνʔϜͰҰॹʹ࣮ࢪ͢Δ ɾ؂ࢹΛΠϯϑϥΤϯδχΞ೚ͤʹͨ͘͠ͳ͍

!27 .BDLFSFM

!28 DevOpsͷத֩ͱͯ͠ Mackerel͸୯ͳΔ؂ࢹαʔϏεͰ͸͋Γ·ͤΜ ༷ʑͳ࢓૊Έͱ؆୯ʹ૊Έ߹Θͤɺ։ൃ΍ӡ༻ϥΠϑαΠΫϧͷத ֩ͱͯ͠ޮ཰ԽΛଅਐ͠·͢ Dev Ops

!29 ΦϯϓϨ͔ΒΫϥ΢υ·Ͱ ɾࣗࣾσʔληϯλʔͱͷϋΠϒϦου؀ڥ ɾϚϧνΫϥ΢υ؀ڥ ɾΫϥ΢υγϑτΛޙԡ͠͠·͢ ɾ΋ͪΖΜΦϯϓϨ͚ͩͰ΋࢖͑·͢

!30 ΞʔΩςΫνϟ Mackerel͔Β ֎෦αʔϏε ΁ͷ௚઀؂ࢹ ΤʔδΣϯτ͔Β ϝτϦοΫΛ౤ߘ HTTPS REST API

!31 ؆୯ɻ͙͢͸͡ΊΒΕ·͢ • ΨΠυʹ͕ͨͬͯ͠ίϚϯυΛ࣮ߦ͢Δ͚ͩ • ਺෼͋Ε͹े෼Ͱ͢ mackerel-agent written in Go

!32 ηΩϡΞͳ؂ࢹ ɾ௨৴͸؂ࢹର৅ϗετ͔Β.BDLFSFMαʔόʔ΁ͷҰํ޲ ɾ5-4ʹΑΔ҉߸Խ ɾ(PͰॻ͔ΕͨBHFOU͸ܰྔʹಈ࡞͠ɺϙʔτ։์΋ߦͳ͍·ͤΜ ɾBHFOUͷιʔείʔυ͸ެ։͞Ε͍ͯ·͢ ɾ*1੍ݶػೳ

!33 ௚ײతͳϢʔβʔମݧ

!34 ௚ײతͳϢʔβʔମݧ

!35 ࣮༻ੑͷߴ͍ΠϯλʔϑΣʔε API CLI GUI User App

!36 ࠷খͷϧʔϧ͕࠷ߴͷརศੑΛ ༧ΊαʔϏεʗϩʔϧʗϗετΛࢦఆ͢Δ͜ͱͰɺࣗಈతʹάϥϑ ΍؂ࢹઃఆͳͲ͕ઃఆ͞ΕΔͨΊɺಛʹେن໛؀ڥͰͷӡ༻ෛՙ ܰݮʹޮՌతͰ͢

!37 ଟ͘ͷ؂ࢹαϙʔτ ɾ-JOVY8JOEPXTରԠ ɾॆ࣮ͷެࣜϓϥάΠϯ ɹɾ֤छϛυϧ΢ΣΞରԠɾϓϩηε؂ࢹɾϩά؂ࢹ  ɾࣗ࡞ϓϥάΠϯͰ؆୯ʹ؂ࢹ߲໨Λ֦ு

!38 ଟ͘ͷ௨஌αϙʔτ • αʔυύʔςΟʔͷ࢓૊Έ΁؆୯ʹ௨஌͢Δ͜ͱ͕Ͱ͖·͢

!39 *OGSBTUSVDUVSFBT$PEF • Dynamic Infrastructure Platforms • AWS • Infrastructure
Orchestration Tools • CloudFormation/Terraform • Configuration Registry • Server Configuration Tools • chef/puppet/Ansible • Infrastructure Services • Monitoring / Log / Deploy

!40 # mkr monitors diff Summary: 1 modify, 0 append,
0 remove   { "name": "loadavg", "type": "host", "metric": "loadavg5", "operator": ">", "warning": 2.000000, - "critical": 4.000000, + "critical": 5.000000, "duration": 5, "url": "", "scopes": [ "stanaka: client", ], "excludeScopes": [ ], },   JSON JSON JSON pull diff push ؂ࢹઃఆΛίʔυͰ؅ཧ • GitϥΠΫͳ࢖͍উख

!41 ͸ͯͳͰͷ૊Έ߹Θͤ droot Monitoring & Host Registry Server Configuration Deploy
Infrastructure Test SSH-Operations Infrastructure DNS Registry tinydns

!42 ૊Έ߹Θͤ͸ແݶେ • ͓࢖͍ͷπʔϧͱ࿈ܞ͠ɺ༻్ʹԠͯ͡૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖·͢ Server Server Server Server Deploy Configuration
API mkr

!43 ৗʹਐԽ͠ଓ͚ΔαʔϏε ຖि৽ػೳͷϦϦʔεΛܧଓத ௠෗ԽͤͣৗʹਐԽ͍ͯ͠·͢ 203ि࿈ଓ ܧ ଓ த

!44 Ϣʔβʔͷ੠ ΞϓϦέʔγϣϯΤϯδχΞ΋ϦιʔεϞχλϦϯάʹහײʹ w σϓϩΠޙʹෛՙ͕ߴ͘ͳͬͨ৔߹ɺ4MBDL΁௨஌͕͞ΕɺσϓϩΠʹ໰୊͕͋ͬ ͨ͜ͱΛൃݟͨ͠ w ࣗ෼͕ͨͪॻ͍ͨίʔυ΍αʔϏεͷಈ࡞ʹରͯ͠ɺࠓ·ͰΑΓ΋ߋʹڵຯɾؔ৺ Λ΋ͭΑ͏ʹͳͬͨ

• !45 Ϣʔβʔͷ੠ w νϟοτπʔϧͱซ༻͢Δ͜ͱͰɺτϥϒϧൃੜ࣌ͷڞ༗΋ਝ଎ʹରԠͰ͖ΔΑ͏ ʹͳͬͨ w 4MBDLʹάϥϑ͕Ұॹʹܝࡌ͞ΕΔͨΊɺ؅ཧը໘Λ։͘͜ͱͳ͘ঢ়گɾਪҠΛαο ͱ೺ѲͰ͖Δ w
αϙʔτɾηʔϧε୲౰ऀ͕ΤϯδχΞʹՔಇঢ়گΛ֬ೝ͢Δ͜ͱ͕ݮΓɺۀ຿ͷ ޮ཰Խʹͭͳ͕ͬͨ ΤϯδχΞҎ֎ͷ৬छͰ΋αʔϏεঢ়گΛؾʹ͢ΔΑ͏ʹͳͬͨ

!46 Ϣʔβʔͷ੠ ΤϯδχΞɾඇΤϯδχΞΛؚΉνʔϜͰͷ,1*֬ೝ΍;ΓฦΓͰ΋ .BDLFSFMΛར༻͍ͯ͠Δ αʔϏεͷ,1*ΛαʔϏεϝτϦοΫͱͯ͠.BDLFSFM΁౤ߘ͠ɺ೔ৗ తʹ,1*Λҙࣝ͢ΔΑ͏ʹͳͬͨ

!47 ಛ௃ػೳ

!48 AWSΠϯςάϨʔγϣϯ ɾAWS CloudWatch API͔Β༷ʑͳϝτϦοΫ΍ΠϕϯτΛ ɹऔಘ͠ՄࢹԽ ɾ1୆ͷϗετͱͯ͠؅ཧ ɾEC2, ELB, RDS,
Elasticache, RedshiftͳͲॱ֦࣍େத

!49 ΦʔτεέʔϧରԠ • αʔό͕ॖୀͯ͠΋άϥϑ͸࢒Γଓ͚·͢ • ॖୀ࣌ͷΞϥʔτ཈੍΋Մೳ • ྉۚ΋࢖ͬͨ෼͚ͩ εέʔϧΞ΢τ εέʔϧΠϯ

!50 63-֎ܗ؂ࢹ • ೚ҙͷ63-ʹରͯ͠ͷ؂ࢹઃఆ͕Մೳ • ೚ҙͷϔομʔɾϝιου෇༩ • ϨεϙϯελΠϜܭଌ

!51 ి࿩௨஌ͷࣗಈԽ ɾ࣮֬ੑͱࣗ༝౓ͷߴ͍௨஌Λ௿ίετͰ࣮ݱ ɾΞϥʔτ௨஌ʹΫϥ΢υՍిαʔϏεͷTwilioΛαϙʔτ ɾྫʣϝϯόʔ΁ॱ൪ʹՍి ɹɹɹରԠՄ൱ΛIVR࿈ಈͰϋϯυϦϯά API Call SMS

!52 ॊೈͳάϥϑදݱʢܭࢉࣜʣ ࠩ෼ લिͱͷॏͶ͋Θͤ ࠷େ ࠷େɾฏۉɾ࠷খ ߹ܭ

!53 ߴ౓ͳΞϥʔτઃఆʢܭࢉࣜʣ wࣜʹΑΔܭࢉ݁ՌΛ؂ࢹ wաڈͱͷࠩ෼ͰΞϥʔτ wώοτ཰ͰΞϥʔτ

!54 কདྷ༧ଌ • աڈͷ܏޲Λ෼ੳ͠ɺকདྷͷ༧ଌʹج͍ͮͯܯࠂ • ྫ͑͹ɺσΟεΫ͕ރׇ͢Δ1ϱ݄લʹ͓஌Βͤ͢Δ͜ͱ΋ Critical > 80% কདྷ
Trends 1ϱ݄લ ݱࡏ

!55 େن໛؀ڥͰ΋ѻ͍΍͍͢ߏ଄ • ΦʔΨχθʔγϣϯʢORGʣຖʹ؅ཧൣғΛ੍ݶ • IPΞυϨε੍ݶ • ॊೈͳϢʔβʔݖݶ User A
User B User C User D Service A Service B Role 1 Role 2 Role 3 Host 1 Host 2 Host 3 Host 4 Read Write Read Only Owner IP Address Authentication A.A.A.A Service A Service B Role 1 Role 2 Role 3 Host 1 Host 2 Host 3 Host 4 Read Write Read Only Owner IP Address Authentication A.A.A.A Service A Service B Role 1 Role 2 Role 3 Host 1 Host 2 Host 3 Host 4 Read Write Read Only Owner IP Address Authentication A.A.A.A ORG X ORG Y ORG Z

!56 ࣌ܥྻσʔλͷ௕ظؒอ࣋ • ෼ִؒͷ··Ͱ೔อଘ

!57 Πϕϯτػೳ • ։ൃ΍ӡ༻ʹ͓͚Δ༷ʑͳࣄ৅ΛτϥοΩϯά • Ξϥʔτɺ.BDLFSFMͷૢ࡞ɺ%FQMPZɺϗετ૿ݮɹͳͲ • ΠϕϯτۦಈͰ֤छػೳ͕࿈ಈ • ௨஌ɺΞϊςʔγϣϯɺΞΫγϣϯ

!58 • .BDLFSFMͷΠϯϕϯτϦ৘ใ͸ɺݶΓͳ͘઱౓͕ߴ͍ҝɺ৴པੑ ͷߴ͍୆ாͱͯ͠ར༻Ͱ͖·͢ ௠෗Խ͠ͳ͍αʔόʔ؅ཧ ؂ࢹσʔλͱͯ͠ৗʹߋ৽ Host1 10.0.0.11 Host2 10.0.0.12
Host3 10.0.0.13 Host4 10.0.0.14 ..... ਖ਼͍͠αʔόʔ؅ཧ୆ா

!59 ೥಺ϦϦʔε༧ఆ

!60 ҟৗݕ஌ • .BDLFSFMͷϩʔϧ֓೦ͱඥ෇͚Δ • BHFOU͕ૹ͍ͬͯΔϝτϦοΫΛݩʹଟ࣍ݩϕΫτϧͷڭࢣͳֶ͠शΛܧଓతʹ ͓͜ͳ͍ɺϩʔϧͷதͷʮ͓͔ͦ͠͏ͳϗετʯΛݕग़͢Δ • $POGJHVSBUJPO'SFFͰΘ͔Γ΍͘͢

!61 ίϯςφରԠڧԽ • ίϯςφͷ֓೦ΛωΠςΟϒαϙʔτ • ΑΓಈతͳΠϯϑϥ؀ڥΛΘ͔Γ΍͘͢

!62 ΧελϜμογϡϘʔυW • ΑΓ௚ײతͰΘ͔Γ΍͍͢ϞμϯͳμογϡϘʔυΛఏڙ

!63 ਐԽΛଓ͚͍͖ͯ·͢

!64 4BB4ʹ͓೚ͤ • ։ൃπʔϧ͸Ϋϥ΢υ΁ • ϝʔϧˠ(NBJM • ιʔείʔυ؅ཧˠ(JU)VC(JUMBC • $*ˠ5SBWJT$*$JSDMF$*
• ࣮ߦ؀ڥ΋Ϋϥ΢υ΁ • "84($&"[VSF*%$'Ϋϥ΢υχϑΫϥ

!65 ӡ༻πʔϧ΋Ϋϥ΢υ΁ .BDLFSFMʹ͓೚͍ͤͩ͘͞

!66 αʔϏεϓϥϯ ϘϦϡʔϜσΟεΧ΢ϯτ͋Γ

Mackerelによるクラウド監視でサービスを加速させる / Essential Mackerel

Mackerelによるクラウド監視でサービスを加速させる / Essential Mackerel

More Decks by mackerelio

Featured

Transcript