$30 off During Our Annual Pro Sale. View Details »

Mackerelによるクラウド監視でサービスを加速させる / Essential Mackerel

mackerelio
August 23, 2018
120

Mackerelによるクラウド監視でサービスを加速させる / Essential Mackerel

mackerelio

August 23, 2018
Tweet

More Decks by mackerelio

Transcript

  1. !1 MackerelʹΑΔΫϥ΢υ؂ࢹͰαʔϏεΛՃ଎ͤ͞Δ 2018/08/23 mackerel.io

  2. !2 ࣗݾ঺հ • দ໦ խ޾ (id:Songmu @songmu) • גࣜձࣾ͸ͯͳ αʔϏεγεςϜ։ൃຊ෦

    • νʔϑΤϯδχΞ݉Mackerelαϒϓϩσϡʔαʔ • 3 Times ISUCON Winner using Perl • ʮΈΜͳͷGoݴޠʯڞஶऀ • ʮMackerelαʔό؂ࢹ࣮ફೖ໳ʯ୅දஶऀ
  3. !3 .BDLFSFM https://mackerel.io

  4. !4 .BDLFSFMͱ͸ ɾ·͢·͢ಈతͰศརʹͳΔΠϯϑϥετϥΫνϟʔ ɾޮ཰తʹෳࡶੑͱ޲͖߹͏͜ͱ͕ٻΊΒΕΔ

  5. !5

  6. !6 .BDLFSFMͱ͸ ɾΫϥ΢υ࣌୅ͷ࣍ੈ୅؂ࢹαʔϏε ɾ%FW0QTͷத֩αʔϏε

  7. !7 Ϣʔβʔ༷اۀ

  8. !8 .BDLFSFMͷ։ൃɾϏδωεମ੍

  9. !9 גࣜձࣾ͸ͯͳ ɾ೥૑ۀ ɾژ౎ຊࣾɾ౦ژຊళ ɾ౦ূϚβʔζ্৔ ɾ࿝ฮͷ8FCϕϯνϟʔ ɾΤϯδχΞͷࢧ͕࣋ߴ͍αʔϏε܈ ɾ͸ͯͳϒϩάɾ͸ͯͳϒοΫϚʔΫ

  10. !10 ͸ͯͳʹ͓͚ΔαʔϏεӡ༻ ɾࣾ಺ج൫Ͱ਺ઍ୆ن໛ͷαʔόʔӡ༻ ɾ໊ऑͷΠϯϑϥΤϯδχΞ ɾݹ͔͘Βαʔόʔ؅ཧɾ؂ࢹαʔϏεΛ಺੡ ɾ4BB4Խ υοάϑʔσΟϯάͷதͰMackerel͸ੜ·Εͨ

  11. !11 .BEFJO+BQBO ɾ.BDLFSFM͸͸ͯͳࣾ಺Ͱ಺੡͍ͯ͠·͢ ɾϏδωεɾ։ൃɾαϙʔτશͯࠃ಺ ɾ೔ຊޠɾӳޠ྆ରԠ ɾւ֎Ϣʔβʔ͸͍·͕͢ڌ఺͸·ͩ

  12. !12 Ϋϥ΢υ࣌୅ͷ࣍ੈ୅؂ࢹαʔϏε

  13. !13 ͳͥΫϥ΢υ͔ ɾΞδϦςΟ ɾΞϕΠϥϏϦςΟ ɾεέʔϥϏϦςΟ ɾηΩϡϦςΟ

  14. !14 ΞδϦςΟ ढ़හੑ ɾૉૣ͍ελʔτΞοϓ ɾมԽʹڧ͍ΞʔΩςΫνϟ

  15. !15 ΞϕΠϥϏϦςΟ Մ༻ੑ ɾো֐଱ੑΛ࣮ݱ͢Δ৑௕ߏ੒͕༰қʹ࣮ݱՄೳ ɾҰ෦ͷγεςϜ͕མͪͯ΋αʔϏεܧଓՄೳʹ ɾαʔϏεͷμ΢ϯλΠϜΛ࠷খݶʹ

  16. !16 εέʔϥϏϦςΟ ֦ுੑ ɾٸͳαʔϏε֦େ΁ͷରԠ ɾௐୡͷૣ͞ ɾػձଛࣦΛ๷͙

  17. !17 ηΩϡϦςΟ ɾύϒϦοΫΫϥ΢υͱಉ౳ͷηΩϡϦςΟϨϕ ϧΛࣗࣾͰ࿫͏ͷ͸࣮͸ࠔ೉ ɾ͓ۚΛࣗࣾͷۚݿʹ͔࣋ͭɺۜߦʹ༬͚Δ͔

  18. !18 Ϋϥ΢υͷϝϦοτ Ϗδωεͷຊ࣭ʹ஫ྗ͢ΔͨΊͷΫϥ΢υ׆༻

  19. !19 Ϋϥ΢υ͸ۜͷ஄ؙʁ

  20. !20 ΤϯδχΞΛऔΓר͘؀ڥ ɾ·͢·͢ಈతͰศརʹͳΔΠϯϑϥετϥΫνϟʔ ɾޮ཰తʹෳࡶੑͱ޲͖߹͏͜ͱ͕ٻΊΒΕΔ ɾطଘͷ؂ࢹख๏͕Ϋϥ΢υͰ͸௨༻͠ͳ͍ ɾΫϥ΢υർΕ

  21. !21 ϞχλϦϯάͷॏཁੑͷ޲্ • γεςϜ΁ͷཁٻͷ޲্ • Ϗδωε੒௕ͷͨΊʹ౷ҰతͳϞχλϦϯά͸ඞਢ • ػձଛࣦΛ๷͙ͨΊʹ • ޮ཰తͳαʔϏεӡ༻ͷͨΊʹ

    ʮ͢΂ͯͷαʔϏεͰ݈શੑͱҰൠతͳ؂ࢹؔ࿈ͷ ϝτϦοΫΛಉ͡Α͏ʹग़ྗ͢Δ͜ͱΛ͓͢͢Ί͠ ·͢ɻ(தུ) ͲΕΛબΜͰ΋ɺඪ४Խ͢ΔΑ͏ʹ͠ ͍ͯͩ͘͞ɻʯ
  22. !22

  23. !23 *OGSBTUSVDUVSFBT$PEF ɾಈతͳΠϯϑϥΛ؅ཧ͢ΔͨΊʹίʔυԽ͢Δ ɾಉ͡༻్ͷαʔόʔΛҰ੪ʹ૿Ճ ɾ৽όʔδϣϯͷΞϓϦέʔγϣϯΛҰ੪ʹ౤ೖ ɾαʔόʔߏங͚ͩͰ͸ͳ͘ӡ༻΋ίʔυԽ͢Δ

  24. !24 Ϋϥ΢υ࣌୅ͷΠϯϑϥӡ༻ ɾγεςϜ͸࡞͓ͬͯ͠·͍Ͱ͸ͳ͍ ɾϏδωεͷͨΊʹӡ༻ʹίϛοτ͢Δ ɾ؂ࢹॏཁ

  25. !25 ςετۦಈ։ൃʹֶͿ ɾςετ͸ςελʔ͕΍Δ͚ͩͷ΋ͷͰ͸ͳ͍ ɾ։ൃऀ͕ςετίʔυΛॻ͖ɺ$*Λ࣮ࢪ͢Δ

  26. !26 ؂ࢹͱ%FW0QT ɾαʔϏε։ൃͱӡ༻͸྆ྠ ɾ؂ࢹΛνʔϜͰҰॹʹ࣮ࢪ͢Δ ɾ؂ࢹΛΠϯϑϥΤϯδχΞ೚ͤʹͨ͘͠ͳ͍

  27. !27 .BDLFSFM

  28. !28 DevOpsͷத֩ͱͯ͠ Mackerel͸୯ͳΔ؂ࢹαʔϏεͰ͸͋Γ·ͤΜ ༷ʑͳ࢓૊Έͱ؆୯ʹ૊Έ߹Θͤɺ։ൃ΍ӡ༻ϥΠϑαΠΫϧͷத ֩ͱͯ͠ޮ཰ԽΛଅਐ͠·͢ Dev Ops

  29. !29 ΦϯϓϨ͔ΒΫϥ΢υ·Ͱ ɾࣗࣾσʔληϯλʔͱͷϋΠϒϦου؀ڥ ɾϚϧνΫϥ΢υ؀ڥ ɾΫϥ΢υγϑτΛޙԡ͠͠·͢ ɾ΋ͪΖΜΦϯϓϨ͚ͩͰ΋࢖͑·͢

  30. !30 ΞʔΩςΫνϟ Mackerel͔Β ֎෦αʔϏε ΁ͷ௚઀؂ࢹ ΤʔδΣϯτ͔Β ϝτϦοΫΛ౤ߘ HTTPS REST API

  31. !31 ؆୯ɻ͙͢͸͡ΊΒΕ·͢ • ΨΠυʹ͕ͨͬͯ͠ίϚϯυΛ࣮ߦ͢Δ͚ͩ • ਺෼͋Ε͹े෼Ͱ͢ mackerel-agent written in Go

  32. !32 ηΩϡΞͳ؂ࢹ ɾ௨৴͸؂ࢹର৅ϗετ͔Β.BDLFSFMαʔόʔ΁ͷҰํ޲ ɾ5-4ʹΑΔ҉߸Խ ɾ(PͰॻ͔ΕͨBHFOU͸ܰྔʹಈ࡞͠ɺϙʔτ։์΋ߦͳ͍·ͤΜ ɾBHFOUͷιʔείʔυ͸ެ։͞Ε͍ͯ·͢ ɾ*1੍ݶػೳ

  33. !33 ௚ײతͳϢʔβʔମݧ

  34. !34 ௚ײతͳϢʔβʔମݧ

  35. !35 ࣮༻ੑͷߴ͍ΠϯλʔϑΣʔε API CLI GUI User App

  36. !36 ࠷খͷϧʔϧ͕࠷ߴͷརศੑΛ ༧ΊαʔϏεʗϩʔϧʗϗετΛࢦఆ͢Δ͜ͱͰɺࣗಈతʹάϥϑ ΍؂ࢹઃఆͳͲ͕ઃఆ͞ΕΔͨΊɺಛʹେن໛؀ڥͰͷӡ༻ෛՙ ܰݮʹޮՌతͰ͢

  37. !37 ଟ͘ͷ؂ࢹαϙʔτ ɾ-JOVY8JOEPXTରԠ ɾॆ࣮ͷެࣜϓϥάΠϯ ɹɾ֤छϛυϧ΢ΣΞରԠɾϓϩηε؂ࢹɾϩά؂ࢹ
 ɾࣗ࡞ϓϥάΠϯͰ؆୯ʹ؂ࢹ߲໨Λ֦ு

  38. !38 ଟ͘ͷ௨஌αϙʔτ • αʔυύʔςΟʔͷ࢓૊Έ΁؆୯ʹ௨஌͢Δ͜ͱ͕Ͱ͖·͢

  39. !39 *OGSBTUSVDUVSFBT$PEF • Dynamic Infrastructure Platforms • AWS • Infrastructure

    Orchestration Tools • CloudFormation/Terraform • Configuration Registry • Server Configuration Tools • chef/puppet/Ansible • Infrastructure Services • Monitoring / Log / Deploy
  40. !40 # mkr monitors diff Summary: 1 modify, 0 append,

    0 remove 
 { "name": "loadavg", "type": "host", "metric": "loadavg5", "operator": ">", "warning": 2.000000, - "critical": 4.000000, + "critical": 5.000000, "duration": 5, "url": "", "scopes": [ "stanaka: client", ], "excludeScopes": [ ], }, 
 JSON JSON JSON pull diff push ؂ࢹઃఆΛίʔυͰ؅ཧ • GitϥΠΫͳ࢖͍উख
  41. !41 ͸ͯͳͰͷ૊Έ߹Θͤ droot Monitoring & Host Registry Server Configuration Deploy

    Infrastructure Test SSH-Operations Infrastructure DNS Registry tinydns
  42. !42 ૊Έ߹Θͤ͸ແݶେ • ͓࢖͍ͷπʔϧͱ࿈ܞ͠ɺ༻్ʹԠͯ͡૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖·͢ Server Server Server Server Deploy Configuration

    API mkr
  43. !43 ৗʹਐԽ͠ଓ͚ΔαʔϏε ຖि৽ػೳͷϦϦʔεΛܧଓத ௠෗ԽͤͣৗʹਐԽ͍ͯ͠·͢ 203ि࿈ଓ ܧ ଓ த

  44. !44 Ϣʔβʔͷ੠ ΞϓϦέʔγϣϯΤϯδχΞ΋ϦιʔεϞχλϦϯάʹහײʹ w σϓϩΠޙʹෛՙ͕ߴ͘ͳͬͨ৔߹ɺ4MBDL΁௨஌͕͞ΕɺσϓϩΠʹ໰୊͕͋ͬ ͨ͜ͱΛൃݟͨ͠ w ࣗ෼͕ͨͪॻ͍ͨίʔυ΍αʔϏεͷಈ࡞ʹରͯ͠ɺࠓ·ͰΑΓ΋ߋʹڵຯɾؔ৺ Λ΋ͭΑ͏ʹͳͬͨ

  45. • !45 Ϣʔβʔͷ੠ w νϟοτπʔϧͱซ༻͢Δ͜ͱͰɺτϥϒϧൃੜ࣌ͷڞ༗΋ਝ଎ʹରԠͰ͖ΔΑ͏ ʹͳͬͨ w 4MBDLʹάϥϑ͕Ұॹʹܝࡌ͞ΕΔͨΊɺ؅ཧը໘Λ։͘͜ͱͳ͘ঢ়گɾਪҠΛαο ͱ೺ѲͰ͖Δ w

    αϙʔτɾηʔϧε୲౰ऀ͕ΤϯδχΞʹՔಇঢ়گΛ֬ೝ͢Δ͜ͱ͕ݮΓɺۀ຿ͷ ޮ཰Խʹͭͳ͕ͬͨ ΤϯδχΞҎ֎ͷ৬छͰ΋αʔϏεঢ়گΛؾʹ͢ΔΑ͏ʹͳͬͨ
  46. !46 Ϣʔβʔͷ੠ ΤϯδχΞɾඇΤϯδχΞΛؚΉνʔϜͰͷ,1*֬ೝ΍;ΓฦΓͰ΋ .BDLFSFMΛར༻͍ͯ͠Δ αʔϏεͷ,1*ΛαʔϏεϝτϦοΫͱͯ͠.BDLFSFM΁౤ߘ͠ɺ೔ৗ తʹ,1*Λҙࣝ͢ΔΑ͏ʹͳͬͨ

  47. !47 ಛ௃ػೳ

  48. !48 AWSΠϯςάϨʔγϣϯ ɾAWS CloudWatch API͔Β༷ʑͳϝτϦοΫ΍ΠϕϯτΛ ɹऔಘ͠ՄࢹԽ ɾ1୆ͷϗετͱͯ͠؅ཧ ɾEC2, ELB, RDS,

    Elasticache, RedshiftͳͲॱ֦࣍େத
  49. !49 ΦʔτεέʔϧରԠ • αʔό͕ॖୀͯ͠΋άϥϑ͸࢒Γଓ͚·͢ • ॖୀ࣌ͷΞϥʔτ཈੍΋Մೳ • ྉۚ΋࢖ͬͨ෼͚ͩ εέʔϧΞ΢τ εέʔϧΠϯ

  50. !50 63-֎ܗ؂ࢹ • ೚ҙͷ63-ʹରͯ͠ͷ؂ࢹઃఆ͕Մೳ • ೚ҙͷϔομʔɾϝιου෇༩ • ϨεϙϯελΠϜܭଌ

  51. !51 ి࿩௨஌ͷࣗಈԽ ɾ࣮֬ੑͱࣗ༝౓ͷߴ͍௨஌Λ௿ίετͰ࣮ݱ ɾΞϥʔτ௨஌ʹΫϥ΢υՍిαʔϏεͷTwilioΛαϙʔτ ɾྫʣϝϯόʔ΁ॱ൪ʹՍి ɹɹɹରԠՄ൱ΛIVR࿈ಈͰϋϯυϦϯά API Call SMS

  52. !52 ॊೈͳάϥϑදݱʢܭࢉࣜʣ ࠩ෼ લिͱͷॏͶ͋Θͤ ࠷େ ࠷େɾฏۉɾ࠷খ ߹ܭ

  53. !53 ߴ౓ͳΞϥʔτઃఆʢܭࢉࣜʣ wࣜʹΑΔܭࢉ݁ՌΛ؂ࢹ wաڈͱͷࠩ෼ͰΞϥʔτ wώοτ཰ͰΞϥʔτ

  54. !54 কདྷ༧ଌ • աڈͷ܏޲Λ෼ੳ͠ɺকདྷͷ༧ଌʹج͍ͮͯܯࠂ • ྫ͑͹ɺσΟεΫ͕ރׇ͢Δ1ϱ݄લʹ͓஌Βͤ͢Δ͜ͱ΋ Critical > 80% কདྷ

    Trends 1ϱ݄લ ݱࡏ
  55. !55 େن໛؀ڥͰ΋ѻ͍΍͍͢ߏ଄ • ΦʔΨχθʔγϣϯʢORGʣຖʹ؅ཧൣғΛ੍ݶ • IPΞυϨε੍ݶ • ॊೈͳϢʔβʔݖݶ User A

    User B User C User D Service A Service B Role 1 Role 2 Role 3 Host 1 Host 2 Host 3 Host 4 Read Write Read Only Owner IP Address Authentication A.A.A.A Service A Service B Role 1 Role 2 Role 3 Host 1 Host 2 Host 3 Host 4 Read Write Read Only Owner IP Address Authentication A.A.A.A Service A Service B Role 1 Role 2 Role 3 Host 1 Host 2 Host 3 Host 4 Read Write Read Only Owner IP Address Authentication A.A.A.A ORG X ORG Y ORG Z
  56. !56 ࣌ܥྻσʔλͷ௕ظؒอ࣋ • ෼ִؒͷ··Ͱ೔อଘ

  57. !57 Πϕϯτػೳ • ։ൃ΍ӡ༻ʹ͓͚Δ༷ʑͳࣄ৅ΛτϥοΩϯά • Ξϥʔτɺ.BDLFSFMͷૢ࡞ɺ%FQMPZɺϗετ૿ݮɹͳͲ • ΠϕϯτۦಈͰ֤छػೳ͕࿈ಈ • ௨஌ɺΞϊςʔγϣϯɺΞΫγϣϯ

  58. !58 • .BDLFSFMͷΠϯϕϯτϦ৘ใ͸ɺݶΓͳ͘઱౓͕ߴ͍ҝɺ৴པੑ ͷߴ͍୆ாͱͯ͠ར༻Ͱ͖·͢ ௠෗Խ͠ͳ͍αʔόʔ؅ཧ ؂ࢹσʔλͱͯ͠ৗʹߋ৽ Host1 10.0.0.11 Host2 10.0.0.12

    Host3 10.0.0.13 Host4 10.0.0.14 ..... ਖ਼͍͠αʔόʔ؅ཧ୆ா
  59. !59 ೥಺ϦϦʔε༧ఆ

  60. !60 ҟৗݕ஌ • .BDLFSFMͷϩʔϧ֓೦ͱඥ෇͚Δ • BHFOU͕ૹ͍ͬͯΔϝτϦοΫΛݩʹଟ࣍ݩϕΫτϧͷڭࢣͳֶ͠शΛܧଓతʹ ͓͜ͳ͍ɺϩʔϧͷதͷʮ͓͔ͦ͠͏ͳϗετʯΛݕग़͢Δ • $POGJHVSBUJPO'SFFͰΘ͔Γ΍͘͢

  61. !61 ίϯςφରԠڧԽ • ίϯςφͷ֓೦ΛωΠςΟϒαϙʔτ • ΑΓಈతͳΠϯϑϥ؀ڥΛΘ͔Γ΍͘͢

  62. !62 ΧελϜμογϡϘʔυW • ΑΓ௚ײతͰΘ͔Γ΍͍͢ϞμϯͳμογϡϘʔυΛఏڙ

  63. !63 ਐԽΛଓ͚͍͖ͯ·͢

  64. !64 4BB4ʹ͓೚ͤ • ։ൃπʔϧ͸Ϋϥ΢υ΁ • ϝʔϧˠ(NBJM • ιʔείʔυ؅ཧˠ(JU)VC(JUMBC • $*ˠ5SBWJT$*$JSDMF$*

    • ࣮ߦ؀ڥ΋Ϋϥ΢υ΁ • "84($&"[VSF*%$'Ϋϥ΢υχϑΫϥ
  65. !65 ӡ༻πʔϧ΋Ϋϥ΢υ΁ .BDLFSFMʹ͓೚͍ͤͩ͘͞

  66. !66 αʔϏεϓϥϯ ϘϦϡʔϜσΟεΧ΢ϯτ͋Γ