Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CloudWatch複合アラームでELBの5XXをいい感じに検知しようとしたらうまくいかなかった話 / cloudwatch alarm elb 5xx

k.goto
May 29, 2023

CloudWatch複合アラームでELBの5XXをいい感じに検知しようとしたらうまくいかなかった話 / cloudwatch alarm elb 5xx

2023/05/29開催「JAWS-UG SRE支部 #6」での発表資料です。

k.goto

May 29, 2023
Tweet

More Decks by k.goto

Other Decks in Technology

Transcript

  1. ࣗݾ঺հ LHPUP w ςοΫϦʔυɾϥʔϝϯ޷͖ w "84$PNNVOJUZ#VJMEFS %FW5PPMT  w าͷςοΫ

    ٕज़ϒϩά  w ࣗ࡞"84ؔ࿈πʔϧͷ044։ൃ w 5XJUUFS!@TUFQ@UFDI ‣ LHPUP าͷςοΫ
  2. ͍͍ײ͡ʁ w &-# "-# ͷΤϥʔ௨஌ $MPVE8BUDI"MBSN  ‣ )551$PEF@&-#@ccc@$PVOU 

    εςʔλε͝ͱʹ௨஌͍ͨ͠ ‣ )551$PEF@&-#@99@$PVOU  cccͷ৔߹͸99௨஌͠ͳ͍ 㾎ॏෳ௨஌Λഉআ͍ͨ͠ ʮ99΋΋དྷͯ͠·͏ʯͷΛආ͚Δ   cccҎ֎ͷΤϥʔ͚ͩ99Ͱ௨஌͢Δ 㾎  ͳͲͷͱ͖͚ͩ௨஌͢Δ w ͦΜͳ΋ͷΛʰ$MPVE8BUDIෳ߹ΞϥʔϜʱͰ࡞ͬͯΈͨ
  3. $MPVE8BUDIෳ߹ΞϥʔϜ w ଞͷෳ਺ͷΞϥʔϜͷঢ়ଶΛ΋ͱʹ w ࣜͰಈతʹ൑அͰ͖ΔΞϥʔϜ (ALARM("CPUUtilizationTooHigh") OR ALARM("DiskReadOpsTooHigh")) AND OK("NetworkOutTooHigh")

    $166UJMJ[BUJPO5PP)JHI͕ΞϥʔϜঢ়ଶ ·ͨ͸ %JTL3FBE0QT5PP)JHI͕ΞϥʔϜঢ়ଶ "/% /FUXPSL0VU5PP)JHI͕ਖ਼ৗঢ়ଶ ΞϥʔϜঢ়ଶ
  4. 

  5. ঢ়گΛ੔ཧͯ͠Έͨ w ෳ߹ΞϥʔϜ͕དྷͨ ‣    Ͱͳ͍99ͷͱ͖ʹ௨஌ ͢Δ͸ͣ 

    ‣    ͷͱ͖͸ɺ͜ͷΞϥʔϜ͸ൃՐ͠ͳ͍  ͔֬ʹͦͷ࣌͸͸ൃՐͤͣ99ͷΈൃՐ͍ͯͨ͠͸ͣ ‣ ͭ·Γͱ͔͕ൃੜͨ͠ʁ w ͦͷ਺ඵޙʹɺΤϥʔ͕དྷͨ ‣ Ͱ΋͖ͬ͞ෳ߹ΞϥʔϜ͕དྷͨͬͯ͜ͱ͸ɺͰ͸ͳ͍͸ͣɾɾɾ
  6. ঢ়گΛ੔ཧͯ͠Έͨ w Τϥʔ͕݅ൃੜʢҙਤతʹʣ w 99୅ΘΓͷෳ߹ΞϥʔϜ͕݅௨஌ ‣    Ͱͳ͍99ͷͱ͖ʹ௨஌

    ͢Δ͸ͣ  w ਺ඵޙ )551$PEF@&-#@͕݅௨஌ ݅ͷΤϥʔʹରͯ͠ ෳ߹ΞϥʔϜ΋ΞϥʔϜ΋ ྆ํ௨஌͕དྷͨ ॏෳ௨஌͞Εͯ͠·ͬͨ
  7. Α͘ݟͯΈͨ w )551$PEF@&-#@99@$PVOU ‣ ൃੜ ঢ়ଶͷߋ৽ ࣌ࠁɿ w )551$PEF@&-#@@$PVOU ‣

    ൃੜ ঢ়ଶͷߋ৽ ࣌ࠁɿ 99͕ઌʹൃՐ͠ɺͦͷޙ͕ൃՐͨ͠ ෳ߹ΞϥʔϜͷλΠϛϯάͰ͸ ͔֬ʹ99͔͠ൃՐ͍ͯ͠ͳ͍ ͸ൃՐ͍ͯ͠ͳ͍ ࣜ௨Γͷڍಈʹͳ͍ͬͯΔ
  8. αϓϨοαʔΞϥʔϜ w બ୒ͨ͠ΞϥʔϜ͕ΞϥʔϜঢ়ଶʹͳΔͱෳ߹ΞϥʔϜͷΞΫγϣϯ͕ແޮʹͳΔ ‣ αϓϨοαʔΞϥʔϜͷൃՐΛࢦఆඵ਺଴ͪɺෳ߹ΞϥʔϜͷൃՐΛ஗Ԇɾ཈੍ w αϓϨοαʔΞϥʔϜ͸͔ͭ͠બ΂ͳ͍ ‣  

     Λશ෦બ͹ͳ͍ͱࠓճͷ໨త͸ୡ੒Ͱ͖ͳ͍  ͜ΕΒ͕ൃՐ͢Δ͔Ͳ͏͔଴͔ͬͯΒෳ߹ΞϥʔϜΛධՁ͍ͤͨ͞  ΋࣌ؒࠩ͠ͰൃՐ͢ΔͳΒෳ߹ΞϥʔϜͷൃՐΛࢭΊ͍ͨ ‣    ͷෳ߹ΞϥʔϜΛ࡞੒ͯ͠ɺͦΕΛબ୒͢Ε͹ྑ͍  UFTUHPUPFMCTVQQSFTTJPO ෳ߹ΞϥʔϜ  ɹɹl"-"3. UFTUHPUPFMC 03"-"3. lUFTUHPUPFMC 03ʜz
  9. ஫ҙ w ࠓճͷख๏Ͱ͸ɺ͏·͍ͬͨ͘ʢ૝ఆ௨Γͷڍಈʹͳͬͨʣͱͯ͠΋ɺ    ͱͦΕҎ֎ͷ99Τϥʔ͕ಉ࣌ʢಉධՁظؒ಺ʣʹൃੜͨ͠৔߹ɺ ෳ߹ΞϥʔϜ ࣮࣭99ϝτϦΫε ͷํͷ௨஌͕དྷ·ͤΜɻ

    ‣ ྫ͑͹ͱ͕ಉ࣌ʹى͖ͨͱ͖ɺʢ۩৅ʣͷ௨஌͚ͩདྷΔ w ͦΕ͸࣮ࡍʹΤϥʔ͕ى͖ͨ࣌͸݁ہ"84ίϯιʔϧͷάϥϑΛݟʹߦ͘ͱ ࢥ͏ͷͰͦΕͰ൑ผ͢Ε͹͍͍͔ͳͱ͍͏ํ޲ͰɺؤுΒͳ͍͜ͱʹ͠·͠ ͨɻ
  10. એ఻ɿࣗ࡞"84ؔ࿈πʔϧ044 w EFMTUBDL"84$MPVE'PSNBUJPOελοΫڧ੍࡟আπʔϧ ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZEFMTUBDL w DMT4όέοτߴ଎࡟আɾۭʹ͢Δπʔϧ όʔδϣχϯάରԠ  ‣

    IUUQTHPUPLIBUFOBCMPHDPNFOUSZDMT w MBNWFS-BNCEBϥϯλΠϜόʔδϣϯݕࡧπʔϧ Ϧʔδϣϯԣஅ  ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZMBNWFS