Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CloudWatch複合アラームでELBの5XXをいい感じに検知しようとしたらうまくいかなかった話 / cloudwatch alarm elb 5xx

k.goto
May 29, 2023

CloudWatch複合アラームでELBの5XXをいい感じに検知しようとしたらうまくいかなかった話 / cloudwatch alarm elb 5xx

2023/05/29開催「JAWS-UG SRE支部 #6」での発表資料です。

k.goto

May 29, 2023
Tweet

More Decks by k.goto

Other Decks in Technology

Transcript


  1. LHPUP าͷςοΫ

    $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ
    ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
    ͏·͍͔͘ͳ͔ͬͨ࿩
    +"846(43&ࢧ෦

    View Slide

  2. ࣗݾ঺հ LHPUP

    w ςοΫϦʔυɾϥʔϝϯ޷͖
    w "84$PNNVOJUZ#VJMEFS %FW5PPMT

    w าͷςοΫ ٕज़ϒϩά

    w ࣗ࡞"84ؔ࿈πʔϧͷ044։ൃ
    w 5XJUUFS!@TUFQ@UFDI
    ‣ LHPUP าͷςοΫ

    View Slide

  3. $MPVE8BUDIෳ߹ΞϥʔϜͰ

    View Slide

  4. &-#ͷ99Λ

    View Slide

  5. ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β

    View Slide

  6. ͏·͍͔͘ͳ͔ͬͨ࿩

    View Slide

  7. ͍͍ײ͡ʁ
    w &-# "-#
    ͷΤϥʔ௨஌ $MPVE8BUDI"MBSN

    ‣ )551$PEF@&-#@ccc@$PVOU
    εςʔλε͝ͱʹ௨஌͍ͨ͠
    ‣ )551$PEF@&-#@99@$PVOU
    cccͷ৔߹͸99௨஌͠ͳ͍
    㾎ॏෳ௨஌Λഉআ͍ͨ͠ ʮ99΋΋དྷͯ͠·͏ʯͷΛආ͚Δ

    cccҎ֎ͷΤϥʔ͚ͩ99Ͱ௨஌͢Δ
    㾎 ͳͲͷͱ͖͚ͩ௨஌͢Δ
    w ͦΜͳ΋ͷΛʰ$MPVE8BUDIෳ߹ΞϥʔϜʱͰ࡞ͬͯΈͨ

    View Slide

  8. $MPVE8BUDIෳ߹ΞϥʔϜͱ͸ʁ

    View Slide

  9. $MPVE8BUDIෳ߹ΞϥʔϜ
    w ଞͷෳ਺ͷΞϥʔϜͷঢ়ଶΛ΋ͱʹ
    w ࣜͰಈతʹ൑அͰ͖ΔΞϥʔϜ
    (ALARM("CPUUtilizationTooHigh") OR
    ALARM("DiskReadOpsTooHigh")) AND
    OK("NetworkOutTooHigh")
    $166UJMJ[BUJPO5PP)JHI͕ΞϥʔϜঢ়ଶ
    ·ͨ͸
    %JTL3FBE0QT5PP)JHI͕ΞϥʔϜঢ়ଶ
    "/% /FUXPSL0VU5PP)JHI͕ਖ਼ৗঢ়ଶ
    ΞϥʔϜঢ়ଶ

    View Slide

  10. ΍ͬͯΈͨ

    View Slide

  11. ΍ͬͯΈͨ
    ෳ߹ΞϥʔϜ 99୅ΘΓͳͲͰൃՐͤ͞Δ༻

    View Slide

  12. ΍ͬͯΈͨ
    YY͕ΞϥʔϜঢ়ଶ "/%
    ͕ΞϥʔϜঢ়ଶ
    ·ͨ͸
    ͕ΞϥʔϜঢ়ଶ
    /05
    ˞ ͕ͳ͍ͷ͸
    ݕূ্ͷཧ༝Ͱ͢
    ͩͱΞϥʔϜ
    ͩͱਖ਼ৗ

    View Slide

  13. ΍ͬͯΈͨ
    ΞϥʔϜ ͳͲ΋ಉ༷ʹ࡞੒

    View Slide

  14. ΍ͬͯΈͨ
    99ΞϥʔϜ
    ࣮ࡍͷൃՐ͸ෳ߹ΞϥʔϜͷํͰߦ͏ͷͰ
    ͪ͜Β͸ΞΫγϣϯແޮʹ
    ͨͩͷΞϥʔϜࡐྉͱͯ͠࡞੒

    View Slide

  15. ΤϥʔΛग़ͯ͠Έͨ

    View Slide

  16. ௨஌͕དྷͨʂ

    View Slide

  17. ෳ߹ΞϥʔϜͷํ͕དྷͨʂ

    View Slide

  18. ͭ·Γ
    ʮ Ͱͳ͍99ʯ
    ͷํ

    View Slide

  19. ͭ·Γɺͱ͔ʂ

    View Slide

  20. Ͱ΋Ͱ΋Ͱ΋Ͱ΋
    ͳ͍εςʔλεͳΜͯ
    ͦΜͳ؆୯ʹى͜Δ͔ɾɾɾʁ

    View Slide

  21. ͦͷ͞Βʹ਺ඵޙ
    ΋͏݅௨஌͕ɾɾɾ

    View Slide


  22. View Slide

  23. ঢ়گΛ੔ཧͯ͠Έͨ
    w ෳ߹ΞϥʔϜ͕དྷͨ
    ‣ Ͱͳ͍99ͷͱ͖ʹ௨஌ ͢Δ͸ͣ

    ‣ ͷͱ͖͸ɺ͜ͷΞϥʔϜ͸ൃՐ͠ͳ͍
    ͔֬ʹͦͷ࣌͸͸ൃՐͤͣ99ͷΈൃՐ͍ͯͨ͠͸ͣ
    ‣ ͭ·Γͱ͔͕ൃੜͨ͠ʁ
    w ͦͷ਺ඵޙʹɺΤϥʔ͕དྷͨ
    ‣ Ͱ΋͖ͬ͞ෳ߹ΞϥʔϜ͕དྷͨͬͯ͜ͱ͸ɺͰ͸ͳ͍͸ͣɾɾɾ

    View Slide

  24. ঢ়گΛ੔ཧͯ͠Έͨ
    w $MPVE8BUDIϝτϦΫεΛݟͯΈͨ
    ‣ )551$PEF@&-#@99@$PVOU
    ݅
    ‣ )551$PEF@&-#@@$PVOU
    ݅
    ͳͲͷɺͦͷଞͷΤϥʔ͸
    ى͖͍ͯ·ͤΜͰͨ͠ɻ

    View Slide

  25. ঢ়گΛ੔ཧͯ͠Έͨ
    w Τϥʔ͕݅ൃੜʢҙਤతʹʣ
    w 99୅ΘΓͷෳ߹ΞϥʔϜ͕݅௨஌
    ‣ Ͱͳ͍99ͷͱ͖ʹ௨஌ ͢Δ͸ͣ

    w ਺ඵޙ
    )551$PEF@&-#@͕݅௨஌
    ݅ͷΤϥʔʹରͯ͠
    ෳ߹ΞϥʔϜ΋ΞϥʔϜ΋
    ྆ํ௨஌͕དྷͨ
    ॏෳ௨஌͞Εͯ͠·ͬͨ

    View Slide

  26. ͏·͍͔͘ͳ͔ͬͨ

    View Slide

  27. Α͘ݟͯΈͨ

    View Slide

  28. Α͘ݟͯΈͨ

    View Slide

  29. Α͘ݟͯΈͨ

    View Slide

  30. Α͘ݟͯΈͨ
    w )551$PEF@&-#@99@$PVOU
    ‣ ൃੜ ঢ়ଶͷߋ৽
    ࣌ࠁɿ
    w )551$PEF@&-#@@$PVOU
    ‣ ൃੜ ঢ়ଶͷߋ৽
    ࣌ࠁɿ
    99͕ઌʹൃՐ͠ɺͦͷޙ͕ൃՐͨ͠
    ෳ߹ΞϥʔϜͷλΠϛϯάͰ͸
    ͔֬ʹ99͔͠ൃՐ͍ͯ͠ͳ͍
    ͸ൃՐ͍ͯ͠ͳ͍

    ࣜ௨Γͷڍಈʹͳ͍ͬͯΔ

    View Slide

  31. 99͕ઌʹൃՐͯ͠
    ͦͷޙ۩৅εςʔλε͕
    ൃՐ͢Δͷ͔ɾɾɾʁ

    View Slide

  32. ೦ͷͨΊ؀ڥΛ࡞Γ௚ͯ͠
    ࠶౓࣮ߦͯ͠Έͨ

    View Slide

  33. ࠓ౓͸
    99ΑΓઌʹ͕ൃՐͨ͠

    View Slide

  34. ൃՐλΠϛϯά ඵ਺
    ͸
    ຖճಉͩͬͨ͡

    View Slide

  35. ൃՐධՁλΠϛϯά͸
    ΞϥʔϜΛ࡞੒ͨ͠
    λΠϛϯά ࣌ࠁ
    ʹΑΔʁ

    View Slide

  36. ඵ਺͸ҧ͏ɾɾɾ

    View Slide

  37. View Slide

  38. ԿʹͤΑ
    ධՁλΠϛϯά͸ຖ෼ಉ͡ඵ਺

    View Slide

  39. ΞϥʔϜͷධՁλΠϛϯάͷલޙͲͪΒͰ
    Τϥʔ͕ൃੜ͢Δ͔࣍ୈͰॱ൪͕มΘΔ
    T T
    T T
    99ධՁ ධՁ
    ͜͜ͰΤϥʔ͕ى͖ͨΒ͕ઌʹൃՐ
    ͜͜ͰΤϥʔ͕ى͖ͨΒ99͕ઌʹൃՐ

    View Slide

  40. ΤϥʔͷൃੜλΠϛϯάͰ
    ΞϥʔϜൃՐॱ൪͕มΘΔͷ͸ࠔΔ

    View Slide

  41. ͜ΕͰऴΘΓ͔ʁ

    View Slide

  42. ͍΍

    View Slide

  43. ·ͩ΍ΕΔ

    View Slide

  44. αϓϨοαʔΞϥʔϜ

    View Slide

  45. αϓϨοαʔΞϥʔϜ
    IUUQTEPDTBXTBNB[PODPNKB@KQ"NB[PO$MPVE8BUDIMBUFTUNPOJUPSJOH$SFBUF@$PNQPTJUF@"MBSNIUNM

    View Slide

  46. αϓϨοαʔΞϥʔϜ
    ࣮ߦ͢·͢

    View Slide

  47. αϓϨοαʔΞϥʔϜ
    w બ୒ͨ͠ΞϥʔϜ͕ΞϥʔϜঢ়ଶʹͳΔͱෳ߹ΞϥʔϜͷΞΫγϣϯ͕ແޮʹͳΔ
    ‣ αϓϨοαʔΞϥʔϜͷൃՐΛࢦఆඵ਺଴ͪɺෳ߹ΞϥʔϜͷൃՐΛ஗Ԇɾ཈੍
    w αϓϨοαʔΞϥʔϜ͸͔ͭ͠બ΂ͳ͍
    ‣ Λશ෦બ͹ͳ͍ͱࠓճͷ໨త͸ୡ੒Ͱ͖ͳ͍
    ͜ΕΒ͕ൃՐ͢Δ͔Ͳ͏͔଴͔ͬͯΒෳ߹ΞϥʔϜΛධՁ͍ͤͨ͞
    ΋࣌ؒࠩ͠ͰൃՐ͢ΔͳΒෳ߹ΞϥʔϜͷൃՐΛࢭΊ͍ͨ
    ‣ ͷෳ߹ΞϥʔϜΛ࡞੒ͯ͠ɺͦΕΛબ୒͢Ε͹ྑ͍
    UFTUHPUPFMCTVQQSFTTJPO ෳ߹ΞϥʔϜ

    ɹɹl"-"3. UFTUHPUPFMC
    03"-"3. lUFTUHPUPFMC
    03ʜz

    View Slide

  48. αϓϨοαʔΞϥʔϜ

    View Slide

  49. ΍ͬͯΈͨ

    View Slide

  50. ΍ͬͯΈͨɿαϓϨοαʔΞϥʔϜ
    UFTUHPUPFMC99
    UFTUHPUPFMC
    UFTUHPUPFMCTVQQSFTTJPO
    ࠷ॳͷݕূ࣌ͱಉ͘͡ΤϥʔΛൃੜͤͨ͞Β
    99͕ઌʹൃՐͨ͠
    ͱෳ߹ͷॏෳൃՐࣦͨ͠ഊύλʔϯ௨Γ

    View Slide

  51. ͢Δͱɾɾɾ

    View Slide

  52. ෳ߹ΞϥʔϜ ࣮࣭99
    ͸
    ൃՐ ௨஌
    ͤͣ

    View Slide

  53. ͚ͩ௨஌͕དྷͨʂʂʂ

    View Slide

  54. ॏෳഉআ੒ޭʂ

    View Slide

  55. ΍ͬͯΈͨɿαϓϨοαʔΞϥʔϜ
    99ൃՐʹΑΓҰ࣌తʹΞϥʔϜঢ়ଶʹ͸ͳ͍ͬͯΔ͕
    ෼ؒ 8BJU1FSJPE
    ͸ΞΫγϣϯ͕཈੍͞ΕΔ
    ͦͷޙൃՐʹΑͬͯ0,ঢ়ଶʹ໭ͬͨ
    ʢ͕ൃՐ͠ͳ͚Ε͹ΞΫγϣϯ͕ى͜Δ͸ͣʁʣ

    View Slide


  56. LHPUP าͷςοΫ

    $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ
    ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
    ͏·͍͔͘ͳ͔ͬͨ࿩
    +"846(43&ࢧ෦

    View Slide


  57. LHPUP าͷςοΫ

    $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ
    ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
    ͏·͍͔͘ͳ͔ͬͨ࿩
    +"846(43&ࢧ෦

    View Slide

  58. ஫ҙ
    w ࣮ࡍʹͳͲͷΤϥʔ Ҏ֎
    Λൃੜͤ͞ΔݕূΛ͍ͯ͠
    ͳ͍ͷͰɺͩͱͪΌΜͱෳ߹ΞϥʔϜ͕ൃՐ͢Δ͔Ͳ͏͔ͷ֬ೝ͸ͯ͠
    ͍·ͤΜ
    ‣ μϝͩͬͨΒ͍͢·ͤΜɾɾɾ

    View Slide

  59. ஫ҙ
    w ࠓճͷख๏Ͱ͸ɺ͏·͍ͬͨ͘ʢ૝ఆ௨Γͷڍಈʹͳͬͨʣͱͯ͠΋ɺ
    ͱͦΕҎ֎ͷ99Τϥʔ͕ಉ࣌ʢಉධՁظؒ಺ʣʹൃੜͨ͠৔߹ɺ
    ෳ߹ΞϥʔϜ ࣮࣭99ϝτϦΫε
    ͷํͷ௨஌͕དྷ·ͤΜɻ
    ‣ ྫ͑͹ͱ͕ಉ࣌ʹى͖ͨͱ͖ɺʢ۩৅ʣͷ௨஌͚ͩདྷΔ
    w ͦΕ͸࣮ࡍʹΤϥʔ͕ى͖ͨ࣌͸݁ہ"84ίϯιʔϧͷάϥϑΛݟʹߦ͘ͱ
    ࢥ͏ͷͰͦΕͰ൑ผ͢Ε͹͍͍͔ͳͱ͍͏ํ޲ͰɺؤுΒͳ͍͜ͱʹ͠·͠
    ͨɻ

    View Slide

  60. ࢀߟɿϒϩάهࣄ าͷςοΫ

    w $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
    ͏·͍͔͘ͳ͔ͬͨ࿩
    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZ

    View Slide

  61. એ఻ɿࣗ࡞"84ؔ࿈πʔϧ044
    w EFMTUBDL"84$MPVE'PSNBUJPOελοΫڧ੍࡟আπʔϧ
    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZEFMTUBDL
    w DMT4όέοτߴ଎࡟আɾۭʹ͢Δπʔϧ όʔδϣχϯάରԠ

    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZDMT
    w MBNWFS-BNCEBϥϯλΠϜόʔδϣϯݕࡧπʔϧ Ϧʔδϣϯԣஅ

    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZMBNWFS

    View Slide

  62. એ఻ɿొஃ
    w ໦
    d"84%FW%BZ5PLZP ೥࿈ଓ

    ‣ ʰ"84$%,ͰֶͿ(P'σβΠϯύλʔϯʙ*B$ʹ΋ίʔυઃܭʙʱ

    View Slide

  63. 5IBOL:PV
    LHPUP าͷςοΫ

    View Slide