CloudWatch複合アラームでELBの5XXをいい感じに検知しようとしたらうまくいかなかった話 / cloudwatch alarm elb 5xx

2023/05/29開催「JAWS-UG SRE支部 #6」での発表資料です。

Transcript

1. 
   LHPUP าͷςοΫ
   
   $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ
   ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
   ͏·͍͔͘ͳ͔ͬͨ࿩
   +"846(43&ࢧ෦
   

   View Slide

2. ࣗݾ঺հ LHPUP
   
   w ςοΫϦʔυɾϥʔϝϯ޷͖
   w "84$PNNVOJUZ#VJMEFS %FW5PPMT
   
   w าͷςοΫ ٕज़ϒϩά
   
   w ࣗ࡞"84ؔ࿈πʔϧͷ044։ൃ
   w 5XJUUFS!@TUFQ@UFDI
   ‣ LHPUP าͷςοΫ
   
   

   View Slide

3. $MPVE8BUDIෳ߹ΞϥʔϜͰ
   

   View Slide

4. &-#ͷ99Λ
   

   View Slide

5. ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
   

   View Slide

6. ͏·͍͔͘ͳ͔ͬͨ࿩
   

   View Slide

7. ͍͍ײ͡ʁ
   w &-# "-#
   ͷΤϥʔ௨஌ $MPVE8BUDI"MBSN
   
   ‣ )551$PEF@&-#@ccc@$PVOU
   εςʔλε͝ͱʹ௨஌͍ͨ͠
   ‣ )551$PEF@&-#@99@$PVOU
   cccͷ৔߹͸99௨஌͠ͳ͍
   㾎ॏෳ௨஌Λഉআ͍ͨ͠ ʮ99΋΋དྷͯ͠·͏ʯͷΛආ͚Δ
   
   cccҎ֎ͷΤϥʔ͚ͩ99Ͱ௨஌͢Δ
   㾎 ͳͲͷͱ͖͚ͩ௨஌͢Δ
   w ͦΜͳ΋ͷΛʰ$MPVE8BUDIෳ߹ΞϥʔϜʱͰ࡞ͬͯΈͨ
   

   View Slide

8. $MPVE8BUDIෳ߹ΞϥʔϜͱ͸ʁ
   

   View Slide

9. $MPVE8BUDIෳ߹ΞϥʔϜ
   w ଞͷෳ਺ͷΞϥʔϜͷঢ়ଶΛ΋ͱʹ
   w ࣜͰಈతʹ൑அͰ͖ΔΞϥʔϜ
   (ALARM("CPUUtilizationTooHigh") OR
   ALARM("DiskReadOpsTooHigh")) AND
   OK("NetworkOutTooHigh")
   $166UJMJ[BUJPO5PP)JHI͕ΞϥʔϜঢ়ଶ
   ·ͨ͸
   %JTL3FBE0QT5PP)JHI͕ΞϥʔϜঢ়ଶ
   "/% /FUXPSL0VU5PP)JHI͕ਖ਼ৗঢ়ଶ
   ΞϥʔϜঢ়ଶ
   

   View Slide

10. ΍ͬͯΈͨ
    

    View Slide

11. ΍ͬͯΈͨ
    ෳ߹ΞϥʔϜ 99୅ΘΓͳͲͰൃՐͤ͞Δ༻
    
    

    View Slide

12. ΍ͬͯΈͨ
    YY͕ΞϥʔϜঢ়ଶ "/%
    ͕ΞϥʔϜঢ়ଶ
    ·ͨ͸
    ͕ΞϥʔϜঢ়ଶ
    /05
    ˞ ͕ͳ͍ͷ͸
    ݕূ্ͷཧ༝Ͱ͢
    ͩͱΞϥʔϜ
    ͩͱਖ਼ৗ
    

    View Slide

13. ΍ͬͯΈͨ
    ΞϥʔϜ ͳͲ΋ಉ༷ʹ࡞੒
    
    

    View Slide

14. ΍ͬͯΈͨ
    99ΞϥʔϜ
    ࣮ࡍͷൃՐ͸ෳ߹ΞϥʔϜͷํͰߦ͏ͷͰ
    ͪ͜Β͸ΞΫγϣϯແޮʹ
    ͨͩͷΞϥʔϜࡐྉͱͯ͠࡞੒
    
    

    View Slide

15. ΤϥʔΛग़ͯ͠Έͨ
    

    View Slide

16. ௨஌͕དྷͨʂ
    

    View Slide

17. ෳ߹ΞϥʔϜͷํ͕དྷͨʂ
    

    View Slide

18. ͭ·Γ
    ʮ Ͱͳ͍99ʯ
    ͷํ
    

    View Slide

19. ͭ·Γɺͱ͔ʂ
    

    View Slide

20. Ͱ΋Ͱ΋Ͱ΋Ͱ΋
    ͳ͍εςʔλεͳΜͯ
    ͦΜͳ؆୯ʹى͜Δ͔ɾɾɾʁ
    

    View Slide

21. ͦͷ͞Βʹ਺ඵޙ
    ΋͏݅௨஌͕ɾɾɾ
    

    View Slide

22. 
    

    View Slide

23. ঢ়گΛ੔ཧͯ͠Έͨ
    w ෳ߹ΞϥʔϜ͕དྷͨ
    ‣ Ͱͳ͍99ͷͱ͖ʹ௨஌ ͢Δ͸ͣ
    
    ‣ ͷͱ͖͸ɺ͜ͷΞϥʔϜ͸ൃՐ͠ͳ͍
    ͔֬ʹͦͷ࣌͸͸ൃՐͤͣ99ͷΈൃՐ͍ͯͨ͠͸ͣ
    ‣ ͭ·Γͱ͔͕ൃੜͨ͠ʁ
    w ͦͷ਺ඵޙʹɺΤϥʔ͕དྷͨ
    ‣ Ͱ΋͖ͬ͞ෳ߹ΞϥʔϜ͕དྷͨͬͯ͜ͱ͸ɺͰ͸ͳ͍͸ͣɾɾɾ
    

    View Slide

24. ঢ়گΛ੔ཧͯ͠Έͨ
    w $MPVE8BUDIϝτϦΫεΛݟͯΈͨ
    ‣ )551$PEF@&-#@99@$PVOU
    ݅
    ‣ )551$PEF@&-#@@$PVOU
    ݅
    ͳͲͷɺͦͷଞͷΤϥʔ͸
    ى͖͍ͯ·ͤΜͰͨ͠ɻ
    

    View Slide

25. ঢ়گΛ੔ཧͯ͠Έͨ
    w Τϥʔ͕݅ൃੜʢҙਤతʹʣ
    w 99୅ΘΓͷෳ߹ΞϥʔϜ͕݅௨஌
    ‣ Ͱͳ͍99ͷͱ͖ʹ௨஌ ͢Δ͸ͣ
    
    w ਺ඵޙ
    )551$PEF@&-#@͕݅௨஌
    ݅ͷΤϥʔʹରͯ͠
    ෳ߹ΞϥʔϜ΋ΞϥʔϜ΋
    ྆ํ௨஌͕དྷͨ
    ॏෳ௨஌͞Εͯ͠·ͬͨ
    

    View Slide

26. ͏·͍͔͘ͳ͔ͬͨ
    

    View Slide

27. Α͘ݟͯΈͨ
    

    View Slide

28. Α͘ݟͯΈͨ
    

    View Slide

29. Α͘ݟͯΈͨ
    

    View Slide

30. Α͘ݟͯΈͨ
    w )551$PEF@&-#@99@$PVOU
    ‣ ൃੜ ঢ়ଶͷߋ৽
    ࣌ࠁɿ
    w )551$PEF@&-#@@$PVOU
    ‣ ൃੜ ঢ়ଶͷߋ৽
    ࣌ࠁɿ
    99͕ઌʹൃՐ͠ɺͦͷޙ͕ൃՐͨ͠
    ෳ߹ΞϥʔϜͷλΠϛϯάͰ͸
    ͔֬ʹ99͔͠ൃՐ͍ͯ͠ͳ͍
    ͸ൃՐ͍ͯ͠ͳ͍
    
    ࣜ௨Γͷڍಈʹͳ͍ͬͯΔ
    

    View Slide

31. 99͕ઌʹൃՐͯ͠
    ͦͷޙ۩৅εςʔλε͕
    ൃՐ͢Δͷ͔ɾɾɾʁ
    

    View Slide

32. ೦ͷͨΊ؀ڥΛ࡞Γ௚ͯ͠
    ࠶౓࣮ߦͯ͠Έͨ
    

    View Slide

33. ࠓ౓͸
    99ΑΓઌʹ͕ൃՐͨ͠
    

    View Slide

34. ൃՐλΠϛϯά ඵ਺
    ͸
    ຖճಉͩͬͨ͡
    

    View Slide

35. ൃՐධՁλΠϛϯά͸
    ΞϥʔϜΛ࡞੒ͨ͠
    λΠϛϯά ࣌ࠁ
    ʹΑΔʁ
    

    View Slide

36. ඵ਺͸ҧ͏ɾɾɾ
    

    View Slide

37. View Slide

38. ԿʹͤΑ
    ධՁλΠϛϯά͸ຖ෼ಉ͡ඵ਺
    

    View Slide

39. ΞϥʔϜͷධՁλΠϛϯάͷલޙͲͪΒͰ
    Τϥʔ͕ൃੜ͢Δ͔࣍ୈͰॱ൪͕มΘΔ
    T T
    T T
    99ධՁ ධՁ
    ͜͜ͰΤϥʔ͕ى͖ͨΒ͕ઌʹൃՐ
    ͜͜ͰΤϥʔ͕ى͖ͨΒ99͕ઌʹൃՐ
    

    View Slide

40. ΤϥʔͷൃੜλΠϛϯάͰ
    ΞϥʔϜൃՐॱ൪͕มΘΔͷ͸ࠔΔ
    

    View Slide

41. ͜ΕͰऴΘΓ͔ʁ
    

    View Slide

42. ͍΍
    

    View Slide

43. ·ͩ΍ΕΔ
    

    View Slide

44. αϓϨοαʔΞϥʔϜ
    

    View Slide

45. αϓϨοαʔΞϥʔϜ
    IUUQTEPDTBXTBNB[PODPNKB@KQ"NB[PO$MPVE8BUDIMBUFTUNPOJUPSJOH$SFBUF@$PNQPTJUF@"MBSNIUNM
    

    View Slide

46. αϓϨοαʔΞϥʔϜ
    ࣮ߦ͢·͢
    

    View Slide

47. αϓϨοαʔΞϥʔϜ
    w બ୒ͨ͠ΞϥʔϜ͕ΞϥʔϜঢ়ଶʹͳΔͱෳ߹ΞϥʔϜͷΞΫγϣϯ͕ແޮʹͳΔ
    ‣ αϓϨοαʔΞϥʔϜͷൃՐΛࢦఆඵ਺଴ͪɺෳ߹ΞϥʔϜͷൃՐΛ஗Ԇɾ཈੍
    w αϓϨοαʔΞϥʔϜ͸͔ͭ͠બ΂ͳ͍
    ‣ Λશ෦બ͹ͳ͍ͱࠓճͷ໨త͸ୡ੒Ͱ͖ͳ͍
    ͜ΕΒ͕ൃՐ͢Δ͔Ͳ͏͔଴͔ͬͯΒෳ߹ΞϥʔϜΛධՁ͍ͤͨ͞
    ΋࣌ؒࠩ͠ͰൃՐ͢ΔͳΒෳ߹ΞϥʔϜͷൃՐΛࢭΊ͍ͨ
    ‣ ͷෳ߹ΞϥʔϜΛ࡞੒ͯ͠ɺͦΕΛબ୒͢Ε͹ྑ͍
    UFTUHPUPFMCTVQQSFTTJPO ෳ߹ΞϥʔϜ
    
    ɹɹl"-"3. UFTUHPUPFMC
    03"-"3. lUFTUHPUPFMC
    03ʜz
    

    View Slide

48. αϓϨοαʔΞϥʔϜ
    

    View Slide

49. ΍ͬͯΈͨ
    

    View Slide

50. ΍ͬͯΈͨɿαϓϨοαʔΞϥʔϜ
    UFTUHPUPFMC99
    UFTUHPUPFMC
    UFTUHPUPFMCTVQQSFTTJPO
    ࠷ॳͷݕূ࣌ͱಉ͘͡ΤϥʔΛൃੜͤͨ͞Β
    99͕ઌʹൃՐͨ͠
    ͱෳ߹ͷॏෳൃՐࣦͨ͠ഊύλʔϯ௨Γ
    
    

    View Slide

51. ͢Δͱɾɾɾ
    

    View Slide

52. ෳ߹ΞϥʔϜ ࣮࣭99
    ͸
    ൃՐ ௨஌
    ͤͣ
    

    View Slide

53. ͚ͩ௨஌͕དྷͨʂʂʂ
    

    View Slide

54. ॏෳഉআ੒ޭʂ
    

    View Slide

55. ΍ͬͯΈͨɿαϓϨοαʔΞϥʔϜ
    99ൃՐʹΑΓҰ࣌తʹΞϥʔϜঢ়ଶʹ͸ͳ͍ͬͯΔ͕
    ෼ؒ 8BJU1FSJPE
    ͸ΞΫγϣϯ͕཈੍͞ΕΔ
    ͦͷޙൃՐʹΑͬͯ0,ঢ়ଶʹ໭ͬͨ
    ʢ͕ൃՐ͠ͳ͚Ε͹ΞΫγϣϯ͕ى͜Δ͸ͣʁʣ
    

    View Slide

56. 
    LHPUP าͷςοΫ
    
    $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ
    ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
    ͏·͍͔͘ͳ͔ͬͨ࿩
    +"846(43&ࢧ෦
    

    View Slide

57. 
    LHPUP าͷςοΫ
    
    $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ
    ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
    ͏·͍͔͘ͳ͔ͬͨ࿩
    +"846(43&ࢧ෦
    

    View Slide

58. ஫ҙ
    w ࣮ࡍʹͳͲͷΤϥʔ Ҏ֎
    Λൃੜͤ͞ΔݕূΛ͍ͯ͠
    ͳ͍ͷͰɺͩͱͪΌΜͱෳ߹ΞϥʔϜ͕ൃՐ͢Δ͔Ͳ͏͔ͷ֬ೝ͸ͯ͠
    ͍·ͤΜ
    ‣ μϝͩͬͨΒ͍͢·ͤΜɾɾɾ
    

    View Slide

59. ஫ҙ
    w ࠓճͷख๏Ͱ͸ɺ͏·͍ͬͨ͘ʢ૝ఆ௨Γͷڍಈʹͳͬͨʣͱͯ͠΋ɺ
    ͱͦΕҎ֎ͷ99Τϥʔ͕ಉ࣌ʢಉධՁظؒ಺ʣʹൃੜͨ͠৔߹ɺ
    ෳ߹ΞϥʔϜ ࣮࣭99ϝτϦΫε
    ͷํͷ௨஌͕དྷ·ͤΜɻ
    ‣ ྫ͑͹ͱ͕ಉ࣌ʹى͖ͨͱ͖ɺʢ۩৅ʣͷ௨஌͚ͩདྷΔ
    w ͦΕ͸࣮ࡍʹΤϥʔ͕ى͖ͨ࣌͸݁ہ"84ίϯιʔϧͷάϥϑΛݟʹߦ͘ͱ
    ࢥ͏ͷͰͦΕͰ൑ผ͢Ε͹͍͍͔ͳͱ͍͏ํ޲ͰɺؤுΒͳ͍͜ͱʹ͠·͠
    ͨɻ
    

    View Slide

60. ࢀߟɿϒϩάهࣄ าͷςοΫ
    
    w $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
    ͏·͍͔͘ͳ͔ͬͨ࿩
    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZ
    

    View Slide

61. એ఻ɿࣗ࡞"84ؔ࿈πʔϧ044
    w EFMTUBDL"84$MPVE'PSNBUJPOελοΫڧ੍࡟আπʔϧ
    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZEFMTUBDL
    w DMT4όέοτߴ଎࡟আɾۭʹ͢Δπʔϧ όʔδϣχϯάରԠ
    
    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZDMT
    w MBNWFS-BNCEBϥϯλΠϜόʔδϣϯݕࡧπʔϧ Ϧʔδϣϯԣஅ
    
    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZMBNWFS
    

    View Slide

62. એ఻ɿొஃ
    w ໦
    d"84%FW%BZ5PLZP ೥࿈ଓ
    
    ‣ ʰ"84$%,ͰֶͿ(P'σβΠϯύλʔϯʙ*B$ʹ΋ίʔυઃܭʙʱ
    

    View Slide

63. 5IBOL:PV
    LHPUP าͷςοΫ
    
    

    View Slide