CloudWatch複合アラームでELBの5XXをいい感じに検知しようとしたらうまくいかなかった話 / cloudwatch alarm elb 5xx

by k.goto

Slide 1

Slide 1 text

LHPUP าͷςοΫ $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β ͏·͍͔͘ͳ͔ͬͨ࿩ +"846(43&ࢧ෦

Slide 2

Slide 2 text

ࣗݾ঺հ LHPUP w ςοΫϦʔυɾϥʔϝϯ޷͖ w "84$PNNVOJUZ#VJMEFS %FW5PPMT w าͷςοΫ ٕज़ϒϩά w ࣗ࡞"84ؔ࿈πʔϧͷ044։ൃ w 5XJUUFS!@TUFQ@UFDI ‣ LHPUP าͷςοΫ

Slide 3

Slide 3 text

$MPVE8BUDIෳ߹ΞϥʔϜͰ

Slide 4

Slide 4 text

&-#ͷ99Λ

Slide 5

Slide 5 text

͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β

Slide 6

Slide 6 text

͏·͍͔͘ͳ͔ͬͨ࿩

Slide 7

Slide 7 text

͍͍ײ͡ʁ w &-# "-# ͷΤϥʔ௨஌ $MPVE8BUDI"MBSN ‣ )551$PEF@&-#@ccc@$PVOU εςʔλε͝ͱʹ௨஌͍ͨ͠ ‣ )551$PEF@&-#@99@$PVOU cccͷ৔߹͸99௨஌͠ͳ͍ 㾎ॏෳ௨஌Λഉআ͍ͨ͠ ʮ99΋΋དྷͯ͠·͏ʯͷΛආ͚Δ cccҎ֎ͷΤϥʔ͚ͩ99Ͱ௨஌͢Δ 㾎 ͳͲͷͱ͖͚ͩ௨஌͢Δ w ͦΜͳ΋ͷΛʰ$MPVE8BUDIෳ߹ΞϥʔϜʱͰ࡞ͬͯΈͨ

Slide 8

Slide 8 text

$MPVE8BUDIෳ߹ΞϥʔϜͱ͸ʁ

Slide 9

Slide 9 text

$MPVE8BUDIෳ߹ΞϥʔϜ w ଞͷෳ਺ͷΞϥʔϜͷঢ়ଶΛ΋ͱʹ w ࣜͰಈతʹ൑அͰ͖ΔΞϥʔϜ (ALARM("CPUUtilizationTooHigh") OR ALARM("DiskReadOpsTooHigh")) AND OK("NetworkOutTooHigh") $166UJMJ[BUJPO5PP)JHI͕ΞϥʔϜঢ়ଶ ·ͨ͸ %JTL3FBE0QT5PP)JHI͕ΞϥʔϜঢ়ଶ "/% /FUXPSL0VU5PP)JHI͕ਖ਼ৗঢ়ଶ ΞϥʔϜঢ়ଶ

Slide 10

Slide 10 text

΍ͬͯΈͨ

Slide 11

Slide 11 text

΍ͬͯΈͨ ෳ߹ΞϥʔϜ 99୅ΘΓͳͲͰൃՐͤ͞Δ༻

Slide 12

Slide 12 text

΍ͬͯΈͨ YY͕ΞϥʔϜঢ়ଶ "/% ͕ΞϥʔϜঢ়ଶ ·ͨ͸ ͕ΞϥʔϜঢ়ଶ /05 ˞ ͕ͳ͍ͷ͸ ݕূ্ͷཧ༝Ͱ͢ ͩͱΞϥʔϜ ͩͱਖ਼ৗ

Slide 13

Slide 13 text

΍ͬͯΈͨ ΞϥʔϜ ͳͲ΋ಉ༷ʹ࡞੒

Slide 14

Slide 14 text

΍ͬͯΈͨ 99ΞϥʔϜ ࣮ࡍͷൃՐ͸ෳ߹ΞϥʔϜͷํͰߦ͏ͷͰ ͪ͜Β͸ΞΫγϣϯແޮʹ ͨͩͷΞϥʔϜࡐྉͱͯ͠࡞੒

Slide 15

Slide 15 text

ΤϥʔΛग़ͯ͠Έͨ

Slide 16

Slide 16 text

௨஌͕དྷͨʂ

Slide 17

Slide 17 text

ෳ߹ΞϥʔϜͷํ͕དྷͨʂ

Slide 18

Slide 18 text

ͭ·Γ ʮ Ͱͳ͍99ʯ ͷํ

Slide 19

Slide 19 text

ͭ·Γɺͱ͔ʂ

Slide 20

Slide 20 text

Ͱ΋Ͱ΋Ͱ΋Ͱ΋ ͳ͍εςʔλεͳΜͯ ͦΜͳ؆୯ʹى͜Δ͔ɾɾɾʁ

Slide 21

Slide 21 text

ͦͷ͞Βʹ਺ඵޙ ΋͏݅௨஌͕ɾɾɾ

Slide 22

Slide 22 text

Slide 23

Slide 23 text

ঢ়گΛ੔ཧͯ͠Έͨ w ෳ߹ΞϥʔϜ͕དྷͨ ‣ Ͱͳ͍99ͷͱ͖ʹ௨஌ ͢Δ͸ͣ ‣ ͷͱ͖͸ɺ͜ͷΞϥʔϜ͸ൃՐ͠ͳ͍ ͔֬ʹͦͷ࣌͸͸ൃՐͤͣ99ͷΈൃՐ͍ͯͨ͠͸ͣ ‣ ͭ·Γͱ͔͕ൃੜͨ͠ʁ w ͦͷ਺ඵޙʹɺΤϥʔ͕དྷͨ ‣ Ͱ΋͖ͬ͞ෳ߹ΞϥʔϜ͕དྷͨͬͯ͜ͱ͸ɺͰ͸ͳ͍͸ͣɾɾɾ

Slide 24

Slide 24 text

ঢ়گΛ੔ཧͯ͠Έͨ w $MPVE8BUDIϝτϦΫεΛݟͯΈͨ ‣ )551$PEF@&-#@99@$PVOU ݅ ‣ )551$PEF@&-#@@$PVOU ݅ ͳͲͷɺͦͷଞͷΤϥʔ͸ ى͖͍ͯ·ͤΜͰͨ͠ɻ

Slide 25

Slide 25 text

ঢ়گΛ੔ཧͯ͠Έͨ w Τϥʔ͕݅ൃੜʢҙਤతʹʣ w 99୅ΘΓͷෳ߹ΞϥʔϜ͕݅௨஌ ‣ Ͱͳ͍99ͷͱ͖ʹ௨஌ ͢Δ͸ͣ w ਺ඵޙ )551$PEF@&-#@͕݅௨஌ ݅ͷΤϥʔʹରͯ͠ ෳ߹ΞϥʔϜ΋ΞϥʔϜ΋ ྆ํ௨஌͕དྷͨ ॏෳ௨஌͞Εͯ͠·ͬͨ

Slide 26

Slide 26 text

͏·͍͔͘ͳ͔ͬͨ

Slide 27

Slide 27 text

Α͘ݟͯΈͨ

Slide 28

Slide 28 text

Α͘ݟͯΈͨ

Slide 29

Slide 29 text

Α͘ݟͯΈͨ

Slide 30

Slide 30 text

Α͘ݟͯΈͨ w )551$PEF@&-#@99@$PVOU ‣ ൃੜ ঢ়ଶͷߋ৽ ࣌ࠁɿ w )551$PEF@&-#@@$PVOU ‣ ൃੜ ঢ়ଶͷߋ৽ ࣌ࠁɿ 99͕ઌʹൃՐ͠ɺͦͷޙ͕ൃՐͨ͠ ෳ߹ΞϥʔϜͷλΠϛϯάͰ͸ ͔֬ʹ99͔͠ൃՐ͍ͯ͠ͳ͍ ͸ൃՐ͍ͯ͠ͳ͍ ࣜ௨Γͷڍಈʹͳ͍ͬͯΔ

Slide 31

Slide 31 text

99͕ઌʹൃՐͯ͠ ͦͷޙ۩৅εςʔλε͕ ൃՐ͢Δͷ͔ɾɾɾʁ

Slide 32

Slide 32 text

೦ͷͨΊ؀ڥΛ࡞Γ௚ͯ͠ ࠶౓࣮ߦͯ͠Έͨ

Slide 33

Slide 33 text

ࠓ౓͸ 99ΑΓઌʹ͕ൃՐͨ͠

Slide 34

Slide 34 text

ൃՐλΠϛϯά ඵ਺ ͸ ຖճಉͩͬͨ͡

Slide 35

Slide 35 text

ൃՐධՁλΠϛϯά͸ ΞϥʔϜΛ࡞੒ͨ͠ λΠϛϯά ࣌ࠁ ʹΑΔʁ

Slide 36

Slide 36 text

ඵ਺͸ҧ͏ɾɾɾ

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

ԿʹͤΑ ධՁλΠϛϯά͸ຖ෼ಉ͡ඵ਺

Slide 39

Slide 39 text

ΞϥʔϜͷධՁλΠϛϯάͷલޙͲͪΒͰ Τϥʔ͕ൃੜ͢Δ͔࣍ୈͰॱ൪͕มΘΔ T T T T 99ධՁ ධՁ ͜͜ͰΤϥʔ͕ى͖ͨΒ͕ઌʹൃՐ ͜͜ͰΤϥʔ͕ى͖ͨΒ99͕ઌʹൃՐ

Slide 40

Slide 40 text

ΤϥʔͷൃੜλΠϛϯάͰ ΞϥʔϜൃՐॱ൪͕มΘΔͷ͸ࠔΔ

Slide 41

Slide 41 text

͜ΕͰऴΘΓ͔ʁ

Slide 42

Slide 42 text

͍΍

Slide 43

Slide 43 text

·ͩ΍ΕΔ

Slide 44

Slide 44 text

αϓϨοαʔΞϥʔϜ

Slide 45

Slide 45 text

αϓϨοαʔΞϥʔϜ IUUQTEPDTBXTBNB[PODPNKB@KQ"NB[PO$MPVE8BUDIMBUFTUNPOJUPSJOH$SFBUF@$PNQPTJUF@"MBSNIUNM

Slide 46

Slide 46 text

αϓϨοαʔΞϥʔϜ ࣮ߦ͢·͢

Slide 47

Slide 47 text

αϓϨοαʔΞϥʔϜ w બ୒ͨ͠ΞϥʔϜ͕ΞϥʔϜঢ়ଶʹͳΔͱෳ߹ΞϥʔϜͷΞΫγϣϯ͕ແޮʹͳΔ ‣ αϓϨοαʔΞϥʔϜͷൃՐΛࢦఆඵ਺଴ͪɺෳ߹ΞϥʔϜͷൃՐΛ஗Ԇɾ཈੍ w αϓϨοαʔΞϥʔϜ͸͔ͭ͠બ΂ͳ͍ ‣ Λશ෦બ͹ͳ͍ͱࠓճͷ໨త͸ୡ੒Ͱ͖ͳ͍ ͜ΕΒ͕ൃՐ͢Δ͔Ͳ͏͔଴͔ͬͯΒෳ߹ΞϥʔϜΛධՁ͍ͤͨ͞ ΋࣌ؒࠩ͠ͰൃՐ͢ΔͳΒෳ߹ΞϥʔϜͷൃՐΛࢭΊ͍ͨ ‣ ͷෳ߹ΞϥʔϜΛ࡞੒ͯ͠ɺͦΕΛબ୒͢Ε͹ྑ͍ UFTUHPUPFMCTVQQSFTTJPO ෳ߹ΞϥʔϜ ɹɹl"-"3. UFTUHPUPFMC 03"-"3. lUFTUHPUPFMC 03ʜz

Slide 48

Slide 48 text

αϓϨοαʔΞϥʔϜ

Slide 49

Slide 49 text

΍ͬͯΈͨ

Slide 50

Slide 50 text

΍ͬͯΈͨɿαϓϨοαʔΞϥʔϜ UFTUHPUPFMC99 UFTUHPUPFMC UFTUHPUPFMCTVQQSFTTJPO ࠷ॳͷݕূ࣌ͱಉ͘͡ΤϥʔΛൃੜͤͨ͞Β 99͕ઌʹൃՐͨ͠ ͱෳ߹ͷॏෳൃՐࣦͨ͠ഊύλʔϯ௨Γ

Slide 51

Slide 51 text

͢Δͱɾɾɾ

Slide 52

Slide 52 text

ෳ߹ΞϥʔϜ ࣮࣭99 ͸ ൃՐ ௨஌ ͤͣ

Slide 53

Slide 53 text

͚ͩ௨஌͕དྷͨʂʂʂ

Slide 54

Slide 54 text

ॏෳഉআ੒ޭʂ

Slide 55

Slide 55 text

΍ͬͯΈͨɿαϓϨοαʔΞϥʔϜ 99ൃՐʹΑΓҰ࣌తʹΞϥʔϜঢ়ଶʹ͸ͳ͍ͬͯΔ͕ ෼ؒ 8BJU1FSJPE ͸ΞΫγϣϯ͕཈੍͞ΕΔ ͦͷޙൃՐʹΑͬͯ0,ঢ়ଶʹ໭ͬͨ ʢ͕ൃՐ͠ͳ͚Ε͹ΞΫγϣϯ͕ى͜Δ͸ͣʁʣ

Slide 56

Slide 56 text

LHPUP าͷςοΫ $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β ͏·͍͔͘ͳ͔ͬͨ࿩ +"846(43&ࢧ෦

Slide 57

Slide 57 text

LHPUP าͷςοΫ $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β ͏·͍͔͘ͳ͔ͬͨ࿩ +"846(43&ࢧ෦

Slide 58

Slide 58 text

஫ҙ w ࣮ࡍʹͳͲͷΤϥʔ Ҏ֎ Λൃੜͤ͞ΔݕূΛ͍ͯ͠ ͳ͍ͷͰɺͩͱͪΌΜͱෳ߹ΞϥʔϜ͕ൃՐ͢Δ͔Ͳ͏͔ͷ֬ೝ͸ͯ͠ ͍·ͤΜ ‣ μϝͩͬͨΒ͍͢·ͤΜɾɾɾ

Slide 59

Slide 59 text

஫ҙ w ࠓճͷख๏Ͱ͸ɺ͏·͍ͬͨ͘ʢ૝ఆ௨Γͷڍಈʹͳͬͨʣͱͯ͠΋ɺ ͱͦΕҎ֎ͷ99Τϥʔ͕ಉ࣌ʢಉධՁظؒ಺ʣʹൃੜͨ͠৔߹ɺ ෳ߹ΞϥʔϜ ࣮࣭99ϝτϦΫε ͷํͷ௨஌͕དྷ·ͤΜɻ ‣ ྫ͑͹ͱ͕ಉ࣌ʹى͖ͨͱ͖ɺʢ۩৅ʣͷ௨஌͚ͩདྷΔ w ͦΕ͸࣮ࡍʹΤϥʔ͕ى͖ͨ࣌͸݁ہ"84ίϯιʔϧͷάϥϑΛݟʹߦ͘ͱ ࢥ͏ͷͰͦΕͰ൑ผ͢Ε͹͍͍͔ͳͱ͍͏ํ޲ͰɺؤுΒͳ͍͜ͱʹ͠·͠ ͨɻ

Slide 60

Slide 60 text

ࢀߟɿϒϩάهࣄ าͷςοΫ w $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β ͏·͍͔͘ͳ͔ͬͨ࿩ ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZ

Slide 61

Slide 61 text

એ఻ɿࣗ࡞"84ؔ࿈πʔϧ044 w EFMTUBDL"84$MPVE'PSNBUJPOελοΫڧ੍࡟আπʔϧ ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZEFMTUBDL w DMT4όέοτߴ଎࡟আɾۭʹ͢Δπʔϧ όʔδϣχϯάରԠ ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZDMT w MBNWFS-BNCEBϥϯλΠϜόʔδϣϯݕࡧπʔϧ Ϧʔδϣϯԣஅ ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZMBNWFS

Slide 62

Slide 62 text

એ఻ɿొஃ w ໦ d"84%FW%BZ5PLZP ೥࿈ଓ ‣ ʰ"84$%,ͰֶͿ(P'σβΠϯύλʔϯʙ*B$ʹ΋ίʔυઃܭʙʱ

Slide 63

Slide 63 text

5IBOL:PV LHPUP าͷςοΫ