8/23 Developers.IOブログに 何が起きたか

AWS東京リージョン障害のあった2019年8月23日、Developers.IOブログに
何が起きたか紹介します。

Transcript

1. ࣮࿥Ϋϥϝι࣌
   
   %FWFMPQFST*0ϒϩάʹ
   Կ͕ى͖͔ͨ "84ࣄۀຊ෦ɹίϯαϧςΟϯά෦
   ླ໦྄ #cmdevio5
2.  ࣗݾ঺հ
3. ࣗݾ঺հ  • ࢯ໊
   
   ླ໦ɹ྄ɹ ͖ͣ͢Γΐ͏
   • 5XJUUFS
   TV[SZP
   • ॴଐ
   Ϋϥεϝιου"84ࣄۀຊ෦ Solutions Architect - Professional DevOps Engineer - Professional Big Data - Specialty Security - Specialty Solutions Architect - Associate Developer - Associate SysOps Administrator - Associate
4. ޷͖ͳ"84αʔϏε  "NB[PO
   3PVUF
   
   "NB[PO
   ,JOFTJT
5. "84
   ΢ϧτϥΫΠζͰ༏উ͖ͯͨ͠
   "84
   4VNNJU
   
   SF.JY  IUUQTEFWDMBTTNFUIPEKQDMPVEBXTTVNNJUUPLZPVMUSBRVJ[
6. ΞδΣϯμ  w %FWFMPQFST*0
   ϒϩάج൫
   ɹߏ੒ɺಛ௃ɺӡ༻
   w 
   ౦ژϦʔδϣϯো֐தʹԿ͕ى͖͔ͨ
   ɹൃੜɺௐࠪɺાஔɺใࠂ
   w ো֐ޙʹ࣮ࢪͨ͠࠶ൃରࡦ
   ɹվળ
   w ·ͱΊ
7. Developers.IO ϒϩάج൫  ߏ੒
8. ϒϩάج൫ͱͯ͠͸  ϨΨγʔͳWordPress؀ڥͰ͢
9. %FWFMPQFST*0
   ϒϩάͷߏ੒ਤ 
10. ೥౰࣌ͷهࣄ  IUUQTEFWDMBTTNFUIPEKQTFSWFSTJEFOHJOYXPSEQSFTTFYQJSF
11. #FTU
    1SBDUJDFT
    GPS
    8PSE1SFTT
    PO
    "84  IUUQTEBXTTUBUJDDPNXIJUFQBQFSTXPSEQSFTTCFTUQSBDUJDFTPOBXTQEG
    
12. 8IPMF
    XFCTJUF
    EFMJWFSZ
    WJB
    "NB[PO
    $MPVE'SPOU  $MPVE'SPOUܦ༝Ͱ8PSE1SFTT؀ڥΛެ։
    
13. 3FGFSFODF
    BSDIJUFDUVSF
    GPS
    IPTUJOH
    8PSE1SFTT
    PO
    "84  "VSPSBɺ&MBTUJ$BDIFɺ&'4Λ׆༻ͨ͠ߴՄ༻ੑߏ੒ͳͲ΋঺հ͞Ε͍ͯ·͢
    
14. Developers.IO ϒϩάج൫  ಛ௃
15. %FWFMPQFST*0
    ϒϩάͷಛ௃  େྔʹྔ࢈͞ΕΔهࣄ
16. %FWFMPQFST*0
    ࣮੷ ݄ؒهࣄܝࡌ਺  ೔ฏۉʙهࣄ
17. Ξ΢τό΢ϯυ௨৴τϥϑΟοΫ ࣌ؒฏۉ  IUUQE  /HJOY  $MPVE'SPOU 
18. ϦΫΤετ݅਺ ࣌ؒฏۉ  IUUQE  /HJOY  $MPVE'SPOU 
19. Developers.IO ϒϩάج൫  ӡ༻
20. Քಇ཰໨ඪ 4-0  Քಇ཰99.95% (ڐ༰μ΢ϯλΠϜ͸݄ؒ20෼)
21.  ؂ࢹ
22. 8PSE1SFTT  8PSE1SFTTͱ%#ʹґଘ͢Δػೳͷҟৗ Developers.IO ϒϩάج൫
23. &MBTUJD#FBOTUBML
    ֦ுϔϧενΣοΫ  ϔϧεʮ0,ʯҎ֎͕ܧଓɺࣗಈ෮چͰճ෮͠ͳ͔ͬͨ৔߹ʹ௨஌ Developers.IO ϒϩάج൫
24. &MBTUJD#FBOTUBML
    ֦ுϔϧενΣοΫϩά  $MPVE8BUDI-PHT
    *OTJHIU͕ศརͰ͢ Developers.IO ϒϩάج൫
25. Ωϟογϡ /HJOYɺ$MPVE'SPOU  /HJOYͱɺ$MPVE'SPOU഑৴ͷਖ਼ৗੑ Developers.IO ϒϩάج൫
26. 3PVUF
    ϔϧενΣοΫ  63-؂ࢹɺ֎ܗ؂ࢹΛ࣮ࢪ Developers.IO ϒϩάج൫
27. %# "VSPSB  ϘτϧωοΫͱͳΔ%#ͷߴෛՙΛݕ஌ Developers.IO ϒϩάج൫
28. ύϑΥʔϚϯεΠϯαΠτ  %#ϩʔυɺ࿈ଓ্ͨ͠ঢΛաෛՙͱͯ͠௨஌ Developers.IO ϒϩάج൫
29. ༗ਓ؂ࢹ  Developers.IO ϒϩάج൫ ৗ࣌୭͔͕ࣥචதɻෆ۩߹࿈བྷ͸ࣾ಺4MBDLʹల։
30. 3PVUF
    ϔϧενΣοΫ
    ͱͷൺֱ
     3PVUFϔϧενΣοΧʔ
    ա൒਺͕ΤϥʔͱͳΔ෼લʹݕग़ Developers.IO ϒϩάج൫
31.  όοΫΞοϓ
32. όοΫΞοϓର৅  "VSPSB هࣄ%# ͱ4 هࣄͷը૾ ͷফࣦ͸ઈରʹճආ Developers.IO ϒϩάج൫
33. 4  όʔδϣχϯά
    
    ΫϩεϦʔδϣϯ
    ϨϓϦέʔγϣϯ ॏԽ Developers.IO ϒϩάج൫
34. "VSPSB  "VSPSB
    ετϨʔδ ॏԽ
    εφοϓγϣοτ
    %#μϯϓͷ4ΤΫεϙʔτ
    ॏԽ Developers.IO ϒϩάج൫
35. "VSPSB
    άϩʔόϧσʔλϕʔε
    ˞ۙ೔ධՁ༧ఆ  "VSPSBάϩʔόϧσʔλϕʔε ౦ژ೥݄ϦϦʔε ʹΑΔɺ
    ౦ژɺΦϨΰϯؒͷ%#ಉظ Developers.IO ϒϩάج൫
36.  ϦΧόϦʔ
37. ࠶ىಈӡ༻  աෛՙ%#ͷ࠶ىಈɺεϖοΫௐ੔ Developers.IO ϒϩάج൫
38. %#
    ϦετΞ  %#ϦετΞ͢Δ৔߹ɺΞϓϦͷ؀ڥม਺Λมߋ Developers.IO ϒϩάج൫
39. %3
    େن໛Ϧʔδϣϯো֐૝ఆ  ผϦʔδϣϯʹ࣮ߦ؀ڥΛల։ɺެ։༻ͷΤϯυϙΠϯτΛௐ੔͠࠶։ Developers.IO ϒϩάج൫
40. མͪͳ͍αΠτͰ͸͋Γ·ͤΜɻ  ෮׆Ͱ͖ΔࣄΛ໨ࢦͨ͠αΠτͰ͢
41. ΞδΣϯμ  w %FWFMPQFST*0
    ϒϩάج൫
    ɹߏ੒ɺಛ௃ɺӡ༻
    w 
    ౦ژϦʔδϣϯো֐தʹԿ͕ى͖͔ͨ
    ɹൃੜɺௐࠪɺાஔɺใࠂ
    w ো֐ޙʹ࣮ࢪͨ͠࠶ൃରࡦ
    ɹվળ
    w ·ͱΊ
42.  8/23 Developers.IO ϒϩά ͳʹ͕ى͖͔ͨ
43. ౦ژϦʔδϣϯো֐தͷ౰ϒϩάՔಇΛ঺հ͠·͢  IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBQOFB[EPXOEFWJP
44. "84
    4FSWJDF
    )FBMUI
    %BTICPBSE
    4)%  IUUQTTUBUVTBXTBNB[PODPN"1@CMPDL
    Developers.IO ϒϩάج൫
45. 4)%
    ܝࡌࣄ߲ൈਮ  Developers.IO ϒϩάج൫ ࣌ࠁ +45 ࣄ৅  ۭௐނোɺΦʔόʔώʔτʹΑΓ&$͕ఀࢭ  &$ͷ"1*ΤϥʔϨʔτ্ঢ  ۭௐ෮چɺΦʔόώʔτͰఀࢭͨ͠&$ճ෮։࢝  େ෦෼ͷ&$͕ճ෮
46. ɹো֐ൃੜ  ELB5XXΤϥʔ཰্ঢ
47. 
    ো֐௨஌  &MBTUJD#FBOTUBMLεςʔλεҟৗΛݕ஌ Developers.IO ϒϩάج൫
48.  ௐࠪ։࢝
49. 
    &MBTUJD#FBOTUBMLμογϡϘʔυ  ࠒΑΓ
    )551
    YY
    ͕ൃੜ͍ͯͨ͠ࣄΛ֬ೝ
    Developers.IO ϒϩάج൫
50. 
    $MPVE8BUDIϝτϦοΫ֬ೝ %BUBEPHμογϡϘʔυ  Developers.IO ϒϩάج൫ &-#ͷ99ΤϥʔҎ֎ɺ%#ɺ&$ͷਖ਼ৗಈ࡞Λ֬ೝ
51. 
    $MPVE8BUDIϝτϦοΫ  &-#99Τϥʔ
    ো֐ใࠂͷ͋ͬͨ"; BQOPSUIFBTUB
    ͷΈͰൃੜ͍ͯ͠ΔࣄΛ֬ೝ
    Developers.IO ϒϩάج൫
52. 
    99Τϥʔৄࡉɺ&-#ͷΞΫηεϩάΑΓௐࠪ։࢝  "-#ͷΞΫηεϩά໿෼ఔ౓ͷλΠϜϥάͰ"UIFOBղੳՄೳʹ͢Δ࢓૊ΈΛ঺հ͠·͢
    IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHTRTEFpSFIPTF
    Developers.IO ϒϩάج൫
53. 
    &-#ΞΫηεϩάʮXBGGBJMEʯݕग़  99Τϥʔͷେ൒͕Τϥʔίʔυ಺෦αʔόΤϥʔ
    ʮXBGGBJMEʯͷग़ྗΑΓ"84
    8"'ؔ܎ͷ໰୊ͱਪఆ Developers.IO ϒϩάج൫
54. 
    &-#ΞΫηεϩάʮXBGGBJMEʯ֘౰Ϩίʔυ  +40/Խͨ͠ΞΫηεϩά֬ೝɺ4
    4FMFDUɺ+2ͳͲซ༻ Developers.IO ϒϩάج൫
55. "84υΩϡϝϯτ
    "QQMJDBUJPO
    -PBE
    #BMBODFS
    ͷτϥϒϧγϡʔςΟϯά  IUUQTEPDTBXTBNB[PODPNKB@KQFMBTUJDMPBECBMBODJOHMBUFTUBQQMJDBUJPOMPBECBMBODFS USPVCMFTIPPUJOHIUNMIUUQJTTVFT Developers.IO ϒϩάج൫
56.  AWSWAF
57. "84
    8"'ηΩϡϦςΟΦʔτϝʔγϣϯ  944ɺ42-Jɺ)551
    'MPPEɺ*1
    3FQVUBUJPO
    ϧʔϧར༻த
    IUUQTBXTBNB[PODPNKQTPMVUJPOTBXTXBGTFDVSJUZBVUPNBUJPOT
    Developers.IO ϒϩάج൫
58. "84
    8"'
    Ϛωʔδυϧʔϧ
    ݕূ  IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBXTSFJOWFOUXBGNBOBHFESVMFUSZ Developers.IO ϒϩάج൫
59. ৄࡉͳΞΫηεϩάճऩ ϦΫΤετϔομ౳  IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBXGXBGDPNQSFIFOTJWFMPHHJOH
    Developers.IO ϒϩάج൫
60. ৵ೖ๷ࢭγεςϜ *%4
    ͸Ҿୀࡁ  IUUQTEFWDMBTTNFUIPEKQDMPVEBXTJETXJUILJOFTJTXBG Developers.IO ϒϩάج൫
61. 8"'ͷϒϥοΫϦετ*1ࣗಈߋ৽͢Δ࢓૊Έ  ηΩϡϦςΟΦʔτϝʔγϣϯ -BNCEB ɺ8"'ͷϨʔτϧʔϧͰ୅ସ Developers.IO ϒϩάج൫
62. 
    8"'
    ಈ࡞ঢ়گΛ֬ೝ  "84
    8"'ϒϩοΫ࣮੷ΑΓɺແޮԽʹΑΔϦεΫΛ൑அ Developers.IO ϒϩάج൫
63.  ાஔ
64. 
    8"'
    ແޮԽ  "84
    8"'
    "$-ͷର৅͔Β"-#Λআ֎ Developers.IO ϒϩάج൫
65. 
    $MPVE8BUDIϝτϦοΫ֬ೝ
    &-#99  8"'
    อޢఀࢭޙɺ99Τϥʔऩଋ Developers.IO ϒϩάج൫
66. 
    ";Bͷ
    "-#ɺ8"'ͰԿΒ͔ͷ໰୊ൃੜͱਪఆ  Developers.IO ϒϩάج൫ "84ޙ೔ൃද  ʹΑΔͱɺ
    8"'΍εςοΩʔηογϣϯ༗ޮͱͨ͠Ұ෦ͷ"-#Ͱൃੜͨ͠໰୊ʹ֘౰
67. 
    $MPVE8BUDIϝτϦοΫ֬ೝ
    %BUBEPHμογϡϘʔυ  Developers.IO ϒϩάج൫ &-#ͷ99Τϥʔͷऩଋͱɺ%#ɺ&$ͷਖ਼ৗಈ࡞Λ֬ೝ
68.  ใࠂ
69.  ৘ใల։(ࣾ಺)
70. 
    ࣾ಺ใࠂ
    4MBDL  Developers.IO ϒϩάج൫
71. 
    ࣾ಺ใࠂ
    4MBDL  "84
    8"'͕༗ޮͳผ؀ڥͰ΋ྨࣅͷࣄ৅Λ֬ೝ Developers.IO ϒϩάج൫
72. 
    ࣾ಺ใࠂ
    4MBDL  Developers.IO ϒϩάج൫
73.  ৘ใల։(AWSαϙʔτ)
74. 
    "84αϙʔτέʔεىථ  Developers.IO ϒϩάج൫
75. 
    "84αϙʔτέʔεىථ಺༰ͷൈਮ  Developers.IO ϒϩάج൫ ɾELBɺWAF࿈ܞͷ໰୊Β͍͠ه࿥͕ɺELBͷΞΫηεϩάɺactions_executed ʹࣔ͞Ε͍ͯΔࣄɻ ɾWAFͷACLઃఆͰɺอޢର৅ͷELBΛআ֎ͨ͠ޙ͸ɺ5XXΤϥʔͷൃੜ͸ऩଋͨ͠ࣄΑΓɺ ԿΒ͔ͷ໰୊͕ELB(ALB)ɺWAFؒͰൃੜ͍ͯͨ͠΋ͷͱਪଌ͞Ε·͢ɻ https://status.aws.amazon.com/#AP_block ɾAmazon Elastic Load Balancing (Tokyo) ɾAWS WAF Service is operating normally ELBɺWAFͷΤϥʔ৘ใ͸ެ։͞Ε͍ͯͳ͍༷ࢠͰ͕ͨ͠ɺ AWSαϙʔτଆͰ೺Ѳ͞Ε͍ͯΔ੍ݶࣄ߲ͳͲ͋Ε͹ڞ༗௖͚Ε͹޾͍Ͱ͢ɻ
76. 
    "84αϙʔτʹఏڙͨ͠"-#ΞΫηεϩά৘ใൈਮ  Developers.IO ϒϩάج൫ zcat log.gz | jq . | grep '"elb_status_code":' | sort | uniq -c | sort -nr 11805 "elb_status_code": "200", 1615 "elb_status_code": "500", 1523 "elb_status_code": "304", 261 "elb_status_code": "403", 166 "elb_status_code": “301", 127 "elb_status_code": "404", 119 "elb_status_code": "400", 48 "elb_status_code": "460", 21 "elb_status_code": "302", 5 "elb_status_code": "401", 2 "elb_status_code": "504",
77. 
    "84αϙʔτʹఏڙͨ͠"-#ΞΫηεϩάൈਮ  Developers.IO ϒϩάج൫ "timestamp": "2019-08-23T05:53:50.767192Z", "elb": "app/awseb-AWSEB-xxxxxxxxxxx/xxxxxxxxxxx", "request_processing_time": -1, "target_processing_time": -1, "response_processing_time": -1, "elb_status_code": "500", "target_status_code": "-", "request": "GET https://dev.classmethod.jp:443/cloud/aws/insufficientinstancecapacity/ HTTP/ "target_group_arn": "arn:aws:elasticloadbalancing:ap-northeast-1:xxxxxxxxxxx:targetgroup/a "trace_id": "Self=1-5d5f7f6c-f8b1df60e11e997011940890;Root=1-5d5f7f6c-fd7ec9703dd7b "request_creation_time": "2019-08-23T05:53:48.766000Z", "actions_executed": "waf-failed",
78. 
    ো֐ൃੜͷཌʑ೔  AWSެࣜ৘ใ
79. ౦ژϦʔδϣϯͰൃੜͨ͠
    "NB[PO
    &$
    ͱ
    "NB[PO
    &#4
    ͷࣄ৅֓ཁ  IUUQTBXTBNB[PODPNKQNFTTBHF
80. ౦ژϦʔδϣϯͰൃੜͨ͠
    "NB[PO
    &$
    ͱ
    "NB[PO
    &#4
    ͷࣄ৅֓ཁ  IUUQTBXTBNB[PODPNKQNFTTBHF
81. 
    ࣌ࣾ಺4MBDL  Developers.IO ϒϩάج൫
82. 
    ਂ໷  Developers.IO ϒϩάج൫
83. 
    ো֐ൃੜ͔Β೔ޙ  AWSެࣜ৘ใߋ৽
84. 
    "84ެࣜϖʔδࣄ৅֓ཁ௥ه  IUUQTBXTBNB[PODPNKQNFTTBHF
85. ΞδΣϯμ  w %FWFMPQFST*0
    ϒϩάج൫
    ɹߏ੒ɺಛ௃ɺӡ༻
    w 
    ౦ژϦʔδϣϯো֐தʹԿ͕ى͖͔ͨ
    ɹൃੜɺௐࠪɺાஔɺใࠂ
    w ো֐ޙʹ࣮ࢪͨ͠࠶ൃରࡦ
    ɹվળ
    w ·ͱΊ
86.  ରࡦ঺հ
87. ෳ਺ͷ"WBJMBCJMJUZ
    ;POFʹϓϩϏδϣχϯάͨ͠&-# "-#
    
    "VUP4DBMJOH
    (SPVQ͔Βಛఆ"WBJMBCJMJUZ
    ;POF্ͷϦιʔεΛύʔδ͢Δ  ";ো֐ͷରࡦखஈͷ֬อͱɺ";ো֐ͷӨڹ؇࿨ͷͨΊʹ࣮ࢪ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTQVSHFSFTPVSDFTTQFDJpDB[
88. "-#ͷΞΫηεϩάʹΞϕΠΞϏϦςΟκʔϯ৘ใΛ௥Ճͯ͠Έͨ  "-#ͷΞΫηεϩάϨίʔυʹ͸&-#ϊʔυ৘ใ͕ແ͍ͨΊɺ
    ϩάϑΝΠϧͷΩʔʹؚ·ΕΔ*1৘ใΛݩʹ-BNCEBͰϊʔυ৘ใΛิ͍ɺ"UIFOBूܭΛ࣮ݱ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBEEB[BMCBDDFTTMPH
89. $MPVE8BUDI
    -PHT
    *OTJHIUTͰ"-#ͷ&-#
    99ൃੜ࣌ͷΞΫηεϩάΛ෼ੳͯ͠Έͨ  &-#99ΤϥʔϩάͷΈ$MPVE8BUDI-PHTʹసૹ͢Δ-BNCEBΛར༻͠ɺ
    "UIFOBΑΓ؆қͳΤϥʔϩά֬ೝखஈΛ֬อ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCYYDXMPHTJOTJHIUT
90. 'JSFIPTFͰ
    1BSRVFUܗࣜʹม׵ͨ͠"-#ͷΞΫηεϩάΛ"UIFOBͰղੳͯ͠Έͨ  'JSFIPTFͷྻࢤ޲ม׵Λར༻ɺ"UIFOBͷॲཧޮ཰Λվળ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHpSFIPTFQBSRVFU
91. 'JSFIPTFͰ
    1BSRVFUܗࣜʹม׵ͨ͠"-#ͷΞΫηεϩάΛ"UIFOBͰղੳͯ͠Έͨ  %FW*0ͷ&-#ϩάॲཧͷઆ໌ࢿྉͰ͢ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHTRTEFpSFIPTF
92. (MPCBM
    "DDFMFSBUPS
    Λར༻ͯ͠#MVF(SFFOͳ&-#ަ׵Λ࣮ࢪͯ͠Έͨ  %FW*0ͷ&-#ɺ";ɺ*1WରԠͷ؀ڥ΁ަ׵ͨ͠खஈͰ͢ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTCMVFHSFFOFMCCZBHB
93. %FWFMPQFST*0͕*1WαϙʔτʹͳΓ·ͨ͠  *1W
    ো֐࣌ͷᷖճܦ࿏ͱͳΔࣄΛظ଴ͯ͠*1W༗ޮԽ͠·ͨ͠ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTEFWJPTVQQPSUJQW
94.  ·ͱΊ
95.  DevelopersIOαΠτͷऔΓ૊Έʹ͍ͭͯ ঺հͤͯ͞௖͖·ͨ͠
96.  ো֐͸վળʹܨ͕Δ ઈ޷ͷػձͱߟ͍͑ͯ·͢
97.  γεςϜͷՌͨ͢΂͖໾ׂɺ ॏཁ౓ɺ՝୊͕໌ྎͳঢ়ଶͰɺ ࢓૊ΈΛߟ͑Δࣄ͕ग़དྷ·͢ɻ
98.  ໰୊Λఆྔతʹࣔ͢ϩά΍ϝτϦοΫɺ ਖ਼͍͠൑அͷཁͱͳΓ·͢ɻ
99.  ໰୊Λڞ༗ɺվળΛଞʹٻΊΔ৔߹ʹ΋ɺ ࠜڌͱͳΔϩά΍ϝτϦοΫ͕ॏཁͰ͢
100.  ໾ཱͭ࢓૊Έ͕ɺ AWSͰ͸਺ଟ͘ར༻ग़དྷ·͢ɻ