8/23 Developers.IOブログに 何が起きたか

B88e743f16c9365d6cd988d787e6b8ff?s=47 suzryo
November 01, 2019

8/23 Developers.IOブログに 何が起きたか

AWS東京リージョン障害のあった2019年8月23日、Developers.IOブログに
何が起きたか紹介します。

B88e743f16c9365d6cd988d787e6b8ff?s=128

suzryo

November 01, 2019
Tweet

Transcript

  1. ࣮࿥Ϋϥϝι࣌ %FWFMPQFST*0ϒϩάʹ Կ͕ى͖͔ͨ "84ࣄۀຊ෦ɹίϯαϧςΟϯά෦ ླ໦྄ #cmdevio5

  2.  ࣗݾ঺հ

  3. ࣗݾ঺հ  • ࢯ໊ླ໦ɹ྄ɹ ͖ͣ͢Γΐ͏  • 5XJUUFSTV[SZP • ॴଐΫϥεϝιου"84ࣄۀຊ෦

    Solutions Architect - Professional DevOps Engineer - Professional Big Data - Specialty Security - Specialty Solutions Architect - Associate Developer - Associate SysOps Administrator - Associate
  4. ޷͖ͳ"84αʔϏε  "NB[PO3PVUF "NB[PO,JOFTJT

  5. "84΢ϧτϥΫΠζͰ༏উ͖ͯͨ͠ "844VNNJUSF.JY  IUUQTEFWDMBTTNFUIPEKQDMPVEBXTTVNNJUUPLZPVMUSBRVJ[

  6. ΞδΣϯμ  w %FWFMPQFST*0ϒϩάج൫ ɹߏ੒ɺಛ௃ɺӡ༻ w ౦ژϦʔδϣϯো֐தʹԿ͕ى͖͔ͨ ɹൃੜɺௐࠪɺાஔɺใࠂ w ো֐ޙʹ࣮ࢪͨ͠࠶ൃରࡦ

    ɹվળ w ·ͱΊ
  7. Developers.IO ϒϩάج൫  ߏ੒

  8. ϒϩάج൫ͱͯ͠͸  ϨΨγʔͳWordPress؀ڥͰ͢

  9. %FWFMPQFST*0ϒϩάͷߏ੒ਤ 

  10. ೥౰࣌ͷهࣄ  IUUQTEFWDMBTTNFUIPEKQTFSWFSTJEFOHJOYXPSEQSFTTFYQJSF

  11. #FTU1SBDUJDFTGPS8PSE1SFTTPO"84  IUUQTEBXTTUBUJDDPNXIJUFQBQFSTXPSEQSFTTCFTUQSBDUJDFTPOBXTQEG

  12. 8IPMFXFCTJUFEFMJWFSZWJB"NB[PO$MPVE'SPOU  $MPVE'SPOUܦ༝Ͱ8PSE1SFTT؀ڥΛެ։

  13. 3FGFSFODFBSDIJUFDUVSFGPSIPTUJOH8PSE1SFTTPO"84  "VSPSBɺ&MBTUJ$BDIFɺ&'4Λ׆༻ͨ͠ߴՄ༻ੑߏ੒ͳͲ΋঺հ͞Ε͍ͯ·͢

  14. Developers.IO ϒϩάج൫  ಛ௃

  15. %FWFMPQFST*0ϒϩάͷಛ௃  େྔʹྔ࢈͞ΕΔهࣄ

  16. %FWFMPQFST*0࣮੷ ݄ؒهࣄܝࡌ਺  ೔ฏۉʙهࣄ

  17. Ξ΢τό΢ϯυ௨৴τϥϑΟοΫ ࣌ؒฏۉ  IUUQE  /HJOY  $MPVE'SPOU 

  18. ϦΫΤετ݅਺ ࣌ؒฏۉ  IUUQE  /HJOY  $MPVE'SPOU 

  19. Developers.IO ϒϩάج൫  ӡ༻

  20. Քಇ཰໨ඪ 4-0  Քಇ཰99.95% (ڐ༰μ΢ϯλΠϜ͸݄ؒ20෼)

  21.  ؂ࢹ

  22. 8PSE1SFTT  8PSE1SFTTͱ%#ʹґଘ͢Δػೳͷҟৗ Developers.IO ϒϩάج൫

  23. &MBTUJD#FBOTUBML֦ுϔϧενΣοΫ  ϔϧεʮ0,ʯҎ֎͕ܧଓɺࣗಈ෮چͰճ෮͠ͳ͔ͬͨ৔߹ʹ௨஌ Developers.IO ϒϩάج൫

  24. &MBTUJD#FBOTUBML֦ுϔϧενΣοΫϩά  $MPVE8BUDI-PHT*OTJHIU͕ศརͰ͢ Developers.IO ϒϩάج൫

  25. Ωϟογϡ /HJOYɺ$MPVE'SPOU  /HJOYͱɺ$MPVE'SPOU഑৴ͷਖ਼ৗੑ Developers.IO ϒϩάج൫

  26. 3PVUFϔϧενΣοΫ  63-؂ࢹɺ֎ܗ؂ࢹΛ࣮ࢪ Developers.IO ϒϩάج൫

  27. %# "VSPSB  ϘτϧωοΫͱͳΔ%#ͷߴෛՙΛݕ஌ Developers.IO ϒϩάج൫

  28. ύϑΥʔϚϯεΠϯαΠτ  %#ϩʔυɺ࿈ଓ্ͨ͠ঢΛաෛՙͱͯ͠௨஌ Developers.IO ϒϩάج൫

  29. ༗ਓ؂ࢹ  Developers.IO ϒϩάج൫ ৗ࣌୭͔͕ࣥචதɻෆ۩߹࿈བྷ͸ࣾ಺4MBDLʹల։

  30. 3PVUFϔϧενΣοΫͱͷൺֱ  3PVUFϔϧενΣοΧʔա൒਺͕ΤϥʔͱͳΔ෼લʹݕग़ Developers.IO ϒϩάج൫

  31.  όοΫΞοϓ

  32. όοΫΞοϓର৅  "VSPSB هࣄ%# ͱ4 هࣄͷը૾ ͷফࣦ͸ઈରʹճආ Developers.IO ϒϩάج൫

  33. 4  όʔδϣχϯά ΫϩεϦʔδϣϯϨϓϦέʔγϣϯ ॏԽ Developers.IO ϒϩάج൫

  34. "VSPSB  "VSPSBετϨʔδ ॏԽ εφοϓγϣοτ %#μϯϓͷ4ΤΫεϙʔτ ॏԽ Developers.IO ϒϩάج൫

  35. "VSPSB άϩʔόϧσʔλϕʔε ˞ۙ೔ධՁ༧ఆ  "VSPSBάϩʔόϧσʔλϕʔε ౦ژ೥݄ϦϦʔε ʹΑΔɺ ౦ژɺΦϨΰϯؒͷ%#ಉظ Developers.IO ϒϩάج൫

  36.  ϦΧόϦʔ

  37. ࠶ىಈӡ༻  աෛՙ%#ͷ࠶ىಈɺεϖοΫௐ੔ Developers.IO ϒϩάج൫

  38. %#ϦετΞ  %#ϦετΞ͢Δ৔߹ɺΞϓϦͷ؀ڥม਺Λมߋ Developers.IO ϒϩάج൫

  39. %3 େن໛Ϧʔδϣϯো֐૝ఆ  ผϦʔδϣϯʹ࣮ߦ؀ڥΛల։ɺެ։༻ͷΤϯυϙΠϯτΛௐ੔͠࠶։ Developers.IO ϒϩάج൫

  40. མͪͳ͍αΠτͰ͸͋Γ·ͤΜɻ  ෮׆Ͱ͖ΔࣄΛ໨ࢦͨ͠αΠτͰ͢

  41. ΞδΣϯμ  w %FWFMPQFST*0ϒϩάج൫ ɹߏ੒ɺಛ௃ɺӡ༻ w ౦ژϦʔδϣϯো֐தʹԿ͕ى͖͔ͨ ɹൃੜɺௐࠪɺાஔɺใࠂ w ো֐ޙʹ࣮ࢪͨ͠࠶ൃରࡦ

    ɹվળ w ·ͱΊ
  42.  8/23 Developers.IO ϒϩά ͳʹ͕ى͖͔ͨ

  43. ౦ژϦʔδϣϯো֐தͷ౰ϒϩάՔಇΛ঺հ͠·͢  IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBQOFB[EPXOEFWJP

  44. "844FSWJDF)FBMUI%BTICPBSE 4)%  IUUQTTUBUVTBXTBNB[PODPN"1@CMPDL Developers.IO ϒϩάج൫

  45. 4)%ܝࡌࣄ߲ൈਮ  Developers.IO ϒϩάج൫ ࣌ࠁ +45 ࣄ৅  ۭௐނোɺΦʔόʔώʔτʹΑΓ&$͕ఀࢭ 

    &$ͷ"1*ΤϥʔϨʔτ্ঢ  ۭௐ෮چɺΦʔόώʔτͰఀࢭͨ͠&$ճ෮։࢝  େ෦෼ͷ&$͕ճ෮
  46. ɹো֐ൃੜ  ELB5XXΤϥʔ཰্ঢ

  47. ো֐௨஌  &MBTUJD#FBOTUBMLεςʔλεҟৗΛݕ஌ Developers.IO ϒϩάج൫

  48.  ௐࠪ։࢝

  49. &MBTUJD#FBOTUBMLμογϡϘʔυ  ࠒΑΓ)551YY͕ൃੜ͍ͯͨ͠ࣄΛ֬ೝ Developers.IO ϒϩάج൫

  50. $MPVE8BUDIϝτϦοΫ֬ೝ %BUBEPHμογϡϘʔυ  Developers.IO ϒϩάج൫ &-#ͷ99ΤϥʔҎ֎ɺ%#ɺ&$ͷਖ਼ৗಈ࡞Λ֬ೝ

  51. $MPVE8BUDIϝτϦοΫ  &-#99Τϥʔো֐ใࠂͷ͋ͬͨ"; BQOPSUIFBTUB  ͷΈͰൃੜ͍ͯ͠ΔࣄΛ֬ೝ Developers.IO ϒϩάج൫

  52. 99Τϥʔৄࡉɺ&-#ͷΞΫηεϩάΑΓௐࠪ։࢝  "-#ͷΞΫηεϩά໿෼ఔ౓ͷλΠϜϥάͰ"UIFOBղੳՄೳʹ͢Δ࢓૊ΈΛ঺հ͠·͢ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHTRTEFpSFIPTF Developers.IO ϒϩάج൫

  53. &-#ΞΫηεϩάʮXBGGBJMEʯݕग़  99Τϥʔͷେ൒͕Τϥʔίʔυ಺෦αʔόΤϥʔ ʮXBGGBJMEʯͷग़ྗΑΓ"848"'ؔ܎ͷ໰୊ͱਪఆ Developers.IO ϒϩάج൫

  54. &-#ΞΫηεϩάʮXBGGBJMEʯ֘౰Ϩίʔυ  +40/Խͨ͠ΞΫηεϩά֬ೝɺ44FMFDUɺ+2ͳͲซ༻ Developers.IO ϒϩάج൫

  55. "84υΩϡϝϯτ"QQMJDBUJPO-PBE#BMBODFSͷτϥϒϧγϡʔςΟϯά  IUUQTEPDTBXTBNB[PODPNKB@KQFMBTUJDMPBECBMBODJOHMBUFTUBQQMJDBUJPOMPBECBMBODFS USPVCMFTIPPUJOHIUNMIUUQJTTVFT Developers.IO ϒϩάج൫

  56.  AWSWAF

  57. "848"'ηΩϡϦςΟΦʔτϝʔγϣϯ  944ɺ42-Jɺ)551'MPPEɺ*13FQVUBUJPOϧʔϧར༻த IUUQTBXTBNB[PODPNKQTPMVUJPOTBXTXBGTFDVSJUZBVUPNBUJPOT Developers.IO ϒϩάج൫

  58. "848"'Ϛωʔδυϧʔϧݕূ  IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBXTSFJOWFOUXBGNBOBHFESVMFUSZ Developers.IO ϒϩάج൫

  59. ৄࡉͳΞΫηεϩάճऩ ϦΫΤετϔομ౳  IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBXGXBGDPNQSFIFOTJWFMPHHJOH Developers.IO ϒϩάج൫

  60. ৵ೖ๷ࢭγεςϜ *%4 ͸Ҿୀࡁ  IUUQTEFWDMBTTNFUIPEKQDMPVEBXTJETXJUILJOFTJTXBG Developers.IO ϒϩάج൫

  61. 8"'ͷϒϥοΫϦετ*1ࣗಈߋ৽͢Δ࢓૊Έ  ηΩϡϦςΟΦʔτϝʔγϣϯ -BNCEB ɺ8"'ͷϨʔτϧʔϧͰ୅ସ Developers.IO ϒϩάج൫

  62. 8"'ಈ࡞ঢ়گΛ֬ೝ  "848"'ϒϩοΫ࣮੷ΑΓɺແޮԽʹΑΔϦεΫΛ൑அ Developers.IO ϒϩάج൫

  63.  ાஔ

  64. 8"'ແޮԽ  "848"'"$-ͷର৅͔Β"-#Λআ֎ Developers.IO ϒϩάج൫

  65. $MPVE8BUDIϝτϦοΫ֬ೝ &-#99  8"'อޢఀࢭޙɺ99Τϥʔऩଋ Developers.IO ϒϩάج൫

  66. ";Bͷ"-#ɺ8"'ͰԿΒ͔ͷ໰୊ൃੜͱਪఆ  Developers.IO ϒϩάج൫ "84ޙ೔ൃද  ʹΑΔͱɺ 8"'΍εςοΩʔηογϣϯ༗ޮͱͨ͠Ұ෦ͷ"-#Ͱൃੜͨ͠໰୊ʹ֘౰

  67. $MPVE8BUDIϝτϦοΫ֬ೝ %BUBEPHμογϡϘʔυ  Developers.IO ϒϩάج൫ &-#ͷ99Τϥʔͷऩଋͱɺ%#ɺ&$ͷਖ਼ৗಈ࡞Λ֬ೝ

  68.  ใࠂ

  69.  ৘ใల։(ࣾ಺)

  70. ࣾ಺ใࠂ 4MBDL  Developers.IO ϒϩάج൫

  71. ࣾ಺ใࠂ 4MBDL  "848"'͕༗ޮͳผ؀ڥͰ΋ྨࣅͷࣄ৅Λ֬ೝ Developers.IO ϒϩάج൫

  72. ࣾ಺ใࠂ 4MBDL  Developers.IO ϒϩάج൫

  73.  ৘ใల։(AWSαϙʔτ)

  74. "84αϙʔτέʔεىථ  Developers.IO ϒϩάج൫

  75. "84αϙʔτέʔεىථ಺༰ͷൈਮ  Developers.IO ϒϩάج൫ ɾELBɺWAF࿈ܞͷ໰୊Β͍͠ه࿥͕ɺELBͷΞΫηεϩάɺactions_executed ʹࣔ͞Ε͍ͯΔࣄɻ ɾWAFͷACLઃఆͰɺอޢର৅ͷELBΛআ֎ͨ͠ޙ͸ɺ5XXΤϥʔͷൃੜ͸ऩଋͨ͠ࣄΑΓɺ ԿΒ͔ͷ໰୊͕ELB(ALB)ɺWAFؒͰൃੜ͍ͯͨ͠΋ͷͱਪଌ͞Ε·͢ɻ https://status.aws.amazon.com/#AP_block ɾAmazon

    Elastic Load Balancing (Tokyo) ɾAWS WAF Service is operating normally ELBɺWAFͷΤϥʔ৘ใ͸ެ։͞Ε͍ͯͳ͍༷ࢠͰ͕ͨ͠ɺ AWSαϙʔτଆͰ೺Ѳ͞Ε͍ͯΔ੍ݶࣄ߲ͳͲ͋Ε͹ڞ༗௖͚Ε͹޾͍Ͱ͢ɻ
  76. "84αϙʔτʹఏڙͨ͠"-#ΞΫηεϩά৘ใൈਮ  Developers.IO ϒϩάج൫ zcat log.gz | jq . |

    grep '"elb_status_code":' | sort | uniq -c | sort -nr 11805 "elb_status_code": "200", 1615 "elb_status_code": "500", 1523 "elb_status_code": "304", 261 "elb_status_code": "403", 166 "elb_status_code": “301", 127 "elb_status_code": "404", 119 "elb_status_code": "400", 48 "elb_status_code": "460", 21 "elb_status_code": "302", 5 "elb_status_code": "401", 2 "elb_status_code": "504",
  77. "84αϙʔτʹఏڙͨ͠"-#ΞΫηεϩάൈਮ  Developers.IO ϒϩάج൫ "timestamp": "2019-08-23T05:53:50.767192Z", "elb": "app/awseb-AWSEB-xxxxxxxxxxx/xxxxxxxxxxx", "request_processing_time": -1,

    "target_processing_time": -1, "response_processing_time": -1, "elb_status_code": "500", "target_status_code": "-", "request": "GET https://dev.classmethod.jp:443/cloud/aws/insufficientinstancecapacity/ HTTP/ "target_group_arn": "arn:aws:elasticloadbalancing:ap-northeast-1:xxxxxxxxxxx:targetgroup/a "trace_id": "Self=1-5d5f7f6c-f8b1df60e11e997011940890;Root=1-5d5f7f6c-fd7ec9703dd7b "request_creation_time": "2019-08-23T05:53:48.766000Z", "actions_executed": "waf-failed",
  78. ো֐ൃੜͷཌʑ೔  AWSެࣜ৘ใ

  79. ౦ژϦʔδϣϯͰൃੜͨ͠"NB[PO&$ͱ"NB[PO&#4ͷࣄ৅֓ཁ  IUUQTBXTBNB[PODPNKQNFTTBHF

  80. ౦ژϦʔδϣϯͰൃੜͨ͠"NB[PO&$ͱ"NB[PO&#4ͷࣄ৅֓ཁ  IUUQTBXTBNB[PODPNKQNFTTBHF

  81. ࣌ࣾ಺4MBDL  Developers.IO ϒϩάج൫

  82. ਂ໷  Developers.IO ϒϩάج൫

  83. ো֐ൃੜ͔Β೔ޙ  AWSެࣜ৘ใߋ৽

  84. "84ެࣜϖʔδࣄ৅֓ཁ௥ه  IUUQTBXTBNB[PODPNKQNFTTBHF

  85. ΞδΣϯμ  w %FWFMPQFST*0ϒϩάج൫ ɹߏ੒ɺಛ௃ɺӡ༻ w ౦ژϦʔδϣϯো֐தʹԿ͕ى͖͔ͨ ɹൃੜɺௐࠪɺાஔɺใࠂ w ো֐ޙʹ࣮ࢪͨ͠࠶ൃରࡦ

    ɹվળ w ·ͱΊ
  86.  ରࡦ঺հ

  87. ෳ਺ͷ"WBJMBCJMJUZ;POFʹϓϩϏδϣχϯάͨ͠&-# "-#  "VUP4DBMJOH(SPVQ͔Βಛఆ"WBJMBCJMJUZ;POF্ͷϦιʔεΛύʔδ͢Δ  ";ো֐ͷରࡦखஈͷ֬อͱɺ";ো֐ͷӨڹ؇࿨ͷͨΊʹ࣮ࢪ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTQVSHFSFTPVSDFTTQFDJpDB[

  88. "-#ͷΞΫηεϩάʹΞϕΠΞϏϦςΟκʔϯ৘ใΛ௥Ճͯ͠Έͨ  "-#ͷΞΫηεϩάϨίʔυʹ͸&-#ϊʔυ৘ใ͕ແ͍ͨΊɺ ϩάϑΝΠϧͷΩʔʹؚ·ΕΔ*1৘ใΛݩʹ-BNCEBͰϊʔυ৘ใΛิ͍ɺ"UIFOBूܭΛ࣮ݱ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBEEB[BMCBDDFTTMPH

  89. $MPVE8BUDI-PHT*OTJHIUTͰ"-#ͷ&-#99ൃੜ࣌ͷΞΫηεϩάΛ෼ੳͯ͠Έͨ  &-#99ΤϥʔϩάͷΈ$MPVE8BUDI-PHTʹసૹ͢Δ-BNCEBΛར༻͠ɺ "UIFOBΑΓ؆қͳΤϥʔϩά֬ೝखஈΛ֬อ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCYYDXMPHTJOTJHIUT

  90. 'JSFIPTFͰ1BSRVFUܗࣜʹม׵ͨ͠"-#ͷΞΫηεϩάΛ"UIFOBͰղੳͯ͠Έͨ  'JSFIPTFͷྻࢤ޲ม׵Λར༻ɺ"UIFOBͷॲཧޮ཰Λվળ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHpSFIPTFQBSRVFU

  91. 'JSFIPTFͰ1BSRVFUܗࣜʹม׵ͨ͠"-#ͷΞΫηεϩάΛ"UIFOBͰղੳͯ͠Έͨ  %FW*0ͷ&-#ϩάॲཧͷઆ໌ࢿྉͰ͢ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHTRTEFpSFIPTF

  92. (MPCBM"DDFMFSBUPSΛར༻ͯ͠#MVF(SFFOͳ&-#ަ׵Λ࣮ࢪͯ͠Έͨ  %FW*0ͷ&-#ɺ";ɺ*1WରԠͷ؀ڥ΁ަ׵ͨ͠खஈͰ͢ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTCMVFHSFFOFMCCZBHB

  93. %FWFMPQFST*0͕*1WαϙʔτʹͳΓ·ͨ͠  *1Wো֐࣌ͷᷖճܦ࿏ͱͳΔࣄΛظ଴ͯ͠*1W༗ޮԽ͠·ͨ͠ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTEFWJPTVQQPSUJQW

  94.  ·ͱΊ

  95.  DevelopersIOαΠτͷऔΓ૊Έʹ͍ͭͯ ঺հͤͯ͞௖͖·ͨ͠

  96.  ো֐͸վળʹܨ͕Δ ઈ޷ͷػձͱߟ͍͑ͯ·͢

  97.  γεςϜͷՌͨ͢΂͖໾ׂɺ ॏཁ౓ɺ՝୊͕໌ྎͳঢ়ଶͰɺ ࢓૊ΈΛߟ͑Δࣄ͕ग़དྷ·͢ɻ

  98.  ໰୊Λఆྔతʹࣔ͢ϩά΍ϝτϦοΫɺ ਖ਼͍͠൑அͷཁͱͳΓ·͢ɻ

  99.  ໰୊Λڞ༗ɺվળΛଞʹٻΊΔ৔߹ʹ΋ɺ ࠜڌͱͳΔϩά΍ϝτϦοΫ͕ॏཁͰ͢

  100.  ໾ཱͭ࢓૊Έ͕ɺ AWSͰ͸਺ଟ͘ར༻ग़དྷ·͢ɻ

  101. None