Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
8/23 Developers.IOブログに 何が起きたか
Search
suzryo
November 01, 2019
Technology
0
1.5k
8/23 Developers.IOブログに 何が起きたか
AWS東京リージョン障害のあった2019年8月23日、Developers.IOブログに
何が起きたか紹介します。
suzryo
November 01, 2019
Tweet
Share
More Decks by suzryo
See All by suzryo
20230630-JAWSUG札幌LT-AWSオンライン試験のコツ
suzryo
1
130
ipv6-aws-20210714-infrastudy-2nd-03
suzryo
0
21
Graviton2を使う理由について語ってみる
suzryo
0
110
ラスベガスへの行き方を調べてみた
suzryo
0
150
AWS Globel Accelerator を導入してみた話 @ JAWS-UG東京 #32 - マイベストヒット2019
suzryo
0
720
Developers.IOを支えるインフラの全て
suzryo
0
89
クラメソのWebサイトを支える技術
suzryo
1
1.9k
Stream Batch Pattern
suzryo
1
2.9k
Ryo Suzuki
suzryo
0
1.5k
Other Decks in Technology
See All in Technology
家族アルバム みてねにおけるGrafana活用術 / Grafana Meetup Japan Vol.1 LT
isaoshimizu
1
720
Azureの基本的な権限管理の勉強会
yhana
0
410
自己改善からチームを動かす! 「セルフエンジニアリングマネージャー」のすゝめ
shoota
6
630
ココがすごいぜ!Playwright Component Test
rakus_fe
0
130
Azure犬駆動開発の記録/GlobalAzureFukuoka2024_20240420
nina01
1
210
生産性向上チームの紹介
cybozuinsideout
PRO
1
870
FrontDoorとWebAppsを組み合わせた際のリダイレクト処理の注意点
kenichirokimura
1
520
require(ESM)とECMAScript仕様
uhyo
3
630
ここが嬉しいABAC ここが辛いよABAC #再解説+補足編
masahirokawahara
1
270
Cloud Native Java with Spring Boot (CNCF Aarhus, April 2024)
thomasvitale
1
170
EMとして2023年度に頑張ったこと / What we did well in FY2023 as a EM
pauli
1
170
地理空間データ可視化・解析・活用ソリューション Pacific Spatial Solutions (PSS)
pacificspatialsolutions
0
270
Featured
See All Featured
Infographics Made Easy
chrislema
238
18k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
125
32k
Become a Pro
speakerdeck
PRO
11
4.5k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
7
1k
Being A Developer After 40
akosma
57
580k
A Modern Web Designer's Workflow
chriscoyier
689
190k
Scaling GitHub
holman
457
140k
Designing Experiences People Love
moore
136
23k
In The Pink: A Labor of Love
frogandcode
138
21k
Mobile First: as difficult as doing things right
swwweet
216
8.6k
The Pragmatic Product Professional
lauravandoore
25
5.8k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
227
16k
Transcript
࣮Ϋϥϝι࣌ %FWFMPQFST*0ϒϩάʹ Կ͕ى͖͔ͨ "84ࣄۀຊ෦ɹίϯαϧςΟϯά෦ ླ྄ #cmdevio5
ࣗݾհ
ࣗݾհ • ࢯ໊ླɹ྄ɹ ͖ͣ͢Γΐ͏ • 5XJUUFSTV[SZP • ॴଐΫϥεϝιου"84ࣄۀຊ෦
Solutions Architect - Professional DevOps Engineer - Professional Big Data - Specialty Security - Specialty Solutions Architect - Associate Developer - Associate SysOps Administrator - Associate
͖ͳ"84αʔϏε "NB[PO3PVUF "NB[PO,JOFTJT
"84ϧτϥΫΠζͰ༏উ͖ͯͨ͠ "844VNNJUSF.JY IUUQTEFWDMBTTNFUIPEKQDMPVEBXTTVNNJUUPLZPVMUSBRVJ[
ΞδΣϯμ w %FWFMPQFST*0ϒϩάج൫ ɹߏɺಛɺӡ༻ w ౦ژϦʔδϣϯোதʹԿ͕ى͖͔ͨ ɹൃੜɺௐࠪɺાஔɺใࠂ w োޙʹ࣮ࢪͨ͠࠶ൃରࡦ
ɹվળ w ·ͱΊ
Developers.IO ϒϩάج൫ ߏ
ϒϩάج൫ͱͯ͠ ϨΨγʔͳWordPressڥͰ͢
%FWFMPQFST*0ϒϩάͷߏਤ
࣌ͷهࣄ IUUQTEFWDMBTTNFUIPEKQTFSWFSTJEFOHJOYXPSEQSFTTFYQJSF
#FTU1SBDUJDFTGPS8PSE1SFTTPO"84 IUUQTEBXTTUBUJDDPNXIJUFQBQFSTXPSEQSFTTCFTUQSBDUJDFTPOBXTQEG
8IPMFXFCTJUFEFMJWFSZWJB"NB[PO$MPVE'SPOU $MPVE'SPOUܦ༝Ͱ8PSE1SFTTڥΛެ։
3FGFSFODFBSDIJUFDUVSFGPSIPTUJOH8PSE1SFTTPO"84 "VSPSBɺ&MBTUJ$BDIFɺ&'4Λ׆༻ͨ͠ߴՄ༻ੑߏͳͲհ͞Ε͍ͯ·͢
Developers.IO ϒϩάج൫ ಛ
%FWFMPQFST*0ϒϩάͷಛ େྔʹྔ࢈͞ΕΔهࣄ
%FWFMPQFST*0࣮ ݄ؒهࣄܝࡌ ฏۉʙهࣄ
Ξτόϯυ௨৴τϥϑΟοΫ ࣌ؒฏۉ IUUQE /HJOY $MPVE'SPOU
ϦΫΤετ݅ ࣌ؒฏۉ IUUQE /HJOY $MPVE'SPOU
Developers.IO ϒϩάج൫ ӡ༻
Քಇඪ 4-0 Քಇ99.95% (ڐ༰μϯλΠϜ݄ؒ20)
ࢹ
8PSE1SFTT 8PSE1SFTTͱ%#ʹґଘ͢Δػೳͷҟৗ Developers.IO ϒϩάج൫
&MBTUJD#FBOTUBML֦ுϔϧενΣοΫ ϔϧεʮ0,ʯҎ֎͕ܧଓɺࣗಈ෮چͰճ෮͠ͳ͔ͬͨ߹ʹ௨ Developers.IO ϒϩάج൫
&MBTUJD#FBOTUBML֦ுϔϧενΣοΫϩά $MPVE8BUDI-PHT*OTJHIU͕ศརͰ͢ Developers.IO ϒϩάج൫
Ωϟογϡ /HJOYɺ$MPVE'SPOU /HJOYͱɺ$MPVE'SPOU৴ͷਖ਼ৗੑ Developers.IO ϒϩάج൫
3PVUFϔϧενΣοΫ 63-ࢹɺ֎ܗࢹΛ࣮ࢪ Developers.IO ϒϩάج൫
%# "VSPSB ϘτϧωοΫͱͳΔ%#ͷߴෛՙΛݕ Developers.IO ϒϩάج൫
ύϑΥʔϚϯεΠϯαΠτ %#ϩʔυɺ࿈ଓ্ͨ͠ঢΛաෛՙͱͯ͠௨ Developers.IO ϒϩάج൫
༗ਓࢹ Developers.IO ϒϩάج൫ ৗ࣌୭͔͕ࣥචதɻෆ۩߹࿈བྷࣾ4MBDLʹల։
3PVUFϔϧενΣοΫͱͷൺֱ 3PVUFϔϧενΣοΧʔա͕ΤϥʔͱͳΔલʹݕग़ Developers.IO ϒϩάج൫
όοΫΞοϓ
όοΫΞοϓର "VSPSB هࣄ%# ͱ4 هࣄͷը૾ ͷফࣦઈରʹճආ Developers.IO ϒϩάج൫
4 όʔδϣχϯά ΫϩεϦʔδϣϯϨϓϦέʔγϣϯ ॏԽ Developers.IO ϒϩάج൫
"VSPSB "VSPSBετϨʔδ ॏԽ εφοϓγϣοτ %#μϯϓͷ4ΤΫεϙʔτ ॏԽ Developers.IO ϒϩάج൫
"VSPSB άϩʔόϧσʔλϕʔε ˞ۙධՁ༧ఆ "VSPSBάϩʔόϧσʔλϕʔε ౦ژ݄ϦϦʔε ʹΑΔɺ ౦ژɺΦϨΰϯؒͷ%#ಉظ Developers.IO ϒϩάج൫
ϦΧόϦʔ
࠶ىಈӡ༻ աෛՙ%#ͷ࠶ىಈɺεϖοΫௐ Developers.IO ϒϩάج൫
%#ϦετΞ %#ϦετΞ͢Δ߹ɺΞϓϦͷڥมΛมߋ Developers.IO ϒϩάج൫
%3 େنϦʔδϣϯোఆ ผϦʔδϣϯʹ࣮ߦڥΛల։ɺެ։༻ͷΤϯυϙΠϯτΛௐ͠࠶։ Developers.IO ϒϩάج൫
མͪͳ͍αΠτͰ͋Γ·ͤΜɻ ෮׆Ͱ͖ΔࣄΛࢦͨ͠αΠτͰ͢
ΞδΣϯμ w %FWFMPQFST*0ϒϩάج൫ ɹߏɺಛɺӡ༻ w ౦ژϦʔδϣϯোதʹԿ͕ى͖͔ͨ ɹൃੜɺௐࠪɺાஔɺใࠂ w োޙʹ࣮ࢪͨ͠࠶ൃରࡦ
ɹվળ w ·ͱΊ
8/23 Developers.IO ϒϩά ͳʹ͕ى͖͔ͨ
౦ژϦʔδϣϯোதͷϒϩάՔಇΛհ͠·͢ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBQOFB[EPXOEFWJP
"844FSWJDF)FBMUI%BTICPBSE 4)% IUUQTTUBUVTBXTBNB[PODPN"1@CMPDL Developers.IO ϒϩάج൫
4)%ܝࡌࣄ߲ൈਮ Developers.IO ϒϩάج൫ ࣌ࠁ +45 ࣄ ۭௐނোɺΦʔόʔώʔτʹΑΓ&$͕ఀࢭ
&$ͷ"1*ΤϥʔϨʔτ্ঢ ۭௐ෮چɺΦʔόώʔτͰఀࢭͨ͠&$ճ෮։࢝ େ෦ͷ&$͕ճ෮
ɹোൃੜ ELB5XXΤϥʔ্ঢ
ো௨ &MBTUJD#FBOTUBMLεςʔλεҟৗΛݕ Developers.IO ϒϩάج൫
ௐࠪ։࢝
&MBTUJD#FBOTUBMLμογϡϘʔυ ࠒΑΓ)551YY͕ൃੜ͍ͯͨ͠ࣄΛ֬ೝ Developers.IO ϒϩάج൫
$MPVE8BUDIϝτϦοΫ֬ೝ %BUBEPHμογϡϘʔυ Developers.IO ϒϩάج൫ &-#ͷ99ΤϥʔҎ֎ɺ%#ɺ&$ͷਖ਼ৗಈ࡞Λ֬ೝ
$MPVE8BUDIϝτϦοΫ &-#99Τϥʔোใࠂͷ͋ͬͨ"; BQOPSUIFBTUB ͷΈͰൃੜ͍ͯ͠ΔࣄΛ֬ೝ Developers.IO ϒϩάج൫
99Τϥʔৄࡉɺ&-#ͷΞΫηεϩάΑΓௐࠪ։࢝ "-#ͷΞΫηεϩάఔͷλΠϜϥάͰ"UIFOBղੳՄೳʹ͢ΔΈΛհ͠·͢ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHTRTEFpSFIPTF Developers.IO ϒϩάج൫
&-#ΞΫηεϩάʮXBGGBJMEʯݕग़ 99Τϥʔͷେ͕Τϥʔίʔυ෦αʔόΤϥʔ ʮXBGGBJMEʯͷग़ྗΑΓ"848"'ؔͷͱਪఆ Developers.IO ϒϩάج൫
&-#ΞΫηεϩάʮXBGGBJMEʯ֘Ϩίʔυ +40/Խͨ͠ΞΫηεϩά֬ೝɺ44FMFDUɺ+2ͳͲซ༻ Developers.IO ϒϩάج൫
"84υΩϡϝϯτ"QQMJDBUJPO-PBE#BMBODFSͷτϥϒϧγϡʔςΟϯά IUUQTEPDTBXTBNB[PODPNKB@KQFMBTUJDMPBECBMBODJOHMBUFTUBQQMJDBUJPOMPBECBMBODFS USPVCMFTIPPUJOHIUNMIUUQJTTVFT Developers.IO ϒϩάج൫
AWSWAF
"848"'ηΩϡϦςΟΦʔτϝʔγϣϯ 944ɺ42-Jɺ)551'MPPEɺ*13FQVUBUJPOϧʔϧར༻த IUUQTBXTBNB[PODPNKQTPMVUJPOTBXTXBGTFDVSJUZBVUPNBUJPOT Developers.IO ϒϩάج൫
"848"'Ϛωʔδυϧʔϧݕূ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBXTSFJOWFOUXBGNBOBHFESVMFUSZ Developers.IO ϒϩάج൫
ৄࡉͳΞΫηεϩάճऩ ϦΫΤετϔομ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBXGXBGDPNQSFIFOTJWFMPHHJOH Developers.IO ϒϩάج൫
৵ೖࢭγεςϜ *%4 Ҿୀࡁ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTJETXJUILJOFTJTXBG Developers.IO ϒϩάج൫
8"'ͷϒϥοΫϦετ*1ࣗಈߋ৽͢ΔΈ ηΩϡϦςΟΦʔτϝʔγϣϯ -BNCEB ɺ8"'ͷϨʔτϧʔϧͰସ Developers.IO ϒϩάج൫
8"'ಈ࡞ঢ়گΛ֬ೝ "848"'ϒϩοΫ࣮ΑΓɺແޮԽʹΑΔϦεΫΛஅ Developers.IO ϒϩάج൫
ાஔ
8"'ແޮԽ "848"'"$-ͷର͔Β"-#Λআ֎ Developers.IO ϒϩάج൫
$MPVE8BUDIϝτϦοΫ֬ೝ &-#99 8"'อޢఀࢭޙɺ99Τϥʔऩଋ Developers.IO ϒϩάج൫
";Bͷ"-#ɺ8"'ͰԿΒ͔ͷൃੜͱਪఆ Developers.IO ϒϩάج൫ "84ޙൃද ʹΑΔͱɺ 8"'εςοΩʔηογϣϯ༗ޮͱͨ͠Ұ෦ͷ"-#Ͱൃੜͨ͠ʹ֘
$MPVE8BUDIϝτϦοΫ֬ೝ %BUBEPHμογϡϘʔυ Developers.IO ϒϩάج൫ &-#ͷ99Τϥʔͷऩଋͱɺ%#ɺ&$ͷਖ਼ৗಈ࡞Λ֬ೝ
ใࠂ
ใల։(ࣾ)
ࣾใࠂ 4MBDL Developers.IO ϒϩάج൫
ࣾใࠂ 4MBDL "848"'͕༗ޮͳผڥͰྨࣅͷࣄΛ֬ೝ Developers.IO ϒϩάج൫
ࣾใࠂ 4MBDL Developers.IO ϒϩάج൫
ใల։(AWSαϙʔτ)
"84αϙʔτέʔεىථ Developers.IO ϒϩάج൫
"84αϙʔτέʔεىථ༰ͷൈਮ Developers.IO ϒϩάج൫ ɾELBɺWAF࿈ܞͷΒ͍͠ه͕ɺELBͷΞΫηεϩάɺactions_executed ʹࣔ͞Ε͍ͯΔࣄɻ ɾWAFͷACLઃఆͰɺอޢରͷELBΛআ֎ͨ͠ޙɺ5XXΤϥʔͷൃੜऩଋͨ͠ࣄΑΓɺ ԿΒ͔ͷ͕ELB(ALB)ɺWAFؒͰൃੜ͍ͯͨ͠ͷͱਪଌ͞Ε·͢ɻ https://status.aws.amazon.com/#AP_block ɾAmazon
Elastic Load Balancing (Tokyo) ɾAWS WAF Service is operating normally ELBɺWAFͷΤϥʔใެ։͞Ε͍ͯͳ͍༷ࢠͰ͕ͨ͠ɺ AWSαϙʔτଆͰѲ͞Ε͍ͯΔ੍ݶࣄ߲ͳͲ͋Εڞ༗͚Ε͍Ͱ͢ɻ
"84αϙʔτʹఏڙͨ͠"-#ΞΫηεϩάใൈਮ Developers.IO ϒϩάج൫ zcat log.gz | jq . |
grep '"elb_status_code":' | sort | uniq -c | sort -nr 11805 "elb_status_code": "200", 1615 "elb_status_code": "500", 1523 "elb_status_code": "304", 261 "elb_status_code": "403", 166 "elb_status_code": “301", 127 "elb_status_code": "404", 119 "elb_status_code": "400", 48 "elb_status_code": "460", 21 "elb_status_code": "302", 5 "elb_status_code": "401", 2 "elb_status_code": "504",
"84αϙʔτʹఏڙͨ͠"-#ΞΫηεϩάൈਮ Developers.IO ϒϩάج൫ "timestamp": "2019-08-23T05:53:50.767192Z", "elb": "app/awseb-AWSEB-xxxxxxxxxxx/xxxxxxxxxxx", "request_processing_time": -1,
"target_processing_time": -1, "response_processing_time": -1, "elb_status_code": "500", "target_status_code": "-", "request": "GET https://dev.classmethod.jp:443/cloud/aws/insufficientinstancecapacity/ HTTP/ "target_group_arn": "arn:aws:elasticloadbalancing:ap-northeast-1:xxxxxxxxxxx:targetgroup/a "trace_id": "Self=1-5d5f7f6c-f8b1df60e11e997011940890;Root=1-5d5f7f6c-fd7ec9703dd7b "request_creation_time": "2019-08-23T05:53:48.766000Z", "actions_executed": "waf-failed",
োൃੜͷཌʑ AWSެࣜใ
౦ژϦʔδϣϯͰൃੜͨ͠"NB[PO&$ͱ"NB[POͷࣄ֓ཁ IUUQTBXTBNB[PODPNKQNFTTBHF
౦ژϦʔδϣϯͰൃੜͨ͠"NB[PO&$ͱ"NB[POͷࣄ֓ཁ IUUQTBXTBNB[PODPNKQNFTTBHF
࣌ࣾ4MBDL Developers.IO ϒϩάج൫
ਂ Developers.IO ϒϩάج൫
োൃੜ͔Βޙ AWSެࣜใߋ৽
"84ެࣜϖʔδࣄ֓ཁه IUUQTBXTBNB[PODPNKQNFTTBHF
ΞδΣϯμ w %FWFMPQFST*0ϒϩάج൫ ɹߏɺಛɺӡ༻ w ౦ژϦʔδϣϯোதʹԿ͕ى͖͔ͨ ɹൃੜɺௐࠪɺાஔɺใࠂ w োޙʹ࣮ࢪͨ͠࠶ൃରࡦ
ɹվળ w ·ͱΊ
ରࡦհ
ෳͷ"WBJMBCJMJUZ;POFʹϓϩϏδϣχϯάͨ͠&-# "-# "VUP4DBMJOH(SPVQ͔Βಛఆ"WBJMBCJMJUZ;POF্ͷϦιʔεΛύʔδ͢Δ ";োͷରࡦखஈͷ֬อͱɺ";োͷӨڹ؇ͷͨΊʹ࣮ࢪ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTQVSHFSFTPVSDFTTQFDJpDB[
"-#ͷΞΫηεϩάʹΞϕΠΞϏϦςΟκʔϯใΛՃͯ͠Έͨ "-#ͷΞΫηεϩάϨίʔυʹ&-#ϊʔυใ͕ແ͍ͨΊɺ ϩάϑΝΠϧͷΩʔʹؚ·ΕΔ*1ใΛݩʹ-BNCEBͰϊʔυใΛิ͍ɺ"UIFOBूܭΛ࣮ݱ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBEEB[BMCBDDFTTMPH
$MPVE8BUDI-PHT*OTJHIUTͰ"-#ͷ&-#99ൃੜ࣌ͷΞΫηεϩάΛੳͯ͠Έͨ &-#99ΤϥʔϩάͷΈ$MPVE8BUDI-PHTʹసૹ͢Δ-BNCEBΛར༻͠ɺ "UIFOBΑΓ؆қͳΤϥʔϩά֬ೝखஈΛ֬อ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCYYDXMPHTJOTJHIUT
'JSFIPTFͰ1BSRVFUܗࣜʹมͨ͠"-#ͷΞΫηεϩάΛ"UIFOBͰղੳͯ͠Έͨ 'JSFIPTFͷྻࢤมΛར༻ɺ"UIFOBͷॲཧޮΛվળ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHpSFIPTFQBSRVFU
'JSFIPTFͰ1BSRVFUܗࣜʹมͨ͠"-#ͷΞΫηεϩάΛ"UIFOBͰղੳͯ͠Έͨ %FW*0ͷ&-#ϩάॲཧͷઆ໌ࢿྉͰ͢ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHTRTEFpSFIPTF
(MPCBM"DDFMFSBUPSΛར༻ͯ͠#MVF(SFFOͳ&-#ަΛ࣮ࢪͯ͠Έͨ %FW*0ͷ&-#ɺ";ɺ*1WରԠͷڥަͨ͠खஈͰ͢ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTCMVFHSFFOFMCCZBHB
%FWFMPQFST*0͕*1WαϙʔτʹͳΓ·ͨ͠ *1Wো࣌ͷᷖճܦ࿏ͱͳΔࣄΛظͯ͠*1W༗ޮԽ͠·ͨ͠ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTEFWJPTVQQPSUJQW
·ͱΊ
DevelopersIOαΠτͷऔΓΈʹ͍ͭͯ հ͖ͤͯ͞·ͨ͠
োվળʹܨ͕Δ ઈͷػձͱߟ͍͑ͯ·͢
γεςϜͷՌׂ͖ͨ͢ɺ ॏཁɺ՝͕໌ྎͳঢ়ଶͰɺ ΈΛߟ͑Δࣄ͕ग़དྷ·͢ɻ
Λఆྔతʹࣔ͢ϩάϝτϦοΫɺ ਖ਼͍͠அͷཁͱͳΓ·͢ɻ
Λڞ༗ɺվળΛଞʹٻΊΔ߹ʹɺ ࠜڌͱͳΔϩάϝτϦοΫ͕ॏཁͰ͢
ཱͭΈ͕ɺ AWSͰଟ͘ར༻ग़དྷ·͢ɻ
None