Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
8/23 Developers.IOブログに 何が起きたか
Search
suzryo
November 01, 2019
Technology
0
1.7k
8/23 Developers.IOブログに 何が起きたか
AWS東京リージョン障害のあった2019年8月23日、Developers.IOブログに
何が起きたか紹介します。
suzryo
November 01, 2019
Tweet
Share
More Decks by suzryo
See All by suzryo
re:Invent2024 事前勉強会 AWS Gameday 参加のポイント
suzryo
0
1k
20230630-JAWSUG札幌LT-AWSオンライン試験のコツ
suzryo
1
250
ipv6-aws-20210714-infrastudy-2nd-03
suzryo
0
43
Graviton2を使う理由について語ってみる
suzryo
0
130
ラスベガスへの行き方を調べてみた
suzryo
0
230
AWS Globel Accelerator を導入してみた話 @ JAWS-UG東京 #32 - マイベストヒット2019
suzryo
0
930
Developers.IOを支えるインフラの全て
suzryo
0
110
クラメソのWebサイトを支える技術
suzryo
1
2.2k
Stream Batch Pattern
suzryo
1
3k
Other Decks in Technology
See All in Technology
MCP とマネージド PaaS で実現する大規模 AI アプリケーションの高速開発
nahokoxxx
1
1.4k
ecspressoの設計思想に至る道 / sekkeinight2025
fujiwara3
10
1.4k
自分がLinc’wellで提供しているプロダクトを理解するためにやったこと
murabayashi
1
160
データエンジニアリング 4年前と変わったこと、 4年前と変わらないこと
tanakarian
2
360
地図と生成AI
nakasho
0
700
2025/07/22_家族アルバム みてねのCRE における生成AI活用事例
masartz
2
110
今日からあなたもGeminiを好きになる
subaruhello
1
560
AI時代にも変わらぬ価値を発揮したい: インフラ・クラウドを切り口にユーザー価値と非機能要件に向き合ってエンジニアとしての地力を培う
netmarkjp
0
220
20150719_Amazon Nova Canvas Virtual try-onアプリ 作成裏話
riz3f7
0
130
Amazon CloudWatchのメトリクスインターバルについて / Metrics interval matters
ymotongpoo
3
210
Turn Your Community into a Fundraising Catalyst for Black Philanthropy Month
auctria
PRO
0
130
複数のGemini CLIが同時開発する狂気 - Jujutsuが実現するAIエージェント協調の新世界
gunta
12
3.2k
Featured
See All Featured
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
Typedesign – Prime Four
hannesfritz
42
2.7k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
760
Side Projects
sachag
455
43k
Gamification - CAS2011
davidbonilla
81
5.4k
A better future with KSS
kneath
238
17k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.7k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
21
1.3k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
The Straight Up "How To Draw Better" Workshop
denniskardys
235
140k
Navigating Team Friction
lara
187
15k
Transcript
࣮Ϋϥϝι࣌ %FWFMPQFST*0ϒϩάʹ Կ͕ى͖͔ͨ "84ࣄۀຊ෦ɹίϯαϧςΟϯά෦ ླ྄ #cmdevio5
ࣗݾհ
ࣗݾհ • ࢯ໊ླɹ྄ɹ ͖ͣ͢Γΐ͏ • 5XJUUFSTV[SZP • ॴଐΫϥεϝιου"84ࣄۀຊ෦
Solutions Architect - Professional DevOps Engineer - Professional Big Data - Specialty Security - Specialty Solutions Architect - Associate Developer - Associate SysOps Administrator - Associate
͖ͳ"84αʔϏε "NB[PO3PVUF "NB[PO,JOFTJT
"84ϧτϥΫΠζͰ༏উ͖ͯͨ͠ "844VNNJUSF.JY IUUQTEFWDMBTTNFUIPEKQDMPVEBXTTVNNJUUPLZPVMUSBRVJ[
ΞδΣϯμ w %FWFMPQFST*0ϒϩάج൫ ɹߏɺಛɺӡ༻ w ౦ژϦʔδϣϯোதʹԿ͕ى͖͔ͨ ɹൃੜɺௐࠪɺાஔɺใࠂ w োޙʹ࣮ࢪͨ͠࠶ൃରࡦ
ɹվળ w ·ͱΊ
Developers.IO ϒϩάج൫ ߏ
ϒϩάج൫ͱͯ͠ ϨΨγʔͳWordPressڥͰ͢
%FWFMPQFST*0ϒϩάͷߏਤ
࣌ͷهࣄ IUUQTEFWDMBTTNFUIPEKQTFSWFSTJEFOHJOYXPSEQSFTTFYQJSF
#FTU1SBDUJDFTGPS8PSE1SFTTPO"84 IUUQTEBXTTUBUJDDPNXIJUFQBQFSTXPSEQSFTTCFTUQSBDUJDFTPOBXTQEG
8IPMFXFCTJUFEFMJWFSZWJB"NB[PO$MPVE'SPOU $MPVE'SPOUܦ༝Ͱ8PSE1SFTTڥΛެ։
3FGFSFODFBSDIJUFDUVSFGPSIPTUJOH8PSE1SFTTPO"84 "VSPSBɺ&MBTUJ$BDIFɺ&'4Λ׆༻ͨ͠ߴՄ༻ੑߏͳͲհ͞Ε͍ͯ·͢
Developers.IO ϒϩάج൫ ಛ
%FWFMPQFST*0ϒϩάͷಛ େྔʹྔ࢈͞ΕΔهࣄ
%FWFMPQFST*0࣮ ݄ؒهࣄܝࡌ ฏۉʙهࣄ
Ξτόϯυ௨৴τϥϑΟοΫ ࣌ؒฏۉ IUUQE /HJOY $MPVE'SPOU
ϦΫΤετ݅ ࣌ؒฏۉ IUUQE /HJOY $MPVE'SPOU
Developers.IO ϒϩάج൫ ӡ༻
Քಇඪ 4-0 Քಇ99.95% (ڐ༰μϯλΠϜ݄ؒ20)
ࢹ
8PSE1SFTT 8PSE1SFTTͱ%#ʹґଘ͢Δػೳͷҟৗ Developers.IO ϒϩάج൫
&MBTUJD#FBOTUBML֦ுϔϧενΣοΫ ϔϧεʮ0,ʯҎ֎͕ܧଓɺࣗಈ෮چͰճ෮͠ͳ͔ͬͨ߹ʹ௨ Developers.IO ϒϩάج൫
&MBTUJD#FBOTUBML֦ுϔϧενΣοΫϩά $MPVE8BUDI-PHT*OTJHIU͕ศརͰ͢ Developers.IO ϒϩάج൫
Ωϟογϡ /HJOYɺ$MPVE'SPOU /HJOYͱɺ$MPVE'SPOU৴ͷਖ਼ৗੑ Developers.IO ϒϩάج൫
3PVUFϔϧενΣοΫ 63-ࢹɺ֎ܗࢹΛ࣮ࢪ Developers.IO ϒϩάج൫
%# "VSPSB ϘτϧωοΫͱͳΔ%#ͷߴෛՙΛݕ Developers.IO ϒϩάج൫
ύϑΥʔϚϯεΠϯαΠτ %#ϩʔυɺ࿈ଓ্ͨ͠ঢΛաෛՙͱͯ͠௨ Developers.IO ϒϩάج൫
༗ਓࢹ Developers.IO ϒϩάج൫ ৗ࣌୭͔͕ࣥචதɻෆ۩߹࿈བྷࣾ4MBDLʹల։
3PVUFϔϧενΣοΫͱͷൺֱ 3PVUFϔϧενΣοΧʔա͕ΤϥʔͱͳΔલʹݕग़ Developers.IO ϒϩάج൫
όοΫΞοϓ
όοΫΞοϓର "VSPSB هࣄ%# ͱ4 هࣄͷը૾ ͷফࣦઈରʹճආ Developers.IO ϒϩάج൫
4 όʔδϣχϯά ΫϩεϦʔδϣϯϨϓϦέʔγϣϯ ॏԽ Developers.IO ϒϩάج൫
"VSPSB "VSPSBετϨʔδ ॏԽ εφοϓγϣοτ %#μϯϓͷ4ΤΫεϙʔτ ॏԽ Developers.IO ϒϩάج൫
"VSPSB άϩʔόϧσʔλϕʔε ˞ۙධՁ༧ఆ "VSPSBάϩʔόϧσʔλϕʔε ౦ژ݄ϦϦʔε ʹΑΔɺ ౦ژɺΦϨΰϯؒͷ%#ಉظ Developers.IO ϒϩάج൫
ϦΧόϦʔ
࠶ىಈӡ༻ աෛՙ%#ͷ࠶ىಈɺεϖοΫௐ Developers.IO ϒϩάج൫
%#ϦετΞ %#ϦετΞ͢Δ߹ɺΞϓϦͷڥมΛมߋ Developers.IO ϒϩάج൫
%3 େنϦʔδϣϯোఆ ผϦʔδϣϯʹ࣮ߦڥΛల։ɺެ։༻ͷΤϯυϙΠϯτΛௐ͠࠶։ Developers.IO ϒϩάج൫
མͪͳ͍αΠτͰ͋Γ·ͤΜɻ ෮׆Ͱ͖ΔࣄΛࢦͨ͠αΠτͰ͢
ΞδΣϯμ w %FWFMPQFST*0ϒϩάج൫ ɹߏɺಛɺӡ༻ w ౦ژϦʔδϣϯোதʹԿ͕ى͖͔ͨ ɹൃੜɺௐࠪɺાஔɺใࠂ w োޙʹ࣮ࢪͨ͠࠶ൃରࡦ
ɹվળ w ·ͱΊ
8/23 Developers.IO ϒϩά ͳʹ͕ى͖͔ͨ
౦ژϦʔδϣϯোதͷϒϩάՔಇΛհ͠·͢ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBQOFB[EPXOEFWJP
"844FSWJDF)FBMUI%BTICPBSE 4)% IUUQTTUBUVTBXTBNB[PODPN"1@CMPDL Developers.IO ϒϩάج൫
4)%ܝࡌࣄ߲ൈਮ Developers.IO ϒϩάج൫ ࣌ࠁ +45 ࣄ ۭௐނোɺΦʔόʔώʔτʹΑΓ&$͕ఀࢭ
&$ͷ"1*ΤϥʔϨʔτ্ঢ ۭௐ෮چɺΦʔόώʔτͰఀࢭͨ͠&$ճ෮։࢝ େ෦ͷ&$͕ճ෮
ɹোൃੜ ELB5XXΤϥʔ্ঢ
ো௨ &MBTUJD#FBOTUBMLεςʔλεҟৗΛݕ Developers.IO ϒϩάج൫
ௐࠪ։࢝
&MBTUJD#FBOTUBMLμογϡϘʔυ ࠒΑΓ)551YY͕ൃੜ͍ͯͨ͠ࣄΛ֬ೝ Developers.IO ϒϩάج൫
$MPVE8BUDIϝτϦοΫ֬ೝ %BUBEPHμογϡϘʔυ Developers.IO ϒϩάج൫ &-#ͷ99ΤϥʔҎ֎ɺ%#ɺ&$ͷਖ਼ৗಈ࡞Λ֬ೝ
$MPVE8BUDIϝτϦοΫ &-#99Τϥʔোใࠂͷ͋ͬͨ"; BQOPSUIFBTUB ͷΈͰൃੜ͍ͯ͠ΔࣄΛ֬ೝ Developers.IO ϒϩάج൫
99Τϥʔৄࡉɺ&-#ͷΞΫηεϩάΑΓௐࠪ։࢝ "-#ͷΞΫηεϩάఔͷλΠϜϥάͰ"UIFOBղੳՄೳʹ͢ΔΈΛհ͠·͢ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHTRTEFpSFIPTF Developers.IO ϒϩάج൫
&-#ΞΫηεϩάʮXBGGBJMEʯݕग़ 99Τϥʔͷେ͕Τϥʔίʔυ෦αʔόΤϥʔ ʮXBGGBJMEʯͷग़ྗΑΓ"848"'ؔͷͱਪఆ Developers.IO ϒϩάج൫
&-#ΞΫηεϩάʮXBGGBJMEʯ֘Ϩίʔυ +40/Խͨ͠ΞΫηεϩά֬ೝɺ44FMFDUɺ+2ͳͲซ༻ Developers.IO ϒϩάج൫
"84υΩϡϝϯτ"QQMJDBUJPO-PBE#BMBODFSͷτϥϒϧγϡʔςΟϯά IUUQTEPDTBXTBNB[PODPNKB@KQFMBTUJDMPBECBMBODJOHMBUFTUBQQMJDBUJPOMPBECBMBODFS USPVCMFTIPPUJOHIUNMIUUQJTTVFT Developers.IO ϒϩάج൫
AWSWAF
"848"'ηΩϡϦςΟΦʔτϝʔγϣϯ 944ɺ42-Jɺ)551'MPPEɺ*13FQVUBUJPOϧʔϧར༻த IUUQTBXTBNB[PODPNKQTPMVUJPOTBXTXBGTFDVSJUZBVUPNBUJPOT Developers.IO ϒϩάج൫
"848"'Ϛωʔδυϧʔϧݕূ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBXTSFJOWFOUXBGNBOBHFESVMFUSZ Developers.IO ϒϩάج൫
ৄࡉͳΞΫηεϩάճऩ ϦΫΤετϔομ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBXGXBGDPNQSFIFOTJWFMPHHJOH Developers.IO ϒϩάج൫
৵ೖࢭγεςϜ *%4 Ҿୀࡁ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTJETXJUILJOFTJTXBG Developers.IO ϒϩάج൫
8"'ͷϒϥοΫϦετ*1ࣗಈߋ৽͢ΔΈ ηΩϡϦςΟΦʔτϝʔγϣϯ -BNCEB ɺ8"'ͷϨʔτϧʔϧͰସ Developers.IO ϒϩάج൫
8"'ಈ࡞ঢ়گΛ֬ೝ "848"'ϒϩοΫ࣮ΑΓɺແޮԽʹΑΔϦεΫΛஅ Developers.IO ϒϩάج൫
ાஔ
8"'ແޮԽ "848"'"$-ͷର͔Β"-#Λআ֎ Developers.IO ϒϩάج൫
$MPVE8BUDIϝτϦοΫ֬ೝ &-#99 8"'อޢఀࢭޙɺ99Τϥʔऩଋ Developers.IO ϒϩάج൫
";Bͷ"-#ɺ8"'ͰԿΒ͔ͷൃੜͱਪఆ Developers.IO ϒϩάج൫ "84ޙൃද ʹΑΔͱɺ 8"'εςοΩʔηογϣϯ༗ޮͱͨ͠Ұ෦ͷ"-#Ͱൃੜͨ͠ʹ֘
$MPVE8BUDIϝτϦοΫ֬ೝ %BUBEPHμογϡϘʔυ Developers.IO ϒϩάج൫ &-#ͷ99Τϥʔͷऩଋͱɺ%#ɺ&$ͷਖ਼ৗಈ࡞Λ֬ೝ
ใࠂ
ใల։(ࣾ)
ࣾใࠂ 4MBDL Developers.IO ϒϩάج൫
ࣾใࠂ 4MBDL "848"'͕༗ޮͳผڥͰྨࣅͷࣄΛ֬ೝ Developers.IO ϒϩάج൫
ࣾใࠂ 4MBDL Developers.IO ϒϩάج൫
ใల։(AWSαϙʔτ)
"84αϙʔτέʔεىථ Developers.IO ϒϩάج൫
"84αϙʔτέʔεىථ༰ͷൈਮ Developers.IO ϒϩάج൫ ɾELBɺWAF࿈ܞͷΒ͍͠ه͕ɺELBͷΞΫηεϩάɺactions_executed ʹࣔ͞Ε͍ͯΔࣄɻ ɾWAFͷACLઃఆͰɺอޢରͷELBΛআ֎ͨ͠ޙɺ5XXΤϥʔͷൃੜऩଋͨ͠ࣄΑΓɺ ԿΒ͔ͷ͕ELB(ALB)ɺWAFؒͰൃੜ͍ͯͨ͠ͷͱਪଌ͞Ε·͢ɻ https://status.aws.amazon.com/#AP_block ɾAmazon
Elastic Load Balancing (Tokyo) ɾAWS WAF Service is operating normally ELBɺWAFͷΤϥʔใެ։͞Ε͍ͯͳ͍༷ࢠͰ͕ͨ͠ɺ AWSαϙʔτଆͰѲ͞Ε͍ͯΔ੍ݶࣄ߲ͳͲ͋Εڞ༗͚Ε͍Ͱ͢ɻ
"84αϙʔτʹఏڙͨ͠"-#ΞΫηεϩάใൈਮ Developers.IO ϒϩάج൫ zcat log.gz | jq . |
grep '"elb_status_code":' | sort | uniq -c | sort -nr 11805 "elb_status_code": "200", 1615 "elb_status_code": "500", 1523 "elb_status_code": "304", 261 "elb_status_code": "403", 166 "elb_status_code": “301", 127 "elb_status_code": "404", 119 "elb_status_code": "400", 48 "elb_status_code": "460", 21 "elb_status_code": "302", 5 "elb_status_code": "401", 2 "elb_status_code": "504",
"84αϙʔτʹఏڙͨ͠"-#ΞΫηεϩάൈਮ Developers.IO ϒϩάج൫ "timestamp": "2019-08-23T05:53:50.767192Z", "elb": "app/awseb-AWSEB-xxxxxxxxxxx/xxxxxxxxxxx", "request_processing_time": -1,
"target_processing_time": -1, "response_processing_time": -1, "elb_status_code": "500", "target_status_code": "-", "request": "GET https://dev.classmethod.jp:443/cloud/aws/insufficientinstancecapacity/ HTTP/ "target_group_arn": "arn:aws:elasticloadbalancing:ap-northeast-1:xxxxxxxxxxx:targetgroup/a "trace_id": "Self=1-5d5f7f6c-f8b1df60e11e997011940890;Root=1-5d5f7f6c-fd7ec9703dd7b "request_creation_time": "2019-08-23T05:53:48.766000Z", "actions_executed": "waf-failed",
োൃੜͷཌʑ AWSެࣜใ
౦ژϦʔδϣϯͰൃੜͨ͠"NB[PO&$ͱ"NB[POͷࣄ֓ཁ IUUQTBXTBNB[PODPNKQNFTTBHF
౦ژϦʔδϣϯͰൃੜͨ͠"NB[PO&$ͱ"NB[POͷࣄ֓ཁ IUUQTBXTBNB[PODPNKQNFTTBHF
࣌ࣾ4MBDL Developers.IO ϒϩάج൫
ਂ Developers.IO ϒϩάج൫
োൃੜ͔Βޙ AWSެࣜใߋ৽
"84ެࣜϖʔδࣄ֓ཁه IUUQTBXTBNB[PODPNKQNFTTBHF
ΞδΣϯμ w %FWFMPQFST*0ϒϩάج൫ ɹߏɺಛɺӡ༻ w ౦ژϦʔδϣϯোதʹԿ͕ى͖͔ͨ ɹൃੜɺௐࠪɺાஔɺใࠂ w োޙʹ࣮ࢪͨ͠࠶ൃରࡦ
ɹվળ w ·ͱΊ
ରࡦհ
ෳͷ"WBJMBCJMJUZ;POFʹϓϩϏδϣχϯάͨ͠&-# "-# "VUP4DBMJOH(SPVQ͔Βಛఆ"WBJMBCJMJUZ;POF্ͷϦιʔεΛύʔδ͢Δ ";োͷରࡦखஈͷ֬อͱɺ";োͷӨڹ؇ͷͨΊʹ࣮ࢪ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTQVSHFSFTPVSDFTTQFDJpDB[
"-#ͷΞΫηεϩάʹΞϕΠΞϏϦςΟκʔϯใΛՃͯ͠Έͨ "-#ͷΞΫηεϩάϨίʔυʹ&-#ϊʔυใ͕ແ͍ͨΊɺ ϩάϑΝΠϧͷΩʔʹؚ·ΕΔ*1ใΛݩʹ-BNCEBͰϊʔυใΛิ͍ɺ"UIFOBूܭΛ࣮ݱ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBEEB[BMCBDDFTTMPH
$MPVE8BUDI-PHT*OTJHIUTͰ"-#ͷ&-#99ൃੜ࣌ͷΞΫηεϩάΛੳͯ͠Έͨ &-#99ΤϥʔϩάͷΈ$MPVE8BUDI-PHTʹసૹ͢Δ-BNCEBΛར༻͠ɺ "UIFOBΑΓ؆қͳΤϥʔϩά֬ೝखஈΛ֬อ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCYYDXMPHTJOTJHIUT
'JSFIPTFͰ1BSRVFUܗࣜʹมͨ͠"-#ͷΞΫηεϩάΛ"UIFOBͰղੳͯ͠Έͨ 'JSFIPTFͷྻࢤมΛར༻ɺ"UIFOBͷॲཧޮΛվળ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHpSFIPTFQBSRVFU
'JSFIPTFͰ1BSRVFUܗࣜʹมͨ͠"-#ͷΞΫηεϩάΛ"UIFOBͰղੳͯ͠Έͨ %FW*0ͷ&-#ϩάॲཧͷઆ໌ࢿྉͰ͢ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTBMCMPHTRTEFpSFIPTF
(MPCBM"DDFMFSBUPSΛར༻ͯ͠#MVF(SFFOͳ&-#ަΛ࣮ࢪͯ͠Έͨ %FW*0ͷ&-#ɺ";ɺ*1WରԠͷڥަͨ͠खஈͰ͢ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTCMVFHSFFOFMCCZBHB
%FWFMPQFST*0͕*1WαϙʔτʹͳΓ·ͨ͠ *1Wো࣌ͷᷖճܦ࿏ͱͳΔࣄΛظͯ͠*1W༗ޮԽ͠·ͨ͠ Developers.IO ϒϩάج൫ IUUQTEFWDMBTTNFUIPEKQDMPVEBXTEFWJPTVQQPSUJQW
·ͱΊ
DevelopersIOαΠτͷऔΓΈʹ͍ͭͯ հ͖ͤͯ͞·ͨ͠
োվળʹܨ͕Δ ઈͷػձͱߟ͍͑ͯ·͢
γεςϜͷՌׂ͖ͨ͢ɺ ॏཁɺ՝͕໌ྎͳঢ়ଶͰɺ ΈΛߟ͑Δࣄ͕ग़དྷ·͢ɻ
Λఆྔతʹࣔ͢ϩάϝτϦοΫɺ ਖ਼͍͠அͷཁͱͳΓ·͢ɻ
Λڞ༗ɺվળΛଞʹٻΊΔ߹ʹɺ ࠜڌͱͳΔϩάϝτϦοΫ͕ॏཁͰ͢
ཱͭΈ͕ɺ AWSͰଟ͘ར༻ग़དྷ·͢ɻ
None