Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
LT_2023-01-26 不要になったAWSリソースを 削除してサービス障害起こした話
Search
tomoki inoue
January 26, 2023
Programming
0
270
LT_2023-01-26 不要になったAWSリソースを 削除してサービス障害起こした話
不要になったAWSリソースを 削除してサービス障害起こした話
tomoki inoue
January 26, 2023
Tweet
Share
Other Decks in Programming
See All in Programming
Direct Style Effect Systems The Print[A] ExampleA Comprehension Aid
philipschwarz
PRO
0
150
Kotlin Multiplatform at Stable and Beyond (Android Makers 2024)
zsmb
0
530
『Railsオワコン』と言われる時代に、なぜブルーモ証券はRailsを選ぶのか
free_world21
1
390
Milestoner
bkuhlmann
1
420
Komplexe Oberflächen mit SVG und der Web Animation API
joergneumann
0
680
OpenAPIを中心に考えるAPI開発入門 / Introduction to API Development with a Focus on OpenAPI
seike460
PRO
2
180
初心者のためのRubyKaigi入門/RubyKaigi Introduction
a_matsuda
10
1.6k
Balkan Ruby 2024 — How and why to run SQLite on Rails in production
fractaledmind
0
100
はてなにおける CSS Modules、及び CSS Modules に足りないもの / CSS Modules in Hatena, and CSS Modules missing parts
mizdra
7
990
Anthropic Cookbook のおすすめレシピ
schroneko
7
1.3k
障害対応を起点としたもっといい開発と運用のサイクル作りのためにできること / Hatena Enginner Seminar #29
polamjag
0
400
Elm 0.19.0 Changes
bkuhlmann
0
510
Featured
See All Featured
How GitHub (no longer) Works
holman
305
140k
Optimising Largest Contentful Paint
csswizardry
13
2.4k
Building an army of robots
kneath
300
41k
Become a Pro
speakerdeck
PRO
13
4.6k
Build The Right Thing And Hit Your Dates
maggiecrowley
25
2k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
34
8.9k
How To Stay Up To Date on Web Technology
chriscoyier
782
250k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
0
17
Happy Clients
brianwarren
92
6.4k
Reflections from 52 weeks, 52 projects
jeffersonlam
345
19k
Optimizing for Happiness
mojombo
370
69k
What's new in Ruby 2.0
geeforr
337
31k
Transcript
不要になったAWSリソースを 削除してサービス障害起こした話 Tomoki Inoue 2023/01/26
• Name: Tomoki Inoue • Join: 2019.3~ • Position: Backend
Engineer • Twitter: @_tmkinoue 自己紹介
経歴 ~2019 : PHP, JavaScript 2019 : PHP, JavaScript, AWS
2020 : Golang, AWS, Kubernetes (Amazon EKS) 2021 : Golang, AWS, Kubernetes (Amazon EKS) 2022 : React Native, AWS, Kubernetes (Amazon EKS)
不要になったと思っていたAWSリソースを 削除してサービス障害起こした話
背景
Feb. 2020 icon by Icons8
Feb. 2020 FE team BE team icon by Icons8
Feb. 2020 ~ BE team icon by Icons8
2021 Summer icon by Icons8 卒業します 異動します
2021 Summer BE team icon by Icons8
2021 Summer ~ icon by Icons8 Kubernetes Ver. Amazon EKS
end of support 1.17 2021.11 1.18 2022.03 1.19 2022.08 1.20 2022.11 1.21 2023.02 Ver. up
やったこと
2021 Summer icon by Icons8 Kubernetes Ver. Amazon EKS end
of support 1.17 2021.11 1.18 2022.03 1.19 2022.08 1.20 2022.11 1.21 2023.02 Ver. up New Cluster (Latest ver.)
Mar.2022 ~May.2022 icon by Icons8 Ver. up New Cluster (Latest
ver.)
AWS Cloud Amazon EKS EC2 instance contents icon by Icons8
User Internet Application Load Balancer Pod (Golang Application) Amazon CloudFront
AWS Cloud icon by Icons8 User Internet Amazon CloudFront Amazon
EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application)
AWS Cloud icon by Icons8 User Internet Amazon CloudFront Amazon
EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application)
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) $ eksctl delete cluster
結果
AWS Cloud icon by Icons8 User Internet Amazon CloudFront Amazon
EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application)
原因
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
AWS Cloud $ eksctl create cluster Amazon EKS Application Load
Balancer Role Amazon EC2 AWS CloudFormation
AWS Cloud $ eksctl delete cluster Amazon EKS Application Load
Balancer Role Amazon EC2 AWS CloudFormation
新しいクラスタを作ったときは?
icon by Icons8 Ver. up New Cluster (Latest ver.)
反省と教訓
反省と教訓 • その作業、実施すると何が起こるかわかってますか? • 手順書の内容を把握しただけで理解した気になっていませんか? • レビュー、ちゃんと見て内容を理解してますか? ◦ そんな余裕はない? ◦
そこにリソースを割くことができない?
反省と教訓 • ちゃんと見てないけど担当者がいい感じにやってくれるはず • でも引き継ぎは突然やってくることもあります ◦ 退職、休職、異動... • 前任者がしっかりと仕事してくれる人だったから大丈夫? •
退職者は責任を取ってくれませんよ?
まとめ
本番環境の作業をするときはくれぐれも慎重に ☆(ゝω・)vキャピ
ご清聴ありがとうございました