LT_2023-01-26 不要になったAWSリソースを 削除してサービス障害起こした話

不要になったAWSリソースを 削除してサービス障害起こした話

Transcript

1. 不要になったAWSリソースを 削除してサービス障害起こした話 Tomoki Inoue 2023/01/26
2. • Name: Tomoki Inoue • Join: 2019.3~ • Position: Backend Engineer • Twitter: @_tmkinoue 自己紹介
3. 経歴 ~2019 : PHP, JavaScript 2019 : PHP, JavaScript, AWS 2020 : Golang, AWS, Kubernetes (Amazon EKS) 2021 : Golang, AWS, Kubernetes (Amazon EKS) 2022 : React Native, AWS, Kubernetes (Amazon EKS)
4. 不要になったと思っていたAWSリソースを 削除してサービス障害起こした話
5. 背景
6. Feb. 2020 icon by Icons8
7. Feb. 2020 FE team BE team icon by Icons8
8. Feb. 2020 ~ BE team icon by Icons8
9. 2021 Summer icon by Icons8 卒業します 異動します
10. 2021 Summer BE team icon by Icons8
11. 2021 Summer ~ icon by Icons8 Kubernetes Ver. Amazon EKS end of support 1.17 2021.11 1.18 2022.03 1.19 2022.08 1.20 2022.11 1.21 2023.02 Ver. up
12. やったこと
13. 2021 Summer icon by Icons8 Kubernetes Ver. Amazon EKS end of support 1.17 2021.11 1.18 2022.03 1.19 2022.08 1.20 2022.11 1.21 2023.02 Ver. up New Cluster (Latest ver.)
14. Mar.2022 ~May.2022 icon by Icons8 Ver. up New Cluster (Latest ver.)
15. AWS Cloud Amazon EKS EC2 instance contents icon by Icons8 User Internet Application Load Balancer Pod (Golang Application) Amazon CloudFront
16. AWS Cloud icon by Icons8 User Internet Amazon CloudFront Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application)
17. AWS Cloud icon by Icons8 User Internet Amazon CloudFront Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application)
18. AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) $ eksctl delete cluster
19. 結果
20. AWS Cloud icon by Icons8 User Internet Amazon CloudFront Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application)
21. 原因
22. AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora
23. AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
24. AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
25. AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
26. AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
27. AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
28. AWS Cloud $ eksctl create cluster Amazon EKS Application Load Balancer Role Amazon EC2 AWS CloudFormation
29. AWS Cloud $ eksctl delete cluster Amazon EKS Application Load Balancer Role Amazon EC2 AWS CloudFormation
30. 新しいクラスタを作ったときは？
31. icon by Icons8 Ver. up New Cluster (Latest ver.)
32. 反省と教訓
33. 反省と教訓 • その作業、実施すると何が起こるかわかってますか？ • 手順書の内容を把握しただけで理解した気になっていませんか？ • レビュー、ちゃんと見て内容を理解してますか？ ◦ そんな余裕はない？ ◦ そこにリソースを割くことができない？
34. 反省と教訓 • ちゃんと見てないけど担当者がいい感じにやってくれるはず • でも引き継ぎは突然やってくることもあります ◦ 退職、休職、異動... • 前任者がしっかりと仕事してくれる人だったから大丈夫？ • 退職者は責任を取ってくれませんよ？
35. まとめ
36. 本番環境の作業をするときはくれぐれも慎重に ☆（ゝω・）vｷｬﾋﾟ
37. ご清聴ありがとうございました