Plan for Success

What happens to your system, when you get lucky and become successful?

1. PLAN FOR SUCCESS
2. RIGHT PLACE
3. RIGHT TIME 27TH JANUARY - First COVID-19 case in Germany 24TH FEBRUARY - Pandemic plan activated 09TH MARCH - First COVID-19 death in Germany 21ST MARCH - Official lockdown called in Bavaria, more Länder to follow
4. REGULAR MONTH ~10K REGISTRATIONS / DAY
5. REGULAR MONTH ~25K TRAINING USERS / DAY
6. PEAK MONTH ~20K REGISTRATIONS / DAY
7. PEAK MONTH ~40K TRAINING USERS / DAY
9. ~50K REGISTRATIONS / DAY +500%
10. ~50K REGISTRATIONS / DAY +250%
11. +400% ~100K TRAINING USERS / DAY
12. +250% ~100K TRAINING USERS / DAY
14. DNS HELL PROBLEM #1
15. DNS HELL • KubeDNS doesn’t cache name resolution on internal calls -> moved to CoreDNS does it • Short DNS Names for internal calls: we used bodyweight.api instead of bodyweight.api.svc.cluster.local, requiring 2 DNS resolution requests for each internal call PROBLEM #1 • Sidekiqs and internal calls • KubeDNS asks AWS for internal calls as well, so we used up our quota for external DNS requests very fast. As there is a high chance that a high error rate on something leads to an increased pile on sidekiq delayed jobs that call dns we increased the pressure here.
16. AUTOSCALER MAX TOO SMALL PROBLEM #2
17. PROBLEM #3 CIRCULAR DEPENDENCIES
18. PROBLEM #4 SYNCRONOUS CALLS
19. INCIDENT MANAGEMENT
20. • Task Force • Calendario & Meeting • Confluence • Miro Board • Tools • NR • Grafana • AWS Dashboard • Statuspage • Slack • Hangout
21. countermeasures
22. some things you’ll never figure out
23. note down everything as it happens
24. interdisciplinary task force
25. communicate towards company
26. use the momentum
27. feedback
28. Thank you MONICA.GIAMBITTO@PM.ME @KFMOLLI - TWITTER @NIRNAETH - GITHUB