Operating Under Uncertainty

In a changing world, it's important that your platform can adapt to new constraints or requirements.

This content comes from a webinar on designing your infrastructure and operations with uncertainty in mind. Attendees learned how to prepare for and manage change, and keep costs in check.

For a period of time, the webinar video is available to view at https://zoom.us/rec/share/y_w2DPby7kxOAauT1EziX6JwLrn4aaa81SccrKYIxJ9HcBV4HRq_jA-XLf8pIeI
Access Password: L9$*G356

Transcript

1. OPERATING UNDER UNCERTAINTY_ JON TOPPER | @jtopper | he/him/his
2. $ whoami Founder/CEO/CTO The Scale Factory Working in hosting/infrastructure for 20 years Infrastructure / AWS / DevOps
4. THE TEAM_
5. OUR CLIENTS_
6. TODAY’S AGENDA_ What is uncertainty? How can we manage risk: Through architecture? Through operations design? Specific risks: COVID-19 Q&A
7. UNCERTAINTY_ “a state of limited knowledge where it is impossible to exactly describe the existing state, a future outcome, or more than one possible outcome”
8. TYPES OF UNCERTAINTY_ Economic Personnel Demand Infrastructure Security
9. UNCERTAINTY = RISK
10. RISK MANAGEMENT_ Avoidance Reduction Transfer Acceptance
11. AWS RISK REDUCTION_ Appropriate architecture choices Good operational approach
12. APPROPRIATE ARCHITECTURE CHOICES_
13. LEVERAGE THE PLATFORM_
15. Visible Invisible Value Chain Evolution Genesis Custom Product Commodity Power Customer MySQL Compute Storage Data Centre HA Scripts Monitoring Config Mgmt Networking
16. Visible Invisible Value Chain Evolution Genesis Custom Product Commodity Customer RDS Aurora
17. Somebody Else's Problem
18. DESIGN FOR FAILURE_
19. “EVERYTHING FAILS ALL THE TIME”_ Werner Vogels, CTO amazon.com
20. VPC (eu-west-1) AZ eu-west-1a AZ eu-west-1b AZ eu-west-1c Public Subnet Public Subnet Public Subnet Private Subnet Private Subnet Private Subnet
22. VPC (eu-west-1) VPC (eu-west-2)
23. AVAILABILITY DESIGN_ Clustering / Failover Autoscaling Multi-AZ (and Multi-Region) operation Caching Asynchronous processing Backpressure / Circuit breakers
24. DESIGN FOR ELASTICITY (UNCERTAINTY OF DEMAND)_
25. DESIGN FOR ELASTICITY_
28. n=1
29. n=6
30. n=3
31. CONTROL SCALING_ Time-based Metric-based Predictive
32. GOOD OPERATIONAL APPROACH_
33. ADOPT DEVOPS PRACTICES_ Infrastructure as Code Deployment Automation (CI/CD) Monitoring / Tracing
34. https:/ /services.google.com/fh/files/misc/state-of-devops-2019.pdf
35. WHAT TO MONITOR_ User behaviour Resource usage Security controls
36. Knowledge Awareness Known Unknowns (known risks) Unknown Unknowns (unknown risks) Unknown Knowns (hidden facts) Known Knowns (facts, requirements)
37. Knowledge Awareness Known Unknowns (known risks) Unknown Unknowns (unknown risks) Unknown Knowns (hidden facts) Known Knowns (facts, requirements) Build hypotheses Measure Iterate Brainstorming Group documentation Research Exploration External consultancy
38. Knowledge Awareness Known Unknowns (known risks) Unknown Unknowns (unknown risks) Unknown Knowns (hidden facts) Known Knowns (facts, requirements) Build hypotheses Measure Iterate Brainstorming Group documentation Research Exploration External consultancy Prepared
40. PLAN FOR FAILURE_ Build a list of potential failure scenarios Understand how your platform will react Game days (tabletop) Mitigate / Document Game days (live failure injection)
44. RUNBOOKS/ PLAYBOOKS_ Preserve institutional knowledge Reduce burden on key personnel Design troubleshooting procedures in advance
45. START DOCUMENTING_ Prioritise: Most frequent procedures Procedures with highest error rates Procedures with significant harmful impact
46. START DOCUMENTING_ Document: Procedure steps Expected outcomes Escalation paths
47. COVID-19 CONSIDERATIONS_
48. COST CONSIDERATIONS_ Economic impact: Reduced sales $ exchange rate Take action: Right-size workload “pilot light” dev environments
49. DEMAND CONSIDERATIONS_ Traffic impact: Education, entertainment - up now Travel - up later Take action: Load testing Performance tuning
50. PERSONNEL CONSIDERATIONS_ Impact: Illness / death Availability/focus due to dependents Take action Documentation / knowledge share Increase team size Engage third party support
51. Leading Well-Architected Partner Worldwide >200 Reviews Completed Since April 2018 Book a Well-Architected review today https:/ /scalefactory.com/services/well-architected/ $5,000 funding available to support improvement work
52. Q&A_
53. KEEP IN TOUCH_ http:/ /www.scalefactory.com/ @scalefactory jon@scalefactory.com