Lessons Learned Reviewing 150 Infrastructures

Since April 2018 we've had the opportunity to perform a structured review of the architectural and operational choices of 150 platform teams. In this talk (given at QCon London 2020) I explore some themes, talk about common mistakes, and give some advice on how to avoid these yourselves.

2. LESSONS LEARNED FROM REVIEWING 150 INFRASTRUCTURES_ JON TOPPER | @jtopper | he/him/his
3. $ whoami Founder/CEO/CTO The Scale Factory Working in hosting/infrastructure for 20 years @jtopper Infrastructure / AWS / DevOps
4. @jtopper
5. 0 45 90 135 180 Mar-2018 May-2018 Jul-2018 Sep-2018 Nov-2018 Jan-2019 Mar-2019 May-2019 Jul-2019 Sep-2019 Nov-2019 Jan-2020 REVIEWS RUN_ @jtopper
6. TODAY’S AGENDA_ What is Well-Architected? What is a Well-Architected Review? Common Review Findings @jtopper
7. WHAT IS WELL-ARCHITECTED?_ @jtopper
8. WELL ARCHITECTED ORIGINS_ Catalogue of emergent good practices Observed by AWS Field Solutions Architects Codified and shared Platform agnostic* @jtopper
9. 
     
    
    
   
    
    
   
    
   
    
   
   
   
    
    ! 
   
   
    
   
   
   
    
    
   !
   "
   
    #   $
   
   
   #
    #
   !  !
   
   
   
   ! %
   
   ! 
   
   # 
   
   %  !
   
    
   
   %
   
    
   
   "
    
    
   $ White Papers Review Tool @jtopper
10. Performance Efficiency Cost Optimisation Operational Excellence Reliability Security @jtopper
11. IoT (Internet of Things) High Performance Computing Serverless Applications Lenses @jtopper
12. USING WELL-ARCHITECTED_ Gap analysis / planning Teaching Team alignment @jtopper
13. WHAT IS A WELL-ARCHITECTED REVIEW?_ @jtopper
14. WELL ARCHITECTED REVIEW_ Foundational questions Up to 4 hours Qualitative @jtopper
15. Performance Efficiency Cost Optimisation Operational Excellence Reliability Security Well Architected Core Serverless Applications High Performance Computing IoT (Internet of Things) 9 11 9 8 9 2 3 2 1 1 4 3 3 4 2 4 11 6 10 4 @jtopper 46 9 16 35
16. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? @jtopper
17. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI WA WA NI NI @jtopper
18. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI WA WA NI NI High Risk @jtopper
19. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI WA WA NI NI Medium Risk @jtopper
20. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI WA WA NI NI Medium Risk @jtopper
21. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI WA WA NI NI Well Architected @jtopper
22. COMMON REVIEW FINDINGS_ @jtopper
23. THE GOOD_ @jtopper
24. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI Well Architected 77% WA WA NI NI 93% 87% 90% 85% 89% 89% 0% WA Rank: 1 @jtopper
25. QUESTION PERF 3_ • Understand storage characteristics and requirements • Evaluate available configuration options • Make decisions based on access patterns and metrics • None of these How do you select your storage solution? WA CI Well Architected 70% NI NI 84% WA Rank: 2 78% 73% 5% @jtopper
26. QUESTION REL 5_ • Deploy changes in a planned manner • Deploy changes with automation • None of these How do you implement change? WA CI Well Architected 63% NI 83% 67% 6% WA Rank: 3 @jtopper
27. THE BAD_ @jtopper
28. QUESTION REL 9_ • Define recovery objectives for downtime and data loss • Use defined recovery strategies to meet the recovery objectives • Test disaster recovery implementation to validate the implementation • Manage configuration drift on all changes • Automate recovery • None of these How do you plan for disaster recovery? WA CI High Risk 79% NI 33% HRI Rank: 1 WA WA NI 33% 25% 39% 16% 31% (87%) @jtopper
29. QUESTION SEC 11_ • Identify key personnel and external resources • Identify tooling • Develop incident response plans • Automate containment capability • Identify forensic capabilities • Pre-provision access • Pre-deploy tools • Run game days • None of these How do you respond to a [security] incident? WA CI High Risk 75% NI 51% HRI Rank: 2 WA WA NI 27% 39% 0% 11% 27% 10% 3% 35% NI NI NI (93%) @jtopper
30. QUESTION SEC 8_ • Define data classification requirements • Define data protection controls • Implement data identification • Automate identification and classification • Identify the types of data • None of these How do you classify your data? WA CI High Risk 75% HRI Rank: 3 WA WA 61% 39% 17% 4% 59% 23% NI NI (88%) @jtopper
31. QUESTION COST 9_ • Establish a cost optimisation function • Develop a workload review process • Review and implement services in an unplanned way • Review and analyse this workload regularly • Keep up to date with new service releases • None of these How do you evaluate new services? WA CI High Risk 71% HRI Rank: 4 WA 34% 26% 84% NI NI (79%) NI 43% 63% 1% @jtopper
32. QUESTION REL 8_ • Use playbooks for unanticipated failures • Conduct root cause analysis and share results • Inject failures to test resiliency • Conduct game days regularly • None of these How do you test resilience? WA CI High Risk 67% HRI Rank: 5 WA 25% NI (92%) NI 73% 6% 0% 16% @jtopper
33. THE NOTABLE_ @jtopper
34. QUESTION OPS 3_ • Use version control • Test and validate changes • Use config management systems • Use build/deploy systems • Perform patch management • Share design standards • Implement practices to improve code quality • Use multiple environments • Make frequent, small, reversible changes • Fully automate integration and deployment • None of these How do you reduce defects, ease remediation, and improve flow into production? NI WA Well Architected 14% WA WA Rank: 23 @jtopper NI NI NI NI NI NI NI CI 90% 87% 78% 82% 37% 57% 83% 81% 63% 52% 3%
35. QUESTION OPS 6_ • Identify key performance indicators • Define workload metrics • Collect and analyse workload metrics • Establish workload metric baselines • Learn expected patterns of activity for workload • Alert when workload outcomes are at risk • Alert when workload anomalies are detected • Validate the achievement of outcomes and the effectiveness of KPIs and metrics • None of these How do you understand the health of your workload? WA Well Architected 46% WA WA Rank: 21 @jtopper NI NI NI NI NI CI 53% 62% 72% 51% 54% 40% 34% 37% WA 14%
36. QUESTION SEC 2_ • Define human access requirements • Grant least privileges • Allocate unique credentials per person • Manage credentials based on lifecycle • Automate credential management • Grant access through roles or federation • None of these How do you control human access? WA CI High Risk 47% HRI Rank: 20 WA WA 70% 58% 90% 70% 13% 62% 3% NI NI (88%) @jtopper NI
37. QUESTION SEC 3_ • Define programmatic access requirements • Grant least privileges • Automate credential management • Allocate unique credentials per component • Grant access through roles or federation • Implement dynamic authentication • None of these How do you control programmatic access? WA CI High Risk 57% HRI Rank: 15 WA 40% 70% 24% 68% 58% 22% 13% NI NI (89%) @jtopper NI NI
38. MAJOR THEMES_ @jtopper
39. TEAMS ARE OK AT CHOOSING CORRECT SERVICES_ Database choices match workload Storage choices match workload Compute choices sometimes not right- sized. @jtopper
40. TEAMS ARE OK AT MAKING SOFTWARE CHANGES_ Automation tools are being used Full CD remains out of reach Change batch sizes need to be smaller @jtopper
41. @jtopper https:/ /services.google.com/fh/files/misc/state-of-devops-2019.pdf
42. TEAMS ARE BAD AT THINKING ABOUT FAILURE MODES_ Not considering business requirements No risk analysis of failure modes Poor documentation Almost no attempt to rehearse outages @jtopper
43. @jtopper
44. TEAMS ARE BAD AT MONITORING FOR FAILURE MODES_ Monitoring happening Data not used for much Tracing almost non-existent @jtopper
45. TEAMS NEED TO DO BETTER AT SECURITY_ Poor hygiene around patching Limited data classification Mediocre human access control Bad programmatic access control Low adoption of security monitoring tools @jtopper
46. TOP BREACH CAUSES_ Using components with known vulnerabilities Security misconfiguration Injection Weak auth / session management Missing function access control https:/ /snyk.io/blog/owasp-top-10-breaches/ @jtopper
47. EVERYONE IS BETTER AT BUILDING PLATFORMS THAN THEY ARE AT SECURING OR RUNNING THEM_
48. WHAT NEXT?_ Read the white papers: https:/ /aws.amazon.com/architecture/well-architected/ Run your own review(s) https:/ /aws.amazon.com/well-architected-tool/ Consider engaging an AWS Well-Architected partner https:/ /scalefactory.com/services/well-architected/ (funding available) @jtopper
49. KEEP IN TOUCH_ http:/ /www.scalefactory.com/ https:/ /github.com/scalefactory @scalefactory jon@scalefactory.com