Supporting SRE with Azure Services

Transcript

1. Supporting SRE with Azure Services
2. YURY NIÑO Chaos Engineering Advocate www.yurynino.com JHONNATAN GIL Site Reliability Engineer
3. www.ingenieriadelcaos.com .
4. www.ingenieriadelcaos.com .
5. Agenda • What is SRE • SRE Principles ◦ Azure Services • SRE Practices ◦ Azure Services • Resources www.ingenieriadelcaos.com .
6. Those systems were not RELIABLE! Reliability describes the ability of a system or component to function under stated conditions for a specified period of time.
7. Site Reliability Engineering is what you get when you treat operations as if it’s a software problem. Our mission is to protect, provide for, and progress the software and systems with an ever-watchful eye on their availability, latency, performance, and capacity.
8. SRE Principles
9. 7 SRE Principles Embracing Risk Service Level Objectives Eliminating Toiling Monitoring Automation Release Engineering Simplicity www.ingenieriadelcaos.com .
10. 7 SRE Principles Embracing Risk Service Level Objectives Eliminating Toiling Monitoring Automation Release Engineering Simplicity www.ingenieriadelcaos.com .
11. Service Level Objectives www.ingenieriadelcaos.com .
12. Monitoring Azure Management Portal Azure provides dashboards that display specific metrics for services that not be available through APM tools or profilers. For example, the average latency of Azure Storage, or the rate for messaging within an Azure IoT Hub. Azure Services Azure helps to maximize the availability and performance of applications and services. Azure delivers a solution for collecting, analyzing, and acting on telemetry from cloud and on-premises environments. www.ingenieriadelcaos.com .
13. Monitoring www.ingenieriadelcaos.com .
14. 7 SRE Principles Embracing Risk Service Level Objectives Eliminating Toiling Monitoring Automation Release Engineering Simplicity www.ingenieriadelcaos.com .
15. Automation Automate ALL the Things! • User account creation. • Software or hardware installation preparation • Runtime configuration changes. • Rollouts of new software versions. Automate yourself? Automate the Resilience! www.ingenieriadelcaos.com .
16. Automation www.ingenieriadelcaos.com . Chaos Engineering It is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. https://principlesofchaos.org
17. Eliminating Toiling Azure DevOps www.ingenieriadelcaos.com .
18. Eliminating Toiling Azure ARM Templates Azure Resource Manager templates (ARM templates) • ARM is Infrastructure as Code for Azure. • A template is a JavaScript Object Notation (JSON) file. • A template uses declarative syntax, which lets you state what you intend to deploy without having to write the sequence of programming commands to create it. • A template specify the resources to deploy and the properties for those resources www.ingenieriadelcaos.com .
19. Eliminating Toiling www.ingenieriadelcaos.com .
20. 7 SRE Principles Embracing Risk Service Level Objectives Eliminating Toiling Monitoring Automation Release Engineering Simplicity www.ingenieriadelcaos.com .
21. Embracing Risk Continuous Build and Deployment! Release engineers have a solid understanding of source code management, compilers, build configuration languages, automated build tools, package managers, and installers. 4 principles: Self-Service Model, High Velocity, Hermetic Builds, Enforcement Policies and Procedures. You expect to build 100% reliable services—ones that never fail. However, increasing reliability is worse for a service rather than better! Extreme reliability comes at a cost! Embrace the Risk! www.ingenieriadelcaos.com .
22. Release Engineering Azure Staging slots One way to implement blue-green deployments is using the staging slots available in Azure App Service to stage a deployment before moving it to production. www.ingenieriadelcaos.com .
23. SRE Practices
24. 18 SRE Practices Effective Troubleshooting Practical Alerting Emergency Response Being On-Call Postmortems Managing Incidents Tracking Outages Testing for Reliability www.ingenieriadelcaos.com .
25. Being On-call www.ingenieriadelcaos.com .
26. Practical Alerting Azure Alerts Using Azure Monitor it is possible to generate alert rules that captures the target and criteria for alerting. Target Resource Defines the scope and signals available for alerting. • Virtual machines. • Storage accounts. • Log Analytics workspace. • Application Insights. Signals can be of the following types: metric, activity log, Application Insights, and log. www.ingenieriadelcaos.com .
27. 18 SRE Practices Effective Troubleshooting Practical Alerting Emergency Response Being On-Call Postmortems Managing Incidents Tracking Outages Testing for Reliability www.ingenieriadelcaos.com .
28. Emergency Response www.ingenieriadelcaos.com .
29. Preparation • Training and support, general knowledge management • Needs constant iteration Analysis & Learning • Understand what happened and build new action plans and training • Siloed apps for reporting and analysis make it hard to summarize incidents, delaying post mortems Detection & Alerting • Monitoring systems send alerts for issues which need to be reviewed and escalated • Alerting systems generate noise which is lost in email, unclear where to escalate Remediation • Resolve the issues • Multiple tools, people and processes need to be coordinated for the implementation of long-term solutions for incidents. Process Overview & Challenges Containment • Data must be reviewed and the current situation assessed • Damage has to be contained quickly and efficiently to minimize the impact. Incident Management
30. Incident Management Azure Security Incident Management It is a critical responsibility for Microsoft and represents an investment that any customer using Microsoft Online Services can count on. Azure implements the five-stage process. Detect Assess Diagnose Stabilize Close www.ingenieriadelcaos.com .
31. 18 SRE Practices Effective Troubleshooting Practical Alerting Emergency Response Being On-Call Postmortems Managing Incidents Tracking Outages Testing for Reliability
32. Effective Troubleshooting www.ingenieriadelcaos.com .
33. Tracking Outage www.ingenieriadelcaos.com .
34. 18 SRE Practices Handling Overload Software Engineering Cascading Failures Load Balancing Distributed Consensus Data Pipelines Data Integrity Distributed Scheduling www.ingenieriadelcaos.com .
35. 18 SRE Practices Handling Overload Software Engineering Cascading Failures Load Balancing Distributed Consensus Data Pipelines Data Integrity Distributed Scheduling www.ingenieriadelcaos.com .
36. Software Engineering Plan Requirements Design Code Deploy Maintain www.ingenieriadelcaos.com .
37. Load Balancing Azure Load Balancer Azure delivers high availability and network performance: • Instantly add scale to your applications • Load balance Internet and private network traffic • Improve application reliability via health checks • Flexible NAT rules for better security • Directly integrated into virtual machines and cloud services www.ingenieriadelcaos.com .
38. 18 SRE Practices Handling Overload Software Engineering Cascading Failures Load Balancing Distributed Consensus Data Pipelines Data Integrity Distributed Scheduling www.ingenieriadelcaos.com .
39. Handling Overload Throttling www.ingenieriadelcaos.com .
40. Handling Overload Priority Queue www.ingenieriadelcaos.com .
41. Handling Overload Circuit Breaker www.ingenieriadelcaos.com .
42. Load Balancing www.ingenieriadelcaos.com .
43. 18 SRE Practices Handling Overload Software Engineering Cascading Failures Load Balancing Distributed Consensus Data Pipelines Data Integrity Distributed Scheduling
44. Other Services DEMO Azure IaC con Terraform Bondades de la IaC Principales opciones: terrafor, Pulumi, Crossplane, Azure Resource Manager, CLI, entre otros…. www.ingenieriadelcaos.com .
45. Well Framework Architecture
46. Resources www.yurynino.com . Azure Reference https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-controlled-chaos https://marketplace.visualstudio.com/items?itemName=proofdock.chaos-platform https://docs.microsoft.com/en-us/azure/architecture/framework/ https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/reque st-limits-and-throttling
47. Swag and more • Claim your attendee Learner Badge here: • 30 Days to learn it: aka.ms/global-azure/30D2L • Virtual background and ANOTHER Badge: blog.globalazure.net/Swag
48. Questions??
49. GRACIAS YURY NIÑO ROA JHONNATAN GIL CHAVES Site Reliability Engineer ADL Digital Labs