Enterprise Monitoring at SCB - Building a Strategy

Building a Monitoring Strategy and Gaining Concensus

Presented by Gregory Parker and Trevor Morgan at Monitorama EU 2018 in Amsterdam

Transcript

1. Gregory Parker and Trevor Morgan Enterprise Monitoring Services Standard Chartered Bank
2. 1 Building an Enterprise Monitoring Strategy Who are we? Greg Parker Head, Enterprise Monitoring Services at Standard Chartered Trevor Morgan Head, Enterprise Monitoring Engineering at Standard Chartered About Standard Chartered Bank • 150 year history • Operates in >70 countries • 1,200 branches • 87,000 employees • 1,200 applications • Multiple global and country DCs • > 45,000 servers and devices
3. 2 Building an Enterprise Monitoring Strategy It’s all too familiar My app is live. How should I monitor it? App team: Is my server monitored? Infra team: *shrug* We don’t do that type of monitoring What’s happening with networks? We’ve love to help set up your monitoring but we’re too busy We never received an alert! Us: What do you mean you’ve bought your own monitoring tools!
4. 3 Building an Enterprise Monitoring Strategy Recipe for Disaster 1) Take one third party vendor product suite 2) Add in a pinch of process to slow down deployment 3) Add in a process for the process 4) Bake for 2 - 4 years and realise the solution has gone stale 5) Repeat steps 1 to 4 until you are locked into millions in contracts and software 6) Realise you have no control and nobody wants to use your solutions Building a floating city
5. 4 Building an Enterprise Monitoring Strategy Going Back to Basics Forget what you have, and focus on what you need SCB Approach • Talk to Business and Tech influencers • Listen to feedback and understand constraints • Publish a clear and detailed first principles based strategy • Get feedback from stakeholders, revise early and often • What do your users want? • Where are their pain points? • Understand your business • Draft and publish your strategy • Put the building blocks in place
6. 5 Building an Enterprise Monitoring Strategy New Monitoring Journey The first question always asked should be, “What are my Business KPIs?” Monitoring policy and tool decisions should only be made once the business needs are understood. • Your monitoring journey starts here • What are my primary Business KPIs? • Issues that will impact customers, reputation, revenue, or other key business considerations Business Monitoring • What monitoring do I need to implement to satisfy the business KPIs? Application Monitoring • What monitoring needs to fundamentally exist to feed the data requirements of application and business monitoring? Infrastructure Monitoring
7. 6 Building an Enterprise Monitoring Strategy Plugging In – Tools and People • What is the Enterprise Monitoring framework and how will it benefit me? • How do I plug in? Get teams to plug in SCB Approach • Allow existing tools • Don’t preach BUT do have a process • Avoid mandates where possible • Encourage modern solutions • Enlist champions in the business • Generate awareness • Support throughout
8. 7 Building an Enterprise Monitoring Strategy Enterprise Monitoring Approach Implementation of the Enterprise Monitoring strategy at SCB Monitoring Data Data Aggregation (Normalization + Enrichment) Notification + Automation Monitoring Data Data Collection • Infra / App Agents • Logs • APM • Cloud / Container Monitoring • SNMP • Others Normalized Data + Alerts Aggregation Layer Event Mgmt. Platform Standard Policies Normalization Enrichment Basic Dashboards Integrations Rest API Normalization + Enrichment Cell Ingest standard alert 3rd party integrations Transform to common event data format Enrich with config data (CMDB / Cloud), knowledge, etc API ITSM Automation Callout Collaboration NoSQL Alerts + Metrics API Metrics + Logs CMDB Cloud Config Others Alerts + Key Metrics AIOps Transform + Enrich Focus on Alerts! API CMDB Transform + Enrich
9. 8 Building an Enterprise Monitoring Strategy The SCB Pattern is Born • Focus on high quality data, but also collect lots of it • Build for the future – you don’t know what tomorrow will bring ▪ Scalable / Extensible / Customisable • Simplify for your users • Define your monitoring KPIs Refine and innovate over time SCB Approach • Provision central data layer • Integrate existing tools / polices day 1 • Implement DevOps to add features • Build an API layer and overlay custom front-end / self-service platform • Select best of breed solution for each new requirement where possible
10. 9 Building an Enterprise Monitoring Strategy Aggregation Approach Monitoring data aggregation at SCB A key piece of SCB’s monitoring strategy is to centralize, aggregate, and normalize all performance and monitoring data, from which we can deliver visualizations, capacity planning, and more. Event / Performance Data 3rd Party Vendor In-house built AGGREGATE | NORMALIZE | DE-DUPLICATE Event Management Capacity Planning Cloud / SaaS Richer Data  Fewer Alerts  Less Noise Agents, streaming data, SaaS integration, public/private cloud, polling
11. 10 Building an Enterprise Monitoring Strategy Common source for data processing Enrichment Analytics Visualisation Normalisation • Categorise • Common format • CMDB • Weighting • Routing • Metadata Filter • Maintenance • Noise • Storm detect • Deduplicate • RCA • Correlation • Anomaly • Storm detect • Trending Automation • Event Management • Metric history • Raw data (API) • Knowledge • Notification • Self-healing • Incident tickets
12. 11 Building an Enterprise Monitoring Strategy Grow bananas, don’t go bananas! • As your company grows, so do the monitoring requirements • Keep up to date with new technologies and be ready with a solution • Review existing solutions frequently. Don’t let your solution become over-ripe • Mature both monitoring and users experience Monitoring makes people’s lives easier if done properly SCB Approach • Regular meetings with your customers • Guided and self-help knowledge articles • Ensure vendors understand your roadmap • Build a maturity model to help teams grow their knowledge in the right way • Publicise successes and praise teams using monitoring effectively
13. 12 Building an Enterprise Monitoring Strategy Lessons Learnt What Went Well? • Engaging our customers throughout the project • Being flexible when new challenges approach • Inclusive approach / extensible data model • Roadshows / Show and Tell / Awareness sessions What Would We Change? • Spend more time gaining consensus in the early stages • Adopt a more DevOps(y) approach earlier on • Work out how to pay for it! • Focus less on legacy technologies (80/20 rule)
14. 13 Building an Enterprise Monitoring Strategy Future of Monitoring at SCB The future is bright and incident free Where do we go from here? • Fully integrated monitoring toolset • AIOps • DevOps / Monitoring as Code • Monitoring Service Charge based funding • On-Prem vs. SaaS Monitoring Platforms Thank you! Questions?