Parker Head, Enterprise Monitoring Services at Standard Chartered Trevor Morgan Head, Enterprise Monitoring Engineering at Standard Chartered About Standard Chartered Bank • 150 year history • Operates in >70 countries • 1,200 branches • 87,000 employees • 1,200 applications • Multiple global and country DCs • > 45,000 servers and devices
My app is live. How should I monitor it? App team: Is my server monitored? Infra team: *shrug* We don’t do that type of monitoring What’s happening with networks? We’ve love to help set up your monitoring but we’re too busy We never received an alert! Us: What do you mean you’ve bought your own monitoring tools!
Take one third party vendor product suite 2) Add in a pinch of process to slow down deployment 3) Add in a process for the process 4) Bake for 2 - 4 years and realise the solution has gone stale 5) Repeat steps 1 to 4 until you are locked into millions in contracts and software 6) Realise you have no control and nobody wants to use your solutions Building a floating city
Forget what you have, and focus on what you need SCB Approach • Talk to Business and Tech influencers • Listen to feedback and understand constraints • Publish a clear and detailed first principles based strategy • Get feedback from stakeholders, revise early and often • What do your users want? • Where are their pain points? • Understand your business • Draft and publish your strategy • Put the building blocks in place
first question always asked should be, “What are my Business KPIs?” Monitoring policy and tool decisions should only be made once the business needs are understood. • Your monitoring journey starts here • What are my primary Business KPIs? • Issues that will impact customers, reputation, revenue, or other key business considerations Business Monitoring • What monitoring do I need to implement to satisfy the business KPIs? Application Monitoring • What monitoring needs to fundamentally exist to feed the data requirements of application and business monitoring? Infrastructure Monitoring
and People • What is the Enterprise Monitoring framework and how will it benefit me? • How do I plug in? Get teams to plug in SCB Approach • Allow existing tools • Don’t preach BUT do have a process • Avoid mandates where possible • Encourage modern solutions • Enlist champions in the business • Generate awareness • Support throughout
of the Enterprise Monitoring strategy at SCB Monitoring Data Data Aggregation (Normalization + Enrichment) Notification + Automation Monitoring Data Data Collection • Infra / App Agents • Logs • APM • Cloud / Container Monitoring • SNMP • Others Normalized Data + Alerts Aggregation Layer Event Mgmt. Platform Standard Policies Normalization Enrichment Basic Dashboards Integrations Rest API Normalization + Enrichment Cell Ingest standard alert 3rd party integrations Transform to common event data format Enrich with config data (CMDB / Cloud), knowledge, etc API ITSM Automation Callout Collaboration NoSQL Alerts + Metrics API Metrics + Logs CMDB Cloud Config Others Alerts + Key Metrics AIOps Transform + Enrich Focus on Alerts! API CMDB Transform + Enrich
Born • Focus on high quality data, but also collect lots of it • Build for the future – you don’t know what tomorrow will bring ▪ Scalable / Extensible / Customisable • Simplify for your users • Define your monitoring KPIs Refine and innovate over time SCB Approach • Provision central data layer • Integrate existing tools / polices day 1 • Implement DevOps to add features • Build an API layer and overlay custom front-end / self-service platform • Select best of breed solution for each new requirement where possible
aggregation at SCB A key piece of SCB’s monitoring strategy is to centralize, aggregate, and normalize all performance and monitoring data, from which we can deliver visualizations, capacity planning, and more. Event / Performance Data 3rd Party Vendor In-house built AGGREGATE | NORMALIZE | DE-DUPLICATE Event Management Capacity Planning Cloud / SaaS Richer Data Fewer Alerts Less Noise Agents, streaming data, SaaS integration, public/private cloud, polling
bananas! • As your company grows, so do the monitoring requirements • Keep up to date with new technologies and be ready with a solution • Review existing solutions frequently. Don’t let your solution become over-ripe • Mature both monitoring and users experience Monitoring makes people’s lives easier if done properly SCB Approach • Regular meetings with your customers • Guided and self-help knowledge articles • Ensure vendors understand your roadmap • Build a maturity model to help teams grow their knowledge in the right way • Publicise successes and praise teams using monitoring effectively
Well? • Engaging our customers throughout the project • Being flexible when new challenges approach • Inclusive approach / extensible data model • Roadshows / Show and Tell / Awareness sessions What Would We Change? • Spend more time gaining consensus in the early stages • Adopt a more DevOps(y) approach earlier on • Work out how to pay for it! • Focus less on legacy technologies (80/20 rule)
SCB The future is bright and incident free Where do we go from here? • Fully integrated monitoring toolset • AIOps • DevOps / Monitoring as Code • Monitoring Service Charge based funding • On-Prem vs. SaaS Monitoring Platforms Thank you! Questions?