LOPUG: Measuring your Platform Engineering impact

Platform Engineering is gaining traction, but its value is often lost in translation. To engineering teams, it means golden paths and reusable pipelines. But how do you make its impact resonate with a CEO, a Head of Product, or even Security and Change Management? In this talk, Christopher will explore how the same platform capabilities, such as environments as a service and Continuous Delivery pipelines, can move the needle for different types of stakeholders. The key is choosing the right metric and telling the right story. He will share real examples from working with platform teams at large enterprises, showing how he's helped:

Explain platform engineering in the language of different stakeholders
Define outcome-based KPIs that align with each sponsor’s priorities
Track those metrics over time to demonstrate value and build accountability
He will walk through:

Strategic metrics such as faster market entry and fewer launch failures
Delivery metrics such as lead time and deployment frequency
Enabling metrics such as pipeline reliability and environment readiness
Governance metrics such as change success rates, policy coverage, and audit time

Speaker: Christopher Batey, CTO @ Core Engineering Consulting Group
Over 20 years Christopher has spent time in most areas of software delivery, product development, JVM library development (concurrency/distributed systems), Infrastructure engineering, architecture & system design and most recently Platform Engineering. He has a passion for speed and stability, focusing on what we need to do next to speed up the software delivery lifecycle. Over the years he’s contributed to many open source projects including Akka & Kubernetes.

Transcript

1. Measuring your platform Christopher Batey engineering impact Christopher Batey
2. Measuring your platform Christopher Batey engineering impact Christopher Batey
3. Platform Engineering is how we scale DevOps… But we need to get buy in at all levels to do it
4. Worked Example Making it all concrete Understanding Your Stakeholders Platform Engineering cuts across the organisation, let’s meet the common stakeholders at medium to large enterprises 1 3 Measure Measure Measure: Understanding Needles See how we measure value at different levels for different stakeholders 2 Transform how we demonstrate platform engineering value to stakeholders across the organisation
5. The goal of platform engineering is to enable change, at speed, with security built in, while increasing reliability
6. Accountability Speed Security Reliability Did the effort enable the business to delivery value to its customers faster? Did the effort enable a measurable improvement in security? Have we increased the reliability (i.e. the reputation) or the customer facing system? Focussing on change keeps us accountable as platform engineers
7. Problems With Platform Engineering • Platforms built with great technical merit • Value remains invisible to leadership • Executives cut funding or prioritize initiatives • Platforms don’t get the recognition they deserve or they don’t provide the value they intended (or at least not quick enough to survive) Many Platform Engineering teams are delivering great platforms while also struggling to maintain their existence
8. Why do platform engineers need to talk to everyone?
9. Organisations VP Engineering Head of QA Head of Engineering Head of Security Head of Change Management Software Engineers QAs & Perf Testers Head of Platform
10. Coordination Tax The friction of change across teams Day 5 Perf Testing Key performance tests are manually executed Day 10 Security Sign Off Security need to check if a new pen test is required. Checks for vulnerabilities. If anything is found we start again Day 1 Dev Complete A product team finished the change Day 2 QA A separate set of QAs then deploy the new version to a test environment to sign it off Day 15 Deploy Finally operations deploy the change
11. The Opportunity Platform Engineering Stakeholders • Understanding the varied perspectives and success criteria across your organisation is essential for effective platform engineering strategy • Always frame reporting and discussions based on the audience, their role, their problems, and their success. Friction Allies
12. The Opportunity Platform Engineering Stakeholders • Understanding the varied perspectives and success criteria across your organisation is essential for effective platform engineering strategy • Always frame reporting and discussions based on the audience, their role, their problems, and their success. Friction Best Friends
13. Avoid at all costs discussing your solutions with stakeholders
14. Large platform engineers initiatives can need buy in at the exec level. We have to frame things in their language, e.g. product launches Exec They care about business velocity, competitive risk, return on investment, cost They want faster market entry, fewer failed product launches, predictable delivery Their problems could be, not understanding why things are going slowly, not able to measure return on engineering investment
15. Head of Engineering for a Product Development Head of Engineering They care about predictable delivery, quality, developer empowerment They want engineers spending maximum time on product development, frictionless infrastructure, seamless onboarding Their problems are lack of visibility into quality, slow onboarding of engineers, waiting on infrastructure
16. Engage with Security, early, and often, making them your best ally Security They care about coverage and visibility. Speed of remediation. Vulnerabilities, endpoint protection, policies and conformance. They want to be included early, have high engagement from engineering, to see the security posture improve over time Their problems are lack of adoption, increasing number of vulnerabilities, unknown assets in production
17. Can be a single role, function or full department. Responsible for the production change process Change Management They care about about safe, predictable change They want higher change rate success, shorter lead time for normal changes, better evidence of change safety, full audit of every change Their problems are lack of engagement from engineering teams, lack of clarity for what is in a change, lack of belief that change evidence is reliable
18. Platform Lead, Infra, whoever builds your platform Head of Infrastructure They care about adoption, consistency, operational excellence, resource efficiency, reliability They want to provide infrastructure and platform capabilities that are used Their problems can be adoption of platform services, cost of infrastructure including visibility, large overhead keeping infrastructure conformant
19. Start With The People Awareness Build trust, set the frame, find out their problem Quantify Attach numbers to pain. Turn a “feeling” into a number. Commit Get buy-in to act Agree Needle Co-own the metric for success Proof Show a credible, low-risk fix, think about external vs internal proof Discover & Bucket Let them surface friction “their problems” and bucket it into a category
20. Start With The People Commit Get buy-in to act
21. Most things platform engineering aims to achieve fall into… Bucketing Measurable Value Speed Delivery velocity, how long it takes to get ideas to production “Nothing ever gets released” “Everything happens so slowly” Reliability / Performance Uptime, quality, # of user facing outages, # concurrent customers “We always go down during big events” Security RBAC, vulnerabilities, endpoint protection, conformance to regulations “I’m worried about a breach” Cost Total cost, cost per service, cost per user “Why is it so expensive?”
22. Needles for Speed Manual Actions # of manual actions to get a change to production Product Launch Time Time taken to launch a product to production from scratch User Feature Releases # of features released per unit of time Deployment Frequency For one environment, rate of deployments Lead Time For one delivery unit, time taken from commit to production Engineer Onboarding Duration for a new engineer to raise first change that makes it to production Service Onboarding Duration to have all environments, pipeline, and first prod deployment Manual Approvals # of manual approvals for a change to enter production
23. Needles for Cost Cost per Customer Granular cost of one customer for one service then aggregated across services Service Cost Granular cost of one service per environment and total Cost per User Journey Granular cost of one user journey across the system Storage Costs Database costs for a service across environments Return on Investment Time taken to launch a product to production from scratch Total Cost of Ownership Per product, per tool Application Hosting Costs Application hosting costs for a service across environments
24. Needles for Reliability Functional Test Coverage What proportion of key business use cases are covered by functional tests Product Uptime What % is the product available overall Outages during key events For key events, how often does the product work SLA Breaches How often do we breach contractual commitments Service Uptime What % of time is our service up SLO Breaches How often do we breanch our SLOs set below our SLAs Service Reliability What % of requests to a service succeed Non-Functional Test Coverage/Reliability Do we have the tests? What coverage do we have? How often do they fail?
25. Needles for Security Public Breaches Time taken to launch a product to production from scratch # of Vulnerabilities How many vulnerabilities Compliant Environments % of compliant environments # of assets without ownership Audit Outcome The number of actions on an external audit
26. Product releases have slowed down, execs are complaining they are falling behind competitors… As the team have tried to speed up, reliability has suffered and there is an increasing number of vulnerabilities in production Worked Example Solution Standard Path to Production, starting with key components that change the most, that makes use of Application Environments as a Service to ensure environments are consistent Diagnosis Manual set of environments Leads to differences in test and production environments Dev teams skipping automated testing as nowhere to run tests (lack of environments) Security are having to go to every service individually as they are deployed differently to slightly different environemnts
28. Application Environments as a Service Fast Feedback Extended Test Production Functional Non-Functional Integration Extended Canary Production Integrated Functional Testing Stubbed Non-Functional Testing Build Push Build Stubbed Functional Testing Integration Testing Extended Test Ready Deploy Test Ready Stubbed Non-Functional Testing Extended Testing 22:00 Production Ready Prod Dev Infrastructure Environments
29. Stakeholders Exec Problem: Product delivery is too slow Quantified: Features delivered monthly Change Management Problem: No trust in the evidence attached to change requests Quantified: 95% of change requests have manually added evidence Security Problem: Vulnerabilities just keep growing Quantified: 35 vulnerabilities in production Head of Engineering Problem: My engineers struggle to get things done Quantified: Service onboarding is 2 weeks. PRs take 3 days on average to merge
30. Measure the impact of change Reduce Lead Time from 1 week to 1 day for service X, Y, Z Reduce Manual Actions from 4 to 0 for a change User Feature Releases Release 2 features per week Reduce Service Onboarding from 2 weeks to 1 hour Reduce PR review time from 3 days to 0.5 days Reduce # of Vulnerabilities from 35 to 5 Increase % of application releases automatically scanned by 100% 100% of changes with automated, reliable evidence
31. Carrots
32. Stakeholder-specific insights inform platform engineers why they are developing a feature beyond its technical merit
33. Seeing every platform feature from the perspective of different stakeholders will subtly, but importantly, change your platform product
34. The goal isn't to measure everything, but to measure the right things that demonstrate clear business value to the people who make funding decisions Key Takeaways Measurement = Translation Accountability Start Simple Measure your platform engineering initiatives with needles that each stakeholder cares about Measuring the right things, at the right levels, keeps us accountable to build the right thing Pick a small set of services that encompass a key user journey to show value
35. cecg.io Don't talk about platform features. Show the right needle moving based on the person’s role