14/09/2023 Ludovic Pourrat Apidays London 2023 API Metrics Matter in APIOps 

1 · Ludovic Pourrat P u b l i c API Architect & API Catalyst | Platform Architect Lombard Odier Group Auxiliary Ski Instructor Ecole Suisse de Ski, Verbier 

2 · Ludovic Pourrat P u b l i c Lombard Odier Group Private Bank in Switzerland since 1796 Main businesses • Private Clients • Asset Management • Technology for Banking Technology • Financial Software Solution Developer • BPO activity «Bank as a service» 

3 · Ludovic Pourrat P u b l i c Banking Platform 4 functional development streams • Market, Front, Tax & Operations, Finance Modular Service oriented solution • ≈ 800 application components GX - Large Modernization Initiative started in 2020 • Functional (e.g. SMF, Order Management) • Technical • OpenShift, GitOps, APIOps - Kong, Kafka, Vault, Consul, … • Angular micro-frontends • Migration Program - SpringBoot and Quarkus • Service Now, APM & OTEL, OIDC • Cloud - development environments 

API Metrics & Discovered Landscape 

5 · Ludovic Pourrat P u b l i c API Metrics Overview A wide range of available metrics Performance Metrics Latency Time Throughput Cache Hit Rate Resource Utilization Reliability and Availability Metrics Uptime Error Code Breakdown & Error Rate SLA Compliance 

6 · Ludovic Pourrat P u b l i c API Metrics Overview A wide range of available metrics Performance Metrics Latency Time Throughput Cache Hit Rate Resource Utilization Reliability and Availability Metrics Uptime Error Code Breakdown & Error Rate SLA Compliance Security Metrics OWASP Top 10 API Conformance Authentication & Authorization Rate Limiting Developer & Consumer Experience Metrics Engagement Rate Adoption Rate Retention & Churn Rate Documentation Quality and Coverage Rate Validation Error Rate 

7 · Ludovic Pourrat P u b l i c API Metrics Overview A wide range of available metrics Standard Conformance Metrics OpenAPI / GraphQL / AsyncAPI / gRPC Conformance Custom API Standard Conformance Change Management Metrics API Version Adoption & Retention Rate Deprecation Effectiveness Deprecation Usage API Version Views & Sunset Compliance SDK Version Usage Performance Metrics Latency Time Throughput Cache Hit Rate Resource Utilization Reliability and Availability Metrics Uptime Error Code Breakdown & Error Rate SLA Compliance Security Metrics OWASP Top 10 API Conformance Authentication & Authorization Rate Limiting Developer & Consumer Experience Metrics Engagement Rate Adoption Rate Retention & Churn Rate Documentation Quality and Coverage Rate Validation Error Rate 

8 · Ludovic Pourrat P u b l i c API Metrics Overview A wide range of available metrics AI Readiness & Usability Metrics AI & bot consumption readiness AI context prompt awareness Data Governance Metrics Data Classification & Tagging Data Privacy Data Quality Standard Conformance Metrics OpenAPI / GraphQL / AsyncAPI / gRPC Conformance Custom API Standard Conformance Change Management Metrics API Version Adoption & Retention Rate Deprecation Effectiveness Deprecation Usage API Version Views & Sunset Compliance SDK Version Usage Performance Metrics Latency Time Throughput Cache Hit Rate Resource Utilization Reliability and Availability Metrics Uptime Error Code Breakdown & Error Rate SLA Compliance Security Metrics OWASP Top 10 API Conformance Authentication & Authorization Rate Limiting Developer & Consumer Experience Metrics Engagement Rate Adoption Rate Retention & Churn Rate Documentation Quality and Coverage Rate Validation Error Rate 

9 · Ludovic Pourrat P u b l i c API Metrics Landscape Identified so far and not exhaustive Reliability and Availability Metrics Standard Conformance Metrics Data Governance Metrics Performance Metrics Change Management Metrics Security Metrics AI Readiness & Usability Metrics Developer Experience Metrics API Gateways API Gateways API Portals OTEL OTEL 

API Metrics Applied to APIOps 

11 · Ludovic Pourrat P u b l i c API Metrics applied to APIOps Design Develop Secure Deploy Publish Monitor Analyze Developer Experience Metrics API documentation via descriptions and attached Markdown files API conformance with the OpenAPI / GraphQL / AsyncAPI standard API conformance with your own API standard API Standard Conformance Metrics API samples explained with a business context Built-in and custom resources constraints definition and documentation API backward compatibility checks & API breaking changes API Security Score OWASP API Security Top 10 2023 Assessment Data Governance Metrics Data classification Data ownership Performance Metrics Resource allocations & limitations Refine Metrics Collections Review metrics accuracy Engage with new metric sensors Focus & Prioritization Review the topmost areas of improvement Data Governance Metrics Data Privacy Nothing Yet ? We discuss that later 

API Specification Overlay 

13 · Ludovic Pourrat P u b l i c Current State Solution Driver But we already got something we invested upon and that is well understood. Problem Statement It looks quite complex to compute all those metrics and to feed them back into your APIOps cycles. Market Readiness There is not yet a single solution or provider to gather all the API metrics you may need. 

14 · Ludovic Pourrat P u b l i c Overlaying API Specification with API Metrics API Overlay API Specification API Extensions The API overlay definition provide a common definition of how to apply modifications on the API specification. The API overlay is a method to patch the API specification to apply the API metrics. 3. API Overlay Allows extending the existing API definitions with custom metadata. The extensions are the placeholder for the API metrics. 2. API Extensions We start from what matter the most, so the API specification. 1. API Specification https://github.com/OAI/Overlay-Specification But we selected an easier approach by using the overlayed result instead. 

15 · Ludovic Pourrat P u b l i c API Specification Overlayed openapi: 3.1.0 info: […] termsOfService: www.lombardodier.com/terms-and-conditions.html contact: name: Lombard Odier url: 'https://www.lombardodier.com/home.html' email: [email protected] x-api-conformance-score: 80 paths: ‘/my-resource’: x-api-operation-latency: 20ms […] asyncapi: 2.6.0 info: […] termsOfService: www.lombardodier.com/terms-and-conditions.html contact: name: Lombard Odier url: 'https://www.lombardodier.com/home.html' email: [email protected] x-api-conformance-score: 40 channels: […] graphql: October 2021 info: […] termsOfService: www.lombardodier.com/terms-and-conditions.html contact: name: Lombard Odier url: 'https://www.lombardodier.com/home.html' email: [email protected] x-api-conformance-score: 60 spec: | type Query { […] } openapi: 3.1.0 info: x-api-conformance-score: 80 paths: ‘/my-resource’: x-api-operation-latency: 20ms Overlay with Extensions 1. OpenAPI 2. AsyncAPI 3. GraphQL 

API Overlay Computation 

17 · Ludovic Pourrat P u b l i c API Specification & API Mock | A Kubernetes Native Automation The API specification and the API mock definition are handled as Kubernetes custom resource definitions. Those can be also created at deployment for upper environments. API Artefacts as CRDs 

18 · Ludovic Pourrat P u b l i c API Specification & API Mock | A Kubernetes Native Automation The API specification is automatically deployed into our API developer portal and an API mock is instantiated within our API sandbox. Kubernetes Controllers 

19 · Ludovic Pourrat P u b l i c API Overlay Computation | A Kubernetes Extension The API overlay is ‘yet another’ Kubernetes custom resource definition. API Artefacts as CRDs We added a Kubernetes controller to compute the API metrics from the various sources and to generate the API overlay.. Kubernetes Controllers 

20 · Ludovic Pourrat P u b l i c API Overlay Computation | A Kubernetes Extension Then the overlay could extend the API specification definition and the API mock definition. Kubernetes Controllers 

21 · Ludovic Pourrat P u b l i c Finally, the API Developer Portal View Once collected the API overlay is published in the API developer portal. The API card can be flipped to display synthetics about the API metrics. API Card 

22 · Ludovic Pourrat P u b l i c API Overlay Overview From the source The API artefacts are collected from the source in Git. API Metrics Collection The API metrics are collected from the different sources. Enhance API Developer Portal The API metrics are visible directly from the API specification. API Overlay The API overlay describe the collected API metrics. API Sandbox The API mocks can be updated according to the API metrics. API Artefacts The API specification and the API mock definition are handled as CRDs. 

23 · Ludovic Pourrat P u b l i c Recap about the process 1 5 4 3 2 All the organization can leverage and benefits from the API metrics and ease decision making The API overlay can be fed into you APIOps cycles along your iterations The API overlay and the API extensions are a simple vehicle to carry-on API metrics The API metrics are computed from their sources The API specifications are the source of truth and ease any API related analysis API Metrics matter in APIOps As an outcome your APIOps cycles are enriched with valuable API metrics to foster better developer and consumer experiences. API Metrics in the API Portal We think that is the sweet spot to display back the collected API metrics. 

API Metrics | Maturity Matrix 

25 · Ludovic Pourrat P u b l i c Implement fundamental metrics to establish a baseline. Focus on performance metrics Level 1 - Basic Expand your metrics with cover usage, utilization, security and start documenting the APIs Level 2 - Intermediate Add the data governance metrics and gather the reliability and availability metrics Level 4 - Expert Dive deeper by adding conformance, consumer and developer experience metrics Level 3 - Advanced Continuously monitor and optimize the metrics. Explore AI consumption and integration metrics Level 5- Optimized API Metrics | Maturity Matrix The five levels of maturity 

26 · Ludovic Pourrat P u b l i c Q&A Wish you some h[API]ness at Apidays ! Better skiers have more fun. 

No content