Autonomous APIs (O’Reilly Software Architecture 2019)

https://conferences.oreilly.com/software-architecture/sa-eu/public/schedule/detail/79145

Services with their APIs are no longer living isolated in silos. Capabilities exposed by these microservices, service meshes, and serverless functions are creating the landscapes. The clients depend on multiple services and are often servers to other clients. These landscapes are forming at every level—team, application, organization, domain, and global—are starting to face out-of-the-bound complexity issues. The current approach to building such systems is imperative programming, human-produced and consumed interface documentation, minimal visibility, and limited automation (“Do you CI? Do you even CD?”). As such, it’s impossible to predict the effects of modifying even one component of the system. The task structure and size complexity is growing beyond our cognitive abilities, and yet we’re still trying to solve the problem by hiring more engineers.

With limited visibility, the capability discovery in these landscapes is limited to institutional knowledge, hand-crafted organizational Wiki pages, or Google searches on the domain and global level. Zdenek Nemec explores the problems with the complexity of forming API landscapes and proposes the autonomy of the components as the solution to the difficulty of managing components centrally. He defines levels of autonomy and examines the building blocks of autonomous systems such as automation, virtualization, uniform interface, bounded context, domain-driven design, declarative programming, semantic profiles, service discovery, and self-driving clients. He also gives special attention to the interface lifecycle, advertising future changes and subscription to it. He details the most important and currently missing piece—the service discovery (API producing and consuming standpoint, not the DevOps virtual machine (VM) and machine instance).

Transcript

1. goodapi.co Autonomous APIs The next chapter begins
2. goodapi.co supermodel.io
3. goodapi.co Pleasure to meet you! Networking encouraged Let’s spend a couple of minutes introducing ourselves to the people next to us, say what company or organization you are from, and what you are looking to learn from this session.
4. goodapi.co Autonomous Cars source: https://www.bmw.com/en/automotive-life/autonomous-driving.html
5. goodapi.co Autonomous APIs?
6. goodapi.co How can we have autonomous cars but not autonomous APIs?
7. goodapi.co API styles over time REST vs. GraphQL @zdne
8. goodapi.co Interest over time
9. goodapi.co BREAKTHROUGH ANNOUNCEMENT Paradigm Shift 1. CUSTOMER-SPECIFIC APIS 2. GENERIC APIS 3. HARMONIZED APIS 4. AUTONOMOUS APIS one-to-many one-to-one many-to-many machine-to-machine l C P l C P l C l C l C P l C l C P P l C P l C l C P P 3P
10. goodapi.co goodapi.co EVERY CHAPTER HAS ITS API ARCHITECTURAL STYLE–API – PARADIGM
11. goodapi.co Customer-specific APIs • Large organizations embracing IT • years 1990-2000s • SOAP, EDI, FTP & ESB • one-to-one, point-to-point 1. CUSTOMER-SPECIFIC APIS one-to-one l C P
12. goodapi.co Problems • API providers face issues with size complexity of point to point integrations and the ESB “failure” • New API consumers find the APIs too complex and cumbersome to use • New API consumers often ignored or misunderstood, and over/under served by SOAP API providers 1. CUSTOMER-SPECIFIC APIS one-to-one l C P
13. goodapi.co Generic APIs one-to-many: many mid- or small- size customers one-to-one: few large established customers l C P l C P l C l C
14. goodapi.co Generic APIs • Ubiquitous internet & software development • years 2005-2019+ • so-called-REST, later also GraphQL • one-to-many 2. GENERIC APIS one-to-many l C P l C l C
15. goodapi.co Problems • API providers are facing complexity issues from increasing number of own APIs, client integrations, server & client deployments and api versions • New consumers find complex and costly to integrate and maintain the integrations with multiple different apis • API consumers often ignored or misunderstood, and over/under served by Fragmented, siloed APIs and its providers 2. GENERIC APIS one-to-many l C P l C l C
16. goodapi.co Harmonized APIs many-to-many: clients consuming multiple APIs one-to-many: many mid- or small- size customers l C P l C l C l C P l C l C P P 3P
17. goodapi.co API Landscapes APIs are no longer in silos goodapi.co
18. goodapi.co HARMONIZED APIS one-to-many Organizations landscapes
19. goodapi.co HARMONIZED APIS one-to-many Organizations landscapes
20. goodapi.co HARMONIZED APIS B C D . . . If I am building an e-shop do I want to integrate with 400 logistics providers? e-shop Domain Landscapes
21. goodapi.co Harmonized APIs Examples
22. goodapi.co –Radek Novotny; CEO & founder MyStay “There are few integrators for hospitality APIs consolidating as many as 40 PMS but they are either not-duplex or slow in reactions” Challenge: Integration with many property management systems (PMS)
23. goodapi.co Accommoda tion B Accommoda tion C Accommoda tion D Accommoda tion E Existing Customer Existing Customer Existing Customer 3rd party New Type of Customer New Type of Customer New Type of Customer Logistics Metapack integrates over 400 carriers Over 100 3rd Party vendors
24. goodapi.co Client Client Client Client Client Client Hospitality & Travel Accommoda tion B Accommoda tion C Accommoda tion D Accommoda tion E
25. goodapi.co Accommoda tion B Accommoda tion C Accommoda tion D Accommoda tion E Existing Customer Existing Customer Existing Customer 3rd party New Type of Customer New Type of Customer New Type of Customer Banking APIS Accelerated by PSD2
26. goodapi.co Existing Customer Existing Customer 3rd party New Type of Customer New Type of Customer New Type of Customer Provider Provider Provider Uniform interface Uniform Interface to incumbents B C D e-shop 3rd party
27. goodapi.co goodapi.co How are incumbents loosing contact with their customers
28. goodapi.co Incumbent Existing Customer Existing Customer Existing Customer Facilitator New Type of Customer Ignored at the start Second wave of API customers Facilitator–Disruptor A 3rd party between API customer and API provider. Unrecognized or ignored it is either an external provider or shadow IT. Second Wave
29. goodapi.co Incumbent Existing Customer Existing Customer Existing Customer 3rd party New Type of Customer New Type of Customer New Type of Customer Second wave of API consumers Second Wave
30. goodapi.co Incumbent Existing Customer Existing Customer Existing Customer 3rd party New Type of Customer New Type of Customer New Type of Customer First chapter customers moving to 3rd Party
31. goodapi.co Unable to adapt to the incumbents are loosing their positions to disruptors
32. goodapi.co The fear of becoming commodity, lack of knowledge lack of understanding and inability to innovate leads to isolation and leaves incumbents once again vulnerable to third parties
33. goodapi.co Website A Client Client Client Client Client Client Are your customers really yours? Website B Website C Website D Website E
34. goodapi.co How to integrate with thousands of IoT devices from different vendors deployed in different environments?
35. goodapi.co IOT
36. goodapi.co goodapi.co Multiple Domains The situation is even worse when you consider…
37. goodapi.co Accommoda tion B Accommoda tion C Accommoda tion D Accommoda tion E API Consumer APIs Accommoda tion B Accommoda tion E consuming multiple APIs Cross-domain
38. goodapi.co Accommoda tion B Accommoda tion C Accommoda tion D Accommoda tion E Customer 3rd party consuming multiple APIs Cross-domain
39. goodapi.co Universe of capabilities This is what Amazon (Alexa skills), Google (Home skills) & VIV (Samsung) are doing. Database of knowledge and services with uniform interface
40. goodapi.co Harmonized APIs • Clients consuming multiple APIs • Micro-services, forming API landscapes • Year 2017+ • Complexity of consuming multiple different APIs is tackled by standardization, design governance to achieve consistency • Harmonization is also often done with “wrapper” APIs 3. HARMONIZED APIS many-to-many l C P l C l C P P 3P
41. goodapi.co Problems • Harmonized APIs providers are facing complexity issues trying to normalize, govern or maintaining many integrations APIs manually • Harmonization, standardization & governance complexity tackled by automation • Complexity external API consumers are facing when consuming multiple domain or cross-domain APIs is still left unchecked 3. HARMONIZED APIS many-to-many l C P l C l C P P 3P
42. goodapi.co Complexity
43. goodapi.co COMPLEXITY Task-structure Complexity How difficult it is understand how to perform a task in a distributed system Unpredictability How difficult is to predict effect of an action in distributed system, amount of entropy Size Complexity Size of the system implies cognitive complexity, large number of concepts and the knowledge required Chaotic Complexity Small variations in a certain part of the system can have large effects on overall system behavior Algorithmic Complexity Both traditional algorithm complexity in the term of time and space and cognitive complexity of understanding algorithm
44. goodapi.co COMPLEXITY API Designs x Versions x Deployment Hosts x Clients x Client Deployments IS HIGHER THAN TWO You are likely to face complexity using, maintaining and evolving your API 130 APIs 3 x 6 (environments x availability zones) 70 versions 1500 integrations own deployments: 3 x 6 on-premise deployments: 1000 apps: 3 000 000 IoT: ????? = 1,10565×10^9 IF THE NUMBER OF
45. goodapi.co Cost of Change 6,1% chance of breaking change every day (java library)
46. goodapi.co Solutions
47. goodapi.co SOLUTIONS TO COMPLEXITY 1. Hire more people 2. Standardization 3. Automation 4. Autonomy of the components
48. goodapi.co Solutions to complexity 1. Hire More People Mechanical turks, eyeballing documentation for changes in design, fixing clients and implementations accordingly. CUSTOMER-SPECIFIC & GENERIC APIS ROLE OF HUMANS IN APIS
49. goodapi.co MACHINE EXPOSES INTERFACE ANOTHER MACHINE LEARNS & USES INTERFACE MACHINES MEET OVER THE NETWORK discovery What we think API is Hire more people
50. goodapi.co Hire more people Reality MACHINE EXPOSES INTERFACE ANOTHER HUMAN READS DOCUMENTATION HUMAN WRITES DOCUMENTATION ANOTHER MACHINE IS TAUGHT TO USE INTERFACE HUMANS MEET discovery
51. goodapi.co Hire more people Reality HUMANS INSIDE VS. • time and money spent • not scaling • chance of misinterpretation • error-prone
52. goodapi.co SOLUTIONS TO COMPLEXITY 2. Standardization HARMONIZED APIS API DESIGN, LIFECYCLE & DEVELOPMENT STANDARDIZATION
53. goodapi.co Standardization Product consistency
54. goodapi.co Standardization API Guidelines & design checks
55. goodapi.co 3. Automation AUTOMATED HARMONIZED APIS Solutions to complexity AUTOMATED CONTRACT-DRIVEN API LIFECYCLE
56. goodapi.co Automation INTERFACE ACTUALLY EXPOSED INTERFACE DOCUMENTED INTERFACED USED are they the same? are they the same? are they the same? Keeping things in check is hard
57. goodapi.co Contract-driven Automation
58. goodapi.co https://goodapi.co/api-lifecycle Automation Entire lifecycle
59. goodapi.co Tools are our limits
60. goodapi.co Tools are our limits • Existing investments (tools, infrastructure & processes) • Bloated API Management Tools • Missing automation • Lifecycle orchestration • Limiting API description formats • Data modeling from the past • Missing tooling for structured data
61. goodapi.co 4. Autonomy AUTONOMOUS APIS AUTONOMY IS THE SOLUTION TO THE DIFFICULTY OF MANAGING MANY COMPONENTS CENTRALLY Components autonomy in an API Landscape reduces the complexity, enables scaling, and promotes the service commoditization. Solutions to complexity
62. goodapi.co Autonomy “Autonomy: freedom from external control or influence; independence; sovereignty in making decisions and changes.” –New Oxford American Dictionary
63. goodapi.co Attacking the complexity with autonomy Self-driving Clients Clients can self- navigate the API landscape, perform self-configuration, self-repair, self- optimization and make choices at various decisions points automatically High-level, declarative programming Reducing the cognitive algorithmic complexity, decoupling from components implementation, programming for domain, not service Automated, contract-driven API lifecycle Handling API provider complexity by keeping, versions, implementations and deployments in check Hierarchical and context-bound knowledge Divide and conquer size complexity with domain definitions and bounded contexts. Reusable representation of domain knowledge– ontologies– published language
64. goodapi.co Autonomous APIs • Machine-to-machine APIs, self-driving, no human intervention, cross-domain landscapes • Rise of IoT & AI • Year 2020+ 4. AUTONOMOUS APIS machine-to-machine l C P l C l C P P
65. goodapi.co Autonomous APIs 1. Consuming more than one API 2. Cross-domain 3. Demanding reduction of complexity 4. Self-driving in API Landscape 5. Autonomous-decoupled from particular API provider NEXT-GEN CONSUMERS
66. goodapi.co Navigation through autonomic units Countries, Regions, Cities, City Quarters are autonomous regions but when you are driving from A to B you need to navigate through their roads A B Design-time & Runtime Navigation You need both to follow a map and traffic signs to navigate effectively Autonomous APIs Navigate safely in API landscape
67. goodapi.co 5 Levels of Autonomy goodapi.co
68. goodapi.co Levels of autonomy –Autonomous Client “The level of autonomy is the ability to safely navigate in the API landscape (where the constraints are time and price)”
69. goodapi.co Levels of Autonomy Autonomy Autonomous APIs Brittle System Documented Brittle System Automated System Decoupled Automated System Autonomous System 1 2 3 4 5
70. goodapi.co Level 1: Brittle system • Tightly coupled components • No semantics, interface or lifecycle data • Pros • Time to delivery for point to point integrations when complexity is low • Cons • Everything breaks when something changes • Not scaling with growing complexity Level 1
71. goodapi.co Level 2: Documented Brittle System • Tightly coupled components • Interface metadata available • Human only-semantics metadata available • No lifecycle metadata (is the documentation documenting the interface implementation?) • Pros • Possible API productization • Reusability • Cons • Lock on changes, unable to evolve Level 2
72. goodapi.co Level 3: Automated system • Mostly tightly coupled components • Metadata are verified, valid and reliable • Metadata are human or machine readable • Advertisement of the interface, semantics and lifecycle metadata • Client subscription to future changes • Pros • Explicit understanding of price and risk of making changes in the API landscape • Quality Control • Cons • Advanced mindset change involving many people • Cultural denial, hard to get buy-in Level 3
73. goodapi.co Level 4: Decoupled automated system • Uniform Interface • Decoupled components • Evolvability of the components in their respective environments • Share understanding-metadata at RUNTIME • Pros • Harmonized within the system • Reducing price and risk of making changes (by design) • Cons • No machine autonomy • Difficult taking everything into account, prone to chaotic & size complexity Level 4
74. goodapi.co Building blocks Level 4 Decoupled Automated System 4
75. goodapi.co Building block 1/3 Uniform Interface A consistent, predictable way to query data and exercise actions regardless of the service provider
76. goodapi.co Uniform interface 1957 2018 Engineering principle of generality
77. goodapi.co Engineering Principle of Generality
78. goodapi.co Uniform interface Engineering principle of generality Existing Customer Existing Customer 3rd party New Type of Customer New Type of Customer New Type of Customer Provider Provider Provider Uniform Interface to incumbents B C D e-shop 3rd party
79. goodapi.co Benefits of UI Engineering principle of generality • Simplifies architecture • Improves visibility of interactions • Decouple components, enables evolution • Improves interoperability, reduces cost of integration • Reduces costs and time to market • Enables autonomous service discovery • ENABLES FOR GENERAL AI • Automatic clients • Autonomous clients
80. goodapi.co Cross-domain Uniform interface Accommoda tion B Accommoda tion C Accommoda tion D Accommoda tion E Customer Customer Customer 3rd party Customer Customer Customer Uniform Interface
81. goodapi.co Path To Uniform interface 1. Apply Modern Architectural Styles (REST) and Linked Data (JSON-LD) 2. “Wrapper APIs”, build or leave it up to 3rd Party Vendors (Metapack, Amazon, Google..) to become the intermediaries to the universe 3. Autonomous APIs (coming later)
82. goodapi.co Do you want to provide an uniform interface or leave it to third-party vendor to do it? Client Client Client Client Client Client Provider Provider Provider Client Client Client Linked Data & Architecural Style Client Client Client Provider Provider Provider - OR - Building your next API
83. goodapi.co Building block 2/3 Decoupled components A component does not duplicate or rely on any other component internal logic neither it relies on any other component information shared ahead of time Abstraction Layer Comp-A Comp-B
84. goodapi.co API TASKS IS TO ABSTRACT AWAY THE INTERNALS OF SERVICE COMPONENT
85. goodapi.co DEGRADED EFFICIENCY UI Transformation Comp-A Comp-B Transformation The Tradeoff of Decoupled components and Uniform interface
86. goodapi.co Building block 3/3 Understanding shared at runtime • Available HATEOAS state transitions • HAL, Siren, RESTful JSON • Non-authoritative parameter validations • JSON Schema, Code on Demand • API Description - map • OpenAPI Specification, API Blueprint … • GraphQL Schema
87. goodapi.co Level 5: Autonomous System • Self driving clients • Landscape discovery • Machine understanding • Vocabulary not data structure • Declarative programming • Autonomy of the components • Leveraging learned data from simulation for infrastructure • Pros • Higher level of abstraction • Implicit harmonization • Full autonomy of components • Independent evolution • Cons • Cognitive (Algorithmic) Complexity • Demanding on computational power • Higher level of abstraction Level 5
88. goodapi.co Building blocks Level 5 Autononomous System 5
89. goodapi.co Building block 1/3 Declarative programming // HTTP Request GET /v1/balance HTTP/1.1 Host: api.stripe.com Authorization: Basic c2tfdGVzdF9CUW9raWtKT3ZCaUkySGxXZ0g0b2xmUTI // Imperative, hard-coded fetch(‘http://api.stripe.com/v1/balance‘, { ‘Authorization': 'Basic ' + base64.encode(username + ":" + password) } ).then.. // Declarative, decoupled if (hasAffordance(‘retrieve-balance’) { exerciseAffordance(‘retrieve-balance’).then.. }
90. goodapi.co Building block 2/3 Machine understanding Protocol Message Application HTTP Protocol Semantics Message Format Semantics (e.g. HAL) Application Domain Semantics (e.g. Hospitality domain) Program for vocabulary not data structure
91. goodapi.co Data models & vocabularies, ontologies
92. goodapi.co Canonical data model is anti-pattern goodapi.co in modern systems architecture
93. goodapi.co Domain-Driven Design Divide and conquer complexity by bounded context and context mapping images from Domain-Driven-Distilled
94. goodapi.co Context Mapping images from Domain-Driven-Distilled
95. goodapi.co Linked data - semantics for Humans & machines
96. goodapi.co –Autonomous Client “I am looking for a person, but I see a user, should I treat it as person?”
97. goodapi.co Shared vocabularies
98. goodapi.co https://supermodel.io
99. goodapi.co Program for vocabulary NOT Data Structure
100. goodapi.co Program for Vocabulary // Programming for data structure display todo.title // Programming for vocabulary if (todo.has('title')) { display todo(‘title') }
101. goodapi.co JSON-LD (linked data) { "@context": "https://json-ld.org/contexts/person.jsonld", "@id": "http://dbpedia.org/resource/John_Lennon", "name": "John Lennon", "born": "1940-10-09", "spouse": "http://dbpedia.org/resource/Cynthia_Lennon" } https://json-ld.org
102. goodapi.co Hydra JSON-LD meets HATEOAS http://www.markus-lanthaler.com/hydra/console/
103. goodapi.co ALPS Profile Application-Level Profile Semantics (ALPS) http://alps.io
104. goodapi.co Building block 3/3 Autonomous service discovery MACHINE EXPOSES INTERFACE ANOTHER MACHINE LEARNS & USES INTERFACE MACHINES MEET OVER THE NETWORK discovery
105. goodapi.co Yahoos of API World Global Landscapes • Rapid API • Any API • APIS.json • Hitch HQ (acquired) • GraphQL Hub Internal Landscapes • Internal Developer Portals • API Repositories (Apiary, SwaggerHub) Human-only, hand picked
106. goodapi.co Micro service discovery • Netflix Zulu / Eureka • Apache / Kafka / Zookeper • Hashicorp / Consul • Linkerd No API Landscape Metadata discovery Services
107. goodapi.co But these are not enough We need publish-subscribe model for API Landscape and API discovery
108. goodapi.co Autonomous service discovery “Registry, look for a service that knows weather in Paris.” “Registry, look for a service that can fulfill the shipment of 6 pallets from Prague to Paris.” “Registry, look for a service that can play every movie by the director of movie Avatar.”
109. goodapi.co Autonomous service discovery // Using a terms from schema.org dictionary, // find services that offers WeatherForecast. services = apiLandscape.find(WeatherForecast, { vocabulary: http:// schema.org}) // Query a service for WeatherForecast at GeoCoordinates. forecast = service.retrieve(WeatherForecast, { GeoCoordinates: … }) // Display Temperature display forecast(Temperature)
110. goodapi.co Building blocks of Autonomous APIs Level 5 Autonomous System Level 4 Decoupled Automated System • Declarative programming • Machine understanding • Autonomous discovery 4 3 • Uniform Interface • Decoupled components • Autonomous navigation
111. goodapi.co goodapi.co Early four-engine jets required flight engineers. Later four-engine jets were designed with sufficient automation to eliminate the position. 747 FLIGHT ENGINEERS ...this will also happen in the world of APIs.
112. goodapi.co
113. goodapi.co API Chapters one-to-many one-to-one many-to-many machine-to-machine Drivers behind the changes API CONSUMERS EXPECTATIONS COMPLEXITY task-structure unpredictability chaotic size algorithmic + GROWING # OF API CONSUMERS + INCREASING DEMAND EVOLVING DEMAND INCREASING PROBLEMS
114. goodapi.co 1. CUSTOMER-SPECIFIC APIS 2. GENERIC APIS 3. HARMONIZED APIS 4. AUTONOMOUS APIS one-to-many one-to-one many-to-many machine-to-machine API Chapters
115. goodapi.co Digital wholesale presence & service commoditization “Look for a service that can fulfill the shipment of 6 pallets from Prague to Paris under 100 EUR.” “Look for a service that provides information about adidas stan smith” “Is there an air condition service in my landscape?” “Look for a service that has gs1:Offer where gs1:itemOffered gs1:gpcCategoryCode is MILK.” “The level of autonomy is the ability to safely navigate in the API landscape (where the constraints are time and price)”
116. superface.io Commoditization CUSTOMER SPECIFIC APIS (SOAP) GENERIC APIS (SO-CALLED-REST) HARMONIZED APIS AUTONOMOUS APIS HARMONIZED APIS (AUTOMATED) Graph by Simon Wardley, “How commodity is something?”
117. goodapi.co
118. IMAGES BY UNSPLASH.COM ICONS BY MARTINFOWLER.COM
119. goodapi.co What would MyStay do? • Define Hospitality Domain • Define Hotel Room subdomain • Train MyStay Mobile app for the subdomain if (room.roomService.available) display room.roomService.offer if (room.ac.available) display acControls • Deploy mobile clients • Provide IoT integration components-hubs for hotels