Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Autonomous APIs

Z
September 25, 2018

Autonomous APIs

Slides used during my talk at API Conference Berlin 2018 https://apiconference.net/api-business-strategy/autonomous-apis/

Abstract:

The exponential growth of APIs, further accelerated by IoT, micro-services, and FaaS, brought many challenges both technical and economical. On the one hand, businesses are struggling with establishing successful API programs to connect with their customers. On the other side, engineering teams are facing out-of-the-scale complexity.

In this talk, we will explore the problems and shortcomings of current APIs, both with regards to the technical aspects and the API economy. We will discuss the possible solutions to these problems, and introduce Autonomous APIs.
In the second part, we will take a look at what Autonomous APIs are and how they can solve the problems for both businesses and engineering. We will introduce the classification of autonomous systems and take a peek at the building blocks of autonomous systems.

Z

September 25, 2018
Tweet

More Decks by Z

Other Decks in Programming

Transcript

  1. goodapi.co APIS: THE FIRST 30 YEARS CUSTOMER-SPECIFIC APIS one-to-one First

    Wave (2000) GENERIC APIS one-to-many Second Wave (2010) HARMONIZED APIS many-to-one, machine-to-machine Third Wave (2020) The Pattern 1990 2000 2010 2020
  2. goodapi.co FIRST WAVE COMPLEXITY FIRST-WAVE PROVIDERS FACING COMPLEXITY ISSUES FROM

    INCREASING NUMBER OF POINT TO POINT INTEGRATIONS AND/OR THE ESB SINGLE POINT OF FAILURE AND INCREASING DEPENDENCIES
  3. goodapi.co FIRST TO SECOND Two Groups of Customers–API Consumers CUSTOMER-

    SPECIFIC APIS GENERIC APIS one-to-many: many mid- or small- size customers one-to-one: few large established customers First Wave (2000) Second Wave (2010)
  4. goodapi.co Fueled by the success, scalability and performance of the

    most successful API to date – Web SO-CALLED-REST & GRAPHQL SECOND WAVE (GENERIC APIS) SECOND WAVE ARCHITECTURE
  5. goodapi.co THE (FORTUNE 500) WORLD STILL RUNS ON SOAP, EDI

    OR FTP, NOT REST …but the the third wave is already coming… However…
  6. goodapi.co SECOND WAVE COMPLEXITY SECOND-WAVE PROVIDERS ARE FACING COMPLEXITY ISSUES

    FROM INCREASING NUMBER OF CLIENT INTEGRATIONS, NUMBER OF HOST AND CLIENT DEPLOYMENTS AND NUMBER OF API VERSIONS
  7. goodapi.co NEW CONSUMERS FIND COMPLEX AND COSTLY TO INTEGRATE AND

    MAINTAIN THE INTEGRATIONS WITH MANY SECOND-WAVE APIS
  8. goodapi.co THERE IS TOO MANY APIS TO INTEGRATE WITH JUST

    ONE! B C D . . . e-shop IF I AM BUILDING AN E-SHOP DO I WANT TO INTEGRATE WITH 400 LOGISTICS PROVIDERS? THE NUMBER OF APIS IS GROWING STRONG
  9. goodapi.co –Radek Novotny; CEO & founder MyStay “There are few

    integrators for hospitality APIs consolidating as many as 40 PMS but they are either not- duplex or slow in reactions” CHALLENGE: INTEGRATION WITH MANY PROPERTY MANAGEMENT SYSTEMS (PMS)
  10. goodapi.co BUT 40+ PMS INTEGRATIONS ARE JUST ONE PART OF

    MYSTAY PROBLEMS. THE BIGGER ONE IS: HOW TO INTEGRATE WITH THOUSANDS OF “SMART HOSPITALITY” IOT DEVICES FROM DIFFERENT VENDORS AND IN DIFFERENT HOTEL CHAINS?
  11. goodapi.co MANY APIS, CROSS-DOMAIN APIS Accommoda tion B Accommoda tion

    C Accommoda tion D Accommoda tion E API Consumer APIs Accommoda tion B Accommoda tion E
  12. goodapi.co UNIVERSE OF CAPABILITIES This is what Amazon (Alexa skills),

    Google (Home skills) & VIV are doing… Database of knowledge and services with uniform interface
  13. goodapi.co THIRD WAVE Primary Drivers Secondary Drivers Non-drivers 1. Consuming

    more than one API 2. Cross-domain 3. Demanding reduction of complexity 4. Self-driving 5. Autonomous-decoupled from particular provider NEXT-GEN CUSTOMERS Partnership through discoverability Innovation driven by AI & IoT
  14. goodapi.co UNIFORM INTERFACE, DECOUPLING, KNOWLEDGE SHARED AT RUNTIME, DECLARATIVE PROGRAMMING,

    AUTOMATION, AUTONOMY THIRD WAVE (AUTONOMOUS APIS) THIRD WAVE ARCHITECTURE
  15. goodapi.co THE THIRD WAVE COMING… CUSTOMER-SPECIFIC APIS one-to-one: few large

    established customers First Wave (2000) GENERIC APIS one-to-many: many mid- or small- size customers Second Wave (2010) AUTONOMOUS APIS many-to-one & machine-to-machine: automatic, later autonomous APIs Third Wave (2020)
  16. goodapi.co Incumbent Existing Customer Existing Customer Existing Customer FIRST WAVE

    First wave of API customers Point-to-point integrations one-to-one: few large established customers
  17. goodapi.co Incumbent Existing Customer Existing Customer Existing Customer New Type

    of Customer Complex, legacy, point-to- point API interfaces difficult to use for new customers one(API)-to- many(consumers): many mid- or small- size customers SECOND WAVE Second wave of API customers
  18. goodapi.co Incumbent Existing Customer Existing Customer Existing Customer Facilitator New

    Type of Customer Ignored at the start Second wave of API customers Facilitator - Disruptor A 3rd party between API customer and API provider. Unrecognized or ignored it is either an external provider or shadow IT. SECOND WAVE
  19. goodapi.co Incumbent Existing Customer Existing Customer Existing Customer 3rd party

    New Type of Customer New Type of Customer New Type of Customer Second wave of API consumers SECOND WAVE
  20. goodapi.co Incumbent Existing Customer Existing Customer Existing Customer 3rd party

    New Type of Customer New Type of Customer New Type of Customer FIRST WAVE CUSTOMERS MOVING TO 3RD PARTY
  21. goodapi.co Incumbent Existing Customer Existing Customer Existing Customer 3rd party

    New Type of Customer New Type of Customer New Type of Customer 2ND WAVE (GENERIC APIS) First wave customers moving to 3rd Party one-to-many: many mid- or small- size customers
  22. goodapi.co Incumbent Existing Customer Existing Customer Existing Customer 3rd party

    New Type of Customer New Type of Customer New Type of Customer 3RD WAVE (HARMONIZED APIS) 3rd party providing integrations with many providers Incumbent Incumbent Incumbent Incumbent many-to-one & machine-to-machine: automatic, later autonomous APIs
  23. goodapi.co THE FEAR OF BECOMING COMMODITY, LACK OF KNOWLEDGE LACK

    OF UNDERSTANDING AND INABILITY TO INNOVATE LEADS TO ISOLATION AND LEAVES INCUMBENTS ONCE AGAIN VULNERABLE TO THIRD PARTIES
  24. goodapi.co Client Client Client Client Client Client HOSPITALITY Accommoda tion

    B Accommoda tion C Accommoda tion D Accommoda tion E
  25. goodapi.co Accommoda tion B Accommoda tion C Accommoda tion D

    Accommoda tion E Existing Customer Existing Customer Existing Customer 3rd party New Type of Customer New Type of Customer New Type of Customer BANKING APIS Accelerated by PSD2
  26. goodapi.co Accommoda tion B Accommoda tion C Accommoda tion D

    Accommoda tion E Existing Customer Existing Customer Existing Customer 3rd party New Type of Customer New Type of Customer New Type of Customer LOGISTICS Metapack integrates over 400 carriers Over 100 (!!!) 3rd Party vendors
  27. goodapi.co Existing Customer Existing Customer 3rd party New Type of

    Customer New Type of Customer New Type of Customer Provider Provider Provider UNIFORM INTERFACE Uniform Interface to incumbents B C D e-shop 3rd party
  28. goodapi.co THERE IS TOO MANY APIS TO INTEGRATE WITH JUST

    ONE! B C D e-shop B C D e-shop 3rd party VS. IF I AM BUILDING AN E-SHOP DO I WANT TO INTEGRATE WITH 400 LOGISTICS PROVIDERS? THE NUMBER OF APIS IS GROWING STRONG
  29. goodapi.co Website A Client Client Client Client Client Client ARE

    YOUR CUSTOMERS REALLY YOURS? Website B Website C Website D Website E
  30. goodapi.co COMPLEXITY OF DISTRIBUTED COMPUTING SYSTEMS Task-structure Complexity How difficult

    it is understand how to perform a task in a distributed system Unpredictability How difficult is to predict effect of an action in distributed system, amount of entropy Size Complexity Size of the system implies cognitive complexity, large number of concepts and the knowledge required Chaotic Complexity Small variations in a certain part of the system can have large effects on overall system behavior Algorithmic Complexity Both traditional algorithm complexity in the term of time and space and cognitive complexity of understanding algorithm
  31. goodapi.co COMPLEXITY EXERCISE 5. # of client deployments Different lifecycle

    of the deployment • Desktop app • Web app • Mobile app • Backend • Scripts / CLI 3. # of API implementations / deployments • Language • Provider • Protocol • API architectural style 1. # of API (designs) for a domain 2. # of versions of one API stripe.com API has over 70 versions 4. # of client integrations Single consumer / API key
  32. goodapi.co API Designs x Versions x Deployment Hosts x Clients

    x Client Deployments IF THE NUMBER OF IS HIGHER THAN TWO You are likely to face complexity using, maintaining and evolving your API 130 APIs 3 x 6 (environments x availability zones) 70 versions 1500 integrations own deployments: 3 x 6 on-premise deployments: 1000 apps: 3 000 000 IoT: ????? = 1,10565×10^9
  33. goodapi.co 1. Hire more people 2. Standardization 3. Apply massive

    automation 4. Start building distributed systems differently– change the approach–autonomy of the components in API landscape SOLUTIONS TO COMPLEXITY
  34. goodapi.co ROLE OF HUMANS IN APIS 1. Hire More People

    Mechanical turks, eyeballing documentation for changes in design, fixing clients and implementations accordingly.
  35. goodapi.co WHAT WE THINK AN API IS MACHINE EXPOSES INTERFACE

    ANOTHER MACHINE LEARNS & USES INTERFACE MACHINES MEET OVER THE NETWORK discovery
  36. goodapi.co REALITY MACHINE EXPOSES INTERFACE ANOTHER HUMAN READS DOCUMENTATION HUMAN

    WRITES DOCUMENTATION ANOTHER MACHINE IS TAUGHT TO USE INTERFACE HUMANS MEET discovery ! !
  37. goodapi.co REALITY ! HUMANS INSIDE VS. • time and money

    spent • not scaling • chance of misinterpretation • error-prone
  38. API IS BY- PRODUCT API DOCUMENTATION GENERATED FROM CODE API

    FIRST API TREATED AS PRODUCT API CONSISTENCY API MATURITY
  39. goodapi.co KEEPING THINGS IN SYNC IS HARD INTERFACE ACTUALLY EXPOSED

    INTERFACE DOCUMENTED INTERFACED USED are they the same? are they the same? are they the same?
  40. goodapi.co TOOLS ARE OUR LIMITS • Existing investments (tools, infrastructure

    & processes) • Bloated API Management Tools • Missing automation • Lifecycle orchestration • Limiting API description formats • Data modeling from the past • Missing tooling for structured data
  41. goodapi.co Components autonomy in an API Landscape reduces the complexity,

    enables scaling, and promotes the service commoditization. Autonomy is the solution to the difficulty of managing many components centrally. 4. Change the approach AUTONOMOUS APIS “Autonomy: freedom from external control or influence; independence; sovereignty in making decisions and changes.” –New Oxford American Dictionary
  42. goodapi.co MANY APIS, CROSS-DOMAIN APIS Accommoda tion B Accommoda tion

    C Accommoda tion D Accommoda tion E API Consumer APIs Accommoda tion B Accommoda tion E
  43. goodapi.co THE GOAL IS TO NAVIGATE SAFELY IN API LANDSCAPE

    Navigation through autonomic units Countries, Regions, Cities, City Quarters are autonomous regions but when you are driving from A to B you need to navigate through their roads A B Design-time & Runtime Navigation You need both to follow a map and traffic signs to navigate effectively
  44. goodapi.co –Autonomous Client “The level of autonomy is the ability

    to safely navigate in the API landscape (where the only constraints are time and price)”
  45. goodapi.co LEVEL 0 • Tightly coupled components • No semantics,

    interface or lifecycle data • Pros • Time to delivery for point to point integrations when complexity is low • Cons • Everything breaks when something changes • Not scaling with growing complexity Brittle System
  46. goodapi.co LEVEL 1 • Tightly coupled components • Interface metadata

    available • Human only-semantics metadata available • No lifecycle metadata (is the documentation documenting the interface implementation?) • Pros • Possible API productization • Reusability • Cons • Lock on changes, unable to evolve Documented Brittle System
  47. goodapi.co LEVEL 2 • Mostly tightly coupled components • Metadata

    are verified, valid and reliable • Metadata are human or machine readable • Advertisement of the interface, semantics and lifecycle metadata • Client subscription to future changes • Pros • Explicit understanding of price and risk of making changes in the API landscape • Quality Control • Cons • Advanced mindset change involving many people • Cultural denial, hard to get buy-in Automated System
  48. goodapi.co LEVEL 3 • Decoupled components • Evolvability of the

    components in their respective environments • Share understanding-metadata at RUNTIME • Pros • Reducing price and risk of making changes (by design) • Cons • Difficult taking everything into account, prone to chaotic & size complexity Decoupled Automated System
  49. goodapi.co LEVEL 4 • Self driving clients • Autonomy of

    the components • Leveraging learned data from simulation for infrastructure • Pros • Higher level of abstraction • Uniform interface • Cons • Cognitive (Algorithmic) Complexity • Demanding on computational power • Higher level of abstraction Autonomous System
  50. goodapi.co LEVELS OF AUTONOMY Brittle System Documented Brittle System Automated

    System Decoupled Automated System Autonomous System Autonomy Autonomous APIs 0 1 2 3 4
  51. goodapi.co BUILDING BLOCKS OF AUTONOMOUS APIS • Uniform Interface •

    Decoupled components • API understanding shared at run-time Level 4 Autonomous System Level 3 Decoupled Automated System • Client components programmed declaratively - self driving clients • Machine understanding • API Landscape understanding shared at run-time 4 3
  52. goodapi.co goodapi.co Early four-engine jets required flight engineers. Later four-engine

    jets were designed with sufficient automation to eliminate the position. 747 FLIGHT ENGINEERS ...this will also happen in the world of APIs.
  53. goodapi.co A CONSISTENT, PREDICTABLE WAY TO QUERY DATA AND EXERCISE

    ACTIONS REGARDLESS OF THE SERVICE PROVIDER
  54. goodapi.co Existing Customer Existing Customer 3rd party New Type of

    Customer New Type of Customer New Type of Customer Provider Provider Provider UNIFORM INTERFACE Uniform Interface to incumbents B C D e-shop 3rd party
  55. goodapi.co UNIFORM INTERFACE HAS MANY BENEFITS • Simplifies architecture •

    Improves visibility of interactions • Decouple components, enables evolution • Improves interoperability, reduces cost of integration • Reduces costs and time to market • Enables autonomous service discovery • ENABLES FOR GENERAL AI • Automatic clients • Autonomous clients
  56. goodapi.co CROSS-DOMAIN UNIFORM INTERFACE Accommoda tion B Accommoda tion C

    Accommoda tion D Accommoda tion E Customer Customer Customer 3rd party Customer Customer Customer Uniform Interface
  57. goodapi.co PATH TO UNIFORM INTERFACE 1. Apply Modern Architectural Styles

    (REST) and Linked Data (JSON-LD)
 
 2. Leave it up to 3rd Party Vendors (Amazon, Google..) to become the intermediaries to the universe
  58. goodapi.co Do you want to provide an uniform interface or

    leave it to third-party vendor to do it? BUILDING YOUR NEXT API Client Client Client Client Client Client Provider Provider Provider Client Client Client Linked Data & Architecural Style Client Client Client Provider Provider Provider - OR -
  59. goodapi.co A COMPONENT DOES NOT DUPLICATE OR RELY ON ANY

    OTHER COMPONENT INTERNAL LOGIC NEITHER IT RELIES ON ANY OTHER COMPONENT INFORMATION SHARED AHEAD OF TIME
  60. goodapi.co API UNDERSTANDING SHARED AT RUN-TIME • Available HATEOAS state

    transitions • HAL, Siren, RESTful JSON • Non-authoritative parameter validations • JSON Schema, CoD • API Description - map • OpenAPI Specification, API Blueprint … • GraphQL Schema
  61. goodapi.co BUILDING BLOCKS OF AUTONOMOUS APIS • Uniform Interface •

    Decoupled components • API understanding shared at run-time Level 3 Decoupled Automated System 3
  62. goodapi.co DECLARATIVE PROGRAMMING // HTTP Request GET /v1/balance HTTP/1.1 Host:

    api.stripe.com Authorization: Basic c2tfdGVzdF9CUW9raWtKT3ZCaUkySGxXZ0g0b2xmUTI // Imperative, hard-coded fetch(‘http://api.stripe.com/v1/balance‘, { ‘Authorization': 'Basic ' + base64.encode(username + ":" + password) } ).then.. // Declarative, decoupled if (hasAffordance(‘retrieve-balance’) { exerciseAffordance(‘retrieve-balance’).then.. }
  63. goodapi.co API SEMANTIC LAYERS Protocol Message Application HTTP Protocol Semantics

    HAL Semantics Application Domain Semantics (e.g. Hospitality domain)
  64. goodapi.co CANONICAL DATA MODEL IS ANTI-PATTERN IN THE WORLD OF

    MICROSERVICES & FAAS Use concern separation, bounded context, DDD techniques
  65. goodapi.co DOMAIN-DRIVEN DESIGN Divide and conquer complexity by bounded context

    and context mapping images from Domain-Driven-Distilled
  66. goodapi.co –Autonomous Client “I am looking for a person, but

    I see an user, should I treat it as person?”
  67. goodapi.co CLIENT EXAMPLE // Programming for data structure if (todo.title)

    { display todo.title } // Programming for vocabulary if (todo.has(‘schemaorg:title’)) { display todo(‘schemaorg:title’) }
  68. goodapi.co • Rapid API • Any API • APIS.json •

    Hitch HQ (acquired) • GraphQL Hub • … YAHOOS OF API WORLD • Internal Developer Portals • API Repositories (Apiary, SwaggerHub) Human-only, hand picked GLOBAL COMPANY-WIDE
  69. goodapi.co • Netflix Zulu / Eureka • Apache / Kafka

    / Zookeper • Hashicorp / Consul • Linkerd MICROSERVICE DISCOVERY NO SEMANTICS METADATA DISCOVERY SERVICES
  70. goodapi.co BUT THESE ARE NOT ENOUGH WE NEED PUBLISH-SUBSCRIBE MODEL

    FOR MACHINE-READABLE SEMANTICS, INTERFACE AND LIFECYCLE METADATA
  71. goodapi.co AUTONOMOUS SERVICE DISCOVERY “Get me a service that knows

    weather in Paris.” “Get me a service that can fulfill the parcel logistics of 6 pallets from Prague to Paris.” “Get me a service that can play every movies by the director of movie Avatar.”
  72. goodapi.co SERVICE DISCOVERY & DECLARATIVE CLIENT PROGRAMMING // Using a

    terms from schema.org dictionary, // find services that offers WeatherForecast. services = apiLandscape.find(WeatherForecast, { vocabulary: http:// schema.org}) // Query a service for WeatherForecast at GeoCoordinates. forecast = service.retrieve(WeatherForecast, { GeoCoordinates: … }) // Display Temperature display forecast(Temperature)
  73. goodapi.co WHOLESALE PRESENCE & COMMODITIZATION Look for a service that

    has gs1:Offer where gs1:itemOffered gs1:gpcCategoryCode is MILK
  74. goodapi.co apiLandscape.do(“Refill paper A4 100pcs in the printer on the

    2nd floor”, “$10”, “24h”) => false => Promise NLP, DISCOVERY AND DECLARATIVE PROGRAMMING
  75. goodapi.co BUILDING BLOCKS OF AUTONOMOUS APIS Level 4 Autonomous System

    4 • Client components programmed declaratively - self driving clients • Machine understanding • API Landscape understanding shared at run-time
  76. goodapi.co ATTACKING THE COMPLEXITY WITH AUTONOMY Self-driving Clients Clients can

    self- navigate the API landscape, perform self- configuration, self- repair, self- optimization and make choices at various decisions points automatically High-level, declarative programming Reducing the cognitive algorithmic complexity, decoupling from components implementation, programming for domain, not service Automated, contract-driven API lifecycle Handling API provider complexity by keeping, versions, implementations and deployments in check Hierarchical and context-bound knowledge Divide and conquer size complexity with domain definitions and bounded contexts. Reusable representation of domain knowledge– ontologies– published language
  77. goodapi.co WHAT WOULD MYSTAY DO? • Define Hospitality Domain •

    Define Hotel Room subdomain • “Train" MyStay Mobile app for the subdomain 
 
 if (room.roomService.available)
 display room.roomService.offer
 
 if (room.ac.available)
 display acControls
 • Deploy mobile clients • Provide IoT integration components-hubs for hotels
  78. goodapi.co AUTONOMOUS APIS • Self-driving clients • Navigation in landscape

    • Self-configuration • Resilience & Self-repair • Self-optimization • Self-decision making • High-level, declarative programming of the clients • Hierarchical and context-bound knowledge • Automated, contract driven API lifecycle
  79. goodapi.co COMPLEXITY OF DISTRIBUTED COMPUTING SYSTEMS • How understandable a

    system is • How difficult it is to perform tasks in the system • Business tasks • Maintenance tasks • Change tasks (cost of making changes) • High complexity • High costs • High and reaction times • Low performance • High mental and cognitive effort to navigate and use • Fault-prone • Prevent evolution
  80. goodapi.co THE NUMBER OF CLIENTS BROUGHT PROBLEMS WITH COMMUNICATING API

    DESIGN CHANGE, REACTING TO IT AND EVOLVING APIS