Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From MoMs to DaDS: The Evolution of EAI

From MoMs to DaDS: The Evolution of EAI

This talk, presented at tid-x 2019 https://www.tid-x.com/, summarises the evolution of integration architectures in the enterprise . It introduces basic concepts, paradigms and architectures, from messaging oriented middleware based architectures to more modern data services based integrations.

Ruben Gonzalez Blanco

March 02, 2019
Tweet

More Decks by Ruben Gonzalez Blanco

Other Decks in Programming

Transcript

  1. From MoMs to DaDS The Evolution of Enterprise Application Integration

  2. Enterprise IT evolution: Breaking Monoliths

  3. Breaking Monoliths: The early years 80´s - 90´s Mainframe Prov

    Billing ERP CRM HR 1 big System Centralized Logic&Data System of Systems Distributed Logic&Data
  4. Breaking Monoliths: from 2012... the history repeats Application Backend Monolith

    Microservices What is Next?: lambda functions (?)
  5. Enterprise App Integration is about making different apps and systems

    collaborate for... Business Processes Realization ⇢ Systems/Apps integration User Interface Realization ⇢ Frontends to Backends integration Data Processing (BI, Analytics) ⇢ Data Sharing between systems
  6. O(n ) The Challenge: Avoid the point to point hell

    Mainframe Prov Billing ERP CRM HR Mainframe Prov Billing ERP CRM HR n*(n-1)/2 n 2 O(n) EAI Tool / Suite
  7. The Reality: Integration paradigms and technologies evolve, but usually never

    replace completely the previous ones Mainframe ERP DWH MOM Prov1 HR ESB Terminal www1 CRM API GW BFF Prov2 www2 Billing2 Data Services Billing1 µ - services API GW MobileApp1 MobileApp2 DFS ETL ETL Data Platform 1 Data Platform2 NFS Data Query GW www3 Prov2 Sync
  8. Enterprise IT can be like urbanisation “out of control” Aerial

    view of urban sprawl and smog November 6, 2013 in Mexico City, Mexico. Photograph: Richard Ellis/Alamy
  9. Addressing the Challenge: Integration at a Glance Integration Mechanisms Basic

    Decoupling Strategies Integration Architectures clarity in the industry + -
  10. Integration at a Glance Integration Mechanisms Basic Decoupling Strategies Integration

    Architectures/Suites Interconnecting parts for Exchanging Data and Messages
  11. File Transfer System A System B File Integration Mechanisms Decoupling

    Strategies Integration Architectures/Suites
  12. Shared Storage System A System B Database or FileSystem System

    C A B C read() / write () query() Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
  13. RPC : Remote Procedure Call System A System B Request

    // invoke Response // return Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
  14. Messaging Channel (async) Messaging Messaging Channel Producer Consumer Async Message

    Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
  15. Messaging Channel Producer Consumer Consumer Consumer Pub/Sub Channel (Topic) 1:N

    Producer Consumer Consumer Conusmer Point To Point Channel (Queue) 1:1 Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
  16. MOM: Messaging Oriented Middleware (the software that provides async messaging

    services) Topics Queues Producer Conusmer MOM API API Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
  17. Streams Producer Consumer Consumer (Event Logs/ infinite append only file)

    0 1 2 3 end read(from:end) write(reccord)==append(reccord) read(from:0) .... Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
  18. Stream vs Topic Channel Producer Consumer1 : client 0 1

    2 3 end read(from:end) append(reccord) .... Consumer2 : client read(from:end) Producer Consumer1 : server Consumer2 : server onMsgCallback(msg) publish(msg) subscribe(consumer1) subscribe(consumer2) stream:server topic:client read(from:offset) == video streaming read(from:end) == live streaming (tail-f) subscribe(consumer) == radio station(topic) tuning Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
  19. Object Storage Producer Consumer (Object=File+Metadata) metadata File put(metadata, file) get(object

    id) :file Bucket Object Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
  20. DaDS : Data as Data Services API API Buckets Service

    Streams Service Producer Producer Consumer Consumer Consumer put(meta, file) get(id) append(reccord) read(from:offset):[]reccord Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
  21. Integration at a Glance Integration Mechanisms Basic Decoupling Strategies Integration

    Architectures Avoiding point2point coupling
  22. Avoiding Coupling: Adapter System A System B Adapter Adapter A

    Integration Mechanisms Basic Decoupling Strategies Integration Architectures Decouple Systems from Integration Mechanism Adapter ≣ Connector ≣ Extractor/Loader B
  23. Dealing with Coupling: Transformers & Routers System A System B

    Adapter Adapter Transformer A2B Typical ETL ....Is it Point2Point coupling Again? Integration Mechanisms Basic Decoupling Strategies Integration Architectures A B
  24. Avoiding Coupling: Canonical Model System A System B AdapterA AdapterB

    Transformer A2C Transformer C2B Integration Mechanisms Basic Decoupling Strategies Integration Architectures Canonical Message System D AdapterD Transformer C2D A B D C
  25. Integration at a Glance Integration Mechanisms Basic Decoupling Strategies Integration

    Architectures
  26. Integration Architectures : Mediators Simplifies communication among objects in a

    system by introducing a single object that manages message distribution Integration Mechanisms Basic Decoupling Strategies Integration Architectures
  27. Integration Architectures: 2 Type of Mediators Interaction -Centric Data -Centric

    Write/Read Data Send/Receive Messages
  28. Integration Architectures : Interaction Centric Mediators Integration Mechanisms Basic Decoupling

    Strategies Integration Architectures
  29. EMS: EAI Messaging Stacks Mainframe Prov Billing ERP CRM HR

    EAI Messaging Stack Async Messaging based mediation Messages Messages Integration Mechanisms Basic Decoupling Strategies Integration Architectures Messages Messages Messages Messages
  30. EMS: EAI Messaging Stacks MOM Adapter Automated Process Engine Transformers

    / Routers Workflow Engine Integration Mechanisms Basic Decoupling Strategies Integration Architectures Sometimes merged into the same product
  31. ESB: Enterprise Service Bus Mainframe Prov Billing ERP CRM HR

    ESB Enterprise Sevice Bus Mediated RPCs, FileTransfers, Messages in a SOA SOAP Messages FileTranfer REST Messages SOAP Integration Mechanisms Basic Decoupling Strategies Integration Architectures
  32. What is an Enterprise Service Bus? MoM Web Services Protocols

    Transformation & Routing Traffic Management Business Processes Orchestration Connectors Transaction Management Business Application Monitoring Business Schemas Management “Anything” you need to mediate your systems and Services? etc..... Integration Mechanisms Basic Decoupling Strategies Integration Architectures
  33. BFF: Backend For Frontend WWW BFF ERP CRM HR Mobile

    BFF Billing Integration Mechanisms Basic Decoupling Strategies Integration Architectures
  34. API Gateway WWW ERP CRM HR Mobile Billing API GW

    Integration Mechanisms Basic Decoupling Strategies Integration Architectures
  35. API Gateway and MicroServices WWW ERP CRM HR Mobile Billing

    API GW µSrvs API GW Micro-services Mediation Integration Mechanisms Basic Decoupling Strategies Integration Architectures
  36. MicroServices and Service Mesh µSrvs API GW Micro-services Mediation Service

    Mesh Proxy Control Plane & Transversal Services Example: Istio Integration Mechanisms Basic Decoupling Strategies Integration Architectures
  37. Integration Architectures : Data Centric Mediators Integration Mechanisms Basic Decoupling

    Strategies Integration Architectures
  38. Data Query Gateway Integration Mechanisms Basic Decoupling Strategies Integration Architectures

    Data Query Gateway A D B Data Model A Data Model B Data Model C Data Model D Data Model ∅ µSrvs API GW API API Mobile Api == DataModel.query() Examples: GraphQL, Falcor... Joins, Caching, Data routing
  39. Data lake based on DFS Transformer / Processor System A

    System B Extractor Loader Loader DataPlatform • Data and Processing Coupled (i.e HDFS) • Governance of Folders and Files (?) • Metadata (?) Datalake DFS
  40. API Data lake based on Data Services API Buckets Service

    Streams Service Transformer / Processor System A System B Adapter Adapter Adapter DataPlatform Transformer / Processor • Decoupled Data and Processing • Governance and Metadata are intrinsic ◦ Service IAM ◦ Policies ◦ Schemas
  41. A (personal) Vision - FA: Functional Architecture Commands GW Query

    GW App Data Model ∅ Materialized Views System A AdapterA AdapterB System B AdapterC System C Commands stream Events stream
  42. Remember...you are not likely to replace completely the previous systems

    and architectures. Evolve from/with them
  43. Q & A