From MoMs to DaDS: The Evolution of EAI

This talk, presented at tid-x 2019 https://www.tid-x.com/, summarises the evolution of integration architectures in the enterprise . It introduces basic concepts, paradigms and architectures, from messaging oriented middleware based architectures to more modern data services based integrations.

Transcript

1. From MoMs to DaDS The Evolution of Enterprise Application Integration
2. Enterprise IT evolution: Breaking Monoliths
3. Breaking Monoliths: The early years 80´s - 90´s Mainframe Prov Billing ERP CRM HR 1 big System Centralized Logic&Data System of Systems Distributed Logic&Data
4. Breaking Monoliths: from 2012... the history repeats Application Backend Monolith Microservices What is Next?: lambda functions (?)
5. Enterprise App Integration is about making different apps and systems collaborate for... Business Processes Realization ⇢ Systems/Apps integration User Interface Realization ⇢ Frontends to Backends integration Data Processing (BI, Analytics) ⇢ Data Sharing between systems
6. O(n ) The Challenge: Avoid the point to point hell Mainframe Prov Billing ERP CRM HR Mainframe Prov Billing ERP CRM HR n*(n-1)/2 n 2 O(n) EAI Tool / Suite
7. The Reality: Integration paradigms and technologies evolve, but usually never replace completely the previous ones Mainframe ERP DWH MOM Prov1 HR ESB Terminal www1 CRM API GW BFF Prov2 www2 Billing2 Data Services Billing1 µ - services API GW MobileApp1 MobileApp2 DFS ETL ETL Data Platform 1 Data Platform2 NFS Data Query GW www3 Prov2 Sync
8. Enterprise IT can be like urbanisation “out of control” Aerial view of urban sprawl and smog November 6, 2013 in Mexico City, Mexico. Photograph: Richard Ellis/Alamy
9. Addressing the Challenge: Integration at a Glance Integration Mechanisms Basic Decoupling Strategies Integration Architectures clarity in the industry + -
10. Integration at a Glance Integration Mechanisms Basic Decoupling Strategies Integration Architectures/Suites Interconnecting parts for Exchanging Data and Messages
11. File Transfer System A System B File Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
12. Shared Storage System A System B Database or FileSystem System C A B C read() / write () query() Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
13. RPC : Remote Procedure Call System A System B Request // invoke Response // return Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
14. Messaging Channel (async) Messaging Messaging Channel Producer Consumer Async Message Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
15. Messaging Channel Producer Consumer Consumer Consumer Pub/Sub Channel (Topic) 1:N Producer Consumer Consumer Conusmer Point To Point Channel (Queue) 1:1 Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
16. MOM: Messaging Oriented Middleware (the software that provides async messaging services) Topics Queues Producer Conusmer MOM API API Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
17. Streams Producer Consumer Consumer (Event Logs/ infinite append only file) 0 1 2 3 end read(from:end) write(reccord)==append(reccord) read(from:0) .... Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
18. Stream vs Topic Channel Producer Consumer1 : client 0 1 2 3 end read(from:end) append(reccord) .... Consumer2 : client read(from:end) Producer Consumer1 : server Consumer2 : server onMsgCallback(msg) publish(msg) subscribe(consumer1) subscribe(consumer2) stream:server topic:client read(from:offset) == video streaming read(from:end) == live streaming (tail-f) subscribe(consumer) == radio station(topic) tuning Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
19. Object Storage Producer Consumer (Object=File+Metadata) metadata File put(metadata, file) get(object id) :file Bucket Object Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
20. DaDS : Data as Data Services API API Buckets Service Streams Service Producer Producer Consumer Consumer Consumer put(meta, file) get(id) append(reccord) read(from:offset):[]reccord Integration Mechanisms Decoupling Strategies Integration Architectures/Suites
21. Integration at a Glance Integration Mechanisms Basic Decoupling Strategies Integration Architectures Avoiding point2point coupling
22. Avoiding Coupling: Adapter System A System B Adapter Adapter A Integration Mechanisms Basic Decoupling Strategies Integration Architectures Decouple Systems from Integration Mechanism Adapter ≣ Connector ≣ Extractor/Loader B
23. Dealing with Coupling: Transformers & Routers System A System B Adapter Adapter Transformer A2B Typical ETL ....Is it Point2Point coupling Again? Integration Mechanisms Basic Decoupling Strategies Integration Architectures A B
24. Avoiding Coupling: Canonical Model System A System B AdapterA AdapterB Transformer A2C Transformer C2B Integration Mechanisms Basic Decoupling Strategies Integration Architectures Canonical Message System D AdapterD Transformer C2D A B D C
25. Integration at a Glance Integration Mechanisms Basic Decoupling Strategies Integration Architectures
26. Integration Architectures : Mediators Simplifies communication among objects in a system by introducing a single object that manages message distribution Integration Mechanisms Basic Decoupling Strategies Integration Architectures
27. Integration Architectures: 2 Type of Mediators Interaction -Centric Data -Centric Write/Read Data Send/Receive Messages
28. Integration Architectures : Interaction Centric Mediators Integration Mechanisms Basic Decoupling Strategies Integration Architectures
29. EMS: EAI Messaging Stacks Mainframe Prov Billing ERP CRM HR EAI Messaging Stack Async Messaging based mediation Messages Messages Integration Mechanisms Basic Decoupling Strategies Integration Architectures Messages Messages Messages Messages
30. EMS: EAI Messaging Stacks MOM Adapter Automated Process Engine Transformers / Routers Workflow Engine Integration Mechanisms Basic Decoupling Strategies Integration Architectures Sometimes merged into the same product
31. ESB: Enterprise Service Bus Mainframe Prov Billing ERP CRM HR ESB Enterprise Sevice Bus Mediated RPCs, FileTransfers, Messages in a SOA SOAP Messages FileTranfer REST Messages SOAP Integration Mechanisms Basic Decoupling Strategies Integration Architectures
32. What is an Enterprise Service Bus? MoM Web Services Protocols Transformation & Routing Traffic Management Business Processes Orchestration Connectors Transaction Management Business Application Monitoring Business Schemas Management “Anything” you need to mediate your systems and Services? etc..... Integration Mechanisms Basic Decoupling Strategies Integration Architectures
33. BFF: Backend For Frontend WWW BFF ERP CRM HR Mobile BFF Billing Integration Mechanisms Basic Decoupling Strategies Integration Architectures
34. API Gateway WWW ERP CRM HR Mobile Billing API GW Integration Mechanisms Basic Decoupling Strategies Integration Architectures
35. API Gateway and MicroServices WWW ERP CRM HR Mobile Billing API GW µSrvs API GW Micro-services Mediation Integration Mechanisms Basic Decoupling Strategies Integration Architectures
36. MicroServices and Service Mesh µSrvs API GW Micro-services Mediation Service Mesh Proxy Control Plane & Transversal Services Example: Istio Integration Mechanisms Basic Decoupling Strategies Integration Architectures
37. Integration Architectures : Data Centric Mediators Integration Mechanisms Basic Decoupling Strategies Integration Architectures
38. Data Query Gateway Integration Mechanisms Basic Decoupling Strategies Integration Architectures Data Query Gateway A D B Data Model A Data Model B Data Model C Data Model D Data Model ∅ µSrvs API GW API API Mobile Api == DataModel.query() Examples: GraphQL, Falcor... Joins, Caching, Data routing
39. Data lake based on DFS Transformer / Processor System A System B Extractor Loader Loader DataPlatform • Data and Processing Coupled (i.e HDFS) • Governance of Folders and Files (?) • Metadata (?) Datalake DFS
40. API Data lake based on Data Services API Buckets Service Streams Service Transformer / Processor System A System B Adapter Adapter Adapter DataPlatform Transformer / Processor • Decoupled Data and Processing • Governance and Metadata are intrinsic ◦ Service IAM ◦ Policies ◦ Schemas
41. A (personal) Vision - FA: Functional Architecture Commands GW Query GW App Data Model ∅ Materialized Views System A AdapterA AdapterB System B AdapterC System C Commands stream Events stream
42. Remember...you are not likely to replace completely the previous systems and architectures. Evolve from/with them
43. Q & A