From MoMs to DaDS: The Evolution of EAI

This talk, presented at tid-x 2019 https://www.tid-x.com/, summarises the evolution of integration architectures in the enterprise . It introduces basic concepts, paradigms and architectures, from messaging oriented middleware based architectures to more modern data services based integrations.

Transcript

1. From MoMs to DaDS
   The Evolution of Enterprise Application Integration
   

   View Slide

2. Enterprise IT evolution: Breaking Monoliths
   

   View Slide

3. Breaking Monoliths: The early years 80´s - 90´s
   Mainframe
   Prov
   Billing
   ERP
   CRM
   HR
   1 big System
   Centralized Logic&Data
   System of Systems
   Distributed Logic&Data
   

   View Slide

4. Breaking Monoliths: from 2012... the history repeats
   Application
   Backend
   Monolith Microservices
   What is Next?: lambda functions (?)
   

   View Slide

5. Enterprise App Integration is about making
   different apps and systems collaborate for...
   Business Processes Realization ⇢ Systems/Apps integration
   User Interface Realization ⇢ Frontends to Backends integration
   Data Processing (BI, Analytics) ⇢ Data Sharing between systems
   

   View Slide

6. O(n )
   The Challenge: Avoid the point to point hell
   Mainframe
   Prov
   Billing
   ERP
   CRM
   HR
   Mainframe
   Prov
   Billing
   ERP
   CRM
   HR
   n*(n-1)/2 n
   2 O(n)
   EAI
   Tool / Suite
   

   View Slide

7. The Reality: Integration paradigms and technologies evolve, but
   usually never replace completely the previous ones
   Mainframe
   ERP
   DWH
   MOM
   Prov1
   HR
   ESB
   Terminal
   www1
   CRM
   API GW
   BFF
   Prov2
   www2
   Billing2
   Data
   Services
   Billing1
   µ - services API GW
   MobileApp1
   MobileApp2
   DFS
   ETL
   ETL
   Data Platform
   1
   Data
   Platform2
   NFS
   Data Query GW
   www3
   Prov2
   Sync
   

   View Slide

8. Enterprise IT can be like urbanisation “out of control”
   Aerial view of urban sprawl and smog November 6, 2013 in Mexico City, Mexico. Photograph: Richard Ellis/Alamy
   

   View Slide

9. Addressing the Challenge: Integration at a Glance
   Integration Mechanisms
   Basic Decoupling Strategies
   Integration Architectures
   clarity in the
   industry
   +
   -
   

   View Slide

10. Integration at a Glance
    Integration Mechanisms
    Basic Decoupling Strategies
    Integration Architectures/Suites
    Interconnecting parts for Exchanging Data and Messages
    

    View Slide

11. File Transfer
    System A System B
    File
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites
    

    View Slide

12. Shared Storage
    System A System B
    Database or FileSystem
    System C
    A
    B
    C
    read() / write ()
    query()
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites
    

    View Slide

13. RPC : Remote Procedure Call
    System A System B
    Request // invoke
    Response // return
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites
    

    View Slide

14. Messaging Channel
    (async) Messaging
    Messaging Channel
    Producer Consumer
    Async Message
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites
    

    View Slide

15. Messaging Channel
    Producer
    Consumer Consumer Consumer
    Pub/Sub Channel (Topic)
    1:N
    Producer
    Consumer
    Consumer
    Conusmer
    Point To Point Channel (Queue)
    1:1
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites
    

    View Slide

16. MOM: Messaging Oriented Middleware
    (the software that provides async messaging services)
    Topics
    Queues
    Producer Conusmer
    MOM
    API API
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites
    

    View Slide

17. Streams
    Producer
    Consumer Consumer
    (Event Logs/ infinite append only file)
    0 1 2 3 end
    read(from:end)
    write(reccord)==append(reccord)
    read(from:0)
    ....
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites
    

    View Slide

18. Stream vs Topic Channel
    Producer
    Consumer1
    :
    client
    0 1 2 3 end
    read(from:end)
    append(reccord)
    ....
    Consumer2
    :
    client
    read(from:end)
    Producer
    Consumer1
    :
    server
    Consumer2
    :
    server
    onMsgCallback(msg)
    publish(msg)
    subscribe(consumer1)
    subscribe(consumer2)
    stream:server
    topic:client
    read(from:offset) == video streaming
    read(from:end) == live streaming
    (tail-f)
    subscribe(consumer) == radio station(topic) tuning
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites
    

    View Slide

19. Object Storage
    Producer Consumer
    (Object=File+Metadata)
    metadata
    File
    put(metadata, file) get(object id) :file
    Bucket
    Object
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites
    

    View Slide

20. DaDS : Data as Data Services
    API
    API
    Buckets Service Streams Service
    Producer
    Producer
    Consumer
    Consumer
    Consumer
    put(meta, file)
    get(id)
    append(reccord)
    read(from:offset):[]reccord
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites
    

    View Slide

21. Integration at a Glance
    Integration Mechanisms
    Basic Decoupling Strategies
    Integration Architectures
    Avoiding point2point coupling
    

    View Slide

22. Avoiding Coupling: Adapter
    System A System B
    Adapter Adapter
    A
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    Decouple Systems from Integration Mechanism
    Adapter ≣ Connector ≣ Extractor/Loader
    B
    

    View Slide

23. Dealing with Coupling: Transformers & Routers
    System A System B
    Adapter Adapter
    Transformer
    A2B
    Typical ETL ....Is it Point2Point coupling Again?
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    A B
    

    View Slide

24. Avoiding Coupling: Canonical Model
    System A
    System B
    AdapterA
    AdapterB
    Transformer
    A2C
    Transformer
    C2B
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    Canonical Message
    System D
    AdapterD
    Transformer
    C2D
    A
    B
    D
    C
    

    View Slide

25. Integration at a Glance
    Integration Mechanisms
    Basic Decoupling Strategies
    Integration Architectures
    

    View Slide

26. Integration Architectures : Mediators
    Simplifies communication among objects in a system by
    introducing a single object that manages message distribution
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    

    View Slide

27. Integration Architectures: 2 Type of Mediators
    Interaction -Centric Data -Centric
    Write/Read Data
    Send/Receive Messages
    

    View Slide

28. Integration Architectures :
    Interaction Centric Mediators
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    

    View Slide

29. EMS: EAI Messaging Stacks
    Mainframe
    Prov
    Billing
    ERP
    CRM
    HR
    EAI
    Messaging
    Stack
    Async Messaging based mediation
    Messages
    Messages
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    Messages
    Messages Messages
    Messages
    

    View Slide

30. EMS: EAI Messaging Stacks
    MOM
    Adapter
    Automated Process Engine
    Transformers / Routers
    Workflow Engine
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    Sometimes merged into the
    same product
    

    View Slide

31. ESB: Enterprise Service Bus
    Mainframe
    Prov
    Billing
    ERP
    CRM
    HR
    ESB
    Enterprise
    Sevice
    Bus
    Mediated RPCs, FileTransfers, Messages in a SOA
    SOAP
    Messages
    FileTranfer REST
    Messages
    SOAP
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    

    View Slide

32. What is an Enterprise Service Bus?
    MoM
    Web Services Protocols
    Transformation & Routing
    Traffic Management
    Business Processes Orchestration
    Connectors
    Transaction Management
    Business Application Monitoring
    Business Schemas Management
    “Anything” you need to mediate your systems and Services?
    etc.....
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    

    View Slide

33. BFF: Backend For Frontend
    WWW
    BFF
    ERP
    CRM
    HR
    Mobile
    BFF
    Billing
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    

    View Slide

34. API Gateway
    WWW
    ERP
    CRM
    HR
    Mobile
    Billing
    API GW
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    

    View Slide

35. API Gateway and MicroServices
    WWW
    ERP
    CRM
    HR
    Mobile
    Billing
    API GW
    µSrvs API GW
    Micro-services
    Mediation
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    

    View Slide

36. MicroServices and Service Mesh
    µSrvs API GW
    Micro-services Mediation
    Service Mesh Proxy
    Control Plane &
    Transversal Services
    Example: Istio
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    

    View Slide

37. Integration Architectures :
    Data Centric Mediators
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    

    View Slide

38. Data Query Gateway
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    Data Query Gateway
    A
    D
    B
    Data Model A
    Data Model B
    Data Model C
    Data Model D
    Data Model ∅
    µSrvs API GW
    API
    API
    Mobile
    Api == DataModel.query()
    Examples: GraphQL, Falcor...
    Joins,
    Caching,
    Data routing
    

    View Slide

39. Data lake based on DFS
    Transformer
    /
    Processor
    System A
    System B
    Extractor
    Loader
    Loader
    DataPlatform
    ● Data and Processing Coupled (i.e HDFS)
    ● Governance of Folders and Files (?)
    ● Metadata (?)
    Datalake
    DFS
    

    View Slide

40. API
    Data lake based on Data Services
    API
    Buckets Service Streams Service
    Transformer
    /
    Processor
    System A
    System B
    Adapter
    Adapter
    Adapter
    DataPlatform
    Transformer
    /
    Processor
    ● Decoupled Data and
    Processing
    ● Governance and Metadata are
    intrinsic
    ○ Service IAM
    ○ Policies
    ○ Schemas
    

    View Slide

41. A (personal) Vision - FA: Functional Architecture
    Commands GW
    Query GW
    App
    Data Model ∅
    Materialized
    Views
    System A
    AdapterA
    AdapterB
    System B
    AdapterC
    System C
    Commands stream
    Events
    stream
    

    View Slide

42. Remember...you are not likely to replace completely the
    previous systems and architectures. Evolve from/with them
    

    View Slide

43. Q & A
    

    View Slide