Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From MoMs to DaDS: The Evolution of EAI

From MoMs to DaDS: The Evolution of EAI

This talk, presented at tid-x 2019 https://www.tid-x.com/, summarises the evolution of integration architectures in the enterprise . It introduces basic concepts, paradigms and architectures, from messaging oriented middleware based architectures to more modern data services based integrations.

Ruben Gonzalez Blanco

March 02, 2019
Tweet

More Decks by Ruben Gonzalez Blanco

Other Decks in Programming

Transcript

  1. From MoMs to DaDS
    The Evolution of Enterprise Application Integration

    View Slide

  2. Enterprise IT evolution: Breaking Monoliths

    View Slide

  3. Breaking Monoliths: The early years 80´s - 90´s
    Mainframe
    Prov
    Billing
    ERP
    CRM
    HR
    1 big System
    Centralized Logic&Data
    System of Systems
    Distributed Logic&Data

    View Slide

  4. Breaking Monoliths: from 2012... the history repeats
    Application
    Backend
    Monolith Microservices
    What is Next?: lambda functions (?)

    View Slide

  5. Enterprise App Integration is about making
    different apps and systems collaborate for...
    Business Processes Realization ⇢ Systems/Apps integration
    User Interface Realization ⇢ Frontends to Backends integration
    Data Processing (BI, Analytics) ⇢ Data Sharing between systems

    View Slide

  6. O(n )
    The Challenge: Avoid the point to point hell
    Mainframe
    Prov
    Billing
    ERP
    CRM
    HR
    Mainframe
    Prov
    Billing
    ERP
    CRM
    HR
    n*(n-1)/2 n
    2 O(n)
    EAI
    Tool / Suite

    View Slide

  7. The Reality: Integration paradigms and technologies evolve, but
    usually never replace completely the previous ones
    Mainframe
    ERP
    DWH
    MOM
    Prov1
    HR
    ESB
    Terminal
    www1
    CRM
    API GW
    BFF
    Prov2
    www2
    Billing2
    Data
    Services
    Billing1
    µ - services API GW
    MobileApp1
    MobileApp2
    DFS
    ETL
    ETL
    Data Platform
    1
    Data
    Platform2
    NFS
    Data Query GW
    www3
    Prov2
    Sync

    View Slide

  8. Enterprise IT can be like urbanisation “out of control”
    Aerial view of urban sprawl and smog November 6, 2013 in Mexico City, Mexico. Photograph: Richard Ellis/Alamy

    View Slide

  9. Addressing the Challenge: Integration at a Glance
    Integration Mechanisms
    Basic Decoupling Strategies
    Integration Architectures
    clarity in the
    industry
    +
    -

    View Slide

  10. Integration at a Glance
    Integration Mechanisms
    Basic Decoupling Strategies
    Integration Architectures/Suites
    Interconnecting parts for Exchanging Data and Messages

    View Slide

  11. File Transfer
    System A System B
    File
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites

    View Slide

  12. Shared Storage
    System A System B
    Database or FileSystem
    System C
    A
    B
    C
    read() / write ()
    query()
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites

    View Slide

  13. RPC : Remote Procedure Call
    System A System B
    Request // invoke
    Response // return
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites

    View Slide

  14. Messaging Channel
    (async) Messaging
    Messaging Channel
    Producer Consumer
    Async Message
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites

    View Slide

  15. Messaging Channel
    Producer
    Consumer Consumer Consumer
    Pub/Sub Channel (Topic)
    1:N
    Producer
    Consumer
    Consumer
    Conusmer
    Point To Point Channel (Queue)
    1:1
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites

    View Slide

  16. MOM: Messaging Oriented Middleware
    (the software that provides async messaging services)
    Topics
    Queues
    Producer Conusmer
    MOM
    API API
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites

    View Slide

  17. Streams
    Producer
    Consumer Consumer
    (Event Logs/ infinite append only file)
    0 1 2 3 end
    read(from:end)
    write(reccord)==append(reccord)
    read(from:0)
    ....
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites

    View Slide

  18. Stream vs Topic Channel
    Producer
    Consumer1
    :
    client
    0 1 2 3 end
    read(from:end)
    append(reccord)
    ....
    Consumer2
    :
    client
    read(from:end)
    Producer
    Consumer1
    :
    server
    Consumer2
    :
    server
    onMsgCallback(msg)
    publish(msg)
    subscribe(consumer1)
    subscribe(consumer2)
    stream:server
    topic:client
    read(from:offset) == video streaming
    read(from:end) == live streaming
    (tail-f)
    subscribe(consumer) == radio station(topic) tuning
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites

    View Slide

  19. Object Storage
    Producer Consumer
    (Object=File+Metadata)
    metadata
    File
    put(metadata, file) get(object id) :file
    Bucket
    Object
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites

    View Slide

  20. DaDS : Data as Data Services
    API
    API
    Buckets Service Streams Service
    Producer
    Producer
    Consumer
    Consumer
    Consumer
    put(meta, file)
    get(id)
    append(reccord)
    read(from:offset):[]reccord
    Integration Mechanisms
    Decoupling Strategies
    Integration
    Architectures/Suites

    View Slide

  21. Integration at a Glance
    Integration Mechanisms
    Basic Decoupling Strategies
    Integration Architectures
    Avoiding point2point coupling

    View Slide

  22. Avoiding Coupling: Adapter
    System A System B
    Adapter Adapter
    A
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    Decouple Systems from Integration Mechanism
    Adapter ≣ Connector ≣ Extractor/Loader
    B

    View Slide

  23. Dealing with Coupling: Transformers & Routers
    System A System B
    Adapter Adapter
    Transformer
    A2B
    Typical ETL ....Is it Point2Point coupling Again?
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    A B

    View Slide

  24. Avoiding Coupling: Canonical Model
    System A
    System B
    AdapterA
    AdapterB
    Transformer
    A2C
    Transformer
    C2B
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    Canonical Message
    System D
    AdapterD
    Transformer
    C2D
    A
    B
    D
    C

    View Slide

  25. Integration at a Glance
    Integration Mechanisms
    Basic Decoupling Strategies
    Integration Architectures

    View Slide

  26. Integration Architectures : Mediators
    Simplifies communication among objects in a system by
    introducing a single object that manages message distribution
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures

    View Slide

  27. Integration Architectures: 2 Type of Mediators
    Interaction -Centric Data -Centric
    Write/Read Data
    Send/Receive Messages

    View Slide

  28. Integration Architectures :
    Interaction Centric Mediators
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures

    View Slide

  29. EMS: EAI Messaging Stacks
    Mainframe
    Prov
    Billing
    ERP
    CRM
    HR
    EAI
    Messaging
    Stack
    Async Messaging based mediation
    Messages
    Messages
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    Messages
    Messages Messages
    Messages

    View Slide

  30. EMS: EAI Messaging Stacks
    MOM
    Adapter
    Automated Process Engine
    Transformers / Routers
    Workflow Engine
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    Sometimes merged into the
    same product

    View Slide

  31. ESB: Enterprise Service Bus
    Mainframe
    Prov
    Billing
    ERP
    CRM
    HR
    ESB
    Enterprise
    Sevice
    Bus
    Mediated RPCs, FileTransfers, Messages in a SOA
    SOAP
    Messages
    FileTranfer REST
    Messages
    SOAP
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures

    View Slide

  32. What is an Enterprise Service Bus?
    MoM
    Web Services Protocols
    Transformation & Routing
    Traffic Management
    Business Processes Orchestration
    Connectors
    Transaction Management
    Business Application Monitoring
    Business Schemas Management
    “Anything” you need to mediate your systems and Services?
    etc.....
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures

    View Slide

  33. BFF: Backend For Frontend
    WWW
    BFF
    ERP
    CRM
    HR
    Mobile
    BFF
    Billing
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures

    View Slide

  34. API Gateway
    WWW
    ERP
    CRM
    HR
    Mobile
    Billing
    API GW
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures

    View Slide

  35. API Gateway and MicroServices
    WWW
    ERP
    CRM
    HR
    Mobile
    Billing
    API GW
    µSrvs API GW
    Micro-services
    Mediation
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures

    View Slide

  36. MicroServices and Service Mesh
    µSrvs API GW
    Micro-services Mediation
    Service Mesh Proxy
    Control Plane &
    Transversal Services
    Example: Istio
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures

    View Slide

  37. Integration Architectures :
    Data Centric Mediators
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures

    View Slide

  38. Data Query Gateway
    Integration Mechanisms
    Basic Decoupling
    Strategies
    Integration
    Architectures
    Data Query Gateway
    A
    D
    B
    Data Model A
    Data Model B
    Data Model C
    Data Model D
    Data Model ∅
    µSrvs API GW
    API
    API
    Mobile
    Api == DataModel.query()
    Examples: GraphQL, Falcor...
    Joins,
    Caching,
    Data routing

    View Slide

  39. Data lake based on DFS
    Transformer
    /
    Processor
    System A
    System B
    Extractor
    Loader
    Loader
    DataPlatform
    ● Data and Processing Coupled (i.e HDFS)
    ● Governance of Folders and Files (?)
    ● Metadata (?)
    Datalake
    DFS

    View Slide

  40. API
    Data lake based on Data Services
    API
    Buckets Service Streams Service
    Transformer
    /
    Processor
    System A
    System B
    Adapter
    Adapter
    Adapter
    DataPlatform
    Transformer
    /
    Processor
    ● Decoupled Data and
    Processing
    ● Governance and Metadata are
    intrinsic
    ○ Service IAM
    ○ Policies
    ○ Schemas

    View Slide

  41. A (personal) Vision - FA: Functional Architecture
    Commands GW
    Query GW
    App
    Data Model ∅
    Materialized
    Views
    System A
    AdapterA
    AdapterB
    System B
    AdapterC
    System C
    Commands stream
    Events
    stream

    View Slide

  42. Remember...you are not likely to replace completely the
    previous systems and architectures. Evolve from/with them

    View Slide

  43. Q & A

    View Slide