Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Enterprise Integration Patterns Revisited (agai...

Kai Waehner
November 06, 2015

Enterprise Integration Patterns Revisited (again) for the Era of Big Data, Internet of Things, Cloud and Microservices

Enterprise Integration Patterns Revisited for the Era of Big Data, Internet of Things, Cloud and Microservices
In 2015, I had two talks about Enterprise Integration Patterns at OOP 2015 in Munich, Germany and at JavaDay 2015 in Kiev, Ukraine. I reused a talk from 2013 and updated it with current trends to show how important Enterprise Integration Patterns (EIP) are everywhere today and in the upcoming years.

Great to see that Gregor Hohpe - founder and author of the EIP book, and currently Head of IT at Allianz - was attending one of the sessions, too:


Introduction to Enterprise Integration Patterns
Data exchanges between companies increase a lot. Hence, the number of applications, which must be integrated increases, too. The realization of these integration scenarios is a complex and time-consuming task because different applications and services do not use the same concepts, interfaces, data formats and technologies.

Originated and published over ten years ago by Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns (EIP) became the world wide de facto standard for describing integration problems. These patterns offer a standardized way to split huge, complex integration scenarios into smaller recurring problems. The following slide deck revisits EIPs and gives an overview about the status quo and its relevance regarding modern concepts such as Big Data, Internet of Things, Cloud or Microservices. The following shows a simple, but powerful diagram using several different EIPs such as an Splitter or Router (source: http://www.enterpriseintegrationpatterns.com/):

Fortunately, EIPs offer more possibilities than just being used for modeling integration problems in a standardized way. Frameworks and tools implement these patterns, already. The developer does not have to implement EIPs on his own.
Trends: Hybrid Integration and the Citizen Integrator
The behavior of end users changes these days. Mission critical systems still need to be integrated on enterprise level. However, business users also want (and need) to be more practical without having to ask IT departments for help (taking a long time usually). Technical people (no developer, but some technical knowledge) want to realize simple integrations with partners by themselves.

While Forrester talks a about “Hybrid Integration”, Gartner names it “Citizen Developers” or “Citizen Integrators”. No matter how you name these new trends – they are on the way to the IT world of most enterprises, already.

TIBCO is following this trend. Take a look at TIBCO NOW Tour in Paris and London in 2015: TIBCO announced different tools for different end users. The future integration portfolio will contain offerings for business users, citizen developers and classical enterprise integration scenarios:
• TIBCO ActiveMatrix BusinessWorks Enterprise Edition keeps TIBCO’s flagship integration product for enterprise integration scenarios. It can be deployed on-premise on physical hardware, virtual machines or in IaaS cloud infrastructures such as Amazon EC2)
• TIBCO ActiveMatrix BusinessWorks Container Edition is similar to BusinessWorks Enterprise Edition, but optimized for cloud deployment (e.g. automatic port configuration, some activities have been changed / removed to be cloud-ready, for example File and FTP activities). The Container Edition is deployed as PaaS in the cloud or on premise, based on Pivotal’s CloudFoundry. It uses CloudFoundry’s flexible container concepts under the hood. Docker support is also planned for next year.
• TIBCO Cloud Integration offers intuitive web-based integration for the “Citizen Integrator”, e.g. to create and deploy REST services or to configure connections and policies of Open APIs. It can be used as “Micro Gateway” in the cloud.
• TIBCO Simplr also offers an intuitive web-based user interface for the Business User, e.g. to synchronize data via self-service from SaaS offerings such as Salesforce and Marketo without technical knowledge.
EIPs for Streaming Analytics, IoT and Microservices
EIPs are important not only for „classical integration“ using an Enterprise Service Bus such as TIBCO ActiveMatrix BusinessWorks or new integration cloud services such as TIBCO Cloud Integration or TIBCO Simplr.

You also (have to) use EIPs with frameworks and tools for Stream Processing (e.g. TIBCO StreamBase), API Management / Application Services Governance (e.g. TIBCO API Exchange or TIBCO Mashery), and any other tool which uses some kind of integration logic, e.g. for Internet of things (IoT) or building business logic via Microservices.

EIPs for Log Analytics and Operational Intelligence
Log Analytics and Operational Intelligence (e.g. TIBCO LogLogic and TIBCO Unity is another hot topic in the market these days, where you use EIPs for processing log data and other events. Take a look at the typical workflow, which translates to several different EIPs:

Figure 1: Workflow for Log Analytics and Operational Intelligence
Slide Deck “Enterprise Integration Patterns Revisited”
Here is the slide deck, which discusses the above content in more detail:

TODO UPLOAD NEW VERSION TO SLIDESHARE

Integration is Key to a Successful Future
The number of different data sources and technologies increases even more than in the past:
• Legacy systems such as CRM, ERP, Mainframe or B2B via EDIFACT, will not disappear
• Big Data clusters such as Data Warehouse, Apache Hadoop or NoSQL databases will store exponentially more data year by year
• Streaming server and In-Memory Data Grids will process more and more data in real time in the future
• Cloud, Mobile, APIs, Big Data, Fast Data, Internet of Things are no option, but everybody’s future!

Therefore: EVERYTHING HAS TO BE INTEGRATED! Enterprise Integration Patterns are a key piece in this puzzle!

Keywords:
Enterprise Integration Patterns, EIP, Big Data, Fast Data, Internet of Things, IoT, Cloud, Streaming Analytics, Integration, Microservices, Enterprise Service Bus, ESB, Open API, Streaming Analytics, Stream Processing, TIBCO, BusinessWorks, StreamBase, LogLogic, Unity, open api, API Management, Cloud Integration, Citizen Developer, Hybrid Integration, iSaaS, iPaaS, Oracle, IBM, Software AG, MuleSoft, Talend, JBoss, Fuse, Spring, Java

Kai Waehner

November 06, 2015
Tweet

More Decks by Kai Waehner

Other Decks in Technology

Transcript

  1. Enterprise Integration Patterns Revisited for an Era of Big Data

    and Internet of Things Kai Wähner [email protected] @KaiWaehner www.kai-waehner.de LinkedIn / Xing à Please connect!
  2. Key Take-Aways –  Integration is the key factor for success

    of business in the future! –  Enterprise Integration Patterns are used everywhere! –  Real time integration is the game changer in a Big Data and IoT world!
  3. Agenda Ø  Application Integration Ø  Enterprise Integration Patterns Ø  Modeling

    Ø  Frameworks and Tools Ø  Big Data, Internet of Things, Microservices Ø  Future
  4. Agenda Ø  Application Integration Ø  Enterprise Integration Patterns Ø  Modeling

    Ø  Frameworks and Tools Ø  Big Data, Internet of Things, Microservices Ø  Future
  5. Growing IT Infrastructure Everywhere •  Applications •  Interfaces •  Technologies

    •  Products •  Cloud •  Mobile •  Internet of Things
  6. AmazonS3  s3  =  new  AmazonS3Client(new  Proper6esCreden6als(        

             S3Sample.class.getResourceAsStream("AwsCreden6als.proper6es")));     String  bucketName  =  "my-­‐first-­‐s3-­‐bucket-­‐"  +  UUID.randomUUID();   String  key  =  "MyObjectKey";     try  {                    s3.createBucket(bucketName);          s3.putObject(new  PutObjectRequest(bucketName,  key,  createSampleFile()));            S3Object  object  =  s3.getObject(new  GetObjectRequest(bucketName,  key));                    ObjectLis6ng  objectLis6ng  =  s3.listObjects(new  ListObjectsRequest()                          .withBucketName(bucketName)                          .withPrefix("My"));                    s3.deleteObject(bucketName,  key);              s3.deleteBucket(bucketName);     }  catch  (AmazonServiceExcep6on  ase)  {        //  error  handling...   }  catch  (AmazonClientExcep6on  ace)  {        //  error  handling...         }   Writing a lot of Glue Code...
  7. Agenda Ø  Application Integration Ø  Enterprise Integration Patterns Ø  Modeling

    Ø  Frameworks and Tools Ø  Big Data, Internet of Things, Microservices Ø  Future
  8. “A software design pattern, in the sense of a template,

    is a general solution to a problem in programming. A design pattern provides a reusable architectural outline that may speed the development of many computer programs.” http://en.wikipedia.org/wiki/Design_pattern What is a Design Pattern?
  9. “The goal of EIPs is to document technology-independent design guidance

    that helps developers and architects describe and develop robust INTEGRATION solutions.” http://www.eaipatterns.com Enterprise Integration Patterns (EIP)
  10. History of EIPs •  Talk from Gregor Hohpe –  WSO2

    Con 2011 –  „History, present, future of EIP“ –  A lot about past (ideas, creation of book, ...) –  http://www.youtube.com/watch? v=Xwi1DU6KoQ4 •  Today‘s Talk –  Not too much about history –  Just about present and future
  11. Status Quo •  Enterprise Integration Patterns do not change • 

    Just technologies change •  Implementations for EIPs are already available
  12. Agenda Ø  Application Integration Ø  Enterprise Integration Patterns Ø  Modeling

    Ø  Frameworks and Tools Ø  Big Data, Internet of Things, Microservices Ø  Future
  13. Agenda Ø  Application Integration Ø  Enterprise Integration Patterns Ø  Modeling

    Ø  Frameworks and Tools Ø  Big Data, Internet of Things, Microservices Ø  Future
  14. Complexity of Integration Integra6on   Products   Integra6on  Suite  

    Low High Integration Framework Alternatives for Application Integration
  15. Kai Wähner - Enterprise Integration Patterns Revisited Complexity of Integration

    Integra6on   Products   Integra6on  Suite   Low High Integration Framework INTEGRATION Connectivity Routing Transformation When to use an Integration Framework?
  16. Many further connectors Easy to create own components HTTP  

    FTP   File   XSLT   EJB   JDBC Akka   TCP   SMTP   RSS   Quartz   jclouds   LDAP   JMS   MQTT   AMQP   Atom   AWS   Bean-Validation   CXF   IRC   Jetty   JMX   Lucene   Netty   RMI   SQL   Twitter MongoDB   Connectivity
  17. Kai Wähner - Enterprise Integration Patterns Revisited Connectivity Routing Transformation

    INTEGRATION Tooling Monitoring Support Complexity of Integration Integra6on   Products   Integra6on  Suite   Low High Integration Framework + When to use an Enterprise Service Bus?
  18. 36 Integration Products © Copyright 2000-2015 TIBCO Software Inc. One

     Integra*on   Product  does  not  fit   all  problems!  
  19. Adaptive Integration Approaches Self-Service Integration for DIY Integrators (iSaaS) Business

    Integration Cloud Integration Integration Platform as a Service (iPaaS) Enterprise Business Users Department Audience Enterprise Integration Platform (on-premise and cloud) Enterprise Service Bus
  20. Integration Concepts Decision Data Conversion Transitions Process Call Process Design

    Tools Integration Processes Connectors to vendor applications Connectors to technologies Connectors (e.g. JMS, java, FTP) (e.g. SAP, SF.com) Enterprise Integration Patterns
  21. Enterprise Service Bus (ESB) Intuitive graphical design environment Streamlines time

    and cost of development and training Graphical Process Modeling Native Standards based XSLT Mapper Drag-n-Drop Access to Resources Fully Integrated Test Environment Adapters, Services, Processes, Deployment, Management
  22. iSaaS: Integration for the “Prosumer” Subscription cloud service that makes

    it easy to automate moving of data between apps on the Web
  23. iSaaS: Integration for the “Prosumer” •  Connectors –  SaaS APIs

    baked in a common user interface •  Recipes –  Combinations of connectors preconfigured as “apps” –  Just configure to your accounts –  Customize as needed •  Drag and Drop ease
  24. Agenda Ø  Application Integration Ø  Enterprise Integration Patterns Ø  Modeling

    Ø  Frameworks and Tools Ø  Big Data, Internet of Things, Microservices Ø  Future
  25. Evolution of Integration Microservices   Web  APIs   Real-­‐6me  

    Web     Oriented     Architecture   Enabling  Technologies   In-­‐Memory,  Mul6core   REST,  JSON   Demand  Drivers   Mobile,  Cloud   Fast  Data,  IoT   Services     Web  Services   Real-­‐6me   Service     Oriented   Architecture   Enabling  Technologies   XML,  SOAP,  WS-­‐*   Process  Modeling   Demand  Drivers   E-­‐Commerce   BPM   Interfaces   Adapters   Real-­‐6me   Enterprise     Applica6on   Integra6on     Enabling  Technologies   Client-­‐Server   Messaging  Middleware   Demand  Drivers   ERP   Analy6cs   Records   Batch  Jobs   Non-­‐real6me   Enabling  Technologies   Mainframe   ETL,  Databases   Data   Integra6on   Demand  Drivers   Data  Processing   MIS   Are  we   there  yet?   Level-­‐Up  by     u=lizing  the   lessons,   assets  and   prac=ces  of   the  previous   Level     Accelera=ng  Produc=vity  &    Agility    
  26. Enterprise Integration Patterns (EIPs) … •  … are needed everywhere!

    •  Not just in an Integration Framework or Integration Product!
  27. Connectivity Routing Transformation Complexity of Integration Integra6on   Products  

    Integra6on  Suite   Low High Integration Framework INTEGRATION Tooling Monitoring Support + API MANAGEMENT BIG DATA INTERNET OF THINGS STREAMING ANALYTICS „YOU NAME IT“ + When to use an Integration Suite?
  28. EIPs for Open API and API Management Your   Enterprise

      Closed  APIs   EDI   Web   B2B  SOA   FTP   Pre-­‐defined  integra6on  points   Limited,  trusted  partners   Strictly  constrained  interac6ons   Enterprise  friendly   Open  APIs   Opportunis6c  access  points     Many  partners,  untrusted   Encourage  new  ideas   Developer  friendly   Your   Enterprise   Partners   Innovators   Known   Par6es   Consumers   Unknown   Par6es   Suppliers   Coope66on   Employees   API   API   API   API   API   API   API   API   API   API   API   API   API   API   API   API   API   API   API   API  
  29. EIPs for API Management (e.g. Transformation or Routing) Composite Consumer

    Domain Target Service Domain Target Facade Consumer Target Provider Facade Target Target Endpoint Facade Operation Target Provider Facade Policies Target Policies Router Routing Policies Partner Policies Facade Endpoint Target Operation © Copyright 2000-2015 TIBCO Software Inc.
  30. © Copyright 2000-2015 TIBCO Software Inc. EIPs for Log Analytics

    and Operational Intelligence Collect   Store   Index  &   Search   Parse  &   Normalize   Visually   Analyze   Format  &   Forward   © Copyright 2000-2014 TIBCO Software Inc.
  31. EIPs for Business Logic E.g. with Microservices: •  Services implementing

    a limited set of business functions •  Services developed, deployed and scaled independently Integration Service Monolith application SOA Integration Service Business Service Business Service Business Service Micro Service API Gateway SaaS Business Service
  32. EIPs for Integration of “Internet of Everything” à Real Time

    Integration with Things, APIs, Business Processes, Cloud, Social, etc. hep://blogs.cisco.com/ioe/how-­‐the-­‐internet-­‐of-­‐everything-­‐will-­‐change-­‐the-­‐worldfor-­‐the-­‐beeer-­‐infographic  
  33. 55 Integration Suite © Copyright 2000-2015 TIBCO Software Inc. Do

     I  have  to  use  one   Integra*on  Suite  for   all  these  use  cases?  
  34. One or more frameworks / vendors? Frameworks or Products from

    different Vendors •  No license costs (for open source frameworks) •  A lot of glue code •  Testing / Bugfixing •  No (combined) support Frameworks or Products from a single Vendor •  All integrated within one framework or product? à Vision! •  Not integrated? à Often reality! •  Loosely coupled, highly integrated frameworks or products? à Best choice!
  35. Agenda Ø  Application Integration Ø  Enterprise Integration Patterns Ø  Modeling

    Ø  Frameworks and Tools Ø  Big Data, Internet of Things, Microservices Ø  Future
  36. Future of EIPs: The BAD News •  New Patterns (since

    release of the book)? ß NOT YET –  Conversations –  Complex Event Processing –  Error Handling •  More Details: „EIP Visions“ ß TO BE DETERMINED –  EIP I: Messaging à done –  EIP II: Conversations à 50 page paper –  EIP III: Process à tbd –  EIP IV: Event Processing à tbd http://www.eaipatterns.com/ramblings/72_eipvolumes.html
  37. Future of EIPs: The GOOD News •  New Patterns not

    necessary! ß OFTEN –  EIPs were based on Messaging already, which is a good thing (usually) –  Conversations == State == Anti Pattern (often) –  Integration scenarios can be solved with existing patterns (usually) •  Better Tooling ß YES –  More frameworks and products (e.g. Stream Processing for Big Data / Fast Data, Internet of Things) –  Stable, mature tools –  New standards (MQTT, OMG's DDS, Eclipse Open IoT Stack, etc.) –  Support for new hardware (iBeacons, OSIsoft Pi, Google Glass, etc.) –  Ease of use (no coding) –  Better scalability
  38. Future of Integration and EIPs Buy  this  book    

    if  you  do  not  own  it  already!   Integration will get even more important in the future than it is today! The number of different data sources and technologies increases even more than in the past: •  CRM, ERP, Mainframe, B2B (EDIFACT), etc. will not disappear •  DWH, Hadoop cluster, event / streaming server, In-Memory DB – all of them have to communicate •  Cloud, Mobile, APIs, Big Data, Internet of Things are no option, but our future! EVERYTHING HAS TO BE INTEGRATED!
  39. Integration is the key factor for success of business in

    the future! Enterprise Integration Patterns are used everywhere! Real time integration is the game changer in a Big Data and IoT world! Recap: Key Take-Aways