Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

The Enterprise Service Bus is Dead! Long Live t...

The Enterprise Service Bus is Dead! Long Live the Enterprise Service Bus.

The enterprise service bus (ESB) is a concept and resulting set of software products that came to fruition in the early 2000s. The ESB was designed to contain the complexity of connecting heterogeneous applications, including the entire organization’s business logic and data. These concepts have fallen out of favor with the advent of microservices and Kafka. This existing definition of ESB’s is no longer useful to us; in this talk, we’ll introduce a modern definition of the ESB, with Kafka at its heart.

Rimas Silkaitis

April 26, 2016
Tweet

More Decks by Rimas Silkaitis

Other Decks in Technology

Transcript

  1. The Enterprise Service Bus is Dead. Long Live the Enterprise

    Service Bus. Kafka Summit 2016 Rimas Silkaitis
  2. ESBs receive service consumption requests, route the requests to the

    correct service provider, transform the requests to a format compatible with the service provider, wait for the results, and deliver them back to the service consumer. IDC http://www.idc.com/2010st/esbcm.html
  3. Uses a bus architecture for scalability and reliability, establishes a

    unified communications channel, provides integration, mediation and communication services and is standards based. http://redmonk.com/public/OpenSourceESBs.pdf Redmonk
  4. Oracle Service Bus transforms complex and brittle architectures into agile

    integration networks by connecting, virtualizing, and managing interactions between services and applications. http://www.oracle.com/technetwork/middleware/service-bus/overview/index.html Oracle
  5. Mule ESB is a lightweight Java-based enterprise service bus (ESB)

    and integration platform that allows developers to connect applications together quickly and easily, enabling them to exchange data. https://www.mulesoft.com/resources/esb/what-mule-esb Mulesoft
  6. $ heroku addons:create heroku-postgresql:standard-0 -a sushi Creating postgresql-reticulated-74861... done, ($50.00/month)

    Adding postgresql-reticulated-74861 to sushi... done Setting HEROKU_POSTGRESQL_GRAY_URL and restarting sushi... done, v249 The database should be available in 3-5 minutes. ! The database will be empty. If upgrading, you can transfer ! data from another database with pg:copy. Use `heroku pg:wait` to track status Use `heroku addons:docs heroku-postgresql` to view documentation.
  7. Problems Emerge • Bottleneck in getting features released that touch

    API • Difficult to get Analytics across the entire Org • Hiring has become difficult • API is down. So is the entire platform.
  8. ESB Redefinition • Pull over Push • Consumers dictate their

    own consumption rate • Independent development, scaling and scheduling • Stop the world processing doesn’t exist
  9. aggregation router aggregation router metrics-ingest route r metrics-ingest route r

    ZK postgres metrics-ingest router aggregation router aggregation router aggregation router sink router aggregation router aggregation router metrics-api router
  10. $ heroku kafka:list -a metaas-addons === Kafka Topics on KAFKA_URL

    Name Messages Traffic ─────────────────────────────────── ────────────── ───────────── ROUTER_REQUEST.v1 132,823.12/sec 14.6 mb/sec DYNO_MEMORY.v1 1,478.97/sec 187 kb/sec DYNO_LOAD.v1 1,468.01/sec 153 kb/sec ROUTER_STATUS_SUMMARY.v1 987.4/sec 28.3 kb/sec SYSLOG.v1 691.9/sec 76.8 kb/sec ROUTER_LATENCY_SUMMARY.v1 678.84/sec 37.5 kb/sec ROUTER_ERROR.v1 476.12/sec 93.9 kb/sec DYNO_ERROR.v1 354.06/sec 38.5 kb/sec DYNO_LOAD_SUMMARY.v1 241.33/sec 9.73 kb/sec DYNO_MEMORY_SUMMARY.v1 219.26/sec 25.1 kb/sec DYNO_ERROR_SUMMARY.v1 73.49/sec 1.83 kb/sec ROUTER_ERROR_SUMMARY.v1 12.35/sec 276 bytes/sec __consumer_offsets 4.1/sec 405 bytes/sec ROUTER_STATUS_PARTIAL_SUMMARY.v1 0.0/sec 0 bytes/sec ROUTER_LATENCY_PARTIAL_HISTOGRAM.v1 0.0/sec 0 bytes/sec ROUTER_REQUESTS.v1 0.0/sec 0 bytes/sec
  11. What we’ve learned • Team by team roll out has

    been a huge help • Teams are able to react much more quickly • Reduced the scope of our API and it’s business logic
  12. Creating kafka-lively-27896... done, (free) Adding kafka-lively-27896 to sushi... done Kafka

    cluster has been created and will be available shortly Use `heroku addons:docs heroku-kafka` to view documentation. === KAFKA_URL Name: kafka-silhouetted-79119 Created: 2016-03-24 21:21 UTC Plan: Beta 3 Status: available Version: 0.9.0.0 Topics: 0 topics (see heroku kafka:list) Connections: 0 consumers (0 applications) Messages: 0.0 messages/sec Traffic: 0 Bytes/sec in / 0 Bytes/s out $ heroku kafka:info -a sushi $ heroku addons:create heroku-kafka:beta-3 -a sushi