Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Enterprise Service Bus is Dead! Long Live the Enterprise Service Bus.

The Enterprise Service Bus is Dead! Long Live the Enterprise Service Bus.

The enterprise service bus (ESB) is a concept and resulting set of software products that came to fruition in the early 2000s. The ESB was designed to contain the complexity of connecting heterogeneous applications, including the entire organization’s business logic and data. These concepts have fallen out of favor with the advent of microservices and Kafka. This existing definition of ESB’s is no longer useful to us; in this talk, we’ll introduce a modern definition of the ESB, with Kafka at its heart.


Rimas Silkaitis

April 26, 2016


  1. The Enterprise Service Bus is Dead. Long Live the Enterprise

    Service Bus. Kafka Summit 2016 Rimas Silkaitis
  2. how Heroku leveraged Kafka to realize the original vision of

    the enterprise service bus (esb)
  3. None
  4. app cloud

  5. None

  7. Rimas Silkaitis Product

  8. Heroku Postgres Over 1 Million Active DBs Over 5 Petabytes

  9. Heroku Redis Over 100K Active Instances Over 100 Terabytes

  10. What is an ESB?

  11. Analyst Definitions

  12. ESBs receive service consumption requests, route the requests to the

    correct service provider, transform the requests to a format compatible with the service provider, wait for the results, and deliver them back to the service consumer. IDC http://www.idc.com/2010st/esbcm.html
  13. Uses a bus architecture for scalability and reliability, establishes a

    unified communications channel, provides integration, mediation and communication services and is standards based. http://redmonk.com/public/OpenSourceESBs.pdf Redmonk
  14. Vendor Definitions

  15. Oracle Service Bus transforms complex and brittle architectures into agile

    integration networks by connecting, virtualizing, and managing interactions between services and applications. http://www.oracle.com/technetwork/middleware/service-bus/overview/index.html Oracle
  16. Mule ESB is a lightweight Java-based enterprise service bus (ESB)

    and integration platform that allows developers to connect applications together quickly and easily, enabling them to exchange data. https://www.mulesoft.com/resources/esb/what-mule-esb Mulesoft
  17. How should we define it?

  18. ¯\_(ϑ)_/¯

  19. App App App App Finance HR BizOps Inventory

  20. ESB Routing Mediation Messaging Event Processing Service Orchestration

  21. ESB Push Centralized

  22. Architecture

  23. API CLI GIT Dashboard Data Runtime BizOps

  24. $ heroku addons:create heroku-postgresql:standard-0 -a sushi Creating postgresql-reticulated-74861... done, ($50.00/month)

    Adding postgresql-reticulated-74861 to sushi... done Setting HEROKU_POSTGRESQL_GRAY_URL and restarting sushi... done, v249 The database should be available in 3-5 minutes. ! The database will be empty. If upgrading, you can transfer ! data from another database with pg:copy. Use `heroku pg:wait` to track status Use `heroku addons:docs heroku-postgresql` to view documentation.
  25. API CLI GIT Dashboard Data Runtime BizOps

  26. API CLI GIT Web Data Runtime BizOps THE API IS

    OUR ESB!
  27. Problems Emerge • Bottleneck in getting features released that touch

    API • Difficult to get Analytics across the entire Org • Hiring has become difficult • API is down. So is the entire platform.
  28. None
  29. PUSH vs PULL http://kafka.apache.org/documentation.html#design_pull

  30. PUSH

  31. PULL

  32. ESB Redefinition • Pull over Push • Consumers dictate their

    own consumption rate • Independent development, scaling and scheduling • Stop the world processing doesn’t exist
  33. CLI GIT Dashboard API Data Runtime BizOps

  34. Other Use Cases Around Heroku

  35. Operational Experience

  36. aggregation router aggregation router metrics-ingest route r metrics-ingest route r

    ZK postgres metrics-ingest router aggregation router aggregation router aggregation router sink router aggregation router aggregation router metrics-api router
  37. $ heroku kafka:list -a metaas-addons === Kafka Topics on KAFKA_URL

    Name Messages Traffic ─────────────────────────────────── ────────────── ───────────── ROUTER_REQUEST.v1 132,823.12/sec 14.6 mb/sec DYNO_MEMORY.v1 1,478.97/sec 187 kb/sec DYNO_LOAD.v1 1,468.01/sec 153 kb/sec ROUTER_STATUS_SUMMARY.v1 987.4/sec 28.3 kb/sec SYSLOG.v1 691.9/sec 76.8 kb/sec ROUTER_LATENCY_SUMMARY.v1 678.84/sec 37.5 kb/sec ROUTER_ERROR.v1 476.12/sec 93.9 kb/sec DYNO_ERROR.v1 354.06/sec 38.5 kb/sec DYNO_LOAD_SUMMARY.v1 241.33/sec 9.73 kb/sec DYNO_MEMORY_SUMMARY.v1 219.26/sec 25.1 kb/sec DYNO_ERROR_SUMMARY.v1 73.49/sec 1.83 kb/sec ROUTER_ERROR_SUMMARY.v1 12.35/sec 276 bytes/sec __consumer_offsets 4.1/sec 405 bytes/sec ROUTER_STATUS_PARTIAL_SUMMARY.v1 0.0/sec 0 bytes/sec ROUTER_LATENCY_PARTIAL_HISTOGRAM.v1 0.0/sec 0 bytes/sec ROUTER_REQUESTS.v1 0.0/sec 0 bytes/sec
  38. What do we need to do next?

  39. Schema Registry

  40. None
  41. Security

  42. What we’ve learned • Team by team roll out has

    been a huge help • Teams are able to react much more quickly • Reduced the scope of our API and it’s business logic
  43. None
  44. Kafka as a Service

  45. Creating kafka-lively-27896... done, (free) Adding kafka-lively-27896 to sushi... done Kafka

    cluster has been created and will be available shortly Use `heroku addons:docs heroku-kafka` to view documentation. === KAFKA_URL Name: kafka-silhouetted-79119 Created: 2016-03-24 21:21 UTC Plan: Beta 3 Status: available Version: Topics: 0 topics (see heroku kafka:list) Connections: 0 consumers (0 applications) Messages: 0.0 messages/sec Traffic: 0 Bytes/sec in / 0 Bytes/s out $ heroku kafka:info -a sushi $ heroku addons:create heroku-kafka:beta-3 -a sushi
  46. None
  47. Early Access https://www.heroku.com/kafka

  48. Come See Us At Our Booth We’re Hiring!