The Enterprise Service Bus is Dead! Long Live the Enterprise Service Bus.

The enterprise service bus (ESB) is a concept and resulting set of software products that came to fruition in the early 2000s. The ESB was designed to contain the complexity of connecting heterogeneous applications, including the entire organization’s business logic and data. These concepts have fallen out of favor with the advent of microservices and Kafka. This existing definition of ESB’s is no longer useful to us; in this talk, we’ll introduce a modern definition of the ESB, with Kafka at its heart.

Transcript

1. The Enterprise Service Bus is Dead. Long Live the Enterprise Service Bus. Kafka Summit 2016 Rimas Silkaitis
2. how Heroku leveraged Kafka to realize the original vision of the enterprise service bus (esb)
4. app cloud
6. DEPLOY MANAGE SCALE
7. Rimas Silkaitis Product
8. Heroku Postgres Over 1 Million Active DBs Over 5 Petabytes
9. Heroku Redis Over 100K Active Instances Over 100 Terabytes
10. What is an ESB?
11. Analyst Definitions
12. ESBs receive service consumption requests, route the requests to the correct service provider, transform the requests to a format compatible with the service provider, wait for the results, and deliver them back to the service consumer. IDC http://www.idc.com/2010st/esbcm.html
13. Uses a bus architecture for scalability and reliability, establishes a unified communications channel, provides integration, mediation and communication services and is standards based. http://redmonk.com/public/OpenSourceESBs.pdf Redmonk
14. Vendor Definitions
15. Oracle Service Bus transforms complex and brittle architectures into agile integration networks by connecting, virtualizing, and managing interactions between services and applications. http://www.oracle.com/technetwork/middleware/service-bus/overview/index.html Oracle
16. Mule ESB is a lightweight Java-based enterprise service bus (ESB) and integration platform that allows developers to connect applications together quickly and easily, enabling them to exchange data. https://www.mulesoft.com/resources/esb/what-mule-esb Mulesoft
17. How should we define it?
18. ¯\_(ϑ)_/¯
19. App App App App Finance HR BizOps Inventory
20. ESB Routing Mediation Messaging Event Processing Service Orchestration
21. ESB Push Centralized
22. Architecture
23. API CLI GIT Dashboard Data Runtime BizOps
24. $ heroku addons:create heroku-postgresql:standard-0 -a sushi Creating postgresql-reticulated-74861... done, ($50.00/month) Adding postgresql-reticulated-74861 to sushi... done Setting HEROKU_POSTGRESQL_GRAY_URL and restarting sushi... done, v249 The database should be available in 3-5 minutes. ! The database will be empty. If upgrading, you can transfer ! data from another database with pg:copy. Use `heroku pg:wait` to track status Use `heroku addons:docs heroku-postgresql` to view documentation.
25. API CLI GIT Dashboard Data Runtime BizOps
26. API CLI GIT Web Data Runtime BizOps THE API IS OUR ESB!
27. Problems Emerge • Bottleneck in getting features released that touch API • Difficult to get Analytics across the entire Org • Hiring has become difficult • API is down. So is the entire platform.
29. PUSH vs PULL http://kafka.apache.org/documentation.html#design_pull
30. PUSH
31. PULL
32. ESB Redefinition • Pull over Push • Consumers dictate their own consumption rate • Independent development, scaling and scheduling • Stop the world processing doesn’t exist
33. CLI GIT Dashboard API Data Runtime BizOps
34. Other Use Cases Around Heroku
35. Operational Experience
36. aggregation router aggregation router metrics-ingest route r metrics-ingest route r ZK postgres metrics-ingest router aggregation router aggregation router aggregation router sink router aggregation router aggregation router metrics-api router
37. $ heroku kafka:list -a metaas-addons === Kafka Topics on KAFKA_URL Name Messages Traffic ─────────────────────────────────── ────────────── ───────────── ROUTER_REQUEST.v1 132,823.12/sec 14.6 mb/sec DYNO_MEMORY.v1 1,478.97/sec 187 kb/sec DYNO_LOAD.v1 1,468.01/sec 153 kb/sec ROUTER_STATUS_SUMMARY.v1 987.4/sec 28.3 kb/sec SYSLOG.v1 691.9/sec 76.8 kb/sec ROUTER_LATENCY_SUMMARY.v1 678.84/sec 37.5 kb/sec ROUTER_ERROR.v1 476.12/sec 93.9 kb/sec DYNO_ERROR.v1 354.06/sec 38.5 kb/sec DYNO_LOAD_SUMMARY.v1 241.33/sec 9.73 kb/sec DYNO_MEMORY_SUMMARY.v1 219.26/sec 25.1 kb/sec DYNO_ERROR_SUMMARY.v1 73.49/sec 1.83 kb/sec ROUTER_ERROR_SUMMARY.v1 12.35/sec 276 bytes/sec __consumer_offsets 4.1/sec 405 bytes/sec ROUTER_STATUS_PARTIAL_SUMMARY.v1 0.0/sec 0 bytes/sec ROUTER_LATENCY_PARTIAL_HISTOGRAM.v1 0.0/sec 0 bytes/sec ROUTER_REQUESTS.v1 0.0/sec 0 bytes/sec
38. What do we need to do next?
39. Schema Registry
41. Security
42. What we’ve learned • Team by team roll out has been a huge help • Teams are able to react much more quickly • Reduced the scope of our API and it’s business logic
44. Kafka as a Service
45. Creating kafka-lively-27896... done, (free) Adding kafka-lively-27896 to sushi... done Kafka cluster has been created and will be available shortly Use `heroku addons:docs heroku-kafka` to view documentation. === KAFKA_URL Name: kafka-silhouetted-79119 Created: 2016-03-24 21:21 UTC Plan: Beta 3 Status: available Version: 0.9.0.0 Topics: 0 topics (see heroku kafka:list) Connections: 0 consumers (0 applications) Messages: 0.0 messages/sec Traffic: 0 Bytes/sec in / 0 Bytes/s out $ heroku kafka:info -a sushi $ heroku addons:create heroku-kafka:beta-3 -a sushi
47. Early Access https://www.heroku.com/kafka
48. Come See Us At Our Booth We’re Hiring!