The Enterprise Service Bus is Dead! Long Live the Enterprise Service Bus.

The enterprise service bus (ESB) is a concept and resulting set of software products that came to fruition in the early 2000s. The ESB was designed to contain the complexity of connecting heterogeneous applications, including the entire organization’s business logic and data. These concepts have fallen out of favor with the advent of microservices and Kafka. This existing definition of ESB’s is no longer useful to us; in this talk, we’ll introduce a modern definition of the ESB, with Kafka at its heart.

Transcript

1. The Enterprise Service Bus is Dead.
   Long Live the Enterprise Service Bus.
   Kafka Summit 2016
   Rimas Silkaitis
   

   View Slide

2. how Heroku leveraged Kafka to realize
   the original vision of the enterprise
   service bus (esb)
   

   View Slide

3. View Slide

4. app cloud
   

   View Slide

5. View Slide

6. DEPLOY MANAGE SCALE
   

   View Slide

7. Rimas Silkaitis
   Product
   

   View Slide

8. Heroku Postgres
   Over 1 Million Active DBs
   Over 5 Petabytes
   

   View Slide

9. Heroku Redis
   Over 100K Active Instances
   Over 100 Terabytes
   

   View Slide

10. What is an ESB?
    

    View Slide

11. Analyst Definitions
    

    View Slide

12. ESBs receive service consumption requests,
    route the requests to the correct service
    provider, transform the requests to a format
    compatible with the service provider, wait for
    the results, and deliver them back to the service
    consumer.
    IDC
    http://www.idc.com/2010st/esbcm.html
    

    View Slide

13. Uses a bus architecture for scalability and
    reliability, establishes a unified communications
    channel, provides integration, mediation and
    communication services and is standards based.
    http://redmonk.com/public/OpenSourceESBs.pdf
    Redmonk
    

    View Slide

14. Vendor Definitions
    

    View Slide

15. Oracle Service Bus transforms complex and
    brittle architectures into agile integration
    networks by connecting, virtualizing, and
    managing interactions between services and
    applications.
    http://www.oracle.com/technetwork/middleware/service-bus/overview/index.html
    Oracle
    

    View Slide

16. Mule ESB is a lightweight Java-based enterprise
    service bus (ESB) and integration platform that
    allows developers to connect applications
    together quickly and easily, enabling them to
    exchange data.
    https://www.mulesoft.com/resources/esb/what-mule-esb
    Mulesoft
    

    View Slide

17. How should we define it?
    

    View Slide

18. ¯\_(ϑ)_/¯
    

    View Slide

19. App
    App
    App
    App
    Finance
    HR
    BizOps
    Inventory
    

    View Slide

20. ESB
    Routing
    Mediation
    Messaging
    Event
    Processing
    Service
    Orchestration
    

    View Slide

21. ESB
    Push
    Centralized
    

    View Slide

22. Architecture
    

    View Slide

23. API
    CLI GIT Dashboard
    Data
    Runtime
    BizOps
    

    View Slide

24. $ heroku addons:create heroku-postgresql:standard-0 -a sushi
    Creating postgresql-reticulated-74861... done, ($50.00/month)
    Adding postgresql-reticulated-74861 to sushi... done
    Setting HEROKU_POSTGRESQL_GRAY_URL and restarting sushi... done, v249
    The database should be available in 3-5 minutes.
    ! The database will be empty. If upgrading, you can transfer
    ! data from another database with pg:copy.
    Use `heroku pg:wait` to track status
    Use `heroku addons:docs heroku-postgresql` to view documentation.
    

    View Slide

25. API
    CLI GIT Dashboard
    Data
    Runtime
    BizOps
    

    View Slide

26. API
    CLI GIT Web
    Data
    Runtime
    BizOps
    THE API IS OUR ESB!
    

    View Slide

27. Problems Emerge
    • Bottleneck in getting features released that touch API
    • Difficult to get Analytics across the entire Org
    • Hiring has become difficult
    • API is down. So is the entire platform.
    

    View Slide

28. View Slide

29. PUSH vs PULL
    http://kafka.apache.org/documentation.html#design_pull
    

    View Slide

30. PUSH
    

    View Slide

31. PULL
    

    View Slide

32. ESB Redefinition
    • Pull over Push
    • Consumers dictate their own consumption rate
    • Independent development, scaling and scheduling
    • Stop the world processing doesn’t exist
    

    View Slide

33. CLI GIT Dashboard
    API Data Runtime BizOps
    

    View Slide

34. Other Use Cases
    Around Heroku
    

    View Slide

35. Operational Experience
    

    View Slide

36. aggregation
    router
    aggregation
    router
    metrics-ingest
    route
    r
    metrics-ingest
    route
    r
    ZK postgres
    metrics-ingest
    router
    aggregation
    router
    aggregation
    router
    aggregation
    router
    sink
    router
    aggregation
    router
    aggregation
    router
    metrics-api
    router
    

    View Slide

37. $ heroku kafka:list -a metaas-addons
    === Kafka Topics on KAFKA_URL
    Name Messages Traffic
    ─────────────────────────────────── ────────────── ─────────────
    ROUTER_REQUEST.v1 132,823.12/sec 14.6 mb/sec
    DYNO_MEMORY.v1 1,478.97/sec 187 kb/sec
    DYNO_LOAD.v1 1,468.01/sec 153 kb/sec
    ROUTER_STATUS_SUMMARY.v1 987.4/sec 28.3 kb/sec
    SYSLOG.v1 691.9/sec 76.8 kb/sec
    ROUTER_LATENCY_SUMMARY.v1 678.84/sec 37.5 kb/sec
    ROUTER_ERROR.v1 476.12/sec 93.9 kb/sec
    DYNO_ERROR.v1 354.06/sec 38.5 kb/sec
    DYNO_LOAD_SUMMARY.v1 241.33/sec 9.73 kb/sec
    DYNO_MEMORY_SUMMARY.v1 219.26/sec 25.1 kb/sec
    DYNO_ERROR_SUMMARY.v1 73.49/sec 1.83 kb/sec
    ROUTER_ERROR_SUMMARY.v1 12.35/sec 276 bytes/sec
    __consumer_offsets 4.1/sec 405 bytes/sec
    ROUTER_STATUS_PARTIAL_SUMMARY.v1 0.0/sec 0 bytes/sec
    ROUTER_LATENCY_PARTIAL_HISTOGRAM.v1 0.0/sec 0 bytes/sec
    ROUTER_REQUESTS.v1 0.0/sec 0 bytes/sec
    

    View Slide

38. What do we need to do next?
    

    View Slide

39. Schema Registry
    

    View Slide

40. View Slide

41. Security
    

    View Slide

42. What we’ve learned
    • Team by team roll out has been a huge help
    • Teams are able to react much more quickly
    • Reduced the scope of our API and it’s business logic
    

    View Slide

43. View Slide

44. Kafka as a Service
    

    View Slide

45. Creating kafka-lively-27896... done, (free)
    Adding kafka-lively-27896 to sushi... done
    Kafka cluster has been created and will be available shortly
    Use `heroku addons:docs heroku-kafka` to view documentation.
    === KAFKA_URL
    Name: kafka-silhouetted-79119
    Created: 2016-03-24 21:21 UTC
    Plan: Beta 3
    Status: available
    Version: 0.9.0.0
    Topics: 0 topics (see heroku kafka:list)
    Connections: 0 consumers (0 applications)
    Messages: 0.0 messages/sec
    Traffic: 0 Bytes/sec in / 0 Bytes/s out
    $ heroku kafka:info -a sushi
    $ heroku addons:create heroku-kafka:beta-3 -a sushi
    

    View Slide

46. View Slide

47. Early Access
    https://www.heroku.com/kafka
    

    View Slide

48. Come See Us At Our Booth
    We’re Hiring!
    

    View Slide