Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Native Streaming and Event Driven Microservices

Cloud Native Streaming and Event Driven Microservices

Slides from the talk delivered at Devoxx Belgium


Marius Bogoevici

November 10, 2016


  1. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Cloud Native Event-Driven and Streaming Microservices

    Marius Bogoevici, Pivotal @mariusbogoevici
  2. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Streaming • Processing large quantities of

    data as fast as possible (near real-time) • Use cases: • predictive maintenance, fraud detection, QoS, IoT • enterprise integration meets ‘Big Data’ • High throughput / low latency • Grouping, ordering, windowing • Transition in messaging middleware design: • Traditional (JMS-based, RabbitMQ) • To log-style: Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs
  3. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Microservices • Preferred model for distributed

    data processing applications • Failure isolation • Independent scaling • Process-specific resource tuning: • memory/CPU/Instance count/network • Typical benefits and drawbacks of microservices: • bounded context as cohesive units • development agility • added complexity
  4. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Event-driven • Messaging vs HTTP/“fire-and-forget” vs.

    request-reply • Decoupling • Discoverability • Availability • Eventual consistency vs distributed transactions • especially over heterogenous resources • Strong similarities between: • data streaming - ‘big data’: ingestion, analytics • event streaming - async interaction, event sourcing, etc
  5. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Data streaming vs Event sourcing

  6. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Cloud Native • Focus on modern

    platform capabilities • Scaling up/down • Resource management (memory, CPU, instance count) • Health monitoring and failover • Routing and load balancing Apache YARN Apache Mesos Kubernetes
  7. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream In a single sentence … dataflow:>

    stream create demo --definition “http | file”
  8. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Stream 2015 Spring XD

    Spring Cloud Data Flow Spring Cloud Task
  9. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Stream • Event-driven microservice

    framework • Built on battle-tested components • Spring Boot: full-stack standalone applications • Spring Integration: Messaging, EIP patterns, connectors • Focusing on scalable, structured distributed applications • Opinionated primitives • Pluggable middleware abstractions
  10. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Stream in a nutshell

    Application Core Spring Boot Spring Integration Spring Messaging Reactive APIs inputs outputs
  11. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream and a 10000 ft nutshell …

  12. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream The binder abstraction

  13. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Programming model @EnableBinding + Binder Implementation

    Apache Kafka JMS Google PubSub Production-ready: Experimental
  14. Demo @mariusbogoevici @springcloud #Devoxx #SpringCloudStream

  15. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Durable Pub Sub

  16. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Consumer Groups

  17. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Partitioning

  18. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Data Flow • Orchestration:

    • DSL for Stream topologies • REST API • Shell • UI • Portable Deployment SPI • OOTB apps for common integration use-cases
  19. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Stream DSL Stream Definition Spring Boot

    applications built on top of Spring ecosystem
  20. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Data Flow Deployment Pla$orm

  21. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Deployer • SPI for

    deploying applications to modern runtimes • Local (for testing) • Cloud Foundry • YARN • Kubernetes • Mesos + Marathon • Different resources supported • Spring Boot Uberjars • Docker Images • Different resource location supported • File system, HDFS, HTTP, Maven, Docker Hub
  22. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Data Flow Streams

  23. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Deployment and orchestration

  24. Demo @mariusbogoevici @springcloud #Devoxx #SpringCloudStream