Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud-native Streaming and Event-Driven Microservices

Cloud-native Streaming and Event-Driven Microservices

Talk delivered at Spring IO in Barcelona (May 20, 2016)

The talk introduces Spring Cloud Stream as a framework for developing event-driven microservices, as well as Spring Cloud Data Flow as an orchestration and deployment layer.

Ee7ff5474c7ecfe0ec209df0eeb531fa?s=128

Marius Bogoevici

May 20, 2016
Tweet

Transcript

  1. CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES Marius Bogoevici @mariusbogoevici

  2. CLOUD-NATIVE STREAMING AND EVENT- DRIVEN MICROSERVICES • Use cases: •

    predictive maintenance, fraud detection, QoS, IoT • enterprise integration meets ‘Big Data’ • Specific requirements • High throughput, low latency • Grouping, ordering, windowing • Middleware adaptations: beyond JMS, Rabbit MQ • Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs • Brokerless: Aeron, ZeroMQ Spring I/O 2016
  3. CLOUD-NATIVE STREAMING AND EVENT- DRIVEN MICROSERVICES • Architectural style advantages:

    • bounded context, development agility • Process-specific resource tuning: • memory, CPU, instance count • Messaging vs. HTTP • Discoverability • Availability • Consistency across heterogenous resources Spring I/O 2016
  4. CLOUD-NATIVE STREAMING AND EVENT- DRIVEN MICROSERVICES • Modern platforms •

    Resource management • memory, CPU, instance count • Scaling up/down • Health monitoring and failover • Routing and load balancing Spring I/O 2016 Apache YARN Apache Mesos Kubernetes
  5. Spring I/O 2016 dataflow:> stream create demo --definition “http |

    file” Goal
  6. Spring Cloud Stream Spring Cloud Data Flow Spring I/O 2016

    Spring Cloud Data Flow Spring Cloud Stream Spring Cloud Task 2015
  7. Spring Cloud Stream • Event-driven microservice framework • Built on

    battle-tested components • Spring Boot: full-stack standalone applications • Spring Integration: Messaging, EIP patterns, connectors • Opinionated primitives • Pluggable middleware abstractions Spring I/O 2016
  8. Spring Cloud Stream in a nutshell Spring I/O 2016

  9. …. 10000 ft nutshell Spring I/O 2016

  10. Durable publish-subscribe • Fits both streaming and event-driven use cases

    Spring I/O 2016
  11. Consumer Groups • Borrowed from Kafka, applied to all binders

    • Groups of competing consumers within the pub-sub destination • Used for scaling and partitioning Spring I/O 2016
  12. Partitioning • Required for stateful processing • Outputs specify a

    partitioning strategy • Inputs consuming from specific partition Spring I/O 2016
  13. Programming model @EnableBinding + Binder implementation Apache Kafka Gemfire

  14. Spring I/O 2016 DEMO

  15. Spring Cloud Data Flow • Orchestration: • DSL for Stream

    topologies • REST API • Shell • UI • Portable Deployment SPI • OOTB apps for common integration use- cases Spring I/O 2016
  16. Stream DSL: pipes and filters Stream definition Boot Apps build

    on top of Spring Ecosystem httpfile = http | file | Spring I/O 2016 ‘pipe’, or how to communicate Rabbit, Kafka, Gemfire
  17. Spring Cloud Deployer • SPI for deploying applications to modern

    runtimes • Local (for testing) • Cloud Foundry • YARN • Kubernetes • Mesos + Marathon • Different resources supported • Spring Boot Uberjars • Docker Images • Different locations • File system, HDFS, HTTP, Maven, Docker Hub Spring I/O 2016
  18. Spring Cloud Data Flow Pla$orm Spring I/O 2016

  19. Spring I/O 2016 DEMO

  20. Timeline • Spring Cloud Stream • 1.0.0.GA May 10, 2016

    • 1.1.0.M1 July 2016 • 1.1.0.GA Q3 2016 • Spring Cloud Data Flow • 1.0.0.M3 May 7, 2016 • 1.0.0.RC1 End of May, 2016 Spring I/O 2016
  21. Spring Cloud Stream 1.1.0.M1 • Reactive APIs: Reactor, RxJava •

    Kafka 0.9 • Schema evolution support
  22. Spring I/O 2016 Q&A Marius Bogoevici twitter: @mariusbogoevici