Cloud-native Streaming and Event-Driven Microservices

Talk delivered at Spring IO in Barcelona (May 20, 2016)

The talk introduces Spring Cloud Stream as a framework for developing event-driven microservices, as well as Spring Cloud Data Flow as an orchestration and deployment layer.

Transcript

1. CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES Marius Bogoevici @mariusbogoevici
2. CLOUD-NATIVE STREAMING AND EVENT- DRIVEN MICROSERVICES • Use cases: • predictive maintenance, fraud detection, QoS, IoT • enterprise integration meets ‘Big Data’ • Specific requirements • High throughput, low latency • Grouping, ordering, windowing • Middleware adaptations: beyond JMS, Rabbit MQ • Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs • Brokerless: Aeron, ZeroMQ Spring I/O 2016
3. CLOUD-NATIVE STREAMING AND EVENT- DRIVEN MICROSERVICES • Architectural style advantages: • bounded context, development agility • Process-specific resource tuning: • memory, CPU, instance count • Messaging vs. HTTP • Discoverability • Availability • Consistency across heterogenous resources Spring I/O 2016
4. CLOUD-NATIVE STREAMING AND EVENT- DRIVEN MICROSERVICES • Modern platforms • Resource management • memory, CPU, instance count • Scaling up/down • Health monitoring and failover • Routing and load balancing Spring I/O 2016 Apache YARN Apache Mesos Kubernetes
5. Spring I/O 2016 dataflow:> stream create demo --definition “http | file” Goal
6. Spring Cloud Stream Spring Cloud Data Flow Spring I/O 2016 Spring Cloud Data Flow Spring Cloud Stream Spring Cloud Task 2015
7. Spring Cloud Stream • Event-driven microservice framework • Built on battle-tested components • Spring Boot: full-stack standalone applications • Spring Integration: Messaging, EIP patterns, connectors • Opinionated primitives • Pluggable middleware abstractions Spring I/O 2016
8. Spring Cloud Stream in a nutshell Spring I/O 2016
9. …. 10000 ft nutshell Spring I/O 2016
10. Durable publish-subscribe • Fits both streaming and event-driven use cases Spring I/O 2016
11. Consumer Groups • Borrowed from Kafka, applied to all binders • Groups of competing consumers within the pub-sub destination • Used for scaling and partitioning Spring I/O 2016
12. Partitioning • Required for stateful processing • Outputs specify a partitioning strategy • Inputs consuming from specific partition Spring I/O 2016
13. Programming model @EnableBinding + Binder implementation Apache Kafka Gemfire
14. Spring I/O 2016 DEMO
15. Spring Cloud Data Flow • Orchestration: • DSL for Stream topologies • REST API • Shell • UI • Portable Deployment SPI • OOTB apps for common integration use- cases Spring I/O 2016
16. Stream DSL: pipes and filters Stream definition Boot Apps build on top of Spring Ecosystem httpfile = http | file | Spring I/O 2016 ‘pipe’, or how to communicate Rabbit, Kafka, Gemfire
17. Spring Cloud Deployer • SPI for deploying applications to modern runtimes • Local (for testing) • Cloud Foundry • YARN • Kubernetes • Mesos + Marathon • Different resources supported • Spring Boot Uberjars • Docker Images • Different locations • File system, HDFS, HTTP, Maven, Docker Hub Spring I/O 2016
18. Spring Cloud Data Flow Pla$orm Spring I/O 2016
19. Spring I/O 2016 DEMO
20. Timeline • Spring Cloud Stream • 1.0.0.GA May 10, 2016 • 1.1.0.M1 July 2016 • 1.1.0.GA Q3 2016 • Spring Cloud Data Flow • 1.0.0.M3 May 7, 2016 • 1.0.0.RC1 End of May, 2016 Spring I/O 2016
21. Spring Cloud Stream 1.1.0.M1 • Reactive APIs: Reactor, RxJava • Kafka 0.9 • Schema evolution support
22. Spring I/O 2016 Q&A Marius Bogoevici twitter: @mariusbogoevici