Cloud-native Streaming and Event-Driven Microservices

Slide 1

Slide 1 text

CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES Marius Bogoevici @mariusbogoevici

Slide 2

Slide 2 text

CLOUD-NATIVE STREAMING AND EVENT- DRIVEN MICROSERVICES • Use cases: • predictive maintenance, fraud detection, QoS, IoT • enterprise integration meets ‘Big Data’ • Specific requirements • High throughput, low latency • Grouping, ordering, windowing • Middleware adaptations: beyond JMS, Rabbit MQ • Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs • Brokerless: Aeron, ZeroMQ Spring I/O 2016

Slide 3

Slide 3 text

CLOUD-NATIVE STREAMING AND EVENT- DRIVEN MICROSERVICES • Architectural style advantages: • bounded context, development agility • Process-specific resource tuning: • memory, CPU, instance count • Messaging vs. HTTP • Discoverability • Availability • Consistency across heterogenous resources Spring I/O 2016

Slide 4

Slide 4 text

CLOUD-NATIVE STREAMING AND EVENT- DRIVEN MICROSERVICES • Modern platforms • Resource management • memory, CPU, instance count • Scaling up/down • Health monitoring and failover • Routing and load balancing Spring I/O 2016 Apache YARN Apache Mesos Kubernetes

Slide 5

Slide 5 text

Spring I/O 2016 dataflow:> stream create demo --definition “http | file” Goal

Slide 6

Slide 6 text

Spring Cloud Stream Spring Cloud Data Flow Spring I/O 2016 Spring Cloud Data Flow Spring Cloud Stream Spring Cloud Task 2015

Slide 7

Slide 7 text

Spring Cloud Stream • Event-driven microservice framework • Built on battle-tested components • Spring Boot: full-stack standalone applications • Spring Integration: Messaging, EIP patterns, connectors • Opinionated primitives • Pluggable middleware abstractions Spring I/O 2016

Slide 8

Slide 8 text

Spring Cloud Stream in a nutshell Spring I/O 2016

Slide 9

Slide 9 text

…. 10000 ft nutshell Spring I/O 2016

Slide 10

Slide 10 text

Durable publish-subscribe • Fits both streaming and event-driven use cases Spring I/O 2016

Slide 11

Slide 11 text

Consumer Groups • Borrowed from Kafka, applied to all binders • Groups of competing consumers within the pub-sub destination • Used for scaling and partitioning Spring I/O 2016

Slide 12

Slide 12 text

Partitioning • Required for stateful processing • Outputs specify a partitioning strategy • Inputs consuming from specific partition Spring I/O 2016

Slide 13

Slide 13 text

Programming model @EnableBinding + Binder implementation Apache Kafka Gemfire

Slide 14

Slide 14 text

Spring I/O 2016 DEMO

Slide 15

Slide 15 text

Spring Cloud Data Flow • Orchestration: • DSL for Stream topologies • REST API • Shell • UI • Portable Deployment SPI • OOTB apps for common integration use- cases Spring I/O 2016

Slide 16

Slide 16 text

Stream DSL: pipes and filters Stream definition Boot Apps build on top of Spring Ecosystem httpfile = http | file | Spring I/O 2016 ‘pipe’, or how to communicate Rabbit, Kafka, Gemfire

Slide 17

Slide 17 text

Spring Cloud Deployer • SPI for deploying applications to modern runtimes • Local (for testing) • Cloud Foundry • YARN • Kubernetes • Mesos + Marathon • Different resources supported • Spring Boot Uberjars • Docker Images • Different locations • File system, HDFS, HTTP, Maven, Docker Hub Spring I/O 2016

Slide 18

Slide 18 text

Spring Cloud Data Flow Pla$orm Spring I/O 2016

Slide 19

Slide 19 text

Spring I/O 2016 DEMO

Slide 20

Slide 20 text

Timeline • Spring Cloud Stream • 1.0.0.GA May 10, 2016 • 1.1.0.M1 July 2016 • 1.1.0.GA Q3 2016 • Spring Cloud Data Flow • 1.0.0.M3 May 7, 2016 • 1.0.0.RC1 End of May, 2016 Spring I/O 2016

Slide 21

Slide 21 text

Spring Cloud Stream 1.1.0.M1 • Reactive APIs: Reactor, RxJava • Kafka 0.9 • Schema evolution support

Slide 22

Slide 22 text

Spring I/O 2016 Q&A Marius Bogoevici twitter: @mariusbogoevici