Cloud Native Streaming and Event Driven Microservices

Slides from the talk delivered at Devoxx Belgium

Transcript

1. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Cloud Native Event-Driven and Streaming Microservices Marius Bogoevici, Pivotal @mariusbogoevici
2. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Streaming • Processing large quantities of data as fast as possible (near real-time) • Use cases: • predictive maintenance, fraud detection, QoS, IoT • enterprise integration meets ‘Big Data’ • High throughput / low latency • Grouping, ordering, windowing • Transition in messaging middleware design: • Traditional (JMS-based, RabbitMQ) • To log-style: Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs
3. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Microservices • Preferred model for distributed data processing applications • Failure isolation • Independent scaling • Process-specific resource tuning: • memory/CPU/Instance count/network • Typical benefits and drawbacks of microservices: • bounded context as cohesive units • development agility • added complexity
4. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Event-driven • Messaging vs HTTP/“fire-and-forget” vs. request-reply • Decoupling • Discoverability • Availability • Eventual consistency vs distributed transactions • especially over heterogenous resources • Strong similarities between: • data streaming - ‘big data’: ingestion, analytics • event streaming - async interaction, event sourcing, etc
5. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Data streaming vs Event sourcing
6. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Cloud Native • Focus on modern platform capabilities • Scaling up/down • Resource management (memory, CPU, instance count) • Health monitoring and failover • Routing and load balancing Apache YARN Apache Mesos Kubernetes
7. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream In a single sentence … dataflow:> stream create demo --definition “http | file”
8. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Stream 2015 Spring XD Spring Cloud Data Flow Spring Cloud Task
9. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Stream • Event-driven microservice framework • Built on battle-tested components • Spring Boot: full-stack standalone applications • Spring Integration: Messaging, EIP patterns, connectors • Focusing on scalable, structured distributed applications • Opinionated primitives • Pluggable middleware abstractions
10. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Stream in a nutshell Application Core Spring Boot Spring Integration Spring Messaging Reactive APIs inputs outputs
11. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream and a 10000 ft nutshell …
12. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream The binder abstraction
13. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Programming model @EnableBinding + Binder Implementation Apache Kafka JMS Google PubSub Production-ready: Experimental
14. Demo @mariusbogoevici @springcloud #Devoxx #SpringCloudStream
15. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Durable Pub Sub
16. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Consumer Groups
17. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Partitioning
18. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Data Flow • Orchestration: • DSL for Stream topologies • REST API • Shell • UI • Portable Deployment SPI • OOTB apps for common integration use-cases
19. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Stream DSL Stream Definition Spring Boot applications built on top of Spring ecosystem
20. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Data Flow Deployment Pla$orm
21. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Deployer • SPI for deploying applications to modern runtimes • Local (for testing) • Cloud Foundry • YARN • Kubernetes • Mesos + Marathon • Different resources supported • Spring Boot Uberjars • Docker Images • Different resource location supported • File system, HDFS, HTTP, Maven, Docker Hub
22. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Spring Cloud Data Flow Streams
23. @mariusbogoevici @springcloud #Devoxx #SpringCloudStream Deployment and orchestration
24. Demo @mariusbogoevici @springcloud #Devoxx #SpringCloudStream