Cloud-native Streaming and Event-Driven Microservices

Talk delivered at Spring IO in Barcelona (May 20, 2016)

The talk introduces Spring Cloud Stream as a framework for developing event-driven microservices, as well as Spring Cloud Data Flow as an orchestration and deployment layer.

Transcript

1. CLOUD-NATIVE STREAMING AND
   EVENT-DRIVEN MICROSERVICES
   Marius Bogoevici
   @mariusbogoevici
   

   View Slide

2. CLOUD-NATIVE STREAMING AND EVENT-
   DRIVEN MICROSERVICES
   • Use cases:
   • predictive maintenance, fraud detection, QoS, IoT
   • enterprise integration meets ‘Big Data’
   • Specific requirements
   • High throughput, low latency
   • Grouping, ordering, windowing
   • Middleware adaptations: beyond JMS, Rabbit MQ
   • Kafka, Amazon Kinesis, Google Pub/Sub, Azure
   Event Hubs
   • Brokerless: Aeron, ZeroMQ
   Spring I/O
   2016
   

   View Slide

3. CLOUD-NATIVE STREAMING AND EVENT-
   DRIVEN MICROSERVICES
   • Architectural style advantages:
   • bounded context, development agility
   • Process-specific resource tuning:
   • memory, CPU, instance count
   • Messaging vs. HTTP
   • Discoverability
   • Availability
   • Consistency across heterogenous resources
   Spring I/O
   2016
   

   View Slide

4. CLOUD-NATIVE STREAMING AND EVENT-
   DRIVEN MICROSERVICES
   • Modern platforms
   • Resource management
   • memory, CPU, instance count
   • Scaling up/down
   • Health monitoring and failover
   • Routing and load balancing
   Spring I/O
   2016
   Apache YARN
   Apache Mesos Kubernetes
   

   View Slide

5. Spring I/O
   2016
   dataflow:> stream create demo --definition “http | file”
   Goal
   

   View Slide

6. Spring Cloud Stream
   Spring Cloud Data Flow
   Spring I/O
   2016
   Spring Cloud Data Flow
   Spring Cloud Stream
   Spring Cloud Task
   2015
   

   View Slide

7. Spring Cloud Stream
   • Event-driven microservice framework
   • Built on battle-tested components
   • Spring Boot: full-stack standalone
   applications
   • Spring Integration: Messaging, EIP
   patterns, connectors
   • Opinionated primitives
   • Pluggable middleware abstractions
   Spring I/O
   2016
   

   View Slide

8. Spring Cloud Stream in a nutshell
   Spring I/O
   2016
   

   View Slide

9. …. 10000 ft nutshell
   Spring I/O
   2016
   

   View Slide

10. Durable publish-subscribe
    • Fits both streaming and event-driven use cases
    Spring I/O
    2016
    

    View Slide

11. Consumer Groups
    • Borrowed from Kafka, applied to all binders
    • Groups of competing consumers within the pub-sub
    destination
    • Used for scaling and partitioning
    Spring I/O
    2016
    

    View Slide

12. Partitioning
    • Required for stateful processing
    • Outputs specify a partitioning strategy
    • Inputs consuming from specific partition
    Spring I/O
    2016
    

    View Slide

13. Programming model
    @EnableBinding + Binder implementation
    Apache Kafka
    Gemfire
    

    View Slide

14. Spring I/O 2016
    DEMO
    

    View Slide

15. Spring Cloud Data Flow
    • Orchestration:
    • DSL for Stream topologies
    • REST API
    • Shell
    • UI
    • Portable Deployment SPI
    • OOTB apps for common integration use-
    cases
    Spring I/O
    2016
    

    View Slide

16. Stream DSL: pipes and filters
    Stream definition
    Boot Apps build
    on top of Spring
    Ecosystem
    httpfile = http | file
    |
    Spring I/O
    2016
    ‘pipe’, or how to communicate
    Rabbit, Kafka, Gemfire
    

    View Slide

17. Spring Cloud Deployer
    • SPI for deploying applications to modern runtimes
    • Local (for testing)
    • Cloud Foundry
    • YARN
    • Kubernetes
    • Mesos + Marathon
    • Different resources supported
    • Spring Boot Uberjars
    • Docker Images
    • Different locations
    • File system, HDFS, HTTP, Maven, Docker Hub
    Spring I/O
    2016
    

    View Slide

18. Spring Cloud Data Flow
    Pla$orm
    Spring I/O
    2016
    

    View Slide

19. Spring I/O 2016
    DEMO
    

    View Slide

20. Timeline
    • Spring Cloud Stream
    • 1.0.0.GA May 10, 2016
    • 1.1.0.M1 July 2016
    • 1.1.0.GA Q3 2016
    • Spring Cloud Data Flow
    • 1.0.0.M3 May 7, 2016
    • 1.0.0.RC1 End of May, 2016
    Spring I/O
    2016
    

    View Slide

21. Spring Cloud Stream 1.1.0.M1
    • Reactive APIs: Reactor, RxJava
    • Kafka 0.9
    • Schema evolution support
    

    View Slide

22. Spring I/O 2016
    Q&A
    Marius Bogoevici
    twitter: @mariusbogoevici
    

    View Slide