$30 off During Our Annual Pro Sale. View Details »

Cloud-native Streaming and Event-Driven Microservices

Cloud-native Streaming and Event-Driven Microservices

Talk delivered at Spring IO in Barcelona (May 20, 2016)

The talk introduces Spring Cloud Stream as a framework for developing event-driven microservices, as well as Spring Cloud Data Flow as an orchestration and deployment layer.

Marius Bogoevici

May 20, 2016
Tweet

More Decks by Marius Bogoevici

Other Decks in Technology

Transcript

  1. CLOUD-NATIVE STREAMING AND
    EVENT-DRIVEN MICROSERVICES
    Marius Bogoevici
    @mariusbogoevici

    View Slide

  2. CLOUD-NATIVE STREAMING AND EVENT-
    DRIVEN MICROSERVICES
    • Use cases:
    • predictive maintenance, fraud detection, QoS, IoT
    • enterprise integration meets ‘Big Data’
    • Specific requirements
    • High throughput, low latency
    • Grouping, ordering, windowing
    • Middleware adaptations: beyond JMS, Rabbit MQ
    • Kafka, Amazon Kinesis, Google Pub/Sub, Azure
    Event Hubs
    • Brokerless: Aeron, ZeroMQ
    Spring I/O
    2016

    View Slide

  3. CLOUD-NATIVE STREAMING AND EVENT-
    DRIVEN MICROSERVICES
    • Architectural style advantages:
    • bounded context, development agility
    • Process-specific resource tuning:
    • memory, CPU, instance count
    • Messaging vs. HTTP
    • Discoverability
    • Availability
    • Consistency across heterogenous resources
    Spring I/O
    2016

    View Slide

  4. CLOUD-NATIVE STREAMING AND EVENT-
    DRIVEN MICROSERVICES
    • Modern platforms
    • Resource management
    • memory, CPU, instance count
    • Scaling up/down
    • Health monitoring and failover
    • Routing and load balancing
    Spring I/O
    2016
    Apache YARN
    Apache Mesos Kubernetes

    View Slide

  5. Spring I/O
    2016
    dataflow:> stream create demo --definition “http | file”
    Goal

    View Slide

  6. Spring Cloud Stream
    Spring Cloud Data Flow
    Spring I/O
    2016
    Spring Cloud Data Flow
    Spring Cloud Stream
    Spring Cloud Task
    2015

    View Slide

  7. Spring Cloud Stream
    • Event-driven microservice framework
    • Built on battle-tested components
    • Spring Boot: full-stack standalone
    applications
    • Spring Integration: Messaging, EIP
    patterns, connectors
    • Opinionated primitives
    • Pluggable middleware abstractions
    Spring I/O
    2016

    View Slide

  8. Spring Cloud Stream in a nutshell
    Spring I/O
    2016

    View Slide

  9. …. 10000 ft nutshell
    Spring I/O
    2016

    View Slide

  10. Durable publish-subscribe
    • Fits both streaming and event-driven use cases
    Spring I/O
    2016

    View Slide

  11. Consumer Groups
    • Borrowed from Kafka, applied to all binders
    • Groups of competing consumers within the pub-sub
    destination
    • Used for scaling and partitioning
    Spring I/O
    2016

    View Slide

  12. Partitioning
    • Required for stateful processing
    • Outputs specify a partitioning strategy
    • Inputs consuming from specific partition
    Spring I/O
    2016

    View Slide

  13. Programming model
    @EnableBinding + Binder implementation
    Apache Kafka
    Gemfire

    View Slide

  14. Spring I/O 2016
    DEMO

    View Slide

  15. Spring Cloud Data Flow
    • Orchestration:
    • DSL for Stream topologies
    • REST API
    • Shell
    • UI
    • Portable Deployment SPI
    • OOTB apps for common integration use-
    cases
    Spring I/O
    2016

    View Slide

  16. Stream DSL: pipes and filters
    Stream definition
    Boot Apps build
    on top of Spring
    Ecosystem
    httpfile = http | file
    |
    Spring I/O
    2016
    ‘pipe’, or how to communicate
    Rabbit, Kafka, Gemfire

    View Slide

  17. Spring Cloud Deployer
    • SPI for deploying applications to modern runtimes
    • Local (for testing)
    • Cloud Foundry
    • YARN
    • Kubernetes
    • Mesos + Marathon
    • Different resources supported
    • Spring Boot Uberjars
    • Docker Images
    • Different locations
    • File system, HDFS, HTTP, Maven, Docker Hub
    Spring I/O
    2016

    View Slide

  18. Spring Cloud Data Flow
    Pla$orm
    Spring I/O
    2016

    View Slide

  19. Spring I/O 2016
    DEMO

    View Slide

  20. Timeline
    • Spring Cloud Stream
    • 1.0.0.GA May 10, 2016
    • 1.1.0.M1 July 2016
    • 1.1.0.GA Q3 2016
    • Spring Cloud Data Flow
    • 1.0.0.M3 May 7, 2016
    • 1.0.0.RC1 End of May, 2016
    Spring I/O
    2016

    View Slide

  21. Spring Cloud Stream 1.1.0.M1
    • Reactive APIs: Reactor, RxJava
    • Kafka 0.9
    • Schema evolution support

    View Slide

  22. Spring I/O 2016
    Q&A
    Marius Bogoevici
    twitter: @mariusbogoevici

    View Slide