Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Native Streaming and Event Driven Microservices

Cloud Native Streaming and Event Driven Microservices

Slides from the talk delivered at Devoxx Belgium

Marius Bogoevici

November 10, 2016
Tweet

More Decks by Marius Bogoevici

Other Decks in Technology

Transcript

  1. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Cloud Native Event-Driven and
    Streaming Microservices
    Marius Bogoevici, Pivotal
    @mariusbogoevici

    View full-size slide

  2. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Streaming
    • Processing large quantities of data as fast as possible (near real-time)
    • Use cases:
    • predictive maintenance, fraud detection, QoS, IoT
    • enterprise integration meets ‘Big Data’
    • High throughput / low latency
    • Grouping, ordering, windowing
    • Transition in messaging middleware design:
    • Traditional (JMS-based, RabbitMQ)
    • To log-style: Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs

    View full-size slide

  3. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Microservices
    • Preferred model for distributed data processing applications
    • Failure isolation
    • Independent scaling
    • Process-specific resource tuning:
    • memory/CPU/Instance count/network
    • Typical benefits and drawbacks of microservices:
    • bounded context as cohesive units
    • development agility
    • added complexity

    View full-size slide

  4. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Event-driven
    • Messaging vs HTTP/“fire-and-forget” vs. request-reply
    • Decoupling
    • Discoverability
    • Availability
    • Eventual consistency vs distributed transactions
    • especially over heterogenous resources
    • Strong similarities between:
    • data streaming - ‘big data’: ingestion, analytics
    • event streaming - async interaction, event sourcing, etc

    View full-size slide

  5. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Data streaming vs Event sourcing

    View full-size slide

  6. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Cloud Native
    • Focus on modern platform capabilities
    • Scaling up/down
    • Resource management (memory, CPU, instance count)
    • Health monitoring and failover
    • Routing and load balancing
    Apache YARN
    Apache Mesos Kubernetes

    View full-size slide

  7. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    In a single sentence …
    dataflow:> stream create demo --definition “http | file”

    View full-size slide

  8. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Spring Cloud Stream
    2015
    Spring XD
    Spring Cloud Data Flow
    Spring Cloud Task

    View full-size slide

  9. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Spring Cloud Stream
    • Event-driven microservice framework
    • Built on battle-tested components
    • Spring Boot: full-stack standalone applications
    • Spring Integration: Messaging, EIP patterns, connectors
    • Focusing on scalable, structured distributed applications
    • Opinionated primitives
    • Pluggable middleware abstractions

    View full-size slide

  10. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Spring Cloud Stream in a nutshell
    Application Core
    Spring Boot
    Spring Integration
    Spring Messaging
    Reactive APIs
    inputs
    outputs

    View full-size slide

  11. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    and a 10000 ft nutshell …

    View full-size slide

  12. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    The binder abstraction

    View full-size slide

  13. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Programming model
    @EnableBinding + Binder Implementation
    Apache Kafka
    JMS Google PubSub
    Production-ready:
    Experimental

    View full-size slide

  14. Demo
    @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream

    View full-size slide

  15. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Durable Pub Sub

    View full-size slide

  16. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Consumer Groups

    View full-size slide

  17. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Partitioning

    View full-size slide

  18. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Spring Cloud Data Flow
    • Orchestration:
    • DSL for Stream topologies
    • REST API
    • Shell
    • UI
    • Portable Deployment SPI
    • OOTB apps for common integration use-cases

    View full-size slide

  19. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Stream DSL
    Stream Definition
    Spring Boot applications
    built on top of Spring
    ecosystem

    View full-size slide

  20. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Spring Cloud Data Flow Deployment
    Pla$orm

    View full-size slide

  21. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Spring Cloud Deployer
    • SPI for deploying applications to modern runtimes
    • Local (for testing)
    • Cloud Foundry
    • YARN
    • Kubernetes
    • Mesos + Marathon
    • Different resources supported
    • Spring Boot Uberjars
    • Docker Images
    • Different resource location supported
    • File system, HDFS, HTTP, Maven, Docker Hub

    View full-size slide

  22. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Spring Cloud Data Flow Streams

    View full-size slide

  23. @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream
    Deployment and orchestration

    View full-size slide

  24. Demo
    @mariusbogoevici @springcloud
    #Devoxx #SpringCloudStream

    View full-size slide