Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Native Streaming and Event-Driven Microservices

Cloud Native Streaming and Event-Driven Microservices

Talk given at Spring One Platform 2016, August 1-4, 2016, Las Vegas

An introduction to Spring Cloud Stream, it's design and new SCSt 1.1 features.

Marius Bogoevici

August 03, 2016
Tweet

More Decks by Marius Bogoevici

Other Decks in Technology

Transcript

  1. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Cloud Native Event-Driven and
    Streaming Microservices
    By Marius Bogoevici, Staff Engineer, Pivotal
    @mariusbogoevici

    View full-size slide

  2. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Safe Harbor Statement
    • The following is intended to outline the general direction of Pivotal's offerings. It
    is intended for information purposes only and may not be incorporated into any
    contract. Any information regarding pre-release of Pivotal offerings, future
    updates or other planned modifications is subject to ongoing evaluation by
    Pivotal and is subject to change. This information is provided without warranty or
    any kind, express or implied, and is not a commitment to deliver any material,
    code, or functionality, and should not be relied upon in making purchasing
    decisions regarding Pivotal's offerings. These purchasing decisions should only
    be based on features currently available. The development, release, and timing
    of any features or functionality described for Pivotal's offerings in this
    presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to
    update forward looking information in this presentation.
    2

    View full-size slide

  3. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES
    • Processing large quantities of data in real-time
    • Use cases:
    • predictive maintenance, fraud detection, QoS, IoT
    • enterprise integration meets ‘Big Data’
    • Specific requirements
    • High throughput, low latency
    • Grouping, ordering, windowing
    • Transition in messaging middleware design:
    • Traditional (JMS-based, RabbitMQ)
    • To log-style: Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs
    3

    View full-size slide

  4. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES
    • Resource management
    • memory, CPU, instance count
    • Scaling up/down
    • Health monitoring and failover
    • Routing and load balancing
    • Applications and deployment strategies designed to target of all above
    4
    Apache YARN
    Apache Mesos Kubernetes

    View full-size slide

  5. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES
    • Typically known benefits and drawbacks of microservices:
    • bounded context
    • development agility
    • added complexity
    • Failure isolation
    • Process-specific resource tuning:
    • memory
    • CPU
    • instance count
    • network
    5

    View full-size slide

  6. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES
    • Emphasize messaging as communication over HTTP
    • Decoupling
    • Discoverability
    • Availability
    • Eventual consistency across heterogenous resources
    • alternative to distributed transactions
    • Strong similarities between:
    • data streaming - ‘big data’: ingestion, analytics
    • event streaming - async interaction, event sourcing, etc
    6

    View full-size slide

  7. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream: The Origin Story
    7
    Spring Cloud Stream
    2015
    Spring XD Spring Cloud Data Flow
    Spring Cloud Task

    View full-size slide

  8. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream
    • Event-driven microservice framework
    • Built on battle-tested components
    • Spring Boot: full-stack standalone applications
    • Spring Integration: Messaging, EIP patterns,
    connectors
    • Opinionated primitives
    • Pluggable middleware abstractions
    8

    View full-size slide

  9. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream in a nutshell
    9
    Application Core
    Spring Boot
    Spring Integration
    Spring Messaging
    Reactive APIs
    inputs
    outputs

    View full-size slide

  10. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    10

    View full-size slide

  11. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    11

    View full-size slide

  12. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    The Binder Abstraction
    12
    Application
    Messaging
    Middleware
    Binder
    channel
    can be
    input or output
    adapts channel
    to target middleware
    created by
    framework
    configured via
    Spring Boot properties

    View full-size slide

  13. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Programming model
    13
    @EnableBinding + Binder Implementation
    Apache Kafka
    JMS Google PubSub
    Production-ready:
    Experimental

    View full-size slide

  14. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Programming model
    14
    @SpringBootApplication

    @EnableBinding(Processor.class)

    public class UppercaseProcessor {


    @StreamListener(Processor.INPUT)

    @SendTo(Processor.OUTPUT)

    public String process(String s) {

    return s.toUpperCase();

    }

    }

    View full-size slide

  15. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Pipes and filters model
    15
    http averages topN
    ingest
    data
    average value
    over last 5s
    hottest sensors
    over last 10 seconds

    View full-size slide

  16. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream: Durable Publish Subscribe
    16
    http
    raw-sensor-data
    averages
    averages
    top-n
    Calculator

    View full-size slide

  17. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream: Durable Publish Subscribe
    17
    http
    raw-sensor-data
    averages
    averages
    top-n
    Calculator
    Failure
    detector
    in Spring Cloud Stream,
    all destinations are pub-sub

    View full-size slide

  18. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Data and event streaming - conceptually similar
    18

    View full-size slide

  19. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream: Consumer groups
    19
    raw-sensor-data
    averages
    HDFS
    http
    pubsub is great for
    interconnecting logical apps
    what if we add more
    instances ?

    View full-size slide

  20. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream: Consumer groups
    20
    raw-sensor-data
    averages
    HDFS
    http
    we want these two
    to compete
    averages
    HDFS
    and these two !

    View full-size slide

  21. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    21

    View full-size slide

  22. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream: Consumer groups
    22
    Averages
    HDDS
    HDFS
    HDFS
    raw-sensor-data
    groups are in a
    pub-sub relationship
    to each other
    Averages
    Averages
    consumers are competing
    within a group

    View full-size slide

  23. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    23

    View full-size slide

  24. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream: Partitioning
    24
    Average calculator
    Average calculator
    http
    raw-sensor-data

    View full-size slide

  25. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream: Partitioning
    25
    Average calculator
    Average calculator
    http
    partition 1
    partition 2
    raw-sensor-data

    View full-size slide

  26. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream: Partitioning
    26
    Average calculator
    Average calculator
    http
    partition 1
    partition 2
    raw-sensor-data

    View full-size slide

  27. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Simple topologies : (relatively) easy to deploy …
    27
    http
    HDFS
    spring.cloud.stream.bindings.output.destination=httphdfs.1
    spring.cloud.stream.bindings.input.destination=httphdfs.1
    spring.cloud.stream.bindings.input.group=httphdfs
    httphdfs.1

    View full-size slide

  28. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    … but how about complex topologies ?
    28
    http
    raw-sensor-data
    averages
    top-n
    Calculator
    Failure
    detector
    averages
    averages
    HDFS
    HDFS
    HDFS

    View full-size slide

  29. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Data Flow
    • Orchestration:
    • DSL for Stream topologies
    • REST API
    • Shell
    • UI
    • Portable Deployment SPI
    • OOTB apps for common integration use-cases
    29

    View full-size slide

  30. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    In a single sentence …
    30
    dataflow:> stream create demo --definition “http | file”

    View full-size slide

  31. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Data Flow - Stream DSL
    31
    Stream definition
    Boot Apps built
    on top of Spring
    Ecosystem
    httpfile = http | file
    |

    View full-size slide

  32. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Data Flow - deployment SPI
    • SPI for deploying applications
    • Local (for testing)
    • Cloud Foundry
    • YARN
    • Kubernetes
    • Mesos + Marathon
    • Different resources supported
    • Spring Boot Uberjars
    • Docker Images
    • Different locations
    • File system, HDFS, HTTP, Maven, Docker Hub
    32

    View full-size slide

  33. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Data Flow deployment
    33
    Pla$orm

    View full-size slide

  34. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream 1.1
    34

    View full-size slide

  35. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Spring Cloud Stream 1.1
    • Under development, currently on master
    • 1.1.0.M1 release August 2016
    • 1.1.0.RELEASE - early Q4 2016
    • Based on Spring Boot 1.4
    • Release train model
    • binders in separate repositories
    • New features
    • Kafka 0.9 support (also intending 0.10)
    • Reactive programming support
    • Schema Registry support
    35

    View full-size slide

  36. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Programming model - imperative/procedural
    36
    @SpringBootApplication

    @EnableBinding(Processor.class)

    public class UppercaseProcessor {


    @StreamListener(“input”)

    @SendTo(“output”)

    public String process(String s) {

    return s.toUpperCase();

    }

    }

    View full-size slide

  37. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Reactive/functional Programming via Reactor
    37
    @SpringBootApplication

    @EnableBinding(Processor.class)

    public class UppercaseProcessor {


    @StreamListener

    @Output(“output”)

    public Flux process(@Input(“input”) Flux s) {

    return f.map(s -> s.toUppercase());
    }

    }

    View full-size slide

  38. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    38

    View full-size slide

  39. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Handling multiple returns (or none at all)
    39
    @SpringBootApplication

    @EnableBinding(Processor.class)

    public class UppercaseProcessor {


    @StreamListener

    @Output(“output”)

    public Flux process(@Input(“input”) Flux s) {

    return f.filter(s->startsWith(“a”)).map(s -> s.toUppercase());
    }

    }

    View full-size slide

  40. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Stateful operators: grouping, windowing
    40
    SpringBootApplication

    @EnableBinding(Processor.class)

    public class WordCountApplication {


    @StreamListener

    @Output("output")

    public Flux> countWords(@Input("input") Flux words) {

    return words.window(ofSeconds(5), ofSeconds(1))

    .flatMap(window -> window.groupBy(word -> word)

    .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1)

    .map(count -> new WordCount(group.key(), count))));

    }
    }

    View full-size slide

  41. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Content-type transformations (since 1.0)
    • Users can specify the intended content type of an output value, e.g. “application/
    json”
    • Outgoing messages are converted to that content type
    • A message header value carries the content type
    • The content type can be used by inputs to convert to expected types (e.g.
    `@StreamListener` method arguments
    • Extensible model, just requires the registration of AbstractMessageConverter
    beans
    41

    View full-size slide

  42. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Schema management and Avro support
    • Newly introduced Avro MessageConverters
    • Static schema-based resources (classpath, filesystem, etc)
    • Schema Registry Client-based
    • Schema registry server
    • Not Spring Cloud Stream specific
    • Allows storing different formats and versions
    • Schema evolution support via Schema Registry Client converter
    42

    View full-size slide

  43. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Heterogenous data sources/consumers
    43
    sensor v1
    sensor v2
    average-temperature v1
    sensor data
    average-pressure v2

    View full-size slide

  44. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Schema evolution in Spring Cloud Stream 1.1
    44

    View full-size slide

  45. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Schema evolution in Spring Cloud Stream 1.1
    45

    View full-size slide

  46. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Schema Registry Server - embeddable in apps
    46
    @SpringBootApplication
    @EnableSchemaRegistryServer
    public class SchemaRegistryServerApplication {
    public static void main(String[] args) {
    SpringApplication
    .run(SchemaRegistryServerApplication.class,args);
    }
    }

    View full-size slide

  47. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Pluggable schema registry client
    47
    @EnableBinding(Sink.class)

    @EnableAutoConfiguration

    @EnableSchemaRegistryClient

    public static class VoteCounterApplication {


    @StreamListener(Sink.INPUT)

    public void listen(Vote vote) {

    voteService.count(vote);
    }

    }

    View full-size slide

  48. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Future plans
    48

    View full-size slide

  49. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Future plans
    • Additional binders
    • JMS
    • Google Pub Sub
    • Kinesis
    • Fully reactive pipelines
    • KStream support
    • First class support for stateful processing
    49

    View full-size slide

  50. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Fully reactive pipelines (planned)
    • Use natively reactive middleware clients for reactive support
    • e.g. Reactive Kafka client
    50
    public interface ReactiveProcessor {
    @Input(Processor.INPUT)

    Flux> output();


    @Output(Processor.OUTPUT)

    Flux> output();


    }
    @EnableBinding(ReactiveProcessor.class)


    View full-size slide

  51. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Kafka Stream Support (planned)
    51
    @StreamListener

    @Output(“output”)

    public KStream addByKey(@Input(“input”)
    KStream input) {

    return input.map((i,s) -> new KeyValue<>(i, s.toUpperCase()))
    .reduceByKey((String value1, String value2) -> value1 +
    value2, TimeWindows.of("windowName", 1000))
    .toStream()
    }


    View full-size slide

  52. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Related talks
    52
    Reactive Kafka
    Rajini Sivaram
    Thursday, August 4, 9:00 AM - 10:05 AM
    Orchestrate All the Things! with Spring Cloud Data Flow
    Eric Bottard, Ilayaperumal Gopinathan
    Thursday, August 4 11:10 AM - 12:15 PM

    View full-size slide

  53. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
    Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
    Learn More. Stay Connected.
    Tell your friends!

    @springcentral
    spring.io/blog
    @pivotal
    pivotal.io/blog
    @pivotalcf
    http://engineering.pivotal.io

    View full-size slide