Slide 1

Slide 1 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cloud Native Event-Driven and Streaming Microservices By Marius Bogoevici, Staff Engineer, Pivotal @mariusbogoevici

Slide 2

Slide 2 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Safe Harbor Statement • The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation. 2

Slide 3

Slide 3 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES • Processing large quantities of data in real-time • Use cases: • predictive maintenance, fraud detection, QoS, IoT • enterprise integration meets ‘Big Data’ • Specific requirements • High throughput, low latency • Grouping, ordering, windowing • Transition in messaging middleware design: • Traditional (JMS-based, RabbitMQ) • To log-style: Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs 3

Slide 4

Slide 4 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES • Resource management • memory, CPU, instance count • Scaling up/down • Health monitoring and failover • Routing and load balancing • Applications and deployment strategies designed to target of all above 4 Apache YARN Apache Mesos Kubernetes

Slide 5

Slide 5 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES • Typically known benefits and drawbacks of microservices: • bounded context • development agility • added complexity • Failure isolation • Process-specific resource tuning: • memory • CPU • instance count • network 5

Slide 6

Slide 6 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES • Emphasize messaging as communication over HTTP • Decoupling • Discoverability • Availability • Eventual consistency across heterogenous resources • alternative to distributed transactions • Strong similarities between: • data streaming - ‘big data’: ingestion, analytics • event streaming - async interaction, event sourcing, etc 6

Slide 7

Slide 7 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: The Origin Story 7 Spring Cloud Stream 2015 Spring XD Spring Cloud Data Flow Spring Cloud Task

Slide 8

Slide 8 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream • Event-driven microservice framework • Built on battle-tested components • Spring Boot: full-stack standalone applications • Spring Integration: Messaging, EIP patterns, connectors • Opinionated primitives • Pluggable middleware abstractions 8

Slide 9

Slide 9 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream in a nutshell 9 Application Core Spring Boot Spring Integration Spring Messaging Reactive APIs inputs outputs

Slide 10

Slide 10 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 10

Slide 11

Slide 11 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 11

Slide 12

Slide 12 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Binder Abstraction 12 Application Messaging Middleware Binder channel can be input or output adapts channel to target middleware created by framework configured via Spring Boot properties

Slide 13

Slide 13 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 13 @EnableBinding + Binder Implementation Apache Kafka JMS Google PubSub Production-ready: Experimental

Slide 14

Slide 14 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 14 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener(Processor.INPUT)
 @SendTo(Processor.OUTPUT)
 public String process(String s) {
 return s.toUpperCase();
 }
 }

Slide 15

Slide 15 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Pipes and filters model 15 http averages topN ingest data average value over last 5s hottest sensors over last 10 seconds

Slide 16

Slide 16 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 16 http raw-sensor-data averages averages top-n Calculator

Slide 17

Slide 17 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 17 http raw-sensor-data averages averages top-n Calculator Failure detector in Spring Cloud Stream, all destinations are pub-sub

Slide 18

Slide 18 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data and event streaming - conceptually similar 18

Slide 19

Slide 19 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Consumer groups 19 raw-sensor-data averages HDFS http pubsub is great for interconnecting logical apps what if we add more instances ?

Slide 20

Slide 20 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Consumer groups 20 raw-sensor-data averages HDFS http we want these two to compete averages HDFS and these two !

Slide 21

Slide 21 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 21

Slide 22

Slide 22 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Consumer groups 22 Averages HDDS HDFS HDFS raw-sensor-data groups are in a pub-sub relationship to each other Averages Averages consumers are competing within a group

Slide 23

Slide 23 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 23

Slide 24

Slide 24 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 24 Average calculator Average calculator http raw-sensor-data

Slide 25

Slide 25 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 25 Average calculator Average calculator http partition 1 partition 2 raw-sensor-data

Slide 26

Slide 26 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 26 Average calculator Average calculator http partition 1 partition 2 raw-sensor-data

Slide 27

Slide 27 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Simple topologies : (relatively) easy to deploy … 27 http HDFS spring.cloud.stream.bindings.output.destination=httphdfs.1 spring.cloud.stream.bindings.input.destination=httphdfs.1 spring.cloud.stream.bindings.input.group=httphdfs httphdfs.1

Slide 28

Slide 28 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ … but how about complex topologies ? 28 http raw-sensor-data averages top-n Calculator Failure detector averages averages HDFS HDFS HDFS

Slide 29

Slide 29 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow • Orchestration: • DSL for Stream topologies • REST API • Shell • UI • Portable Deployment SPI • OOTB apps for common integration use-cases 29

Slide 30

Slide 30 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ In a single sentence … 30 dataflow:> stream create demo --definition “http | file”

Slide 31

Slide 31 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow - Stream DSL 31 Stream definition Boot Apps built on top of Spring Ecosystem httpfile = http | file |

Slide 32

Slide 32 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow - deployment SPI • SPI for deploying applications • Local (for testing) • Cloud Foundry • YARN • Kubernetes • Mesos + Marathon • Different resources supported • Spring Boot Uberjars • Docker Images • Different locations • File system, HDFS, HTTP, Maven, Docker Hub 32

Slide 33

Slide 33 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow deployment 33 Pla$orm

Slide 34

Slide 34 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream 1.1 34

Slide 35

Slide 35 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream 1.1 • Under development, currently on master • 1.1.0.M1 release August 2016 • 1.1.0.RELEASE - early Q4 2016 • Based on Spring Boot 1.4 • Release train model • binders in separate repositories • New features • Kafka 0.9 support (also intending 0.10) • Reactive programming support • Schema Registry support 35

Slide 36

Slide 36 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model - imperative/procedural 36 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener(“input”)
 @SendTo(“output”)
 public String process(String s) {
 return s.toUpperCase();
 }
 }

Slide 37

Slide 37 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Reactive/functional Programming via Reactor 37 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener
 @Output(“output”)
 public Flux process(@Input(“input”) Flux s) {
 return f.map(s -> s.toUppercase()); }
 }

Slide 38

Slide 38 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 38

Slide 39

Slide 39 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Handling multiple returns (or none at all) 39 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener
 @Output(“output”)
 public Flux process(@Input(“input”) Flux s) {
 return f.filter(s->startsWith(“a”)).map(s -> s.toUppercase()); }
 }

Slide 40

Slide 40 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Stateful operators: grouping, windowing 40 SpringBootApplication
 @EnableBinding(Processor.class)
 public class WordCountApplication {
 
 @StreamListener
 @Output("output")
 public Flux> countWords(@Input("input") Flux words) {
 return words.window(ofSeconds(5), ofSeconds(1))
 .flatMap(window -> window.groupBy(word -> word)
 .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1)
 .map(count -> new WordCount(group.key(), count))));
 } }

Slide 41

Slide 41 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Content-type transformations (since 1.0) • Users can specify the intended content type of an output value, e.g. “application/ json” • Outgoing messages are converted to that content type • A message header value carries the content type • The content type can be used by inputs to convert to expected types (e.g. `@StreamListener` method arguments • Extensible model, just requires the registration of AbstractMessageConverter beans 41

Slide 42

Slide 42 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema management and Avro support • Newly introduced Avro MessageConverters • Static schema-based resources (classpath, filesystem, etc) • Schema Registry Client-based • Schema registry server • Not Spring Cloud Stream specific • Allows storing different formats and versions • Schema evolution support via Schema Registry Client converter 42

Slide 43

Slide 43 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Heterogenous data sources/consumers 43 sensor v1 sensor v2 average-temperature v1 sensor data average-pressure v2

Slide 44

Slide 44 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema evolution in Spring Cloud Stream 1.1 44

Slide 45

Slide 45 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema evolution in Spring Cloud Stream 1.1 45

Slide 46

Slide 46 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema Registry Server - embeddable in apps 46 @SpringBootApplication @EnableSchemaRegistryServer public class SchemaRegistryServerApplication { public static void main(String[] args) { SpringApplication .run(SchemaRegistryServerApplication.class,args); } }

Slide 47

Slide 47 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Pluggable schema registry client 47 @EnableBinding(Sink.class)
 @EnableAutoConfiguration
 @EnableSchemaRegistryClient
 public static class VoteCounterApplication {
 
 @StreamListener(Sink.INPUT)
 public void listen(Vote vote) {
 voteService.count(vote); }
 }

Slide 48

Slide 48 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Future plans 48

Slide 49

Slide 49 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Future plans • Additional binders • JMS • Google Pub Sub • Kinesis • Fully reactive pipelines • KStream support • First class support for stateful processing 49

Slide 50

Slide 50 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Fully reactive pipelines (planned) • Use natively reactive middleware clients for reactive support • e.g. Reactive Kafka client 50 public interface ReactiveProcessor { @Input(Processor.INPUT)
 Flux> output();
 
 @Output(Processor.OUTPUT)
 Flux> output();
 
 } @EnableBinding(ReactiveProcessor.class)


Slide 51

Slide 51 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Kafka Stream Support (planned) 51 @StreamListener
 @Output(“output”)
 public KStream addByKey(@Input(“input”) KStream input) {
 return input.map((i,s) -> new KeyValue<>(i, s.toUpperCase())) .reduceByKey((String value1, String value2) -> value1 + value2, TimeWindows.of("windowName", 1000)) .toStream() }


Slide 52

Slide 52 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Related talks 52 Reactive Kafka Rajini Sivaram Thursday, August 4, 9:00 AM - 10:05 AM Orchestrate All the Things! with Spring Cloud Data Flow Eric Bottard, Ilayaperumal Gopinathan Thursday, August 4 11:10 AM - 12:15 PM

Slide 53

Slide 53 text

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Learn More. Stay Connected. Tell your friends! @springcentral spring.io/blog @pivotal pivotal.io/blog @pivotalcf http://engineering.pivotal.io