S1P2018.pdf

The critical part of becoming a data-centric business is to act on real-time events as and when they happen. Event-streams and its "state" are at the core of it - repeating it consistently and continuously is the key.

For example, stateful stream processing brings tremendous new opportunities for businesses, who with the real-time insights can interact with their customers in more meaningful ways. Kafka Streams APIs provide the primitives to interact with distributed data sets. As an event-driven microservice framework, Spring Cloud Stream provides the primitives to build cloud-native streaming applications with either imperative or functional programming models. By combining the both, we can create stateful streaming solutions to be orchestrated as Spring Boot applications in modern platforms such as Kubernetes or Cloud Foundry.

Given this flexibility, businesses can scale, upgrade, rollback, or continuously deliver data-centric business functions seamlessly. In this talk, we will explore how Spring Cloud Stream and Kafka Streams can support Event Sourcing and CQRS patterns.

Lastly, we will walk-through a practical approach to apply cloud-native patterns for data-intensive applications using Spring Cloud Data Flow and Spring Cloud Skipper.

Transcript

1. Building Cloud-Native Data-Intensive Applications with Spring Soby Chacko [@sobychacko] Sabby Anandan [@sabbyanandan]
2. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Disclaimer 2
3. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Safe Harbor Statement The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation. 3
4. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Plot Events in the Enterprise Events and Data-intensive Workloads A Practical Take on Data-Intensive Applications in Cloud Q+A 4
5. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Events in the Enterprise 1 Day 15 Million Rides 3 Million Drivers 78 Countries 5
6. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Event-centric Thinking 6 Profound Customer Engagement
7. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Real-time Matching Efficient Pickups/ Drops Lower Prices Improved Ride Opportunities More Income More Tips Customer: Driver Convenient Costs Less Flexible Customer: Rider(s) 7
8. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Events in Reality Unpredictable Traffic Patterns Resource Contention Faults Rewind / Replay Ordering Guarantees Slow Processing 8
9. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Events in Reality Unpredictable Traffic Patterns Resource Contention Faults Rewind / Replay Ordering Guarantees Slow Processing 1 Day 15 Million Rides! 9
10. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data Intensity Unpredictable Traffic Patterns Resource Contention Faults Rewind / Replay Ordering Guarantees Slow Processing 1 Day 15 Million Rides! Maintainability Scalability Portability Reliability 10
11. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ “We call an application data- intensive if data is its primary challenge—the quantity of data, the complexity of data, or the speed at which it is changing.” 11
12. !12
13. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ You have data from disparate systems 13
14. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 14 You have data of different types
15. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 15 You have data of varying speed, size and shape
16. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 16 You have data that evolves
17. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ But, we are not Uber! 17
18. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Every company must find ways to learn, adapt, and grow. 18
19. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Stream Partitions Consumer Groups Message Headers Testing Framework Content-type Negotiation Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; } 19
20. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Opportunities: Same code; Same tests; Drop-in replacement for a variety of Message Brokers 20 Rabbit MQ Apache Kafka Google PubSub Amazon Kinesis Azure Event Hubs Solace
21. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Stream Partitions Consumer Groups Message Headers Testing Framework Content-type Negotiation Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; } 21
22. Demo #1 A practical take on Events in Data Intensive Applications. 22
23. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ :UserCreated :UserCreated :UserNameChanged :UserActivated 23
24. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Driver’s nearby Surge Pricing: Passengers headed to the same event Split fare with friends 24
25. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 25 ? John Doe User userCreated() userActivated() userDeactivated() Commands // passenger getUserByGeoLocation() // drivers getUsersNearby() Queries :Eve Append-only Log Application State Event Storming CQRS Event Sourcing Architecture Methods / Patterns
26. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 26 Use Case: Trending User Activity by Region
27. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream + Kafka Streams 27 Bindings support not only for Channels, but also for Kafka Streams types. Hide the infrastructure and configuration overhead from the developers. Stateless and Stateful as options.
28. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream + Kafka Streams Topic = foo Partition n Partition 1 Partition 0 Spring Cloud Stream App Instance 0 KTable KStream GlobalKTable 28 Spring Cloud Stream App Instance 1
29. Demo #2 Let’s interact with the Data Intensive Applications. 29
30. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 30 Use Case: Trending User Activity by Region on a Moving Window
31. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Unpredictable Traffic Patterns Resource Contention Faults Rewind / Replay Ordering Guarantees Slow Processing 1 Day 15 Million Rides! Maintainability Scalability Portability Reliability 31 Data Intensity
32. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Maintainability Build, test, and iterate as frequently as needed. CI/CD as first-class thinking for data-centric workloads. Data processing guarantees in the event of rolling-upgrades. Scalability Auto-scale up/down based on throughput demands.
 Linear throughput characteristics as you scale applications. Same app runs locally in the laptop or in any cloud platform where there’s JVM. Portability Reliability Focus on the business logic and the unit-level, integration, and acceptance tests.
 Depend on platform runtime (Kubernetes or Cloud Foundry) for reliability and resiliency guarantees.
 32 Data Intensive Design
33. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 33 A toolkit for building data integration, real- time streaming, and batch data processing pipelines. WHAT IS SPRING CLOUD DATA FLOW?
34. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data pipelines consist of Spring Boot apps, using Spring Cloud Stream for event-streaming or Spring Cloud Task for batch processes. Ready for Data Integration with >60 out-of-the- box streaming and batch Apps. DSL, GUI, and REST-APIs to build and orchestrate data pipelines onto platforms like Kubernetes and Cloud Foundry. Continuous delivery for streaming data pipelines using Spring Cloud Skipper. Cron-job scheduler for batch data pipelines using Spring Cloud Scheduler. 34 A toolkit for building data integration, real- time streaming, and batch data processing pipelines. WHAT IS SPRING CLOUD DATA FLOW?
35. Demo #3 Run all the things in the Cloud. 35
36. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 1 3 2 36 Putting it all together 1 2 3 Websocket Connection REST request/response REST request/response
37. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Next First-class support for Apps with Multiple Input & Output Channels (eg: KStream) Apps as Composable Functions Audit Trail: Who did what and when? 37
38. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Resources Spring Cloud Stream Samples | Gitter | StackOverflow Spring Cloud Data Flow Samples | Gitter | StackOverflow Demo: https://github.com/sabbyanandan/eda 38
39. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Q+A 39