Getting Started with Spring Cloud Data Flow

Slide 1

Slide 1 text

Getting Started with Spring Cloud Data Flow Sabby Anandan (@sabbyanandan) Product Manager

Slide 2

Slide 2 text

Housekeeping .. - Mix of presentation + walkthrough (30 mins) - 3 hands-on labs (~15 mins each) - Questions? raise hand; ask questions in chat Covering today .. 2 Spring Cloud Data Flow - Overview - Features / Use-cases Hands-on Labs - SCDF on Kubernetes - Event Streaming - Cloud-native Batch

Slide 3

Slide 3 text

Enterprise Integration “communication between people, machines, computers, and devices and their efficient coordination” 3

Slide 4

Slide 4 text

Enterprise Integration Patterns “communication between people, machines, computers, and devices and their efficient coordination” messaging routing aggregation transformation 4

Slide 5

Slide 5 text

Enterprise Integration in Reality “communication between people, machines, computers, and devices and their efficient coordination” Ordering Guarantees Data Loss Unpredictable Traffic Patterns Schema Changes Slow Dependencies 5

Slide 6

Slide 6 text

Spring to Rescue! 6

Slide 7

Slide 7 text

Use-cases: Enterprise data integration (EAI/EIP) Event-driven architectures Real-time predictive analytics (e.g., IoT) Use-cases: Scheduled data migration jobs Extract, Transform, and Load (ETL) Offline machine learning and model training Cloud-native Enterprise Integration Short-lived Spring Boot microservices for batch data processing Event-driven Spring Boot microservices for real-time data processing Spring Cloud Task Spring Cloud Stream 7

Slide 8

Slide 8 text

A microservices based Streaming and Batch data processing in the cloud Spring Cloud Data Flow 8

Slide 9

Slide 9 text

Cloud-native Data Pipelines Spring Cloud Data Flow Kubernetes Deploy Streams Launch Batch Jobs Track Execution Monitor Performance 9

Slide 10

Slide 10 text

Lab #1: Set up SCDF on Kubernetes 10 15 Mins 15 Mins

Slide 11

Slide 11 text

1 2 3 4 Start Minikube Cluster In Strigo, Click: “My Lab” Test Environment Install SCDF 11

Slide 12

Slide 12 text

Cloud-native Data Pipelines - Extend cloud-native patterns to data processing use-cases - CI/CD-centered microservices architecture - Elastically scale Event Streaming and Batch, on-demand Spring Cloud Data Flow See: Features 12

Slide 13

Slide 13 text

Lab #2: Event Streaming 13 15 Mins 15 Mins

Slide 14

Slide 14 text

11:30:30 11:30:40 11:30:50 11:30:60 11:31:10 11:31:20 Brake Temperature Analysis Time: 50°F 90°F 50°F 90°F 50°F 90°F 50°F 90°F 50°F 90°F 50°F 90°F Temp:

Slide 15

Slide 15 text

Topic: trucks Topic: trucks Topic: truck-logs Topic: truck-logs Consumer log() { return result -> System.out.println(result); } Results Supplier generateTruck() { return () -> randomTruck(); } Function,?> processBrakeTemperature() { return input -> input .map(..) .groupByKey(..) .windowedBy(10) // 10-secs moving avg. .aggregate(..) .map(..); } Moving Window Avg. Brake Temperature 15

Slide 16

Slide 16 text

Lab #2: Cloud-native ETL 16 15 Mins 15 Mins

Slide 17

Slide 17 text

Job 2 Job 1 Original Image File Create Thumbnail File Move Thumbnail File ➜ ~/tmp |____dataflow-icon.png |____ready | |____dataflow-icon-thumb-ready.png |____draft | |____dataflow-icon-thumb-draft.png Print Original & Thumbnail File Sizes Results 1 Extract Step 2 Transform Step 3 Load Step 17 1 Status Step

Slide 18

Slide 18 text

Game of Streams : How to Tame and Get the Most from Your Messaging Platforms Mark Heckler, VMware Resources Spring Cloud Data Flow: https://dataﬂow.spring.io/ Source Code: https://github.com/sabbyanandan/SpringOne2020 Lab Instructions: https://hackmd.io/@sabbyanandan/B1bDf74fv Slides: Speaker Deck 18 Batching for the Modern Enterprise Michael Minella, VMware Walking Through Spring Cloud Data Flow Glenn Renfro, VMware Ilayaperumal Gopinathan, VMware IoT Scale Event-Stream Processing for Connected Fleet at Penske Krishna Gogineni, VMware Shruti Modi, Penske Sep 2nd Sep 3rd Sep 3rd Sep 2nd