Data Integration and Real-time Data Processing with Spring Boot Sabby Anandan @sabbyanandan

No content

- Introduce Spring projects - Discuss how they address Data Integration / Data Processing Challenges Goal:

A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow

A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow

No content

- Decentralization (no ESB) - Lightweight applications that contain integration logic - Loose coupling through message channels Spring Integration

A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow

Stream processing is much more than just EIP

Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public String replaceStringMsgHandler(String payload) { return StringUtils.replace(payload, "foo", "bar"); } } B I N D I N G E V E N T S foo channel B I N D I N G channel bar C O N S U M E R S Programming Model: Message Channel Abstraction

Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public KStream handler(KStream input){ return . .; } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Native Kafka Streams

Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public Flux sensorAverage(Flux data) { return . .; } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Native Reactor Flux’s

Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @Bean public Function toUpperCase() { return s -> s.toUpperCase(); } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Plain Old Java Functions

Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Stream Partitions Consumer Groups Message Headers Testing Framework Content-type Negotiation Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; }

Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Rabbit MQ Apache Kafka Google PubSub Amazon Kinesis Azure Event Hubs Solace Same code + Same test-harness Drop-in replacement for a variety of Messaging Systems Opportunities:

Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Stream Partitions Consumer Groups Message Headers Testing Framework Content-type Negotiation Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; }

Demo #1 Events and Data Intensive Applications

:UserCreated :UserCreated :UserNameChanged :UserActivated

User activity in the last 30s Users created in the last 2 mins User interaction by region in the last 1hr window

1 3 2 1 2 3 Websocket Connection REST request/response REST request/response

A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow

JSR 352

- Closely related processing steps that perform a discrete business process - Deployable unit comprised of one or more job steps - Lifecycle management for jobs/steps Spring Batch JSR 352

Spring Cloud Task a short-lived microservices framework @EnableTask @EnableBatchProcessing public class BatchJobApplication { @Bean public Step extractStep() { // extract business logic } @Bean public Step transformStep() { // transformation logic } @Bean public Step loadStep() { // persistence logic } @Bean public Job etlJob() { return this.jobBuilderFactory.get("etlJob") .start(extractStep()) .next(transformStep()) .next(loadStep()) .build(); } } Database R E P O S I T O R Y Programming Model: Spring Batch Job as Short-lived Application

Spring Cloud Task a short-lived microservices framework @EnableTask public class TimestampTask { @Bean public TimestampTask timeStampTask() { return new TimestampTask(); } public static class TimestampTask implements CommandLineRunner { @Override public void run(String... strings) throws Exception { DateFormat dateFormat = . . logger.info(dateFormat.format(new Date())); } } } Database R E P O S I T O R Y Programming Model: An arbitrary business-logic as Short-lived Application

Spring Cloud Task a short-lived microservices framework Lifecycle Management Transactions Bookkeeping for Restarts/Replay Historical Representation Remote Partitions

Demo #2 24/7 ETL: Cloud-native File Ingest

SFTP Source TaskLauncher ETL Job/Task Database Orchestrated by Spring Cloud Data Flow SFTP Server poll for new files publish each file launch task for each file persist parsed data

A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow

A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow But wait, there’s more!

Developer Focus Ideation Implementation Production

Demo #3 CI/CD for Data Pipelines

Mask each Payload 111-22-3333 444-55-6666 777-88-9999 . . . The Security Number = xxx-xx-3333 The Security Number = xxx-xx-6666 The Security Number = xxx-xx-9999 . . . Don’t Disturb Don’t Disturb Fix This!

No content

Let’s Recap

Spring Cloud Stream Build highly scalable event-driven microservices connected with shared messaging systems. Spring Cloud Task Build short-lived microservices to perform data processing locally or in the cloud. Spring Cloud Skipper Discover applications and manage their lifecycle on multiple Cloud Platforms. Spring Cloud Data Flow Orchestrate data pipelines made of Spring Cloud Stream or Spring Cloud Task microservices. Consolidate Development and Testing Practices Standardize CI/CD Tooling & Automation Opportunities:

Next Spring Boot 2.1 Compatibility: Stream, Task, Skipper, and SCDF Function Composition / Function Chaining OAuth2 + OpenID Connect by Default Deeper Integration with Micrometer for Metrics/Monitoring New Data Integration Apps

Resources Spring Cloud Stream Samples | Gitter | StackOverflow Spring Cloud Task Samples | Gitter | StackOverflow Spring Cloud Skipper Samples | Gitter | StackOverflow Spring Cloud Data Flow Samples | Gitter | StackOverflow Demo #1: Events + Kafka Streams Demo #2: File-ingest Demo #3: CI/CD for Data Pipelines

Q+A