Data Integration and Real-time Data Processing with Spring Boot

Data Integration and Real-time Data Processing with Spring Boot Sabby
Anandan @sabbyanandan

- Introduce Spring projects - Discuss how they address Data
Integration / Data Processing Challenges Goal:

A toolkit for building data integration, real- time streaming, and
batch data processing pipelines. Spring Cloud Data Flow

- Decentralization (no ESB) - Lightweight applications that contain integration
logic - Loose coupling through message channels Spring Integration

Stream processing is much more than just EIP

Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class
Application { @StreamListener("foo") @SendTo("bar") public String replaceStringMsgHandler(String payload) { return StringUtils.replace(payload, "foo", "bar"); } } B I N D I N G E V E N T S foo channel B I N D I N G channel bar C O N S U M E R S Programming Model: Message Channel Abstraction

Application { @StreamListener("foo") @SendTo("bar") public KStream<Object, Foo> handler(KStream<Object, Event> input){ return . .; } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Native Kafka Streams

Application { @StreamListener("foo") @SendTo("bar") public Flux<Average> sensorAverage(Flux<Sensor> data) { return . .; } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Native Reactor Flux’s

Application { @Bean public Function<String, String> toUpperCase() { return s -> s.toUpperCase(); } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Plain Old Java Functions

Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations
Stream Partitions Consumer Groups Message Headers Testing Framework Content-type Negotiation Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; }

Rabbit MQ Apache Kafka Google PubSub Amazon Kinesis Azure Event Hubs Solace Same code + Same test-harness Drop-in replacement for a variety of Messaging Systems Opportunities:

Stream Partitions Consumer Groups Message Headers Testing Framework Content-type Negotiation Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; }

Demo #1 Events and Data Intensive Applications

:UserCreated :UserCreated :UserNameChanged :UserActivated

User activity in the last 30s Users created in the
last 2 mins User interaction by region in the last 1hr window

1 3 2 1 2 3 Websocket Connection REST request/response
REST request/response

JSR 352

- Closely related processing steps that perform a discrete business
process - Deployable unit comprised of one or more job steps - Lifecycle management for jobs/steps Spring Batch JSR 352

Spring Cloud Task a short-lived microservices framework @EnableTask @EnableBatchProcessing public
class BatchJobApplication { @Bean public Step extractStep() { // extract business logic } @Bean public Step transformStep() { // transformation logic } @Bean public Step loadStep() { // persistence logic } @Bean public Job etlJob() { return this.jobBuilderFactory.get("etlJob") .start(extractStep()) .next(transformStep()) .next(loadStep()) .build(); } } Database R E P O S I T O R Y Programming Model: Spring Batch Job as Short-lived Application

Spring Cloud Task a short-lived microservices framework @EnableTask public class
TimestampTask { @Bean public TimestampTask timeStampTask() { return new TimestampTask(); } public static class TimestampTask implements CommandLineRunner { @Override public void run(String... strings) throws Exception { DateFormat dateFormat = . . logger.info(dateFormat.format(new Date())); } } } Database R E P O S I T O R Y Programming Model: An arbitrary business-logic as Short-lived Application

Spring Cloud Task a short-lived microservices framework Lifecycle Management Transactions
Bookkeeping for Restarts/Replay Historical Representation Remote Partitions

Demo #2 24/7 ETL: Cloud-native File Ingest

SFTP Source TaskLauncher ETL Job/Task Database Orchestrated by Spring Cloud
Data Flow SFTP Server poll for new files publish each file launch task for each file persist parsed data

batch data processing pipelines. Spring Cloud Data Flow But wait, there’s more!

Developer Focus Ideation Implementation Production

Demo #3 CI/CD for Data Pipelines

Mask each Payload 111-22-3333 444-55-6666 777-88-9999 . . . The
Security Number = xxx-xx-3333 The Security Number = xxx-xx-6666 The Security Number = xxx-xx-9999 . . . Don’t Disturb Don’t Disturb Fix This!

Let’s Recap

Spring Cloud Stream Build highly scalable event-driven microservices connected with
shared messaging systems. Spring Cloud Task Build short-lived microservices to perform data processing locally or in the cloud. Spring Cloud Skipper Discover applications and manage their lifecycle on multiple Cloud Platforms. Spring Cloud Data Flow Orchestrate data pipelines made of Spring Cloud Stream or Spring Cloud Task microservices. Consolidate Development and Testing Practices Standardize CI/CD Tooling & Automation Opportunities:

Next Spring Boot 2.1 Compatibility: Stream, Task, Skipper, and SCDF
Function Composition / Function Chaining OAuth2 + OpenID Connect by Default Deeper Integration with Micrometer for Metrics/Monitoring New Data Integration Apps

Data Integration and Real-time Data Processing ...

Data Integration and Real-time Data Processing with Spring Boot

More Decks by Sabby Anandan

Other Decks in Technology

Featured

Transcript