Integrating Spring Batch and Spring Integration

Talk given at the Atlanta Java Users Group on Aug-20, 2013

Abstract

This talk is for everyone who wants to efficiently use Spring Batch and Spring Integration together. Users of Spring Batch often have the requirements to interact with other systems, to schedule the periodic execution Batch jobs and to monitor the execution of Batch jobs. Conversely, Spring Integration users periodically have Big Data processing requirements, be it for example the handling of large traditional batch files or the execution of Apache Hadoop jobs. For these scenarios, Spring Batch is the ideal solution. This session will introduce Spring Batch Integration, a project that provides support to easily tie Spring Batch and Spring Integration together. We will cover the following scenarios:

Launch Batch Jobs through Spring Integration Messages
Generate Informational Messages
Externalize Batch Process Execution using Spring Integration
Create Big Data Pipelines with Spring Batch and Spring Integration

Transcript

1. © 2013 SpringOne 2GX. All rights reserved. Do not distribute without permission. Integrating Spring Batch and Spring Integration By Gunnar Hillert Twitter: @ghillert Atlanta Java Users Group 20 Aug 2013 Wednesday, August 21, 13
2. What we will cover... • Spring Batch • Spring Integration • Spring Batch Integration • Spring XD 2 Wednesday, August 21, 13
3. 3 Wednesday, August 21, 13
4. The Pivotal One Platform • Application Fabric – Languages, Frameworks, Services, Analytics • Data Fabric – High Capacity, Real-time, Ingest & Query, Scale-out, Storage • Cloud Fabric – Automation, Service Registry, Cloud Independence 4 GemFire Wednesday, August 21, 13
5. Spring Stack DI AOP TX JMS JDBC MVC Testing ORM OXM Scheduling JMX REST Caching Profiles Expression Spring Framework HATEOAS JPA 2.0 JSF 2.0 JSR-250 JSR-330 JSR-303 JTA JDBC 4.1 Java EE 1.4+/SE5+ JMX 1.0+ WebSphere 6.1+ WebLogic 9+ GlassFish 2.1+ Tomcat 5+ OpenShift Google App Eng. Heroku AWS Beanstalk Cloud Foundry Spring Web Flow Spring Security Spring Batch Spring Integration Spring Security OAuth Spring Social Twitter LinkedIn Facebook Spring Web Services Spring AMQP Spring Data Redis HBase MongoDB JDBC JPA QueryDSL Neo4j GemFire Solr Splunk HDFS MapReduce Hive Pig Cascading Spring for Apache Hadoop SI/Batch Spring XD Wednesday, August 21, 13
6. 6 Spring Batch http://www.springsource.org/spring-batch Wednesday, August 21, 13
7. 7 Batch processing ... is defined as the processing of data without interaction or interruption. “ Michael T. Minella, Pro Spring Batch Wednesday, August 21, 13
8. Batch Jobs • Long-running – Often outside office hours • Non-interactive – Often include logic for handling errors or restarts • Process large volumes of data – More than fits in memory or a single transaction 8 Wednesday, August 21, 13
9. Batch and offline processing • Close of business processing – Order processing, Business reporting, Account reconciliation • Import/export handling – a.k.a. ETL jobs (Extract-Transform-Load) – Instrument/position import – Data warehouse synchronization • Large-scale output jobs – Loyalty scheme emails, Bank statements 9 Wednesday, August 21, 13
10. Features • Transaction management • Chunk based processing • Declarative I/O • Start/Restart/Skip capabilities • Web administration interface • Based on the Spring framework • JSR 352: Batch Applications for the Java Platform 10 Wednesday, August 21, 13
11. Demo 11 Hello Spring Batch https://github.com/ghillert/spring-batch-sample-hello Wednesday, August 21, 13
12. Concepts • Job • Step • Item • Chunk 12 Repeat | Retry | Skip | Restart Wednesday, August 21, 13
13. Chunk-Oriented Processing • Input-output can be grouped together • Input collects Items before outputting: Chunk-Oriented Processing • Optional ItemProcessor 13 Wednesday, August 21, 13
14. Chunk-Oriented Processing 14 Wednesday, August 21, 13
15. JobLauncher 15 Wednesday, August 21, 13
16. Simple File Load Job 16 Wednesday, August 21, 13
17. ItemReaders and ItemWriters • Flat File • XML (StAX) • Multi-File Input • Database – JDBC, JPA/Hibernate, Stored Procedures • Implement your own... 17 Wednesday, August 21, 13
18. Job Repository 18 Wednesday, August 21, 13
19. Spring Batch Admin • Sub project of Spring Batch • Provides Web UI and REST interface to manage batch processes • Manager, Resources, Sample WAR – Deployed with batch job(s) as single app to be able to control & monitor jobs – Or monitors external jobs only via shared database 19 Wednesday, August 21, 13
20. 20 Spring Integration http://www.springsource.org/spring-integration Wednesday, August 21, 13
21. Integration Styles • Business to Business Integration (B2B) • Inter Application Integration (EAI) • Intra Application Integration 21 JVM JVM EAI External Business Partner B2B Core Messaging Wednesday, August 21, 13
22. Integration Styles • File Transfer • Shared Database • Remoting • Messaging 22 Wednesday, August 21, 13
23. Common Patterns 23 Retrieve Parse Transform Transmit Wednesday, August 21, 13
24. Enterprise Integration Patterns • By Gregor Hohpe & Bobby Woolf • Published 2003 • Collection of well-known patterns • Icon library provided 24 http://www.eaipatterns.com/eaipatterns.html Wednesday, August 21, 13
25. 25 Spring Integration provides an extension of the Spring programming model to support the well-known enterprise integration patterns. “ Spring Integration Website Wednesday, August 21, 13
26. Spring Integration Components 26 • Claim Check (In/Out) • Content Enricher • Header Enricher • Payload Enricher • Control Bus • Delayer • JMX Support • Message Handler Chain • Messaging Bridge • Resequencer • Service Activator • Scripting support (JSR 223) • Ruby/JRuby, Javascript ... • Groovy • Message History • Message Store • JDBC, Redis, MongoDB, Gemfire • Wire Tap • ... Wednesday, August 21, 13
27. Adapters 27 • AMQP/RabbitMQ • AWS* • File/Resource • FTP/FTPS/SFTP • GemFire • HTTP (REST) • JDBC • JMS • JMX • JPA • MongoDB • POP3/IMAP/SMTP • Print* • Redis • RMI • RSS/Atom • SMB* • Splunk* • Spring Application Events • Stored Procedures • TCP/UDP • Twitter • Web Services • XMPP • XPath • XQuery* • ... Wednesday, August 21, 13
28. 28 Spring Batch Integration https://github.com/SpringSource/spring-batch-admin/ Wednesday, August 21, 13
29. Launching batch jobs through messages • Event-Driven execution of the JobLauncher • Spring Integration retrieves the data (e.g. file system, FTP, ...) • Easy to support separate input sources simultaneously 29 D C FTP Inbound Channel Adapter JobLauncher Transformer File JobLaunchRequest Wednesday, August 21, 13
30. JobLaunchRequest 30 public class FileMessageToJobRequest { private Job job; private String fileParameterName; ... @Transformer public JobLaunchRequest toRequest(Message<File> message) { JobParametersBuilder jobParametersBuilder = new JobParametersBuilder(); jobParametersBuilder.addString(fileParameterName, message.getPayload().getAbsolutePath()); return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters()); } } Wednesday, August 21, 13
31. DefaultJobParametersConverter • Convert (textual) Properties/Maps to JobParameters • Provide Typed Parameters – Date – String – Long – Double • Provide Date+Number Format • Define Identifying / Non-Identifying Parameters 31 myDateParam(date)=2013/08/20 aStringParameter=Hello AJUG -stringParamNOTIdentifying=Hello AJUG aNumberParameter(Long)=123456 Wednesday, August 21, 13
32. JobLaunchRequest 32 <batch-int:job-launching-gateway request-channel="requestChannel" reply-channel="replyChannel" job-launcher="jobLauncher"/> Wednesday, August 21, 13
33. Get feedback with informational messages • Spring Batch provides support for listeners: – StepListener – ChunkListener – JobExecutionListener 33 Wednesday, August 21, 13
34. Get feedback with informational messages 34 <batch:job id="importPayments"> ... <batch:listeners> <batch:listener ref="notificationExecutionsListener"/> </batch:listeners> </batch:job> <int:gateway id="notificationExecutionsListener" service-interface="o.s.batch.core.JobExecutionListener" default-request-channel="jobExecutions"/> Wednesday, August 21, 13
35. Externalizing batch process execution • Use Spring Integration inside of Batch jobs – e.g. ItemProcessor + ItemWriter • Offload complex processing • Asynchronous processing support: – AsyncItemProcessor – AsyncItemWriter • Externalize chunk processing using ChunkMessageChannelItemWriter 35 Wednesday, August 21, 13
36. Remote Chunking 36 ItemReader ItemWriter ItemProcessor Step4 ItemReader ItemWriter ItemProcessor Step2b ItemReader ItemWriter Step2 ItemReader ItemWriter ItemProcessor Step1 ItemReader ItemWriter ItemProcessor Step2a ItemReader ItemWriter ItemProcessor Step2c Wednesday, August 21, 13
37. Asynchronous Processors • AsyncItemWriter • AsyncItemProcessor 37 Reader Gateway Output Input Processor Writer Result Item Item Result Wednesday, August 21, 13
38. Remote Partitioning 38 ItemReader ItemWriter ItemProcessor Step3 ItemReader ItemWriter ItemProcessor Slave 2 Master ItemReader ItemWriter ItemProcessor Step1 ItemReader ItemWriter ItemProcessor Slave 1 ItemReader ItemWriter ItemProcessor Slave 3 Partitioner Wednesday, August 21, 13
39. Remote Partitioning 39 Partition Handler Remote Step Master Slave request gateway staging aggregator reply gateway staging request serviceActivator Wednesday, August 21, 13
40. Demo 40 Payment Import https://github.com/ghillert/spring-batch-integration-sample Wednesday, August 21, 13
41. 41 Spring XD http://www.springsource.org/spring-xd Wednesday, August 21, 13
42. Tackling Big Data Complexity • Unified agile experience for • Data Ingestion • Real-time Analytics • Workflow Orchestration • Data Export 42 Wednesday, August 21, 13
43. Tackling Big Data Complexity cont. • Built on existing assets – Spring Integration – Spring Batch – Spring Data • Redis, GemFire, Hadoop • XD = 'eXtreme Data’ 43 Wednesday, August 21, 13
44. Data Ingestion Streams • DSL based on Unix pipes and filters syntax • Modules are parameterizable • Simple logic can be added via expressions or scripts 44 http | file twittersearch --query=spring | file --dir=/spring http | filter --expression=”payload?.customerCode matches ‘GOLD[0-9]+’” | hdfs Wednesday, August 21, 13
45. Hadoop workflow managed by Spring Batch • Reuse Batch infrastructure and features to manage Hadoop workflows – Job state management, launching, monitoring, restart/retry policies, etc. • Step can be any Hadoop job type or HDFS script • Can mix and match with other Batch readers/writers – (e.g. JDBC for import/export use-cases) 45 Wednesday, August 21, 13
46. Demo 46 Spring XD Batch Word-count Sample https://github.com/SpringSource/spring-xd-samples Wednesday, August 21, 13
47. Books 47 Wednesday, August 21, 13
48. Learn More. Stay Connected. Twitter: twitter.com/springframework YouTube: youtube.com/user/SpringSourceDev Google +: plus.google.com/+springframework LinkedIn: springsource.org/linkedin Facebook: facebook.com/groups/springsource Questions? Thank You!! Wednesday, August 21, 13