Enterprise Grails - Spring Batch

Batch processing in Grails with Spring Batch

Transcript

1. ENTERPRISE GRAILS Batch Processing in Grails with Spring Batch John Engelman @johnrengelman Principal Consultant, Object Partners
2. WHAT IS BATCH PROCESSING?
3. WHAT IS BATCH PROCESSING? "Execution of a series of programs ("jobs") on a computer without manual intervention" (Wikipedia)
4. WHAT IS BATCH PROCESSING? "Time based events, periodic application of complex business rules processed repetitively across very large data sets, or integration of information that..requires formatting, validation, and processing in a transactional manner" (Spring Source)
5. APPLICATIONS OF BATCH PROCESSING Data import Extract - Transform - Load (ETL) Data restore Data export Report Generation Data transmission Backups Data conversion/processing Long running processes
6. BATCH PROCESSING FRAMEWORKS Cron Actually a scheduler z/OS Mainframe Spring Batch bmap4j Hadoop MapReduce
7. SPRING BATCH FEATURES Item/Chunk Oriented Processing Job Retry/Repeat Various core reader/write implementations Transaction Management Exception Handling Chunking & Partioning Meta-Data Persistence Based on Spring Framework Job & Step Attribution Binding XML/Annotation configuration
8. ANATOMY OF A SPRING BATCH JOB JOB Encapsulates the batch process Consists of at least 1 Step STEP Encapsulates a single (simple or complex) task within the batch process Has exactly 1 ItemReader & 1 ItemWriter Optionally has 1 ItemProcessor
9. ANATOMY OF A SPRING BATCH JOB Job Parameters The input data to configure/execute the job JobInstance The execution of a Job for given set of JobParameters JobInstance = Job + JobParameters JobExecution A single attempt at executing a JobInstance Many executions for a single instance Instance is complete when an execution completes StepExecution A single attempt at executing a Step within a Job. [ h t t p : / / w w w . s l i d e s h a r e . n e t / g s c h m u t z / s p r i n g - b a t c h - 2 0 - 2 7 9 4 0 1 5 ]
10. ANATOMY OF A SPRING BATCH JOB [ h t t p s : / / r a w . g i t h u b . c o m / S p r i n g S o u r c e / s p r i n g - b a t c h / m a s t e r / s r c / s i t e / d o c b o o k / r e f e r e n c e / i m a g e s / j o b H e i r a r c h y W i t h S t e p s . p n g ]
11. ANATOMY OF A SPRING BATCH JOB ITEMREADER API for retrieving input data for a step Retrieves 1 item at a time Returns null when input is exhausted < f o n t s t y l e = " f o n t - s i z e : 2 4 p x ; " > p a c k a g e o r g . s p r i n g f r a m e w o r k . b a t c h . i t e m ; p u b l i c i n t e r f a c e I t e m R e a d e r < T > { T r e a d ( ) t h r o w s E x c e p t i o n , U n e x p e c t e d I n p u t E x c e p t i o n , P a r s e E x c e p t i o n , N o n T r a n s i e n t R e s o } < / f o n t >
12. ANATOMY OF A SPRING BATCH JOB ITEMPROCESSOR API for transforming/processing input item to output item Processes 1 item at a time Returns null to indicate that result should not be saved < f o n t s t y l e = " f o n t - s i z e : 2 4 p x ; " > p a c k a g e o r g . s p r i n g f r a m e w o r k . b a t c h . i t e m ; p u b l i c i n t e r f a c e I t e m P r o c e s s o r < I , O > { O p r o c e s s ( I i ) t h r o w s E x c e p t i o n ; } < / f o n t >
13. ANATOMY OF A SPRING BATCH JOB ITEMWRITER API representing output of Step Chunk-Oriented Transaction is committed for a series of ItemWriter calls Receives a List of read or processed items List size is size of commit-interval < f o n t s t y l e = " f o n t - s i z e : 2 4 p x ; " > p a c k a g e o r g . s p r i n g f r a m e w o r k . b a t c h . i t e m ; p u b l i c i n t e r f a c e I t e m W r i t e r < T > { v o i d w r i t e ( L i s t < ? e x t e n d s T > i t e m s ) t h r o w s E x c e p t i o n ; } < / f o n t >
14. ANATOMY OF A SPRING BATCH JOB [ h t t p s : / / r a w . g i t h u b . c o m / S p r i n g S o u r c e / s p r i n g - b a t c h / m a s t e r / s r c / s i t e / d o c b o o k / r e f e r e n c e / i m a g e s / c h u n k - o r i e n t e d - p r o c e s s i n g . p n g ]
15. ANATOMY OF A SPRING BATCH JOB Listeners JobExecution, StepExecution, ChunkExecution ItemReader, ItemWrite, ItemProcessor Skips Exception Handling Skip, Retry, Rollback JobParameters Validation Flow Control Sequential, Parallel, Conditional, etc.
16. PROVIDED READERS Ampq MongoDB, Neo4j, Spring Data Repository (as of v2.2.0) Hibernate Cursor Hibernate Paging Ibatis Paging JDBC Cursor JDBC Paging JPA Paging Stored Procedure Flat File Spring Resource JMS StAX Event
17. PROVIDED WRITERS Ampq MongoDB, Neo4j, Spring Data Repository (as of v2.2.0) Gemfire (as of v2.2.0) Hibernate Ibatis Batch JDBC Batch JPA Flat File Simple Mail MIME Message StAX Event
18. JSR-352 BATCH APPLICATIONS FOR THE JAVA PLATFORM v1.0 Final Release - May 24, 2013 Designed for both Java SE and Java EE platforms Requires Java 6+ Designed with DI in mind without specifying a DI implementation Based on the Chunk-Oriented Processing and Reader- Processor-Writer patterns of Spring Batch [ h t t p : / / j c p . o r g / a b o u t J a v a / c o m m u n i t y p r o c e s s / f i n a l / j s r 3 5 2 / i n d e x . h t m l ]
19. DIFFERENCES? JSR-352 batchlet => Spring Batch tasklet Typically non-item-oriented tasks JSR-352 requires Job XML documents in META-INF/batch- jobs JSR-352 item count or time based commit-intervals JSR-352 batchlet and chunk are mutually exclusive, Spring Batch chunk is encapsulated by tasklet [ h t t p : / / w w w . i n f o q . c o m / n e w s / 2 0 1 3 / 0 6 / e e 7 - s p r i n g - b a t c h ]
20. GRAILS SPRING BATCH Current Version: 1.0 Github URL: Spring Batch v2.1.9.RELEASE Spring Batch Admin v1.2.2.RELEASE Compatible with Grails 2.x+ http://github.com/johnrengelman/grails-spring- batch
21. FEATURES Defined job definitions using Spring BeanBuilder Groovy syntax Automatic detection of batch configuration files (grails- app/batch) Automatic configuration reloading during development JMX integration Automatic deployment of Spring Batch DDL Specify the database type UI Services for providing access to Spring Batch data Base views that mimic Spring Batch Admin views
22. INSTALLING PLUGIN Add to BuildConfig.groovy in 'plugins' section c o m p i l e " : s p r i n g - b a t c h : 1 . 0 "
23. CONFIGURING PLUGIN Every configuration value has sane default for simple application. Modify configuration in Config.groovy p l u g i n { s p r i n g B a t c h { d a t a b a s e = ' m y s q l ' l o a d T a b l e s = f a l s e } } < b r > e n v i r o n m e n t s { t e s t { p l u g i n . s p r i n g B a t c h . d a t a b a s e = ' h 2 ' p l u g i n . s p r i n g B a t c h . l o a d T a b l e s = t r u e < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n >
24. CONFIGURING THE DATASOURCE Specify which datasource to use in multi-database application Be aware of cross datasource transactions when using this mode. No XA management by default in Grails Specify a prefix for Spring Batch tables All spring batch tables are in UPPERCASE. Be aware for case sensitive DBs & File Systems p l u g i n . s p r i n g B a t c h . d a t a S o u r c e = ' b a t c h D a t a S o u r c e ' p l u g i n . s p r i n g B a t c h . t a b l e P r e f i x = ' S P R I N G B A T C H _ '
25. DEFINING BATCH JOBS Define jobs in 'grails-app/batch' Files end with BatchConfig.groovy g r a i l s - a p p / b a t c h / S i m p l e J o b B a t c h C o n f i g . g r o o v y
26. DEFINING BATCH JOBS Define jobs in 'grails-app/batch' Files end with BatchConfig.groovy Wrap definition with a 'beans' closure g r a i l s - a p p / b a t c h / S i m p l e J o b B a t c h C o n f i g . g r o o v y < f o n t c o l o r = " # f f 0 0 0 0 " > b e a n s < / f o n t > { }
27. DEFINING BATCH JOBS Define jobs in 'grails-app/batch' Files end with BatchConfig.groovy Wrap definition with a 'beans' closure Define jobs using the 'batch' prefix g r a i l s - a p p / b a t c h / S i m p l e J o b B a t c h C o n f i g . g r o o v y < f o n t c o l o r = " # f f 0 0 0 0 " > b e a n s < / f o n t > { } b e a n s { < f o n t c o l o r = " # f f 0 0 0 0 " > b a t c h . j o b < / f o n t > ( i d : ' s i m p l e J o b ' ) { < f o n t c o l o r = " # f f 0 0 0 0 " > b a t c h . s t e p < / f o n t > ( i d : ' l o g S t a r t ' ) { < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > . . . < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n >
28. DEFINING BATCH JOBS Utilize Grails service by autowiring them into Job components b e a n s { b a t c h . j o b ( i d : ' s i m p l e J o b ' ) b a t c h . s t e p ( i d : ' l o g S t a r t ' ) { b a t c h . t a s k l e t ( r e f : ' l o g S t a r t ' ) } } < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > < b r > < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > l o g S t a r t ( P r i n t S t a r t M e s s a g e T a s k l e t ) { b e a n - > < / s p a < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > b e a n . a u t o w i r e = ' b y N a m e ' < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n >
29. EXECUTING BATCH JOBS Plugin automatically registers a JobLauncher Spring bean Each Job is also registered as a Spring bean Use Grails DI to access and call < s p a n > c l a s s S i m p l e J o b S e r v i c e { < / s p a n > < b r > J o b L a u n c h e r j o b L a u n c h e r / / r e g i s t e r e d b y p l u g i n }
30. EXECUTING BATCH JOBS Plugin automatically registers a JobLauncher Spring bean Each Job is also registered as a Spring bean Use Grails DI to access and call < s p a n > c l a s s S i m p l e J o b S e r v i c e { < / s p a n > < b r > J o b L a u n c h e r j o b L a u n c h e r / / r e g i s t e r e d b y p l u g i n J o b s i m p l e J o b / / r e g i s t e r e d b y d e f i n i n g t h e J o b }
31. EXECUTING BATCH JOBS Plugin automatically registers a JobLauncher Spring bean Each Job is also registered as a Spring bean Use Grails DI to access and call < s p a n > c l a s s S i m p l e J o b S e r v i c e { < / s p a n > < b r > J o b L a u n c h e r j o b L a u n c h e r / / r e g i s t e r e d b y p l u g i n J o b s i m p l e J o b / / r e g i s t e r e d b y d e f i n i n g t h e J o b < b r > L o n g e x e c u t e ( ) { } }
32. EXECUTING BATCH JOBS Plugin automatically registers a JobLauncher Spring bean Each Job is also registered as a Spring bean Use Grails DI to access and call < s p a n > c l a s s S i m p l e J o b S e r v i c e { < / s p a n > < b r > J o b L a u n c h e r j o b L a u n c h e r / / r e g i s t e r e d b y p l u g i n J o b s i m p l e J o b / / r e g i s t e r e d b y d e f i n i n g t h e J o b < b r > L o n g e x e c u t e ( ) { J o b P a r a m e t e r s B u i l d e r b u i l d e r = n e w J o b P a r a m e t e r s B u i l d e r ( ) b u i l d e r . a d d S t r i n g ( ' b a s e M e s s a g e ' , ' s o m e b a s e t e x t ' ) } }
33. EXECUTING BATCH JOBS Plugin automatically registers a JobLauncher Spring bean Each Job is also registered as a Spring bean Use Grails DI to access and call < s p a n > c l a s s S i m p l e J o b S e r v i c e { < / s p a n > < b r > J o b L a u n c h e r j o b L a u n c h e r / / r e g i s t e r e d b y p l u g i n J o b s i m p l e J o b / / r e g i s t e r e d b y d e f i n i n g t h e J o b < b r > L o n g e x e c u t e ( ) { J o b P a r a m e t e r s B u i l d e r b u i l d e r = n e w J o b P a r a m e t e r s B u i l d e r ( ) b u i l d e r . a d d S t r i n g ( ' b a s e M e s s a g e ' , ' s o m e b a s e t e x t ' ) < b r > J o b E x e c u t i o n j o b E x e c u t i o n = j o b L a u n c h e r . r u n ( s i m p l e J o b , b u i l d e r . t o J o b P a r a m e t e r s ( ) r e t u r n j o b E x e c u t i o n . j o b I d } }
34. SAMPLE APPLICATION B a s e d o n S p r i n g B a t c h F o o t b a l l h t t p s : / / g i t h u b . c o m / j o h n r e n g e l m a n / g r a i l s - s p r i n g - b a t c h - e x a m p l e h t t p s : / / g i t h u b . c o m / s p r i n g - p r o j e c t s / s p r i n g - b a t c h / b l o b / m a s t e r / s p r i n g - b a t c h - s a m p l e s / s r c / m a i n / r e s o u r c e s / j o b s / f o o t b a l l J o b . x m l
35. PLUGIN ROADMAP Spring Batch 2.2.0.RELEASE Requires minimum Grails v2.1.1 Drive towards Spring Batch annotations and builder patterns Binary releases Verification with Grails 2.3