Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Enterprise Grails - Spring Batch

Enterprise Grails - Spring Batch

Batch processing in Grails with Spring Batch

John Engelman

January 14, 2014
Tweet

More Decks by John Engelman

Other Decks in Technology

Transcript

  1. ENTERPRISE GRAILS Batch Processing in Grails with Spring Batch John

    Engelman @johnrengelman Principal Consultant, Object Partners
  2. WHAT IS BATCH PROCESSING?

  3. WHAT IS BATCH PROCESSING? "Execution of a series of programs

    ("jobs") on a computer without manual intervention" (Wikipedia)
  4. WHAT IS BATCH PROCESSING? "Time based events, periodic application of

    complex business rules processed repetitively across very large data sets, or integration of information that..requires formatting, validation, and processing in a transactional manner" (Spring Source)
  5. APPLICATIONS OF BATCH PROCESSING Data import Extract - Transform -

    Load (ETL) Data restore Data export Report Generation Data transmission Backups Data conversion/processing Long running processes
  6. BATCH PROCESSING FRAMEWORKS Cron Actually a scheduler z/OS Mainframe Spring

    Batch bmap4j Hadoop MapReduce
  7. SPRING BATCH FEATURES Item/Chunk Oriented Processing Job Retry/Repeat Various core

    reader/write implementations Transaction Management Exception Handling Chunking & Partioning Meta-Data Persistence Based on Spring Framework Job & Step Attribution Binding XML/Annotation configuration
  8. ANATOMY OF A SPRING BATCH JOB JOB Encapsulates the batch

    process Consists of at least 1 Step STEP Encapsulates a single (simple or complex) task within the batch process Has exactly 1 ItemReader & 1 ItemWriter Optionally has 1 ItemProcessor
  9. ANATOMY OF A SPRING BATCH JOB Job Parameters The input

    data to configure/execute the job JobInstance The execution of a Job for given set of JobParameters JobInstance = Job + JobParameters JobExecution A single attempt at executing a JobInstance Many executions for a single instance Instance is complete when an execution completes StepExecution A single attempt at executing a Step within a Job. [ h t t p : / / w w w . s l i d e s h a r e . n e t / g s c h m u t z / s p r i n g - b a t c h - 2 0 - 2 7 9 4 0 1 5 ]
  10. ANATOMY OF A SPRING BATCH JOB [ h t t

    p s : / / r a w . g i t h u b . c o m / S p r i n g S o u r c e / s p r i n g - b a t c h / m a s t e r / s r c / s i t e / d o c b o o k / r e f e r e n c e / i m a g e s / j o b H e i r a r c h y W i t h S t e p s . p n g ]
  11. ANATOMY OF A SPRING BATCH JOB ITEMREADER API for retrieving

    input data for a step Retrieves 1 item at a time Returns null when input is exhausted < f o n t s t y l e = " f o n t - s i z e : 2 4 p x ; " > p a c k a g e o r g . s p r i n g f r a m e w o r k . b a t c h . i t e m ; p u b l i c i n t e r f a c e I t e m R e a d e r < T > { T r e a d ( ) t h r o w s E x c e p t i o n , U n e x p e c t e d I n p u t E x c e p t i o n , P a r s e E x c e p t i o n , N o n T r a n s i e n t R e s o } < / f o n t >
  12. ANATOMY OF A SPRING BATCH JOB ITEMPROCESSOR API for transforming/processing

    input item to output item Processes 1 item at a time Returns null to indicate that result should not be saved < f o n t s t y l e = " f o n t - s i z e : 2 4 p x ; " > p a c k a g e o r g . s p r i n g f r a m e w o r k . b a t c h . i t e m ; p u b l i c i n t e r f a c e I t e m P r o c e s s o r < I , O > { O p r o c e s s ( I i ) t h r o w s E x c e p t i o n ; } < / f o n t >
  13. ANATOMY OF A SPRING BATCH JOB ITEMWRITER API representing output

    of Step Chunk-Oriented Transaction is committed for a series of ItemWriter calls Receives a List of read or processed items List size is size of commit-interval < f o n t s t y l e = " f o n t - s i z e : 2 4 p x ; " > p a c k a g e o r g . s p r i n g f r a m e w o r k . b a t c h . i t e m ; p u b l i c i n t e r f a c e I t e m W r i t e r < T > { v o i d w r i t e ( L i s t < ? e x t e n d s T > i t e m s ) t h r o w s E x c e p t i o n ; } < / f o n t >
  14. ANATOMY OF A SPRING BATCH JOB [ h t t

    p s : / / r a w . g i t h u b . c o m / S p r i n g S o u r c e / s p r i n g - b a t c h / m a s t e r / s r c / s i t e / d o c b o o k / r e f e r e n c e / i m a g e s / c h u n k - o r i e n t e d - p r o c e s s i n g . p n g ]
  15. ANATOMY OF A SPRING BATCH JOB Listeners JobExecution, StepExecution, ChunkExecution

    ItemReader, ItemWrite, ItemProcessor Skips Exception Handling Skip, Retry, Rollback JobParameters Validation Flow Control Sequential, Parallel, Conditional, etc.
  16. PROVIDED READERS Ampq MongoDB, Neo4j, Spring Data Repository (as of

    v2.2.0) Hibernate Cursor Hibernate Paging Ibatis Paging JDBC Cursor JDBC Paging JPA Paging Stored Procedure Flat File Spring Resource JMS StAX Event
  17. PROVIDED WRITERS Ampq MongoDB, Neo4j, Spring Data Repository (as of

    v2.2.0) Gemfire (as of v2.2.0) Hibernate Ibatis Batch JDBC Batch JPA Flat File Simple Mail MIME Message StAX Event
  18. JSR-352 BATCH APPLICATIONS FOR THE JAVA PLATFORM v1.0 Final Release

    - May 24, 2013 Designed for both Java SE and Java EE platforms Requires Java 6+ Designed with DI in mind without specifying a DI implementation Based on the Chunk-Oriented Processing and Reader- Processor-Writer patterns of Spring Batch [ h t t p : / / j c p . o r g / a b o u t J a v a / c o m m u n i t y p r o c e s s / f i n a l / j s r 3 5 2 / i n d e x . h t m l ]
  19. DIFFERENCES? JSR-352 batchlet => Spring Batch tasklet Typically non-item-oriented tasks

    JSR-352 requires Job XML documents in META-INF/batch- jobs JSR-352 item count or time based commit-intervals JSR-352 batchlet and chunk are mutually exclusive, Spring Batch chunk is encapsulated by tasklet [ h t t p : / / w w w . i n f o q . c o m / n e w s / 2 0 1 3 / 0 6 / e e 7 - s p r i n g - b a t c h ]
  20. GRAILS SPRING BATCH Current Version: 1.0 Github URL: Spring Batch

    v2.1.9.RELEASE Spring Batch Admin v1.2.2.RELEASE Compatible with Grails 2.x+ http://github.com/johnrengelman/grails-spring- batch
  21. FEATURES Defined job definitions using Spring BeanBuilder Groovy syntax Automatic

    detection of batch configuration files (grails- app/batch) Automatic configuration reloading during development JMX integration Automatic deployment of Spring Batch DDL Specify the database type UI Services for providing access to Spring Batch data Base views that mimic Spring Batch Admin views
  22. INSTALLING PLUGIN Add to BuildConfig.groovy in 'plugins' section c o

    m p i l e " : s p r i n g - b a t c h : 1 . 0 "
  23. CONFIGURING PLUGIN Every configuration value has sane default for simple

    application. Modify configuration in Config.groovy p l u g i n { s p r i n g B a t c h { d a t a b a s e = ' m y s q l ' l o a d T a b l e s = f a l s e } } < b r > e n v i r o n m e n t s { t e s t { p l u g i n . s p r i n g B a t c h . d a t a b a s e = ' h 2 ' p l u g i n . s p r i n g B a t c h . l o a d T a b l e s = t r u e < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n >
  24. CONFIGURING THE DATASOURCE Specify which datasource to use in multi-database

    application Be aware of cross datasource transactions when using this mode. No XA management by default in Grails Specify a prefix for Spring Batch tables All spring batch tables are in UPPERCASE. Be aware for case sensitive DBs & File Systems p l u g i n . s p r i n g B a t c h . d a t a S o u r c e = ' b a t c h D a t a S o u r c e ' p l u g i n . s p r i n g B a t c h . t a b l e P r e f i x = ' S P R I N G B A T C H _ '
  25. DEFINING BATCH JOBS Define jobs in 'grails-app/batch' Files end with

    BatchConfig.groovy g r a i l s - a p p / b a t c h / S i m p l e J o b B a t c h C o n f i g . g r o o v y
  26. DEFINING BATCH JOBS Define jobs in 'grails-app/batch' Files end with

    BatchConfig.groovy Wrap definition with a 'beans' closure g r a i l s - a p p / b a t c h / S i m p l e J o b B a t c h C o n f i g . g r o o v y < f o n t c o l o r = " # f f 0 0 0 0 " > b e a n s < / f o n t > { }
  27. DEFINING BATCH JOBS Define jobs in 'grails-app/batch' Files end with

    BatchConfig.groovy Wrap definition with a 'beans' closure Define jobs using the 'batch' prefix g r a i l s - a p p / b a t c h / S i m p l e J o b B a t c h C o n f i g . g r o o v y < f o n t c o l o r = " # f f 0 0 0 0 " > b e a n s < / f o n t > { } b e a n s { < f o n t c o l o r = " # f f 0 0 0 0 " > b a t c h . j o b < / f o n t > ( i d : ' s i m p l e J o b ' ) { < f o n t c o l o r = " # f f 0 0 0 0 " > b a t c h . s t e p < / f o n t > ( i d : ' l o g S t a r t ' ) { < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > . . . < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n >
  28. DEFINING BATCH JOBS Utilize Grails service by autowiring them into

    Job components b e a n s { b a t c h . j o b ( i d : ' s i m p l e J o b ' ) b a t c h . s t e p ( i d : ' l o g S t a r t ' ) { b a t c h . t a s k l e t ( r e f : ' l o g S t a r t ' ) } } < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > < b r > < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > l o g S t a r t ( P r i n t S t a r t M e s s a g e T a s k l e t ) { b e a n - > < / s p a < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > b e a n . a u t o w i r e = ' b y N a m e ' < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n > < s p a n s t y l e = " f o n t - f a m i l y : m o n o s p a c e ; " > } < / s p a n >
  29. EXECUTING BATCH JOBS Plugin automatically registers a JobLauncher Spring bean

    Each Job is also registered as a Spring bean Use Grails DI to access and call < s p a n > c l a s s S i m p l e J o b S e r v i c e { < / s p a n > < b r > J o b L a u n c h e r j o b L a u n c h e r / / r e g i s t e r e d b y p l u g i n }
  30. EXECUTING BATCH JOBS Plugin automatically registers a JobLauncher Spring bean

    Each Job is also registered as a Spring bean Use Grails DI to access and call < s p a n > c l a s s S i m p l e J o b S e r v i c e { < / s p a n > < b r > J o b L a u n c h e r j o b L a u n c h e r / / r e g i s t e r e d b y p l u g i n J o b s i m p l e J o b / / r e g i s t e r e d b y d e f i n i n g t h e J o b }
  31. EXECUTING BATCH JOBS Plugin automatically registers a JobLauncher Spring bean

    Each Job is also registered as a Spring bean Use Grails DI to access and call < s p a n > c l a s s S i m p l e J o b S e r v i c e { < / s p a n > < b r > J o b L a u n c h e r j o b L a u n c h e r / / r e g i s t e r e d b y p l u g i n J o b s i m p l e J o b / / r e g i s t e r e d b y d e f i n i n g t h e J o b < b r > L o n g e x e c u t e ( ) { } }
  32. EXECUTING BATCH JOBS Plugin automatically registers a JobLauncher Spring bean

    Each Job is also registered as a Spring bean Use Grails DI to access and call < s p a n > c l a s s S i m p l e J o b S e r v i c e { < / s p a n > < b r > J o b L a u n c h e r j o b L a u n c h e r / / r e g i s t e r e d b y p l u g i n J o b s i m p l e J o b / / r e g i s t e r e d b y d e f i n i n g t h e J o b < b r > L o n g e x e c u t e ( ) { J o b P a r a m e t e r s B u i l d e r b u i l d e r = n e w J o b P a r a m e t e r s B u i l d e r ( ) b u i l d e r . a d d S t r i n g ( ' b a s e M e s s a g e ' , ' s o m e b a s e t e x t ' ) } }
  33. EXECUTING BATCH JOBS Plugin automatically registers a JobLauncher Spring bean

    Each Job is also registered as a Spring bean Use Grails DI to access and call < s p a n > c l a s s S i m p l e J o b S e r v i c e { < / s p a n > < b r > J o b L a u n c h e r j o b L a u n c h e r / / r e g i s t e r e d b y p l u g i n J o b s i m p l e J o b / / r e g i s t e r e d b y d e f i n i n g t h e J o b < b r > L o n g e x e c u t e ( ) { J o b P a r a m e t e r s B u i l d e r b u i l d e r = n e w J o b P a r a m e t e r s B u i l d e r ( ) b u i l d e r . a d d S t r i n g ( ' b a s e M e s s a g e ' , ' s o m e b a s e t e x t ' ) < b r > J o b E x e c u t i o n j o b E x e c u t i o n = j o b L a u n c h e r . r u n ( s i m p l e J o b , b u i l d e r . t o J o b P a r a m e t e r s ( ) r e t u r n j o b E x e c u t i o n . j o b I d } }
  34. SAMPLE APPLICATION B a s e d o n S

    p r i n g B a t c h F o o t b a l l h t t p s : / / g i t h u b . c o m / j o h n r e n g e l m a n / g r a i l s - s p r i n g - b a t c h - e x a m p l e h t t p s : / / g i t h u b . c o m / s p r i n g - p r o j e c t s / s p r i n g - b a t c h / b l o b / m a s t e r / s p r i n g - b a t c h - s a m p l e s / s r c / m a i n / r e s o u r c e s / j o b s / f o o t b a l l J o b . x m l
  35. PLUGIN ROADMAP Spring Batch 2.2.0.RELEASE Requires minimum Grails v2.1.1 Drive

    towards Spring Batch annotations and builder patterns Binary releases Verification with Grails 2.3