Batch Processing

Deck from the Batch Processing in Action presentation

Transcript

1. BATCH PROCESSING IN ACTION HILLMER CHONA RODRIGO GRACIANO
2. WHAT SPRING BATCH IS? Spring Batch is a lightweight, comprehensive batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
3. WHAT SPRING BATCH IS NOT? Spring Batch is not a scheduler https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
4. BATCH VOCABULARY
5. THE BASICS
6. SHOW ME SOME CODE: BASIC DEMO This Photo by Unknown Author is licensed under CC BY-NC-ND
7. JOB STEP REPOSITORY RUNNING
8. BATCH ARCHITECTURE (SIMPLIFIED ) https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader Writer 1 * 1 1 1 1
9. JOB REPOSITORY •It’s how Job, Step, JobParameters, JobExecution, and StepExecution are persisted
10. • JobRepository • JobLauncher • JobRegistry • JobExplorer • PlatformTransactionManager • JobBuilderFactory • StepBuilderFactory @EnableBatchProcessing
11. RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within a web container
12. SHOW ME SOME CODE: RUNNING JOBS This Photo by Unknown Author is licensed under CC BY-NC-ND
13. RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within a web container • Synchronous x Asynchronous
14. PROCESSORS CHUNKS WE CAN DO BETTER
15. STEP SEQUENCE Process Step Read Write
16. BATCH ARCHITECTURE https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader Writer 1 * 1 1 1 1 Processor 1 1
17. PROCESSORS •Transforming •Filtering •Validations
18. STEP SEQUENCE Process Step Read Write
19. CHUNK SEQUENCE Process Step Read Write Read Process
20. MORE CODE MORE CODE: PROCESSORS & CHUNKS This Photo by Unknown Author is licensed under CC BY-NC-ND
21. +READERS +WRITERS TASKLETS
22. READERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file
23. MULTI-FILE READER •Read multiple file in a single step •Files must have the same format •Works with XML, JSON and flat files
24. SHOW ME SOME CODE: MULTI-READERS This Photo by Unknown Author is licensed under CC BY-NC-ND
25. WRITERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file • Email
26. TASKLETS • 1 time operation instead of read/write • Run a DB query • Copy Files • Zip/unzip files
27. BEYOND THE BASICS LISTENERS STEP FLOW SCALING AND PARALLEL PROCESSING
28. LISTENERS JobExecutionListener StepExecutionListener ItemReadListener ItemWriteListener ChunkListener ItemProcessListener SkipListener
29. JOB EXECUTION LISTENERS
30. STEP LISTENERS • before/after step StepExecutionListener • before/after read • onReadError ItemReadListener • before/after write • onWriteError ItemWriteListener
31. STEP LISTENERS • before/after/after error chunk ChunkListener • before/after process • onProcessError ItemProcessListener • onSkipInRead • onSkipInProcess • onSkipInWrite SkipListener
32. STEP FLOW • Sequential
33. STEP FLOW: SEQUENTIAL Step 1 Step 2 Step 3
34. STEP FLOW • Sequential • Conditional (ExitStatus) • Decider
35. STEP FLOW: CONDITIONAL & DECIDER Condition Step 1 Step 3 Step 2
36. STEP FLOW • Sequential • Conditional (ExitStatus) • Decider • Split
37. STEP FLOW: SPLIT Step 1 Step 2 Step 3 Start Step 4
38. SCALING AND PARALLEL PROCESSING • Multi-threaded (Async Executor) • Parallel steps (Split) • Remote Chunking (Data processed by slaves – Real data sent not a reference) • Remote Partitioning (Data in chunks and local to slaves)
39. MORE FUNCTIONALITIES RESTARTING SKIPING FAILURES RETRY
40. USEFUL FUNCTIONALITIES - START •startLimit(n) •allowStartIfComplete(boolean)
41. USEFUL FUNCTIONALITIES – SKIPPING FAILURES •faultTolerant() •skip(SomeException.class) •skipLimit(n) •noSkip(SomeException.class) •noRollback(SomeException.class)
42. USEFUL FUNCTIONALITIES - RETRY •faultTolerant() •retry(SomeException.class) •retryLimit(n)
43. CODE CODE This Photo by Unknown Author is licensed under CC BY-NC-ND
44. QUESTIONS? https://github.com/rodrigolgraciano/batch-demo.git https://speakerdeck.com/graciano/batch-processing @rodrigograciano @ HillmerCh
45. CHUNK-ORIENTED STEP https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png
46. JOB Job JobInstance JobExecution JobParameters Present at Devnexus title = Batch Processing in Action Present Batch Processing in Action at Devnexus First try to present – hopefully, it will work
47. …STEP Job JobInstance JobExecution Step StepExecution Read count Write count Filter count…
48. CHUNK-ORIENTED STEP + PROCESSORS https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png