Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Batch Processing
Rodrigo Graciano
April 11, 2022
Programming
0
97
Batch Processing
Deck from the Batch Processing in Action presentation
Rodrigo Graciano
April 11, 2022
Tweet
Share
More Decks by Rodrigo Graciano
See All by Rodrigo Graciano
Java 9 ao 17 - Oracle no TDC
graciano
0
34
KnoxJava - Java 9-17
graciano
0
93
Of concurrency and other demons
graciano
0
53
Java 9 to Java 16: A review of recent changes to the language
graciano
0
82
Java 9 ao 15 - Evolução da Linguagem Java
graciano
0
170
Other Decks in Programming
See All in Programming
アジャイルで不確実性に向き合うための開発タスクの切り方
tanden
4
1.1k
Licences open source : entre guerre de clochers et radicalité
pylapp
2
550
FutureCon 2022 FlutterアプリのPerformance測定
harukafujita
0
130
Lookerとdbtの共存
ttccddtoki
0
610
Rector, time to refactor your code easily
guikingone
2
140
WindowsコンテナDojo:第6回 Red Hat OpenShift入門
oniak3ibm
PRO
0
130
Amazon Lookout for Visionで 筆跡鑑定してみた
cmnakamurashogo
0
120
僕が便利だと感じる Snow Monkey の特徴/20220723_Gifu_WordPress_Meetup
oleindesign
0
100
JetpackCompose 導入半年で感じた 改善点
spbaya0141
0
140
Recap CDN, Edge, WebAssembly | ワインと鍋.js#1
sadnessojisan
2
1.2k
Google I/O 2022 Android関連概要 / Google I/O 2022 Android summary
phicdy
0
350
There's an API for that!
mariatta
PRO
0
100
Featured
See All Featured
Docker and Python
trallard
27
1.6k
GitHub's CSS Performance
jonrohan
1020
420k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
351
21k
The MySQL Ecosystem @ GitHub 2015
samlambert
239
11k
Atom: Resistance is Futile
akmur
255
20k
Agile that works and the tools we love
rasmusluckow
319
19k
Become a Pro
speakerdeck
PRO
3
900
Writing Fast Ruby
sferik
612
57k
Designing Experiences People Love
moore
130
22k
Building Flexible Design Systems
yeseniaperezcruz
310
34k
A Modern Web Designer's Workflow
chriscoyier
689
180k
Pencils Down: Stop Designing & Start Developing
hursman
113
9.8k
Transcript
BATCH PROCESSING IN ACTION HILLMER CHONA RODRIGO GRACIANO
WHAT SPRING BATCH IS? Spring Batch is a lightweight, comprehensive
batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
WHAT SPRING BATCH IS NOT? Spring Batch is not a
scheduler https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
BATCH VOCABULARY
THE BASICS
SHOW ME SOME CODE: BASIC DEMO This Photo by Unknown
Author is licensed under CC BY-NC-ND
JOB STEP REPOSITORY RUNNING
BATCH ARCHITECTURE (SIMPLIFIED ) https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader
Writer 1 * 1 1 1 1
JOB REPOSITORY •It’s how Job, Step, JobParameters, JobExecution, and StepExecution
are persisted
• JobRepository • JobLauncher • JobRegistry • JobExplorer • PlatformTransactionManager
• JobBuilderFactory • StepBuilderFactory @EnableBatchProcessing
RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within
a web container
SHOW ME SOME CODE: RUNNING JOBS This Photo by Unknown
Author is licensed under CC BY-NC-ND
RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within
a web container • Synchronous x Asynchronous
PROCESSORS CHUNKS WE CAN DO BETTER
STEP SEQUENCE Process Step Read Write
BATCH ARCHITECTURE https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader Writer 1
* 1 1 1 1 Processor 1 1
PROCESSORS •Transforming •Filtering •Validations
STEP SEQUENCE Process Step Read Write
CHUNK SEQUENCE Process Step Read Write Read Process
MORE CODE MORE CODE: PROCESSORS & CHUNKS This Photo by
Unknown Author is licensed under CC BY-NC-ND
+READERS +WRITERS TASKLETS
READERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat
Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file
MULTI-FILE READER •Read multiple file in a single step •Files
must have the same format •Works with XML, JSON and flat files
SHOW ME SOME CODE: MULTI-READERS This Photo by Unknown Author
is licensed under CC BY-NC-ND
WRITERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat
Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file • Email
TASKLETS • 1 time operation instead of read/write • Run
a DB query • Copy Files • Zip/unzip files
BEYOND THE BASICS LISTENERS STEP FLOW SCALING AND PARALLEL PROCESSING
LISTENERS JobExecutionListener StepExecutionListener ItemReadListener ItemWriteListener ChunkListener ItemProcessListener SkipListener
JOB EXECUTION LISTENERS
STEP LISTENERS • before/after step StepExecutionListener • before/after read •
onReadError ItemReadListener • before/after write • onWriteError ItemWriteListener
STEP LISTENERS • before/after/after error chunk ChunkListener • before/after process
• onProcessError ItemProcessListener • onSkipInRead • onSkipInProcess • onSkipInWrite SkipListener
STEP FLOW • Sequential
STEP FLOW: SEQUENTIAL Step 1 Step 2 Step 3
STEP FLOW • Sequential • Conditional (ExitStatus) • Decider
STEP FLOW: CONDITIONAL & DECIDER Condition Step 1 Step 3
Step 2
STEP FLOW • Sequential • Conditional (ExitStatus) • Decider •
Split
STEP FLOW: SPLIT Step 1 Step 2 Step 3 Start
Step 4
SCALING AND PARALLEL PROCESSING • Multi-threaded (Async Executor) • Parallel
steps (Split) • Remote Chunking (Data processed by slaves – Real data sent not a reference) • Remote Partitioning (Data in chunks and local to slaves)
MORE FUNCTIONALITIES RESTARTING SKIPING FAILURES RETRY
USEFUL FUNCTIONALITIES - START •startLimit(n) •allowStartIfComplete(boolean)
USEFUL FUNCTIONALITIES – SKIPPING FAILURES •faultTolerant() •skip(SomeException.class) •skipLimit(n) •noSkip(SomeException.class) •noRollback(SomeException.class)
USEFUL FUNCTIONALITIES - RETRY •faultTolerant() •retry(SomeException.class) •retryLimit(n)
CODE CODE This Photo by Unknown Author is licensed under
CC BY-NC-ND
QUESTIONS? https://github.com/rodrigolgraciano/batch-demo.git https://speakerdeck.com/graciano/batch-processing @rodrigograciano @ HillmerCh
CHUNK-ORIENTED STEP https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png
JOB Job JobInstance JobExecution JobParameters Present at Devnexus title =
Batch Processing in Action Present Batch Processing in Action at Devnexus First try to present – hopefully, it will work
…STEP Job JobInstance JobExecution Step StepExecution Read count Write count
Filter count…
CHUNK-ORIENTED STEP + PROCESSORS https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png