Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Batch Processing
Search
Rodrigo Graciano
April 11, 2022
Programming
0
290
Batch Processing
Deck from the Batch Processing in Action presentation
Rodrigo Graciano
April 11, 2022
Tweet
Share
More Decks by Rodrigo Graciano
See All by Rodrigo Graciano
Your Java isn't the same
graciano
0
150
Cloud Batch
graciano
0
130
Java 9 ao 17 - Oracle no TDC
graciano
0
96
KnoxJava - Java 9-17
graciano
0
140
Of concurrency and other demons
graciano
0
94
Java 9 to Java 16: A review of recent changes to the language
graciano
0
110
Java 9 ao 15 - Evolução da Linguagem Java
graciano
0
240
Other Decks in Programming
See All in Programming
Nuxtベースの「WXT」でChrome拡張を作成する | Vue Fes 2024 ランチセッション
moshi1121
1
520
cXML という電子商取引の トランザクションを支える プロトコルと向きあっている話
phigasui
3
2.3k
ECSのサービス間通信 4つの方法を比較する 〜Canary,Blue/Greenも添えて〜
tkikuc
11
2.3k
Piniaの現状と今後
waka292
5
1.5k
現場で役立つモデリング 超入門
masuda220
PRO
13
2.9k
リリース8年目のサービスの1800個のERBファイルをViewComponentに移行した方法とその結果
katty0324
5
3.6k
Java ジェネリクス入門 2024
nagise
0
610
Android 15 でアクションバー表示時にステータスバーが白くなってしまう問題
tonionagauzzi
0
140
Webの技術スタックで マルチプラットフォームアプリ開発を可能にするElixirDesktopの紹介
thehaigo
2
920
WEBエンジニア向けAI活用入門
sutetotanuki
0
300
Identifying User Idenity
moro
6
7.9k
Vitest Browser Mode への期待 / Vitest Browser Mode
odanado
PRO
2
1.7k
Featured
See All Featured
A Tale of Four Properties
chriscoyier
156
23k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
167
49k
Designing for humans not robots
tammielis
249
25k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
3
370
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Learning to Love Humans: Emotional Interface Design
aarron
272
40k
Fireside Chat
paigeccino
32
3k
Six Lessons from altMBA
skipperchong
26
3.5k
A designer walks into a library…
pauljervisheath
202
24k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
27
1.9k
GitHub's CSS Performance
jonrohan
1030
460k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
Transcript
BATCH PROCESSING IN ACTION HILLMER CHONA RODRIGO GRACIANO
WHAT SPRING BATCH IS? Spring Batch is a lightweight, comprehensive
batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
WHAT SPRING BATCH IS NOT? Spring Batch is not a
scheduler https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
BATCH VOCABULARY
THE BASICS
SHOW ME SOME CODE: BASIC DEMO This Photo by Unknown
Author is licensed under CC BY-NC-ND
JOB STEP REPOSITORY RUNNING
BATCH ARCHITECTURE (SIMPLIFIED ) https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader
Writer 1 * 1 1 1 1
JOB REPOSITORY •It’s how Job, Step, JobParameters, JobExecution, and StepExecution
are persisted
• JobRepository • JobLauncher • JobRegistry • JobExplorer • PlatformTransactionManager
• JobBuilderFactory • StepBuilderFactory @EnableBatchProcessing
RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within
a web container
SHOW ME SOME CODE: RUNNING JOBS This Photo by Unknown
Author is licensed under CC BY-NC-ND
RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within
a web container • Synchronous x Asynchronous
PROCESSORS CHUNKS WE CAN DO BETTER
STEP SEQUENCE Process Step Read Write
BATCH ARCHITECTURE https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader Writer 1
* 1 1 1 1 Processor 1 1
PROCESSORS •Transforming •Filtering •Validations
STEP SEQUENCE Process Step Read Write
CHUNK SEQUENCE Process Step Read Write Read Process
MORE CODE MORE CODE: PROCESSORS & CHUNKS This Photo by
Unknown Author is licensed under CC BY-NC-ND
+READERS +WRITERS TASKLETS
READERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat
Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file
MULTI-FILE READER •Read multiple file in a single step •Files
must have the same format •Works with XML, JSON and flat files
SHOW ME SOME CODE: MULTI-READERS This Photo by Unknown Author
is licensed under CC BY-NC-ND
WRITERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat
Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file • Email
TASKLETS • 1 time operation instead of read/write • Run
a DB query • Copy Files • Zip/unzip files
BEYOND THE BASICS LISTENERS STEP FLOW SCALING AND PARALLEL PROCESSING
LISTENERS JobExecutionListener StepExecutionListener ItemReadListener ItemWriteListener ChunkListener ItemProcessListener SkipListener
JOB EXECUTION LISTENERS
STEP LISTENERS • before/after step StepExecutionListener • before/after read •
onReadError ItemReadListener • before/after write • onWriteError ItemWriteListener
STEP LISTENERS • before/after/after error chunk ChunkListener • before/after process
• onProcessError ItemProcessListener • onSkipInRead • onSkipInProcess • onSkipInWrite SkipListener
STEP FLOW • Sequential
STEP FLOW: SEQUENTIAL Step 1 Step 2 Step 3
STEP FLOW • Sequential • Conditional (ExitStatus) • Decider
STEP FLOW: CONDITIONAL & DECIDER Condition Step 1 Step 3
Step 2
STEP FLOW • Sequential • Conditional (ExitStatus) • Decider •
Split
STEP FLOW: SPLIT Step 1 Step 2 Step 3 Start
Step 4
SCALING AND PARALLEL PROCESSING • Multi-threaded (Async Executor) • Parallel
steps (Split) • Remote Chunking (Data processed by slaves – Real data sent not a reference) • Remote Partitioning (Data in chunks and local to slaves)
MORE FUNCTIONALITIES RESTARTING SKIPING FAILURES RETRY
USEFUL FUNCTIONALITIES - START •startLimit(n) •allowStartIfComplete(boolean)
USEFUL FUNCTIONALITIES – SKIPPING FAILURES •faultTolerant() •skip(SomeException.class) •skipLimit(n) •noSkip(SomeException.class) •noRollback(SomeException.class)
USEFUL FUNCTIONALITIES - RETRY •faultTolerant() •retry(SomeException.class) •retryLimit(n)
CODE CODE This Photo by Unknown Author is licensed under
CC BY-NC-ND
QUESTIONS? https://github.com/rodrigolgraciano/batch-demo.git https://speakerdeck.com/graciano/batch-processing @rodrigograciano @ HillmerCh
CHUNK-ORIENTED STEP https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png
JOB Job JobInstance JobExecution JobParameters Present at Devnexus title =
Batch Processing in Action Present Batch Processing in Action at Devnexus First try to present – hopefully, it will work
…STEP Job JobInstance JobExecution Step StepExecution Read count Write count
Filter count…
CHUNK-ORIENTED STEP + PROCESSORS https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png