Batch Processing - Speaker Deck

Slide 1

Slide 1 text

BATCH PROCESSING IN ACTION HILLMER CHONA RODRIGO GRACIANO

Slide 2

Slide 2 text

WHAT SPRING BATCH IS? Spring Batch is a lightweight, comprehensive batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro

Slide 3

Slide 3 text

WHAT SPRING BATCH IS NOT? Spring Batch is not a scheduler https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro

Slide 4

Slide 4 text

BATCH VOCABULARY

Slide 5

Slide 5 text

THE BASICS

Slide 6

Slide 6 text

SHOW ME SOME CODE: BASIC DEMO This Photo by Unknown Author is licensed under CC BY-NC-ND

Slide 7

Slide 7 text

JOB STEP REPOSITORY RUNNING

Slide 8

Slide 8 text

BATCH ARCHITECTURE (SIMPLIFIED ) https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader Writer 1 * 1 1 1 1

Slide 9

Slide 9 text

JOB REPOSITORY •It’s how Job, Step, JobParameters, JobExecution, and StepExecution are persisted

Slide 10

Slide 10 text

• JobRepository • JobLauncher • JobRegistry • JobExplorer • PlatformTransactionManager • JobBuilderFactory • StepBuilderFactory @EnableBatchProcessing

Slide 11

Slide 11 text

RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within a web container

Slide 12

Slide 12 text

SHOW ME SOME CODE: RUNNING JOBS This Photo by Unknown Author is licensed under CC BY-NC-ND

Slide 13

Slide 13 text

RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within a web container • Synchronous x Asynchronous

Slide 14

Slide 14 text

PROCESSORS CHUNKS WE CAN DO BETTER

Slide 15

Slide 15 text

STEP SEQUENCE Process Step Read Write

Slide 16

Slide 16 text

BATCH ARCHITECTURE https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader Writer 1 * 1 1 1 1 Processor 1 1

Slide 17

Slide 17 text

PROCESSORS •Transforming •Filtering •Validations

Slide 18

Slide 18 text

STEP SEQUENCE Process Step Read Write

Slide 19

Slide 19 text

CHUNK SEQUENCE Process Step Read Write Read Process

Slide 20

Slide 20 text

MORE CODE MORE CODE: PROCESSORS & CHUNKS This Photo by Unknown Author is licensed under CC BY-NC-ND

Slide 21

Slide 21 text

+READERS +WRITERS TASKLETS

Slide 22

Slide 22 text

READERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file

Slide 23

Slide 23 text

MULTI-FILE READER •Read multiple file in a single step •Files must have the same format •Works with XML, JSON and flat files

Slide 24

Slide 24 text

SHOW ME SOME CODE: MULTI-READERS This Photo by Unknown Author is licensed under CC BY-NC-ND

Slide 25

Slide 25 text

WRITERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file • Email

Slide 26

Slide 26 text

TASKLETS • 1 time operation instead of read/write • Run a DB query • Copy Files • Zip/unzip files

Slide 27

Slide 27 text

BEYOND THE BASICS LISTENERS STEP FLOW SCALING AND PARALLEL PROCESSING

Slide 28

Slide 28 text

LISTENERS JobExecutionListener StepExecutionListener ItemReadListener ItemWriteListener ChunkListener ItemProcessListener SkipListener

Slide 29

Slide 29 text

JOB EXECUTION LISTENERS

Slide 30

Slide 30 text

STEP LISTENERS • before/after step StepExecutionListener • before/after read • onReadError ItemReadListener • before/after write • onWriteError ItemWriteListener

Slide 31

Slide 31 text

STEP LISTENERS • before/after/after error chunk ChunkListener • before/after process • onProcessError ItemProcessListener • onSkipInRead • onSkipInProcess • onSkipInWrite SkipListener

Slide 32

Slide 32 text

STEP FLOW • Sequential

Slide 33

Slide 33 text

STEP FLOW: SEQUENTIAL Step 1 Step 2 Step 3

Slide 34

Slide 34 text

STEP FLOW • Sequential • Conditional (ExitStatus) • Decider

Slide 35

Slide 35 text

STEP FLOW: CONDITIONAL & DECIDER Condition Step 1 Step 3 Step 2

Slide 36

Slide 36 text

STEP FLOW • Sequential • Conditional (ExitStatus) • Decider • Split

Slide 37

Slide 37 text

STEP FLOW: SPLIT Step 1 Step 2 Step 3 Start Step 4

Slide 38

Slide 38 text

SCALING AND PARALLEL PROCESSING • Multi-threaded (Async Executor) • Parallel steps (Split) • Remote Chunking (Data processed by slaves – Real data sent not a reference) • Remote Partitioning (Data in chunks and local to slaves)

Slide 39

Slide 39 text

MORE FUNCTIONALITIES RESTARTING SKIPING FAILURES RETRY

Slide 40

Slide 40 text

USEFUL FUNCTIONALITIES - START •startLimit(n) •allowStartIfComplete(boolean)

Slide 41

Slide 41 text

USEFUL FUNCTIONALITIES – SKIPPING FAILURES •faultTolerant() •skip(SomeException.class) •skipLimit(n) •noSkip(SomeException.class) •noRollback(SomeException.class)

Slide 42

Slide 42 text

USEFUL FUNCTIONALITIES - RETRY •faultTolerant() •retry(SomeException.class) •retryLimit(n)

Slide 43

Slide 43 text

CODE CODE This Photo by Unknown Author is licensed under CC BY-NC-ND

Slide 44

Slide 44 text

QUESTIONS? https://github.com/rodrigolgraciano/batch-demo.git https://speakerdeck.com/graciano/batch-processing @rodrigograciano @ HillmerCh

Slide 45

Slide 45 text

CHUNK-ORIENTED STEP https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png

Slide 46

Slide 46 text

JOB Job JobInstance JobExecution JobParameters Present at Devnexus title = Batch Processing in Action Present Batch Processing in Action at Devnexus First try to present – hopefully, it will work

Slide 47

Slide 47 text

…STEP Job JobInstance JobExecution Step StepExecution Read count Write count Filter count…

Slide 48

Slide 48 text

CHUNK-ORIENTED STEP + PROCESSORS https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png