Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Batch Processing

Batch Processing

Deck from the Batch Processing in Action presentation

1d2a6c7cfa9e96ecffe3c10a5018b1b8?s=128

Rodrigo Graciano

April 11, 2022
Tweet

More Decks by Rodrigo Graciano

Other Decks in Programming

Transcript

  1. BATCH PROCESSING IN ACTION HILLMER CHONA RODRIGO GRACIANO

  2. WHAT SPRING BATCH IS? Spring Batch is a lightweight, comprehensive

    batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
  3. WHAT SPRING BATCH IS NOT? Spring Batch is not a

    scheduler https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
  4. BATCH VOCABULARY

  5. THE BASICS

  6. SHOW ME SOME CODE: BASIC DEMO This Photo by Unknown

    Author is licensed under CC BY-NC-ND
  7. JOB STEP REPOSITORY RUNNING

  8. BATCH ARCHITECTURE (SIMPLIFIED ) https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader

    Writer 1 * 1 1 1 1
  9. JOB REPOSITORY •It’s how Job, Step, JobParameters, JobExecution, and StepExecution

    are persisted
  10. • JobRepository • JobLauncher • JobRegistry • JobExplorer • PlatformTransactionManager

    • JobBuilderFactory • StepBuilderFactory @EnableBatchProcessing
  11. RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within

    a web container
  12. SHOW ME SOME CODE: RUNNING JOBS This Photo by Unknown

    Author is licensed under CC BY-NC-ND
  13. RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within

    a web container • Synchronous x Asynchronous
  14. PROCESSORS CHUNKS WE CAN DO BETTER

  15. STEP SEQUENCE Process Step Read Write

  16. BATCH ARCHITECTURE https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader Writer 1

    * 1 1 1 1 Processor 1 1
  17. PROCESSORS •Transforming •Filtering •Validations

  18. STEP SEQUENCE Process Step Read Write

  19. CHUNK SEQUENCE Process Step Read Write Read Process

  20. MORE CODE MORE CODE: PROCESSORS & CHUNKS This Photo by

    Unknown Author is licensed under CC BY-NC-ND
  21. +READERS +WRITERS TASKLETS

  22. READERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat

    Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file
  23. MULTI-FILE READER •Read multiple file in a single step •Files

    must have the same format •Works with XML, JSON and flat files
  24. SHOW ME SOME CODE: MULTI-READERS This Photo by Unknown Author

    is licensed under CC BY-NC-ND
  25. WRITERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat

    Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file • Email
  26. TASKLETS • 1 time operation instead of read/write • Run

    a DB query • Copy Files • Zip/unzip files
  27. BEYOND THE BASICS LISTENERS STEP FLOW SCALING AND PARALLEL PROCESSING

  28. LISTENERS JobExecutionListener StepExecutionListener ItemReadListener ItemWriteListener ChunkListener ItemProcessListener SkipListener

  29. JOB EXECUTION LISTENERS

  30. STEP LISTENERS • before/after step StepExecutionListener • before/after read •

    onReadError ItemReadListener • before/after write • onWriteError ItemWriteListener
  31. STEP LISTENERS • before/after/after error chunk ChunkListener • before/after process

    • onProcessError ItemProcessListener • onSkipInRead • onSkipInProcess • onSkipInWrite SkipListener
  32. STEP FLOW • Sequential

  33. STEP FLOW: SEQUENTIAL Step 1 Step 2 Step 3

  34. STEP FLOW • Sequential • Conditional (ExitStatus) • Decider

  35. STEP FLOW: CONDITIONAL & DECIDER Condition Step 1 Step 3

    Step 2
  36. STEP FLOW • Sequential • Conditional (ExitStatus) • Decider •

    Split
  37. STEP FLOW: SPLIT Step 1 Step 2 Step 3 Start

    Step 4
  38. SCALING AND PARALLEL PROCESSING • Multi-threaded (Async Executor) • Parallel

    steps (Split) • Remote Chunking (Data processed by slaves – Real data sent not a reference) • Remote Partitioning (Data in chunks and local to slaves)
  39. MORE FUNCTIONALITIES RESTARTING SKIPING FAILURES RETRY

  40. USEFUL FUNCTIONALITIES - START •startLimit(n) •allowStartIfComplete(boolean)

  41. USEFUL FUNCTIONALITIES – SKIPPING FAILURES •faultTolerant() •skip(SomeException.class) •skipLimit(n) •noSkip(SomeException.class) •noRollback(SomeException.class)

  42. USEFUL FUNCTIONALITIES - RETRY •faultTolerant() •retry(SomeException.class) •retryLimit(n)

  43. CODE CODE This Photo by Unknown Author is licensed under

    CC BY-NC-ND
  44. QUESTIONS? https://github.com/rodrigolgraciano/batch-demo.git https://speakerdeck.com/graciano/batch-processing @rodrigograciano @ HillmerCh

  45. CHUNK-ORIENTED STEP https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png

  46. JOB Job JobInstance JobExecution JobParameters Present at Devnexus title =

    Batch Processing in Action Present Batch Processing in Action at Devnexus First try to present – hopefully, it will work
  47. …STEP Job JobInstance JobExecution Step StepExecution Read count Write count

    Filter count…
  48. CHUNK-ORIENTED STEP + PROCESSORS https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png