Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Batch Processing

Batch Processing

Deck from the Batch Processing in Action presentation

Rodrigo Graciano

April 11, 2022
Tweet

More Decks by Rodrigo Graciano

Other Decks in Programming

Transcript

  1. BATCH
    PROCESSING
    IN ACTION
    HILLMER CHONA
    RODRIGO GRACIANO

    View full-size slide

  2. WHAT SPRING BATCH IS?
    Spring Batch is a lightweight, comprehensive batch framework designed to enable
    the development of robust batch applications vital for the daily operations of
    enterprise systems.
    https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro

    View full-size slide

  3. WHAT SPRING BATCH IS NOT?
    Spring Batch is not a scheduler
    https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro

    View full-size slide

  4. BATCH VOCABULARY

    View full-size slide

  5. SHOW ME SOME
    CODE:
    BASIC DEMO
    This Photo by Unknown Author is licensed under CC BY-NC-ND

    View full-size slide

  6. JOB STEP REPOSITORY RUNNING

    View full-size slide

  7. BATCH ARCHITECTURE (SIMPLIFIED
    )
    https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png
    JobRepository
    JobLauncher Job Step
    Reader
    Writer
    1 *
    1
    1
    1
    1

    View full-size slide

  8. JOB REPOSITORY
    •It’s how Job, Step, JobParameters,
    JobExecution, and StepExecution are persisted

    View full-size slide

  9. • JobRepository
    • JobLauncher
    • JobRegistry
    • JobExplorer
    • PlatformTransactionManager
    • JobBuilderFactory
    • StepBuilderFactory
    @EnableBatchProcessing

    View full-size slide

  10. RUNNING JOBS
    • From the command Line (CommandLineJobRunner)
    • Within a web container

    View full-size slide

  11. SHOW ME SOME
    CODE:
    RUNNING JOBS
    This Photo by Unknown Author is licensed under CC BY-NC-ND

    View full-size slide

  12. RUNNING JOBS
    • From the command Line (CommandLineJobRunner)
    • Within a web container
    • Synchronous x Asynchronous

    View full-size slide

  13. PROCESSORS CHUNKS
    WE CAN DO BETTER

    View full-size slide

  14. STEP SEQUENCE
    Process
    Step
    Read Write

    View full-size slide

  15. BATCH ARCHITECTURE
    https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png
    JobRepository
    JobLauncher Job Step
    Reader
    Writer
    1 *
    1
    1
    1
    1
    Processor
    1 1

    View full-size slide

  16. PROCESSORS
    •Transforming
    •Filtering
    •Validations

    View full-size slide

  17. STEP SEQUENCE
    Process
    Step
    Read Write

    View full-size slide

  18. CHUNK SEQUENCE
    Process
    Step
    Read Write
    Read Process

    View full-size slide

  19. MORE CODE MORE
    CODE:
    PROCESSORS
    & CHUNKS
    This Photo by Unknown Author is licensed under CC BY-NC-ND

    View full-size slide

  20. +READERS +WRITERS TASKLETS

    View full-size slide

  21. READERS
    • Database (JDBC, Mongo, Neo4j, Hibernate, etc.)
    • Flat Files (delimited, fixed-length)
    • XML, JSON
    • JMS, Kafka, Amqp
    • Multi-file

    View full-size slide

  22. MULTI-FILE READER
    •Read multiple file in a single step
    •Files must have the same format
    •Works with XML, JSON and flat files

    View full-size slide

  23. SHOW ME SOME
    CODE:
    MULTI-READERS
    This Photo by Unknown Author is licensed under CC BY-NC-ND

    View full-size slide

  24. WRITERS
    • Database (JDBC, Mongo, Neo4j, Hibernate, etc.)
    • Flat Files (delimited, fixed-length)
    • XML, JSON
    • JMS, Kafka, Amqp
    • Multi-file
    • Email

    View full-size slide

  25. TASKLETS
    • 1 time operation instead of read/write
    • Run a DB query
    • Copy Files
    • Zip/unzip files

    View full-size slide

  26. BEYOND THE BASICS
    LISTENERS STEP FLOW SCALING AND
    PARALLEL PROCESSING

    View full-size slide

  27. LISTENERS
    JobExecutionListener StepExecutionListener ItemReadListener ItemWriteListener
    ChunkListener ItemProcessListener SkipListener

    View full-size slide

  28. JOB EXECUTION LISTENERS

    View full-size slide

  29. STEP LISTENERS
    • before/after step
    StepExecutionListener
    • before/after read
    • onReadError
    ItemReadListener
    • before/after write
    • onWriteError
    ItemWriteListener

    View full-size slide

  30. STEP LISTENERS
    • before/after/after error chunk
    ChunkListener
    • before/after process
    • onProcessError
    ItemProcessListener
    • onSkipInRead
    • onSkipInProcess
    • onSkipInWrite
    SkipListener

    View full-size slide

  31. STEP FLOW
    • Sequential

    View full-size slide

  32. STEP FLOW: SEQUENTIAL
    Step 1 Step 2 Step 3

    View full-size slide

  33. STEP FLOW
    • Sequential
    • Conditional (ExitStatus)
    • Decider

    View full-size slide

  34. STEP FLOW: CONDITIONAL & DECIDER
    Condition
    Step
    1
    Step
    3
    Step
    2

    View full-size slide

  35. STEP FLOW
    • Sequential
    • Conditional (ExitStatus)
    • Decider
    • Split

    View full-size slide

  36. STEP FLOW: SPLIT
    Step
    1
    Step
    2
    Step
    3
    Start
    Step
    4

    View full-size slide

  37. SCALING AND PARALLEL PROCESSING
    • Multi-threaded (Async Executor)
    • Parallel steps (Split)
    • Remote Chunking (Data processed by slaves – Real
    data sent not a reference)
    • Remote Partitioning (Data in chunks and local to slaves)

    View full-size slide

  38. MORE FUNCTIONALITIES
    RESTARTING SKIPING FAILURES RETRY

    View full-size slide

  39. USEFUL FUNCTIONALITIES - START
    •startLimit(n)
    •allowStartIfComplete(boolean)

    View full-size slide

  40. USEFUL FUNCTIONALITIES – SKIPPING FAILURES
    •faultTolerant()
    •skip(SomeException.class)
    •skipLimit(n)
    •noSkip(SomeException.class)
    •noRollback(SomeException.class)

    View full-size slide

  41. USEFUL FUNCTIONALITIES - RETRY
    •faultTolerant()
    •retry(SomeException.class)
    •retryLimit(n)

    View full-size slide

  42. CODE CODE
    This Photo by Unknown Author is licensed under CC BY-NC-ND

    View full-size slide

  43. QUESTIONS?
    https://github.com/rodrigolgraciano/batch-demo.git
    https://speakerdeck.com/graciano/batch-processing
    @rodrigograciano @ HillmerCh

    View full-size slide

  44. CHUNK-ORIENTED STEP
    https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png

    View full-size slide

  45. JOB
    Job JobInstance JobExecution
    JobParameters
    Present at Devnexus
    title = Batch Processing in Action
    Present Batch Processing
    in Action at Devnexus
    First try to present –
    hopefully, it will work

    View full-size slide

  46. …STEP
    Job JobInstance JobExecution
    Step StepExecution
    Read count
    Write count
    Filter count…

    View full-size slide

  47. CHUNK-ORIENTED STEP + PROCESSORS
    https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png

    View full-size slide