Batch Processing

Deck from the Batch Processing in Action presentation

Transcript

1. BATCH
   PROCESSING
   IN ACTION
   HILLMER CHONA
   RODRIGO GRACIANO
   

   View Slide

2. WHAT SPRING BATCH IS?
   Spring Batch is a lightweight, comprehensive batch framework designed to enable
   the development of robust batch applications vital for the daily operations of
   enterprise systems.
   https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
   

   View Slide

3. WHAT SPRING BATCH IS NOT?
   Spring Batch is not a scheduler
   https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
   

   View Slide

4. BATCH VOCABULARY
   

   View Slide

5. THE BASICS
   

   View Slide

6. SHOW ME SOME
   CODE:
   BASIC DEMO
   This Photo by Unknown Author is licensed under CC BY-NC-ND
   

   View Slide

7. JOB STEP REPOSITORY RUNNING
   

   View Slide

8. BATCH ARCHITECTURE (SIMPLIFIED
   )
   https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png
   JobRepository
   JobLauncher Job Step
   Reader
   Writer
   1 *
   1
   1
   1
   1
   

   View Slide

9. JOB REPOSITORY
   •It’s how Job, Step, JobParameters,
   JobExecution, and StepExecution are persisted
   

   View Slide

10. • JobRepository
    • JobLauncher
    • JobRegistry
    • JobExplorer
    • PlatformTransactionManager
    • JobBuilderFactory
    • StepBuilderFactory
    @EnableBatchProcessing
    

    View Slide

11. RUNNING JOBS
    • From the command Line (CommandLineJobRunner)
    • Within a web container
    

    View Slide

12. SHOW ME SOME
    CODE:
    RUNNING JOBS
    This Photo by Unknown Author is licensed under CC BY-NC-ND
    

    View Slide

13. RUNNING JOBS
    • From the command Line (CommandLineJobRunner)
    • Within a web container
    • Synchronous x Asynchronous
    

    View Slide

14. PROCESSORS CHUNKS
    WE CAN DO BETTER
    

    View Slide

15. STEP SEQUENCE
    Process
    Step
    Read Write
    

    View Slide

16. BATCH ARCHITECTURE
    https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png
    JobRepository
    JobLauncher Job Step
    Reader
    Writer
    1 *
    1
    1
    1
    1
    Processor
    1 1
    

    View Slide

17. PROCESSORS
    •Transforming
    •Filtering
    •Validations
    

    View Slide

18. STEP SEQUENCE
    Process
    Step
    Read Write
    

    View Slide

19. CHUNK SEQUENCE
    Process
    Step
    Read Write
    Read Process
    

    View Slide

20. MORE CODE MORE
    CODE:
    PROCESSORS
    & CHUNKS
    This Photo by Unknown Author is licensed under CC BY-NC-ND
    

    View Slide

21. +READERS +WRITERS TASKLETS
    

    View Slide

22. READERS
    • Database (JDBC, Mongo, Neo4j, Hibernate, etc.)
    • Flat Files (delimited, fixed-length)
    • XML, JSON
    • JMS, Kafka, Amqp
    • Multi-file
    

    View Slide

23. MULTI-FILE READER
    •Read multiple file in a single step
    •Files must have the same format
    •Works with XML, JSON and flat files
    

    View Slide

24. SHOW ME SOME
    CODE:
    MULTI-READERS
    This Photo by Unknown Author is licensed under CC BY-NC-ND
    

    View Slide

25. WRITERS
    • Database (JDBC, Mongo, Neo4j, Hibernate, etc.)
    • Flat Files (delimited, fixed-length)
    • XML, JSON
    • JMS, Kafka, Amqp
    • Multi-file
    • Email
    

    View Slide

26. TASKLETS
    • 1 time operation instead of read/write
    • Run a DB query
    • Copy Files
    • Zip/unzip files
    

    View Slide

27. BEYOND THE BASICS
    LISTENERS STEP FLOW SCALING AND
    PARALLEL PROCESSING
    

    View Slide

28. LISTENERS
    JobExecutionListener StepExecutionListener ItemReadListener ItemWriteListener
    ChunkListener ItemProcessListener SkipListener
    

    View Slide

29. JOB EXECUTION LISTENERS
    

    View Slide

30. STEP LISTENERS
    • before/after step
    StepExecutionListener
    • before/after read
    • onReadError
    ItemReadListener
    • before/after write
    • onWriteError
    ItemWriteListener
    

    View Slide

31. STEP LISTENERS
    • before/after/after error chunk
    ChunkListener
    • before/after process
    • onProcessError
    ItemProcessListener
    • onSkipInRead
    • onSkipInProcess
    • onSkipInWrite
    SkipListener
    

    View Slide

32. STEP FLOW
    • Sequential
    

    View Slide

33. STEP FLOW: SEQUENTIAL
    Step 1 Step 2 Step 3
    

    View Slide

34. STEP FLOW
    • Sequential
    • Conditional (ExitStatus)
    • Decider
    

    View Slide

35. STEP FLOW: CONDITIONAL & DECIDER
    Condition
    Step
    1
    Step
    3
    Step
    2
    

    View Slide

36. STEP FLOW
    • Sequential
    • Conditional (ExitStatus)
    • Decider
    • Split
    

    View Slide

37. STEP FLOW: SPLIT
    Step
    1
    Step
    2
    Step
    3
    Start
    Step
    4
    

    View Slide

38. SCALING AND PARALLEL PROCESSING
    • Multi-threaded (Async Executor)
    • Parallel steps (Split)
    • Remote Chunking (Data processed by slaves – Real
    data sent not a reference)
    • Remote Partitioning (Data in chunks and local to slaves)
    

    View Slide

39. MORE FUNCTIONALITIES
    RESTARTING SKIPING FAILURES RETRY
    

    View Slide

40. USEFUL FUNCTIONALITIES - START
    •startLimit(n)
    •allowStartIfComplete(boolean)
    

    View Slide

41. USEFUL FUNCTIONALITIES – SKIPPING FAILURES
    •faultTolerant()
    •skip(SomeException.class)
    •skipLimit(n)
    •noSkip(SomeException.class)
    •noRollback(SomeException.class)
    

    View Slide

42. USEFUL FUNCTIONALITIES - RETRY
    •faultTolerant()
    •retry(SomeException.class)
    •retryLimit(n)
    

    View Slide

43. CODE CODE
    This Photo by Unknown Author is licensed under CC BY-NC-ND
    

    View Slide

44. QUESTIONS?
    https://github.com/rodrigolgraciano/batch-demo.git
    https://speakerdeck.com/graciano/batch-processing
    @rodrigograciano @ HillmerCh
    

    View Slide

45. CHUNK-ORIENTED STEP
    https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png
    

    View Slide

46. JOB
    Job JobInstance JobExecution
    JobParameters
    Present at Devnexus
    title = Batch Processing in Action
    Present Batch Processing
    in Action at Devnexus
    First try to present –
    hopefully, it will work
    

    View Slide

47. …STEP
    Job JobInstance JobExecution
    Step StepExecution
    Read count
    Write count
    Filter count…
    

    View Slide

48. CHUNK-ORIENTED STEP + PROCESSORS
    https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png
    

    View Slide