Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Batch

Cloud Batch

Deck for the Batch Processing Cloud Apps with Java

Rodrigo Graciano

October 07, 2022
Tweet

More Decks by Rodrigo Graciano

Other Decks in Programming

Transcript

  1. BATCH
    PROCESSING
    CLOUD APPS
    WITH JAVA
    ELDER MORAES
    RODRIGO GRACIANO
    OUTUBRO 2022

    View Slide

  2. WHAT SPRING BATCH IS?
    Spring Batch is a lightweight, comprehensive batch framework designed to enable
    the development of robust batch applications vital for the daily operations of
    enterprise systems.
    https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro

    View Slide

  3. WHAT SPRING BATCH IS NOT?
    Spring Batch is not a scheduler
    https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro

    View Slide

  4. RODRIGO GRACIANO
    • Principal Software Engineer - NY
    • Líder do JUG de NY - NYJavaSIG
    • graciano.dev
    • Twitter: @rodrigograciano

    View Slide

  5. ELDER MORAES
    • Java Champion
    • Autor de diversos livros
    • Developer Advocate na Red Hat
    • Líder do SouJava
    • eldermoraes.com
    • Twitter: @elderjava

    View Slide

  6. THE BASICS

    View Slide

  7. BATCH VOCABULARY

    View Slide

  8. BATCH COMPONENTS
    JOB

    View Slide

  9. BATCH COMPONENTS
    STEP

    View Slide

  10. BATCH ARCHITECTURE
    https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png
    JobRepository
    JobLauncher Job Step
    1 *

    View Slide

  11. BATCH COMPONENTS
    TASKLET

    View Slide

  12. TASKLETS
    • 1 time operation instead of read/write
    • Run a DB query
    • Copy Files
    • Zip/unzip files

    View Slide

  13. CHUNK TASKLET - STEP SEQUENCE
    Process
    Step
    Read Write

    View Slide

  14. CHUNK-ORIENTED STEP + PROCESSORS
    https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png

    View Slide

  15. SHOW ME SOME
    CODE:
    BASIC DEMO
    This Photo by Unknown Author is licensed under CC BY-NC-ND

    View Slide

  16. • JobRepository
    • JobLauncher
    • JobRegistry
    • JobExplorer
    • PlatformTransactionManager
    • JobBuilderFactory
    • StepBuilderFactory
    @EnableBatchProcessing

    View Slide

  17. JOB REPOSITORY
    •It’s how Job, Step, JobParameters,
    JobExecution, and StepExecution are persisted

    View Slide

  18. READERS
    • Database (JDBC, Mongo, Neo4j, Hibernate, etc.)
    • Flat Files (delimited, fixed-length)
    • XML, JSON
    • JMS, Kafka, Amqp
    • Multi-file

    View Slide

  19. WRITERS
    • Database (JDBC, Mongo, Neo4j, Hibernate, etc.)
    • Flat Files (delimited, fixed-length)
    • XML, JSON
    • JMS, Kafka, Amqp
    • Multi-file
    • Email

    View Slide

  20. PROCESSORS
    •Transforming
    •Filtering
    •Validations

    View Slide

  21. LISTENERS
    JobExecutionListener StepExecutionListener ItemReadListener ItemWriteListener
    ChunkListener ItemProcessListener SkipListener

    View Slide

  22. JOB EXECUTION LISTENERS

    View Slide

  23. BEYOND THE BASICS
    SCALING AND PARALLEL
    PROCESSING

    View Slide

  24. SCALING AND PARALLEL PROCESSING
    •Parallel steps
    •Multi-threaded
    •Async Processor and AsyncWriter
    •Remote Partitioning
    •Remote Chunking

    View Slide

  25. SCALING AND PARALLEL PROCESSING
    •Parallel steps
    •Multi-threaded
    •Async Processor and AsyncWriter
    •Remote Partitioning
    •Remote Chunking

    View Slide

  26. PARALLEL PROCESSING
    Step
    1
    Step
    2
    Step
    3
    Start
    Step
    4

    View Slide

  27. SCALING AND PARALLEL PROCESSING
    •Parallel steps
    •Multi-threaded
    •Async Processor and AsyncWriter
    •Remote Partitioning
    •Remote Chunking

    View Slide

  28. SCALING AND PARALLEL PROCESSING
    •Parallel steps
    •Multi-threaded
    •Async Processor and AsyncWriter
    •Remote Partitioning
    •Remote Chunking

    View Slide

  29. SCALING AND PARALLEL PROCESSING
    •Parallel steps
    •Multi-threaded
    •Async Processor and AsyncWriter
    •Remote Partitioning
    •Remote Chunking

    View Slide

  30. View Slide

  31. WHAT
    ABOUT THE
    CLOUD?

    View Slide

  32. QUESTIONS?
    https://github.com/rodrigolgraciano/cloud-batch.git
    https://speakerdeck.com/graciano/cloud-batch
    @rodrigograciano @ elderjava

    View Slide