BeJUG_2026_Data_Batch_Processing_NiFi_vs_Spring_Batch

Transcript

1. Data Batch Processing With OpenSource Solutions: Apache NiFi vs Spring Batch. Eddie Cho 21.10.2026
2. About me Backend Engineer Open Source Contributor(Spring Integration, Spring AMQP, RabbitMQ-website) Software Communities(BeJUG, BDEN, FOSDEM, PgBE, TwJUG, JCConf, COSCUP…) 02
3. Spring Batch - Overview Job Step 1 Step 2 Step 3 03
4. Spring Batch - Overview Chunk Based Step v.s. Tasklets Step Chunk Based Step ItemReader ItemProcessor Tasklets Step ItemWriter 04
5. Spring Batch - Overview The Metadata Tables (The Schema) • BATCH_JOB_INSTANCE: Represents a unique job definition (e.g., "importUserJob"). If you run the job with the same parameters, it maps to the same instance. • BATCH_JOB_EXECUTION: A new record is inserted every single time a job is triggered. It tracks whether the job is STARTED, COMPLETED, FAILED, or STOPPED, along with start/end times and exit codes. • BATCH_JOB_EXECUTION_PARAMS: Stores the parameters passed to that specific job execution. • BATCH_STEP_EXECUTION: Tracks the status of individual steps within a job (e.g., your read-process-write step). • BATCH_STEP_EXECUTION_CONTEXT & BATCH_JOB_EXECUTION_CONTEXT: Stores the "ExecutionContext" (key-value pairs) used for state management and fault tolerance (allowing jobs to restart from where they left off if they fail). 05
6. Spring Batch - Overview Conditional Flow 06
7. Apache NiFi - Overview NiFi runs as an independent, self-contained Java application. Low-code, create/manage data pipeline through web UI. Cluster mode with Zookeeper 07
8. Apache NiFi - Overview Processor Process Group FlowFile 08
9. Apache NiFi - Overview Connections configured with Relationships 09
10. Apache NiFi - Overview 10
11. Apache NiFi - Overview Customized processor ExecuteScript processor(e.g., Groovy Script) Implement your own with Java/Python. 11
12. Apache NiFi - Overview Repository What it holds Storage Location Content Repository The actual binary payload/data. Disk FlowFile Repository Metadata, queue state, and pointers. Uses a Write-Ahead Log (WAL) for reliability. Disk (WAL) + InMemory Index Provenance Repository Historical log of every action taken on data. Disk 12
13. Fault Tolerant Retry Transient failures. Wait a moment and try the exact same operation again, it will likely succeed. (e.g., shared resource limitations) Skip Permanent failures, and skipping them have no or minor business impact compared to the cost of failing the entire job. (e.g., dirty data) 13
14. Fault Tolerant - Spring Batch - Config Retry Logic FaultTolerantStepBuilder<I,O> retry(Class<? extends Throwable> type): Explicitly ask for an exception (and subclasses) to be retried. retryLimit(int retryLimit): The maximum number of times to try a failed item. retryPolicy(org.springframework.retry.RetryPolicy retryPolicy): Provide an explicit retry policy instead of using the retryLimit(int) and retryable exceptions provided elsewhere. 14
15. Fault Tolerant - Spring Batch - Config Skip Logic FaultTolerantStepBuilder<I,O> skip(Class<? extends Throwable> type): Explicitly request certain exceptions (and subclasses) to be skipped. skipLimit(int skipLimit): Sets the maximum number of failed items to skip before the step fails. skipPolicy(SkipPolicy skipPolicy): Provide an explicit policy for managing skips. 15
16. Fault Tolerant - Apache NiFi - Retry with RetryFlowFile 16
17. Fault Tolerant - Apache NiFi - Retry with RetryFlowFile 17
18. Fault Tolerant - Apache NiFi - Config Skip Relationship 18
19. Summary Spring Batch Apache NiFi Solved Problems Data Batching Data Batching & Streaming Developer Experience Java/Kotlin, Spring Boot ecosystem Low-code visual workflow management. Java, Groovy, Python Primary Unit of Work Chunk (e.g., 1,000 records per DB commit) FlowFile Resource Usage Heap-Bound & GC-Dependent Disk-First & Streams via WAL 19
20. THANK YOU Q&A Contact me Bluesky: https://bsky.app/profile/eddie-cho.bsky.socia Linkedin: https://www.linkedin.com/in/eddie-chien-an-cho/ Email: chochoforwork@gmail.com