Slide 1

Slide 1 text

Simplifiez vos traitements par lots en Java avec Easy Batch Mahmoud Ben Hassine @b_e_n_a_s https://benas.github.io #EasyBatch

Slide 2

Slide 2 text

Agenda #EasyBatch • Introduction • State of the art • Motivation • Easy Batch • Overview • Examples • Wrap-up 2 @b_e_n_a_s

Slide 3

Slide 3 text

Introduction #EasyBatch 3 Batch processing Stream processing Bounded data set Unbounded data stream High latency Low latency Static data set Dynamic data stream @b_e_n_a_s

Slide 4

Slide 4 text

Introduction #EasyBatch 3 Batch processing Stream processing Bounded data set Unbounded data stream High latency Low latency Static data set Dynamic data stream @b_e_n_a_s

Slide 5

Slide 5 text

State of the art JSR 352 #EasyBatch 4 4 @b_e_n_a_s

Slide 6

Slide 6 text

State of the art Excellent solutions! But .. JSR 352 #EasyBatch 4 4 @b_e_n_a_s

Slide 7

Slide 7 text

State of the art Excellent solutions! But .. • Steep learning curve JSR 352 #EasyBatch 4 4 @b_e_n_a_s

Slide 8

Slide 8 text

State of the art Excellent solutions! But .. • Steep learning curve • Complex configuration JSR 352 #EasyBatch 4 4 @b_e_n_a_s

Slide 9

Slide 9 text

State of the art Excellent solutions! But .. • Steep learning curve • Complex configuration • Mandatory components that you may not need JSR 352 #EasyBatch 4 4 @b_e_n_a_s

Slide 10

Slide 10 text

Agenda • Introduction • State of the art • Motivation • Easy Batch • Overview • Examples • Wrap-up #EasyBatch 5 5 @b_e_n_a_s

Slide 11

Slide 11 text

Motivation #id,name,description,price,published,lastUpdate 0001,product1,description1,2500,true,2014-01-01 000x,product2,description2,2400,true,2014-01-01 0003,,description3,2300,true,2014-01-01 0004,product4,description4,-2200,true,2014-01-01 0005,product5,description5,2100,true,2024-01-01 0006,product6,description6,2000,true,2014-01-01,Blah! 0007,product7,description7,2100,true,2024-01-01 public class Product { private long id; private String name; private String description; private double price; private boolean published; private Date lastUpdate; } products.csv Common requirements: - Read file line by line - Filter header record - Parse and map data to the Product bean - Validate product data - Do something with the product (business logic) - Log errors - Report statistics Product.java #EasyBatch 6 @b_e_n_a_s

Slide 12

Slide 12 text

Motivation #id,name,description,price,published,lastUpdate 0001,product1,description1,2500,true,2014-01-01 000x,product2,description2,2400,true,2014-01-01 0003,,description3,2300,true,2014-01-01 0004,product4,description4,-2200,true,2014-01-01 0005,product5,description5,2100,true,2024-01-01 0006,product6,description6,2000,true,2014-01-01,Blah! 0007,product7,description7,2100,true,2024-01-01 public class Product { private long id; private String name; private String description; private double price; private boolean published; private Date lastUpdate; } products.csv Common requirements: - Read file line by line - Filter header record - Parse and map data to the Product bean - Validate product data - Do something with the product (business logic) - Log errors - Report statistics Boilerplate Product.java #EasyBatch 6 @b_e_n_a_s

Slide 13

Slide 13 text

Motivation #id,name,description,price,published,lastUpdate 0001,product1,description1,2500,true,2014-01-01 000x,product2,description2,2400,true,2014-01-01 0003,,description3,2300,true,2014-01-01 0004,product4,description4,-2200,true,2014-01-01 0005,product5,description5,2100,true,2024-01-01 0006,product6,description6,2000,true,2014-01-01,Blah! 0007,product7,description7,2100,true,2024-01-01 public class Product { private long id; private String name; private String description; private double price; private boolean published; private Date lastUpdate; } products.csv Common requirements: - Read file line by line - Filter header record - Parse and map data to the Product bean - Validate product data - Do something with the product (business logic) - Log errors - Report statistics The goal is to keep focus on business logic! Boilerplate Product.java #EasyBatch 6 @b_e_n_a_s

Slide 14

Slide 14 text

Agenda • Introduction • State of the art • Motivation • Easy Batch • Overview • Examples • Wrap-up #EasyBatch 7 @b_e_n_a_s

Slide 15

Slide 15 text

Easy Batch? Kesako? • Name: Easy Batch • Date of birth: 13/08/2012 • Weight: 124 Kb • DNA: https://github.com/j-easy/easy-batch #EasyBatch 8 @b_e_n_a_s

Slide 16

Slide 16 text

The Job abstraction #EasyBatch 9 public interface Job 
 extends Callable { String getName(); } class BatchJob implements Job { } @b_e_n_a_s

Slide 17

Slide 17 text

The Job abstraction #EasyBatch 9 public interface Job 
 extends Callable { String getName(); } class BatchJob implements Job { } • Synchronous execution
 JobReport report = jobExecutor.execute(job); @b_e_n_a_s

Slide 18

Slide 18 text

The Job abstraction #EasyBatch 9 public interface Job 
 extends Callable { String getName(); } class BatchJob implements Job { } • Synchronous execution
 JobReport report = jobExecutor.execute(job); • Asynchronous execution
 Future report = jobExecutor.submit(job); @b_e_n_a_s

Slide 19

Slide 19 text

The Job abstraction #EasyBatch 9 public interface Job 
 extends Callable { String getName(); } class BatchJob implements Job { } • Synchronous execution
 JobReport report = jobExecutor.execute(job); • Asynchronous execution
 Future report = jobExecutor.submit(job); • Parallel executions
 jobExecutor
 .submitAll(job1, job2); @b_e_n_a_s

Slide 20

Slide 20 text

The Job abstraction #EasyBatch 9 public interface Job 
 extends Callable { String getName(); } class BatchJob implements Job { } • Synchronous execution
 JobReport report = jobExecutor.execute(job); • Asynchronous execution
 Future report = jobExecutor.submit(job); • Parallel executions
 jobExecutor
 .submitAll(job1, job2); • Job scheduling
 scheduledExecutorService
 .schedule(job, 2, MINUTES); @b_e_n_a_s

Slide 21

Slide 21 text

The Record abstraction Header
 (No, Source, etc) Payload (Raw Data) Record Record.java public interface Record

{ /** Header of the record */ Header getHeader(); /** Payload of the record */ P getPayload(); } #EasyBatch 10 @b_e_n_a_s

Slide 22

Slide 22 text

The Record abstraction Header
 (No, Source, etc) Payload (Raw Data) Record Multiple implementations: 
 FlatFileRecord, XmlRecord, JsonRecord, JdbcRecord, JmsRecord, etc.. Record.java public interface Record

{ /** Header of the record */ Header getHeader(); /** Payload of the record */ P getPayload(); } #EasyBatch 10 @b_e_n_a_s

Slide 23

Slide 23 text

The Batch abstraction { record 1, record 2, ... record n } Batch public class Batch 
 implements Iterable { private List records; } Batch.java #EasyBatch 11 @b_e_n_a_s

Slide 24

Slide 24 text

Batch Jobs #EasyBatch 12 @b_e_n_a_s

Slide 25

Slide 25 text

Batch Jobs • Read records in sequence #EasyBatch 12 @b_e_n_a_s

Slide 26

Slide 26 text

Batch Jobs • Read records in sequence • Process records in pipeline #EasyBatch 12 @b_e_n_a_s

Slide 27

Slide 27 text

Batch Jobs • Read records in sequence • Process records in pipeline • Write records in batches #EasyBatch 12 @b_e_n_a_s

Slide 28

Slide 28 text

Agenda • Introduction • State of the art • Motivation • Easy Batch • Overview • Examples • Wrap-up #EasyBatch 13 @b_e_n_a_s

Slide 29

Slide 29 text

Agenda • Introduction • State of the art • Motivation • Easy Batch • Overview • Examples • Wrap-up #EasyBatch 14 @b_e_n_a_s

Slide 30

Slide 30 text

Wrap up • No retry on failure • No flows • No data partitioning • No remote chunking #EasyBatch 15 • Lightweight, free and open source • Easy to learn, configure and use • Flexible & extensible API • Modular architecture • Declarative data validation • Real-time monitoring The good ones The not so good ones @b_e_n_a_s

Slide 31

Slide 31 text

Final word: Be pragmatic! #EasyBatch 16 @b_e_n_a_s

Slide 32

Slide 32 text

Thank you! • Slides: http://speakerdeck.com/benas/easybatch-devoxxma-2016 • Code: https://github.com/j-easy/easy-batch #EasyBatch @b_e_n_a_s