Mark Fisher And Dave Syer on Spring Batch and Spring Integration

Transcript

1. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. Automating Operations with Spring Batch and Spring Integration Dave Syer and Mark Fisher – SpringSource, a division of VMware
2. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 2 Spring Batch Application Batch Core Batch Infrastructure Re-usable low level stuff: flat files, XML files, database keys Quality of service, auditability, management information Business Logic
3. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 3 Item Oriented Processing Input-output can be grouped together = Item- Oriented Processing ItemReader read() ExitStatus ItemProcessor process(item) Step item execute() repeat, retry, etc. ItemWriter write(items)
4. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 4 JobLauncher JobLauncher start() JobExecution Job execute() Business ExitStatus Client With ExitStatus.FINISHED or FAILED
5. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 5 Item Oriented Processing Example ItemReader input file ItemWriter Step iterates over Items: Database item ItemProcessor
6. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 6 ItemStream ItemStream open(executionContext) save(executionContext) close() Called before commit Step JobRepository ExitStatus execute() repeat, retry, etc. update(executionContext)
7. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 7 Job Configuration and Execution Job JobInstance JobExecution * * The EndOfDay Job The EndOfDay Job for 2007/05/05 The first attempt at EndOfDay Job for 2007/05/05 JobParameters schedule.date = 2007/05/05
8. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 8 Sequential execution footballJob playerLoad gamesLoad playerSummarize
9. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 9 Sequential execution <job> <step id="playerLoad" next="gamesLoad"/> <step id="gamesLoad" next="playerSummarize"/> <step id="playerSummarize" /> </job> <job> <step id="playerLoad" next="gamesLoad"/> <step id="gamesLoad" next="playerSummarize"/> <step id="playerSummarize" /> </job>
10. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 10 Spring Batch Roadmap • 2.1 released 2010Q1 • Spring 3.0; configuration convenience • In use at 100+ projects with scores in production • 3.0 planned 2011Q1 • Distributed JobRepository
11. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 11 Demo Spring Batch
12. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 12 What is Spring Integration? • At the core, an embedded Message Bus – Runs within any Spring ApplicationContext – All components are Spring-managed objects • Also, an Application Integration Framework – Connects to other systems via adapters • Unidirectional Channel Adapters • Bidirectional Messaging Gateways
13. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 13 The Big Picture
14. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 14 Message • Payload can be any object • Header values are stored in a Map Headers Payload
15. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 15 Message and Headers public interface Message<T> { MessageHeaders getHeaders(); T getPayload(); } public interface Message<T> { MessageHeaders getHeaders(); T getPayload(); } MessageHeaders headers = message.getHeaders(); String value = headers.get(“key”, String.class); Object id = headers.getId(); long timestamp = headers.getTimestamp(); MessagePriority priority = headers.getPriority(); MessageHeaders headers = message.getHeaders(); String value = headers.get(“key”, String.class); Object id = headers.getId(); long timestamp = headers.getTimestamp(); MessagePriority priority = headers.getPriority();
16. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 16 Message Channel • Decouples Producers from Consumers • Provides extension point for interceptors • May be Point-to-Point • Or Publish/Subscribe
17. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 17 Message Channel Types <channel id="sync-p2p"/> <channel id="async-p2p"><queue capacity="50"/></channel> <publish-subscribe-channel id="pubsub"/> <channel id="priorityChannel"> <priority-queue comparator="someComparator"/> </channel> <channel id="rendezvousChannel"> <rendezvous-queue/> </channel>
18. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 18 JMS-backed Message Channels • Drop-in replacement for <channel/> elements • Enables full power of underlying JMS provider – Persistence, transactions, etc. • Point-to-Point delegates to a JMS Queue • Publish/Subscribe delegates to a Topic <jms:channel id="p2p" queue-name="someQueueName"/> <jms:publish-subscribe-channel id="pubsub" topic="someTopicRef"/>
19. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 19 Message Endpoint • Producers send Messages to a Message Channel • Depending on their type, Message Channels may have Polling Consumers • Or Event-Driven Consumers
20. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 20 Service Activator • Similar in concept to Spring's JMS support – delegate to any Spring-managed object – the framework handles message reception and method invocation • Clean separation of Code and Configuration <service-activator input-channel="requests" ref="loanBroker" method="processRequest" output-channel="quotes"/>
21. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 21 Other Message Endpoints • Transformer – Convert payload or modify headers • Filter – Discard messages based on boolean evaluation • Router – Determine next channel based on content • Splitter – Generate multiple messages from one • Aggregator – Assemble a single message from multiple
22. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 22 Expression Language Support • Alternative option for ref/method in endpoints <filter input-channel="customers" expression="payload.vip" output-channel="vips" discard-channel="nonVips"/> • Mapping between Messages and Methods public void createAccount( @Payload("firstName") String firstName, @Payload("lastName") String lastName, @Header("account.id") String accountId) {...}
23. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 23 Channel Adapters and Messaging Gateways • File • JMS • HTTP (REST) • Web Services (SOAP or “plain” XML) • Mail (POP3/IMAP and SMTP) • RMI • TCP/UDP • JMX • Spring ApplicationEvents
24. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 24 Demo Spring Integration
25. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 25 Combining Batch and Integration • Adding messaging to a batch process enables automation of operations – Message triggers Jobs – Job completion or failure triggers Message • Messaging can also be embedded within a Job – Reading or writing items via channels • The line is not always clear, but there are guidelines – Think about granularity – Apply common patterns
26. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 26 Pattern: Message Trigger • Event in normal message pipeline triggers batch job • Example: File Polling
27. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 27 Pattern: Message Driven Job Configuration • Job configuration sometimes needs to be updated • Example: change the timeout for a remote process JobRegistry Job configuration
28. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 28 Pattern: Embedded Messaging • Batch job implemented as message- oriented process Item or Chunk
29. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 29 Pattern: Asynchronous Item Processing Gateway
30. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 30 Pattern: Monitor Channel • Informational messages about status of system • Can be adapter to multiple outbound endpoints – e.g. E-mail, SMS, XMPP, etc. XMPP
31. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 31 Pattern: Channel Reuse • More than one source for the same event • Efficient because of modularity • Example: File Polling and inbound HTTP Transformation might be necessary
32. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 32 Example: Job Channels job-launches job-restarts Job Launcher Message Handler job-executions Message-Publishing Interceptor job-operator
33. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 33 Integration: Adapters job-launches job-restarts lead-files HTTP /job-requests HTTP /job-restarts HTTP /leads/files File /target/data polling
34. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 34 Demo Spring Batch Admin
35. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 35 Application Context Architecture Parent Extensions Framework Batch Services Integration Batch Admin UI Child Jobs
36. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 36 Application Context Locations META-INF/batch/*.xml META-INF/servlet/*.xml META-INF/servlet/*.xml META-INF/bootstrap/*.xml META-INF/spring/*.xml META-INF/servlet/*.xml
37. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 37 Spring Batch Admin Roadmap • 1.0 planned 2010Q2 • Web UI, monitor and manage batch executions • Upload input files and configuration files • Add custom jobs • STS template project • Extensible UI • Later… • Persistent work queues • Prioritisation • Roo support + Batch shell
38. Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited. 38 Q&A