Integration Nirvana

presented alongside with Nilanjan Raychaudhuri (@nraychaudhuri) @ COJUG (http://www.cojug.org/), July 2011, Dublin, OH

Transcript

1. Integration Nirvana with Camel
2. Who are we? Boring...
3. Integration Challenges...
5. Networks are unreliable
6. Networks are slow
7. Any two applications are different
8. Change is inevitable
9. Lets face it Integration is HARD
10. So why Integrate?
11. Interesting applications rarely live in isolation Gregor Hohpe
12. Where does Camel fit?
13. Enterprise integration patterns + DSL = Camel
14. Choose integration pattern
17. Choose your DSL
18. Choose your component Whats included in the box? ! 70+ Components activemq crypto flatpack irc ldap activemq-journal cxf freemarker javaspace mail/imap/pop3 amqp cxfrs ftp/ftps/sftp jbi mina atom dataset gae jcr mock bean direct hdfs jdbc msv bean validation esper hibernate jetty nagios browse event hl7 jms netty cache exec http jpa nmr cometd file ibatis jt/400 printer
19. Choose your component hats included in the box? 70+ Components properties scalate stream xslt quartz seda string-template ejb quickfix servlet test ref smooks timer restlet smpp validation rmi snmp velocity rnc spring-integration vm rng spring-security xmpp rss sql xquery
20. Choose Data format 18 Data Formats bindy protobuf castor serialization csv soap crypto tidy markup flatpack xml beans gzip xml security hl7 xstream jaxb zip json dozer
21. Nirvana
23. An example would be handy right about now
24. car buying process Email order Process order new car used car Billing
25. Part 1 - Handling mail orders
26. jms: Incoming Orders Email Consumer Order Processor Message Gmail
27. Part 2 - Routing mail orders
28. jms:topic: newCarOrders Message jms:topic: usedCarOrders jms: Incoming Orders Email Consumer Gmail Order Processor Polling Consumer Content Based Router Message
29. Part 3 - Processing orders
30. jms: Incoming Orders Email Consumer Gmail Order Processor Polling Consumer 'new' jms:topic: newCarOrders jms:topic: usedCarOrders Content Based Router Event-Driven Consumer jms:queue: usedCarOrder jms:queue: newCarOrder Durable Subscriber Event-Driven Consumer Topic Consumer ('new') Topic Consumer ('used') Queue Consumer UsedCarOrder Processor Queue Consumer NewCarOrder Processor Event-Driven Consumer Integration Nirvana Flow Subject?
31. ... demo
32. Part 4 - Send to Billing
33. jms: Incoming Orders Email Consumer Gmail Order Processor Polling Consumer 'new' jms:topic: newCarOrders jms:topic: usedCarOrders Content Based Router Event-Driven Consumer jms:queue: usedCarOrder jms:queue: newCarOrder Durable Subscriber Event-Driven Consumer Topic Consumer ('new') Topic Consumer ('used') Queue Consumer UsedCarOrd er Processor Queue Consumer NewCarOrde r Processor Event-Driven Consumer Integration Nirvana Flow jms:queue: billing Queue Consumer Billing Processor Subject?
34. ... demo w/ Billing
35. Oh no...change
37. You might win a car integrationnirvana@gmail.com subject : “winner” Amazon Gift Card
38. Part 4 -Add the winner
39. Complete process jms:queue: winners jms: Incoming Orders Email Consumer Gmail Order Processor Polling Consumer 'new' jms:topic: newCarOrders jms:topic: usedCarOrders Content Based Router Event-Driven Consumer jms:queue: usedCarOrder jms:queue: newCarOrder Durable Subscriber Event-Driven Consumer Topic Consumer ('new') Topic Consumer ('used') Queue Consumer UsedCarOrd er Processor Queue Consumer NewCarOrde r Processor Event-Driven Consumer Integration Nirvana Flow Queue Consumer Billing Processor jms:topic: winner Topic Consumer ('winner') Queue Consumer Winning Processor Subject? jms:queue: winners
40. ... Winner Announced Later!
42. High Level View of Camel Architecture
43. Message Sender Receiver Message
44. CamelContext view out of the way, let’s take a closer look at the individual ny new concepts, so let’s take some time to go over them one by e CamelContext, which is Camel’s runtime. hat the Camel- of sorts, judging n think of it as m, which keeps e most notable Context keeps igure 1.7, there or the Camel- of. These are Components CamelContext Endpoints Routes Type converters Registry Data formats Languages Figure 1.7 The CamelContext provides access to many useful services, the most notable being components, type converters, a registry, endpoints, routes, data formats, and languages.
45. Routing Content Based Router in out out Message Message
46. Endpoint discuss this in chapter 11. From a programming point of view, components are fairly simple: ated with a name that’s used in a URI, and they act as a factory of endpoi ple, a FileComponent is referred to by file in a URI, and it creates F The endpoint is perhaps an even more fundamental concept in Camel. ENDPOINT An endpoint is the Camel abstraction that models the end of a cha which a system can send or receive messages. This is illustrated in figure ` Data Message endpoint Sender application ` Receiver application Message endpoint Data Message Channel Figure 1 An endpo a neutra allowing to integr Licensed to DJ Daugherty <ddaugher@gmail.com> example: jms:incomingOrders
47. When to use Camel?
48. Every integration uses same concepts
49. Testing is really easy
51. Event Driven Architecture
52. Rube Goldberg Machine
53. Why we prefer messaging?
54. Decoupled from("jms:incomingOrders") .choice() .when(header("Subject").contains("winner")) .to("jms:topic:winner") .when(header("Subject").contains("new")) .to("jms:topic:newCarOrders") .otherwise() .to("jms:topic:usedCarOrders");
55. Scalable from("file:target/inventory?noop=true") .log("Starting to process big file: ${header.CamelFileName}") .split(body().tokenize("\n")).streaming().parallelProcessing() .bean(InventoryService.class, "csvToObject") .to("direct:update") .end() .log("Done processing big file: ${header.CamelFileName}");
56. Fault tolerant errorHandler(defaultErrorHandler() .maximumRedeliveries(2) .redeliveryDelay(1000) .retryAttemptedLogLevel(LoggingLevel.WARN));
57. Changeable from("file://target/rider/orders") .wireTap("seda:audit") .bean(OrderCsvToXmlBean.class) .to("jms:queue:orders"); from("seda:audit") .bean(AuditService.class, "auditFile");
59. Starbucks does not use two-phase commit
60. Event sourcing
61. Real world example
62. Clean Infected Dasient (1000 req/sec) Infected? Submit Domain for Scan DB High Level Process New DB schedule for clean in 60 days schedule for clean tomorrow
63. Clean Infected Dasient (1000 req/sec) Infected? Submit Domain for Scan DB New DB www.me.com -> index.html (1 sec.) www.they.com -> 700 files (45 sec.) schedule for clean in 60 days schedule for clean tomorrow blocking call Event Driven Process Legacy Process
64. Clean Infected Dasient (1000 req/sec) Infected? Submit Domain for Scan DB New DB www.me.com -> index.html (1 sec.) www.they.com -> 700 files (45 sec.) schedule for clean in 60 days schedule for clean tomorrow Event Driven Process immediate response w/ token Response Servlet token exists? reply queue scan queue Processor
65. *note : ack msg's are not consumed by the MDB, Dasient Interface Servlet consumes msg's after result is returned dasient interface servlet scan domain queue dasient result queue scan domain augmented queue dasient ack queue + requestId + statusURL + messageType 'put' msg to queue, with correlation id = request id Dasient Result Consumer DasientDB store scanHistory process scan queue PostScan Operations PostScan Consumer DasientDB store all msg's 'get' by consumer DasientDB (clean message) cleaning agent pull available work cleaning script cleaning result servlet send results update clean request domain cleaned queue PostClean Operations PostClean Consumer DasientDB store end start persist 'IGNORED' ScanHistory entity DasientDB persist 'ACK' ScanHistory entity DasientDB persist 'COMPLETED' ScanHistory entity Dasient Integration Flow timeToLive = 5 mins dasient ack expiry queue DasientAckExpiry Consumer 'reaper' process consumes expired messages Email update scanHistory status = 'EXPIRED' + messageType + domainName + controlPanel + startTime Message + messageType + domainName + controlPanel + startTime Dasient Message + messagetype + domain name + control panel + start time + webserver + request type (top-level-scan) + host ip + requestId + jsonPayload + infectionId Message + messagetype + domain name + control panel + start time + webserver + request type (top-level-scan) + host ip + requestId + jsonPayload + infectionId Processor Dasient Augmented Consumer Ack? 'get' by msg selector = request id all msg's 'get' by consumer create Process Scan Msg ignored msg 'put' msg to queue, with correlation id = request id yes PreScan Operations ScanDomain Consumer domain found? DasientDB mark domain 'deleted' Message + messagetype + domain name + control panel + start time + webserver + request type (top-level-scan) + host ip + requestId + jsonPayload + infectionId result from dasient includes: +request id + completed At + status + hostname + on blacklist + scanned urls + malicious urls + snippets Message + messagetype + domain name + control panel + start time + webserver + request type (top-level-scan) + host ip + requestId + jsonPayload + messagetype + domain name + control panel + start time + request type (top-level-scan) + host ip + webserver
66. ... TDD!
67. ... and the Winner is!
68. Key Message(s) 1. Real world is very asynchronous 2. EDA at times is the right way
69. Camel Books
70. Get the Code https://github.com/ddaugher/integrationNirvana
71. Thanks DJ Daugherty @ddaugher Nilanjan Raychaudhuri @nraychaudhuri