Tweet
Tweet

Slide 1

Slide 1 text

Integration Nirvana with Camel

Slide 2

Slide 2 text

Who are we? Boring...

Slide 3

Slide 3 text

Integration Challenges...

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Networks are unreliable

Slide 6

Slide 6 text

Networks are slow

Slide 7

Slide 7 text

Any two applications are different

Slide 8

Slide 8 text

Change is inevitable

Slide 9

Slide 9 text

Lets face it Integration is HARD

Slide 10

Slide 10 text

So why Integrate?

Slide 11

Slide 11 text

Interesting applications rarely live in isolation Gregor Hohpe

Slide 12

Slide 12 text

Where does Camel fit?

Slide 13

Slide 13 text

Enterprise integration patterns + DSL = Camel

Slide 14

Slide 14 text

Choose integration pattern

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Choose your DSL

Slide 18

Slide 18 text

Choose your component Whats included in the box? ! 70+ Components activemq crypto flatpack irc ldap activemq-journal cxf freemarker javaspace mail/imap/pop3 amqp cxfrs ftp/ftps/sftp jbi mina atom dataset gae jcr mock bean direct hdfs jdbc msv bean validation esper hibernate jetty nagios browse event hl7 jms netty cache exec http jpa nmr cometd file ibatis jt/400 printer

Slide 19

Slide 19 text

Choose your component hats included in the box? 70+ Components properties scalate stream xslt quartz seda string-template ejb quickfix servlet test ref smooks timer restlet smpp validation rmi snmp velocity rnc spring-integration vm rng spring-security xmpp rss sql xquery

Slide 20

Slide 20 text

Choose Data format 18 Data Formats bindy protobuf castor serialization csv soap crypto tidy markup flatpack xml beans gzip xml security hl7 xstream jaxb zip json dozer

Slide 21

Slide 21 text

Nirvana

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

An example would be handy right about now

Slide 24

Slide 24 text

car buying process Email order Process order new car used car Billing

Slide 25

Slide 25 text

Part 1 - Handling mail orders

Slide 26

Slide 26 text

jms: Incoming Orders Email Consumer Order Processor Message Gmail

Slide 27

Slide 27 text

Part 2 - Routing mail orders

Slide 28

Slide 28 text

jms:topic: newCarOrders Message jms:topic: usedCarOrders jms: Incoming Orders Email Consumer Gmail Order Processor Polling Consumer Content Based Router Message

Slide 29

Slide 29 text

Part 3 - Processing orders

Slide 30

Slide 30 text

jms: Incoming Orders Email Consumer Gmail Order Processor Polling Consumer 'new' jms:topic: newCarOrders jms:topic: usedCarOrders Content Based Router Event-Driven Consumer jms:queue: usedCarOrder jms:queue: newCarOrder Durable Subscriber Event-Driven Consumer Topic Consumer ('new') Topic Consumer ('used') Queue Consumer UsedCarOrder Processor Queue Consumer NewCarOrder Processor Event-Driven Consumer Integration Nirvana Flow Subject?

Slide 31

Slide 31 text

... demo

Slide 32

Slide 32 text

Part 4 - Send to Billing

Slide 33

Slide 33 text

jms: Incoming Orders Email Consumer Gmail Order Processor Polling Consumer 'new' jms:topic: newCarOrders jms:topic: usedCarOrders Content Based Router Event-Driven Consumer jms:queue: usedCarOrder jms:queue: newCarOrder Durable Subscriber Event-Driven Consumer Topic Consumer ('new') Topic Consumer ('used') Queue Consumer UsedCarOrd er Processor Queue Consumer NewCarOrde r Processor Event-Driven Consumer Integration Nirvana Flow jms:queue: billing Queue Consumer Billing Processor Subject?

Slide 34

Slide 34 text

... demo w/ Billing

Slide 35

Slide 35 text

Oh no...change

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

You might win a car [email protected] subject : “winner” Amazon Gift Card

Slide 38

Slide 38 text

Part 4 -Add the winner

Slide 39

Slide 39 text

Complete process jms:queue: winners jms: Incoming Orders Email Consumer Gmail Order Processor Polling Consumer 'new' jms:topic: newCarOrders jms:topic: usedCarOrders Content Based Router Event-Driven Consumer jms:queue: usedCarOrder jms:queue: newCarOrder Durable Subscriber Event-Driven Consumer Topic Consumer ('new') Topic Consumer ('used') Queue Consumer UsedCarOrd er Processor Queue Consumer NewCarOrde r Processor Event-Driven Consumer Integration Nirvana Flow Queue Consumer Billing Processor jms:topic: winner Topic Consumer ('winner') Queue Consumer Winning Processor Subject? jms:queue: winners

Slide 40

Slide 40 text

... Winner Announced Later!

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

High Level View of Camel Architecture

Slide 43

Slide 43 text

Message Sender Receiver Message

Slide 44

Slide 44 text

CamelContext view out of the way, let’s take a closer look at the individual ny new concepts, so let’s take some time to go over them one by e CamelContext, which is Camel’s runtime. hat the Camel- of sorts, judging n think of it as m, which keeps e most notable Context keeps igure 1.7, there or the Camel- of. These are Components CamelContext Endpoints Routes Type converters Registry Data formats Languages Figure 1.7 The CamelContext provides access to many useful services, the most notable being components, type converters, a registry, endpoints, routes, data formats, and languages.

Slide 45

Slide 45 text

Routing Content Based Router in out out Message Message

Slide 46

Slide 46 text

Endpoint discuss this in chapter 11. From a programming point of view, components are fairly simple: ated with a name that’s used in a URI, and they act as a factory of endpoi ple, a FileComponent is referred to by file in a URI, and it creates F The endpoint is perhaps an even more fundamental concept in Camel. ENDPOINT An endpoint is the Camel abstraction that models the end of a cha which a system can send or receive messages. This is illustrated in figure ` Data Message endpoint Sender application ` Receiver application Message endpoint Data Message Channel Figure 1 An endpo a neutra allowing to integr Licensed to DJ Daugherty example: jms:incomingOrders

Slide 47

Slide 47 text

When to use Camel?

Slide 48

Slide 48 text

Every integration uses same concepts

Slide 49

Slide 49 text

Testing is really easy

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

Event Driven Architecture

Slide 52

Slide 52 text

Rube Goldberg Machine

Slide 53

Slide 53 text

Why we prefer messaging?

Slide 54

Slide 54 text

Decoupled from("jms:incomingOrders") .choice() .when(header("Subject").contains("winner")) .to("jms:topic:winner") .when(header("Subject").contains("new")) .to("jms:topic:newCarOrders") .otherwise() .to("jms:topic:usedCarOrders");

Slide 55

Slide 55 text

Scalable from("file:target/inventory?noop=true") .log("Starting to process big file: ${header.CamelFileName}") .split(body().tokenize("\n")).streaming().parallelProcessing() .bean(InventoryService.class, "csvToObject") .to("direct:update") .end() .log("Done processing big file: ${header.CamelFileName}");

Slide 56

Slide 56 text

Fault tolerant errorHandler(defaultErrorHandler() .maximumRedeliveries(2) .redeliveryDelay(1000) .retryAttemptedLogLevel(LoggingLevel.WARN));

Slide 57

Slide 57 text

Changeable from("file://target/rider/orders") .wireTap("seda:audit") .bean(OrderCsvToXmlBean.class) .to("jms:queue:orders"); from("seda:audit") .bean(AuditService.class, "auditFile");

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

Starbucks does not use two-phase commit

Slide 60

Slide 60 text

Event sourcing

Slide 61

Slide 61 text

Real world example

Slide 62

Slide 62 text

Clean Infected Dasient (1000 req/sec) Infected? Submit Domain for Scan DB High Level Process New DB schedule for clean in 60 days schedule for clean tomorrow

Slide 63

Slide 63 text

Clean Infected Dasient (1000 req/sec) Infected? Submit Domain for Scan DB New DB www.me.com -> index.html (1 sec.) www.they.com -> 700 files (45 sec.) schedule for clean in 60 days schedule for clean tomorrow blocking call Event Driven Process Legacy Process

Slide 64

Slide 64 text

Clean Infected Dasient (1000 req/sec) Infected? Submit Domain for Scan DB New DB www.me.com -> index.html (1 sec.) www.they.com -> 700 files (45 sec.) schedule for clean in 60 days schedule for clean tomorrow Event Driven Process immediate response w/ token Response Servlet token exists? reply queue scan queue Processor

Slide 65

Slide 65 text

*note : ack msg's are not consumed by the MDB, Dasient Interface Servlet consumes msg's after result is returned dasient interface servlet scan domain queue dasient result queue scan domain augmented queue dasient ack queue + requestId + statusURL + messageType 'put' msg to queue, with correlation id = request id Dasient Result Consumer DasientDB store scanHistory process scan queue PostScan Operations PostScan Consumer DasientDB store all msg's 'get' by consumer DasientDB (clean message) cleaning agent pull available work cleaning script cleaning result servlet send results update clean request domain cleaned queue PostClean Operations PostClean Consumer DasientDB store end start persist 'IGNORED' ScanHistory entity DasientDB persist 'ACK' ScanHistory entity DasientDB persist 'COMPLETED' ScanHistory entity Dasient Integration Flow timeToLive = 5 mins dasient ack expiry queue DasientAckExpiry Consumer 'reaper' process consumes expired messages Email update scanHistory status = 'EXPIRED' + messageType + domainName + controlPanel + startTime Message + messageType + domainName + controlPanel + startTime Dasient Message + messagetype + domain name + control panel + start time + webserver + request type (top-level-scan) + host ip + requestId + jsonPayload + infectionId Message + messagetype + domain name + control panel + start time + webserver + request type (top-level-scan) + host ip + requestId + jsonPayload + infectionId Processor Dasient Augmented Consumer Ack? 'get' by msg selector = request id all msg's 'get' by consumer create Process Scan Msg ignored msg 'put' msg to queue, with correlation id = request id yes PreScan Operations ScanDomain Consumer domain found? DasientDB mark domain 'deleted' Message + messagetype + domain name + control panel + start time + webserver + request type (top-level-scan) + host ip + requestId + jsonPayload + infectionId result from dasient includes: +request id + completed At + status + hostname + on blacklist + scanned urls + malicious urls + snippets Message + messagetype + domain name + control panel + start time + webserver + request type (top-level-scan) + host ip + requestId + jsonPayload + messagetype + domain name + control panel + start time + request type (top-level-scan) + host ip + webserver

Slide 66

Slide 66 text

... TDD!

Slide 67

Slide 67 text

... and the Winner is!

Slide 68

Slide 68 text

Key Message(s) 1. Real world is very asynchronous 2. EDA at times is the right way

Slide 69

Slide 69 text

Camel Books

Slide 70

Slide 70 text

Get the Code https://github.com/ddaugher/integrationNirvana

Slide 71

Slide 71 text

Thanks DJ Daugherty @ddaugher Nilanjan Raychaudhuri @nraychaudhuri