Slide 1

Slide 1 text

Building distributed applications with NServiceBus Software Architect at ServiziCGN @manuelscapolan Manuel Scapolan ServiziCGN – a case study

Slide 2

Slide 2 text

Servizi CGN • Fiscal and tax advice • E-learning and class learning • Dedicated software to fill tax return statements and to run the companies accounting Leader company in offering services to accountants

Slide 3

Slide 3 text

Servizi CGN • More than 1.260.000 tax return statements sent only this year • Thanks to a network of 35.000 professionals that use its software free to fill and send the tax return statements of their clients 1° CAF in Italy as private business

Slide 4

Slide 4 text

“Status quo” • Many applications to be integrated (> 20) • Different platforms (COBOL, .NET) • Different architectures (client-server, n-tier, …) • Different technologies • Many databases (semantic duplication)

Slide 5

Slide 5 text

Where we live • High number of requests in a short interval of time (data-entry activity) • High number of concurrent users • Short and long running operations

Slide 6

Slide 6 text

Where we started Billing/Accou nting system Tax return applications Management system Revenue agency Two-way channel uses custom protocol built over TCP/IP (on-demand communication started by the client) Standalone and off-line applications built on cobol platform Data-sharing between file transfer

Slide 7

Slide 7 text

Synchronize data Challenge #1

Slide 8

Slide 8 text

Synchronize data • Every system has its data stores to store customer related information • When a customer is created (or updated) in one system all the others need to change their copy of data Data synchronization

Slide 9

Slide 9 text

Solving at the database level • ETL process (run nightly) using bulk data modification techniques – Pro: • High control over data manipulation • Easy to load huge amounts of data – Cons: • Strong dependence on data stores availability • The data is not updated in real-time

Slide 10

Slide 10 text

Asynchronous messaging • One application publishes a message (with the updated information) to a common channel • Other systems can read the message from the channel at a later time • The systems must agree on a channel as well as on the format of the message • The communication is asynchronous

Slide 11

Slide 11 text

Benefits of async messaging • No temporal coupling, not require both systems to be up and ready at the same time • “Real-time” data synchronization • Reliability of store-and-forward • “Fine-grained” log out of the box

Slide 12

Slide 12 text

What we need • A Message Bus: NServiceBus • A transport: MSMQ • Producer/consumer components: endpoint • And obviously, a message: event end-point

Slide 13

Slide 13 text

NServiceBus • The most developer-friendly service bus for .NET • Supporting one-way, full-duplex, publish/subscribe patterns • Easy to extend and configure • Long running processes definition and management capabilities using Sagas • Out of the box transports (MSMQ, RabbitMQ, SQL Server, Windows Azure Queues,Windows Azure Service Bus) • Easy to integrate with existing system

Slide 14

Slide 14 text

MSMQ • Max 4MB message size • Persists messages to file • Supports transactional queues • Supports store and forward

Slide 15

Slide 15 text

Message endpoint • A client of the messaging system that the application can then use to send or receive messages • NServiceBus.Host hosted in a Windows Service

Slide 16

Slide 16 text

Message-based architecture • Application development model: Front-end (Composite UI) Web application / API gateway CommandHandlers EventHandlers Message bus

Slide 17

Slide 17 text

Communication mechanisms • RESTful API – In the ApiController of the API gateway the call is translated to a command which is sent to the commandhandlers endpoint synchronously • Messages (Commands or Events) – Asynchronous communication

Slide 18

Slide 18 text

Anti-Corruption Layer (ACL) • Layer built to integrate the bounded context with an existing system where we don’t have control (i.e. legacy or external system) • Translate conceptual objects or actions from one model to another • It can be developed as a façade or an adapter Legacy Application Message- based system

Slide 19

Slide 19 text

Communicating with the client • The ACL is a file adapter that picks a file from a file system and turn into a message to deliver onto a channel and vice versa Message bus Membership Service ACL CreateUser() UserCreated UserCreated Client applications Management system

Slide 20

Slide 20 text

Integrate applications Challenge #2

Slide 21

Slide 21 text

Integrate applications • The tax payment application (F24 model in Italy) is “interested” to the tax amount • The tax amount is a calculated value based on the data of the tax return statement Tax payment service Tax return statement applications Tax return statement applications Revenue agency amount total

Slide 22

Slide 22 text

Publish/subscribe • The tax payment application subscribes to the event TaxAmountChanged • When in a tax return statement application a tax amount is changed a new event of type TaxAmountChanged is published to the message bus • The handler in the tax payment service endpoint receives the message and updates the total amount

Slide 23

Slide 23 text

Scalability Challenge #3

Slide 24

Slide 24 text

Scalability • The life of the accountant will often be on the cutting edge • The unavailability of the software or a high latency in those moments can bring serious harm to our clients • Not all the operations are equals…

Slide 25

Slide 25 text

Scaling out with NServiceBus • In our case the print of a tax return statement is a good candidate to scale out Frontend / web application CommandHandlers endpoint Print Queue NServiceBus Host #1 NServiceBus Host #2 Competing consumer

Slide 26

Slide 26 text

Long-running “transactions” Challenge #4

Slide 27

Slide 27 text

Long-running “transactions” • Avoid locks on non-local resources • Use compensation to handle failures • Potentially aggregate smaller ACID transactions (also referred to as atomic transactions) • Typically use a coordinator to complete or abort the transaction

Slide 28

Slide 28 text

Long-running “transactions” • There are a few different types of statements that must be delivered together • We are in a “all-or-none” situation Revenue agency

Slide 29

Slide 29 text

Long-running “transactions” • Implemented with an NServiceBus saga

Slide 30

Slide 30 text

Lessons Learned

Slide 31

Slide 31 text

Lessons learned • Thinking asynchronously • You can’t have functional scalability without bounded contexts • For integration prefer asynchronous messaging over synchronous http calls (loose coupling) • Keep handlers small and idempotent (single responsability principle)

Slide 32

Slide 32 text

To learn more about Bounded context, Messaging system, DevOps,…

Slide 33

Slide 33 text

Resources • NServiceBus introduction to a message based distributed architecture [http://www.slideshare.net/mauroservienti/nservicebus-introduction-to-a-message-based-distributed-architecture?related=6] • Message Oriented Architecture using NServiceBus [http://www.slideshare.net/kindblad/2013-november-message-oriented-architecture-using-n-servicebus?related=1] • Distributed Application Development with NServiceBus [http://www.slideshare.net/davidboike/distributed-application-development-with-nservicebus] • Introduction to NServiceBus [http://www.slideshare.net/danielmarbach50/introduction-to-nservicebus-14752472?related=1] • Slide5 [https://www.flickr.com/photos/archi-bald/3780921448/]

Slide 34

Slide 34 text

Thank you @manuelscapolan