Building distributed applications with NServiceBus

In this session we will see an example of how very different services for both technology and age can coexist and collaborate in a distributed messaging system. We will see how in CGN we have used NServiceBus to achieve sustainable architecture in a constantly evolving ecosystem.

Transcript

1. Building distributed applications with NServiceBus Software Architect at ServiziCGN @manuelscapolan Manuel Scapolan ServiziCGN – a case study
2. Servizi CGN • Fiscal and tax advice • E-learning and class learning • Dedicated software to fill tax return statements and to run the companies accounting Leader company in offering services to accountants
3. Servizi CGN • More than 1.260.000 tax return statements sent only this year • Thanks to a network of 35.000 professionals that use its software free to fill and send the tax return statements of their clients 1° CAF in Italy as private business
4. “Status quo” • Many applications to be integrated (> 20) • Different platforms (COBOL, .NET) • Different architectures (client-server, n-tier, …) • Different technologies • Many databases (semantic duplication)
5. Where we live • High number of requests in a short interval of time (data-entry activity) • High number of concurrent users • Short and long running operations
6. Where we started Billing/Accou nting system Tax return applications Management system Revenue agency Two-way channel uses custom protocol built over TCP/IP (on-demand communication started by the client) Standalone and off-line applications built on cobol platform Data-sharing between file transfer
7. Synchronize data Challenge #1
8. Synchronize data • Every system has its data stores to store customer related information • When a customer is created (or updated) in one system all the others need to change their copy of data Data synchronization
9. Solving at the database level • ETL process (run nightly) using bulk data modification techniques – Pro: • High control over data manipulation • Easy to load huge amounts of data – Cons: • Strong dependence on data stores availability • The data is not updated in real-time
10. Asynchronous messaging • One application publishes a message (with the updated information) to a common channel • Other systems can read the message from the channel at a later time • The systems must agree on a channel as well as on the format of the message • The communication is asynchronous
11. Benefits of async messaging • No temporal coupling, not require both systems to be up and ready at the same time • “Real-time” data synchronization • Reliability of store-and-forward • “Fine-grained” log out of the box
12. What we need • A Message Bus: NServiceBus • A transport: MSMQ • Producer/consumer components: endpoint • And obviously, a message: event end-point
13. NServiceBus • The most developer-friendly service bus for .NET • Supporting one-way, full-duplex, publish/subscribe patterns • Easy to extend and configure • Long running processes definition and management capabilities using Sagas • Out of the box transports (MSMQ, RabbitMQ, SQL Server, Windows Azure Queues,Windows Azure Service Bus) • Easy to integrate with existing system
14. MSMQ • Max 4MB message size • Persists messages to file • Supports transactional queues • Supports store and forward
15. Message endpoint • A client of the messaging system that the application can then use to send or receive messages • NServiceBus.Host hosted in a Windows Service
16. Message-based architecture • Application development model: Front-end (Composite UI) Web application / API gateway CommandHandlers EventHandlers Message bus
17. Communication mechanisms • RESTful API – In the ApiController of the API gateway the call is translated to a command which is sent to the commandhandlers endpoint synchronously • Messages (Commands or Events) – Asynchronous communication
18. Anti-Corruption Layer (ACL) • Layer built to integrate the bounded context with an existing system where we don’t have control (i.e. legacy or external system) • Translate conceptual objects or actions from one model to another • It can be developed as a façade or an adapter Legacy Application Message- based system
19. Communicating with the client • The ACL is a file adapter that picks a file from a file system and turn into a message to deliver onto a channel and vice versa Message bus Membership Service ACL CreateUser() UserCreated UserCreated Client applications Management system
20. Integrate applications Challenge #2
21. Integrate applications • The tax payment application (F24 model in Italy) is “interested” to the tax amount • The tax amount is a calculated value based on the data of the tax return statement Tax payment service Tax return statement applications Tax return statement applications Revenue agency amount total
22. Publish/subscribe • The tax payment application subscribes to the event TaxAmountChanged • When in a tax return statement application a tax amount is changed a new event of type TaxAmountChanged is published to the message bus • The handler in the tax payment service endpoint receives the message and updates the total amount
23. Scalability Challenge #3
24. Scalability • The life of the accountant will often be on the cutting edge • The unavailability of the software or a high latency in those moments can bring serious harm to our clients • Not all the operations are equals…
25. Scaling out with NServiceBus • In our case the print of a tax return statement is a good candidate to scale out Frontend / web application CommandHandlers endpoint Print Queue NServiceBus Host #1 NServiceBus Host #2 Competing consumer
26. Long-running “transactions” Challenge #4
27. Long-running “transactions” • Avoid locks on non-local resources • Use compensation to handle failures • Potentially aggregate smaller ACID transactions (also referred to as atomic transactions) • Typically use a coordinator to complete or abort the transaction
28. Long-running “transactions” • There are a few different types of statements that must be delivered together • We are in a “all-or-none” situation Revenue agency
29. Long-running “transactions” • Implemented with an NServiceBus saga
30. Lessons Learned
31. Lessons learned • Thinking asynchronously • You can’t have functional scalability without bounded contexts • For integration prefer asynchronous messaging over synchronous http calls (loose coupling) • Keep handlers small and idempotent (single responsability principle)
32. To learn more about Bounded context, Messaging system, DevOps,…
33. Resources • NServiceBus introduction to a message based distributed architecture [http://www.slideshare.net/mauroservienti/nservicebus-introduction-to-a-message-based-distributed-architecture?related=6] • Message Oriented Architecture using NServiceBus [http://www.slideshare.net/kindblad/2013-november-message-oriented-architecture-using-n-servicebus?related=1] • Distributed Application Development with NServiceBus [http://www.slideshare.net/davidboike/distributed-application-development-with-nservicebus] • Introduction to NServiceBus [http://www.slideshare.net/danielmarbach50/introduction-to-nservicebus-14752472?related=1] • Slide5 [https://www.flickr.com/photos/archi-bald/3780921448/]
34. Thank you @manuelscapolan