.NET Day 2023: Messaging: The fine line between awesome and awful (and how to stay on the right side of it)

Distributed systems are becoming increasingly common in today's systems landscape, and messaging is often used to coordinate between components in these systems safely. The benefits of using messaging are numerous, including increased reliability, better performance, easy scalability, and simpler decoupling of components, at which point you might think, "Shut up and take my money!". However, as with any architectural choice, the other side of the coin surfaces challenges and pitfalls that we must consider: diagnosing problems, understanding the system's state, handling duplicate messages or idempotency. To address these problems, we will discuss best practices for designing and implementing durable messaging, including techniques for ensuring reliability, consistency, and decoupling. Finally, we must prepare for and mitigate any unknowns that may arise that we can't anticipate. By the end of this session, you will have a comprehensive understanding of the problem space that comes with messaging and will be better equipped to make informed decisions about whether and how to incorporate messaging into your distributed systems.

Transcript

1. Messaging: The fine line between awesome and awful LAILA BOUGRIA And how to stay on the right side of it…
3. When I was young…
4. The monolith
5. Symptoms
6. Let’s use messaging!
7. Messaging benefits Easier decoupling Increased resilience Better performance Granular scalability
8. Challenges Out of order Hard to troubleshoot Duplicates System is slower UI is inconsistent
9. Distributed, big ball of mud
10. Laila Bougria @lailabougria @noctovis @lailabougria @noctovis@hachyderm.io
11. System is slower
12. Request-response
13. Even slower… Producer Request queue Response queue Consumer Producer Consumer
14. Sync vs async request-response
15. One-way Request-response Messaging communication patterns Publish-subscribe
16. Decoupling
17. Place order process 1 Store order Charge credit card 2 Package order 3 Ship the order 4 Bill order 5 Adjust stock 6 Verify customer status 8 Order more stock? 7
18. Place order process Store order Charge credit card Package order Ship the order Bill order Adjust stock Verify customer status Order more stock?
19. Finding logical boundaries  Talk this through with the business  Event storming  Uncover false assumptions  By asking many, many questions!  Rinse and repeat to keep untangling things
20. We end up somewhere here… Customer places order Store order Charge order Package order Backorder order items Package order Ship order Bill order Order more stock? Verify customer status In stock? Stock arrived? Refund order Yes No No Yes Sales Payments Packaging Invoicing Shipping Stock Marketing
21. Communicate between boundaries  One way communication  Request-reply  Publish-Subscribe
22. The passive-aggressive publisher  Publish-subscribe is not a good fit when:  You expect something specific to be done  You need a response with data to continue  You need control over who receives the event
23. Recap  Decouple components by finding logical boundaries  Decouple data by understanding which data depends on each other and which data changes together  Choose the right communication pattern to communicate across components
24. UI is inconsistent
25. Glitches
26. Adjust the language
27.  Ensure message delivery  Show action details  Local storage  Used in several social media applications Illusion of progress
28. Defining SLAs
29. Future messages  Delayed delivery  Supported in Azure Service Bus, Rabbit MQ, Amazon SQS  Calculate the SLA expiration, use it as the delivery time  Verify whether the SLA was violated  Take appropriate action
30. Recap  Adjust language in the UI  Create the illusion of progress  Define Service Level Agreements  Enforce Service Level Agreements with delayed messages
31. Things occur out of order
32. Do we need ordering?
33. Order workflow Customer places order Store order Charge order Package order Backorder order items Package order Ship order Bill order Order more stock? Verify customer status In stock? Stock arrived? Refund order Yes No No Yes
34. Send out messages in the right order
35. Causes of ordering issues Latency Processing time Concurrency Load increase Retries Service availability
36. Let’s make it ordered Store order Charge credit card Package order Ship the order Bill order Adjust stock Verify customer status Order more stock? Sales Payments Shipments Invoices Stock Customer status
37.  Central component drives the business process  Knows and stores the state of the process  Decides which step is next and when  Recreates order Orchestration
38. Orchestration Order Orchestrator Message broker Sales Payments Shipping Invoicing Stock Customer status Packaging
39. Orchestration Order Orchestrator Message broker Sales Payments Shipping Invoicing Stock Customer status Packaging Stock Orchestrator
40. Orchestration pros & cons Order Orchestrator Message broker Sales Payments Shipping Invoicing Packaging
41.  Expect out-of-order messages  Test out-of-order cases  Use orchestration to model workflows  Guard the coupling within a single workflow  Choreography is an alternative for orchestration Recap
42. It’s impossible to troubleshoot failures
43. Message processing failed, but what’s the root cause?
44. We did not expect duplicates
45. And then there were more
46. Idempotency
47. Message deduplication  Detect whether a message was processed before  If so, discard the message  Deterministic Message ID  Deduplication windows
48.  No distributed transactions  Different types of infrastructure within a single system  Transactional guarantees are not cross-infrastructure  Outbox pattern: consistency across database and broker Atomicity problem
49.  Failure management pattern  Ensure data consistency, step-by-step  Sequence of local transactions  Roll back / compensate if one of the operations fail Saga distributed transactions pattern
50. Saga distributed transaction pattern Saga Service A Service B
51.  Make the change idempotent  Use the saga pattern for distributed transactions  Prefer communication over queues instead of REST APIs  Ensure REST APIs are idempotent  Investigate compensating requests Recap
52.  Messaging buys you options to solve scalability, reliability, coupling and performance problems  Commit to decoupling your logic and data  Think how out-of-order processing affects your workflows  Invest in observability  Make idempotency a key pattern in your design Conclusion
53. @lailabougria @noctovis @lailabougria @noctovis@hachyderm.io