.NET Day 22 - Message processing failed! But what's the root cause? by Laila Bourgia

Teams that are new to building and maintaining distributed systems with messaging are faced with challenges when it comes to investigating failure scenarios. With messages flowing through the system, it's hard to quickly identify where a failure originates from and why it's occurring. Without a call stack to guide you, we need other techniques to keep a good overview. In this session, we'll navigate how modeling techniques, integration testing, and instrumentation with OpenTelemetry can help you create visibility for your entire distributed system.

Transcript

1. Message processing failed…. But what’s the root cause? LAILA BOUGRIA @noctovis @lailabougria
2. Message processing failed…. But what’s the root cause? LAILA BOUGRIA @noctovis @lailabougria
3.  Retail business  Decouple the system  Distributed system  Team Lorem, Ipsum and Dolor  Messaging Once upon a time..
4. Place order process 1 Store order Charge credit card 2 Package order 3 Ship the order 4 Bill order 5 Adjust stock 6 Verify customer status 8 Order more stock? 7 Team Lorem Team Ipsum Team Dolor
5. Billing OrderBilled Inventory UpdateStock StockUpdated Shipping OrderPackaged OrderShipped Marketing VerifyCustomerStatus CreateDiscountCode Messages Sales PlaceOrder OrderPlaced Finance ChargeOrder OrderPaid
6. Interactions Billing Marketing Inventory PlaceOrder Finance OrderCharged When OrderPlaced BillOrder OrderBilled When OrderPlaced StockUpdated Sales OrderPlaced When OrderCharged When OrderPlaced VerifyCustomerStatus Shipping When OrderPlaced OrderPackaged When OrderPackaged OrderShipped UpdateStock
7.  Autonomous teams  Operate in isolation  Interactions are based on a contract  Evolve independently Autonomous components
8. Something fails Immediate Retry is going to retry message 'abfbbdb5-15fe-4236-a52c-ae92013ec1c9' because of an exception: System.InvalidOperationException: Sequence contains no matching element at System.Linq.ThrowHelper.ThrowNoMatchException() at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source, Func`2 predicate) at Inventory.UpdateProductStockHandler.Handle(UpdateProductStock message, IMessageHandlerContext context) in \src\Inventory\UpdateProductStockHandler.cs:line 14
9. Something fails What happened? Where? In what context? Why?
10. More questions…
11. Debugging a distributed system… Call stack F11 Start entire solution Start / Attach to process Logging Is this message mine? Forgot a breakpoint
12. Why debug?
14. Rethinking the AAA-syntax  Arrange: prepare message  Act: invoke the message handler  Assert:  Desired outcome in any data modifications  Outgoing messages
15. What about the order?
16. An example Inventory ChargeOrder Finance OrderCharged When OrderPlaced UpdateStock StockUpdated Sales OrderPlaced When OrderCharged Shipping When OrderPlaced OrderPackaged PlaceOrder ReserveStock
17. An example ChargeOrder Finance OrderCharged Sales OrderPlaced When OrderCharged UpdateStock PlaceOrder ReserveStock Inventory When OrderPackaged StockUpdated StockReserved InsufficientStock Shipping OrderPackaged When StockReserved When InsufficientStock SendPartiallyRefund OrderPartiallyRefunded RefundPartOfOrder
18. Recap  Test, test, and then test, again  Assert outgoing messages  Be aware of assumptions  Expect out-of-order messages
19. A message just failed processing on production…
20. Example UpdateStock InvalidOperationException Product does not exist OrderPackaged foreach (var orderline in order.orderlines) Send<UpdateStock>( orderLine.ProductId); OrderPlaced Product Ids were incorrectly mapped
21.  Looking at a single step in insufficient  Follow business flow through:  All participating & interacting components  Including context propagation We need the big picture
22. Distributed tracing  Early 2000’s  Independent of architectural styles  Decoupled systems  Root cause of problems
23.  Many frameworks and tools  Vendor-specific  Need for standardization Distributed tracing = +
24.  Observability framework  Open source  Tools, APIs and SDKs  Collect and export telemetry data  Cross-platform, cross-runtime OpenTelemetry
25.  Inner workings  Expected and unexpected system state  Without changing any code  External tools Observability
26. Observability signals Traces Metrics Logs
27.  Tracks progression of a request  Tree of spans  So, what’s a span?  A unit of work in the trace  Has a context What is a trace?
28. TraceId: 84b2b44f79e4434fab19ea9362f6536c Client Sales Finance PlaceOrder 9093d82f33eb41d0 PlaceOrder b7b82293d1356b8a ChargeOrder 8d44929e2cc48fe3 ChargeOrder b8dc83c0dee60327 OrderCharged 921e321d61a5ea0e OrderPlaced a4004dd28ae952af orders/create 8a381a70bd9d6d42 Payment Bc0ea953da2d03c9 OrderPlaced 8b63e7af24719e6c OrderPlaced 9276846f6f5e0506 OrderBilled a74700ae9c1fa359 OrderPackaged 8d133f57cf7a5475 Billing Shipping OrderCharged a4004dd28ae952af PackageOrder 9f94fdf6c6871368
29. TraceId: 84b2b44f79e4434fab19ea9362f6536c Client Sales Finance PlaceOrder 9093d82f33eb41d0 PlaceOrder b7b82293d1356b8a ChargeOrder 8d44929e2cc48fe3 ChargeOrder b8dc83c0dee60327 OrderCharged 921e321d61a5ea0e OrderPlaced a4004dd28ae952af orders/create 8a381a70bd9d6d42 Payment Bc0ea953da2d03c9 OrderPlaced 8b63e7af24719e6c OrderPlaced 9276846f6f5e0506 OrderBilled a74700ae9c1fa359 OrderPackaged 8d133f57cf7a5475 Billing Shipping OrderCharged a4004dd28ae952af PackageOrder 9f94fdf6c6871368
30. TraceId: 84b2b44f79e4434fab19ea9362f6536c Client Sales Finance PlaceOrder 9093d82f33eb41d0 PlaceOrder b7b82293d1356b8a ChargeOrder 8d44929e2cc48fe3 ChargeOrder b8dc83c0dee60327 OrderCharged 921e321d61a5ea0e OrderPlaced a4004dd28ae952af orders/create 8a381a70bd9d6d42 Payment Bc0ea953da2d03c9 OrderPlaced 8b63e7af24719e6c OrderPlaced 9276846f6f5e0506 OrderBilled a74700ae9c1fa359 OrderPackaged 8d133f57cf7a5475 Billing Shipping OrderCharged a4004dd28ae952af PackageOrder 9f94fdf6c6871368 Service: Sales | Duration: 1.69s Tags messaging.message_id 848af954-cae1-426a-a101-ae92013e1310 messaging.destination Finance messaging.destination_kind Queue order.id f2c0ada9-a148-4390-b488-1c94a5960fc9 order.amount 35.99 € order.paymentmethod CreditCard …
31. TraceId: 84b2b44f79e4434fab19ea9362f6536c Sales Finance OrderCharged 921e321d61a5ea0e OrderPlaced a4004dd28ae952af OrderPlaced 8b63e7af24719e6c OrderPlaced 9276846f6f5e0506 OrderBilled a74700ae9c1fa359 OrderPackaged 8d133f57cf7a5475 Billing Shipping OrderCharged a4004dd28ae952af PackageOrder 9f94fdf6c6871368 Inventory OrderPackaged 8ca6599733263101 UpdateProductStock 9201210e48d25d39 UpdateProductStock 9129e854d8597dda UpdateProductStock af46300f5ae8ddad UpdateProductStock 9ede452a93abd3e4 Service: Sales | Duration: 1.69s Tags messaging.conversation_id fb2c6eba-1d59-4dc6-9304-ae92013e1311 messaging.message_id 848af954-cae1-426a-a101-ae92013e1310 messaging.destination Sales order.details {"OrderId" : "f2c0ada9-a148-4390-b488-1c94a5960fc9", "Lines" : [ { "ProductId" : "f2c0ada9-a148-4390-b488-1c94a5960fc9", "Quantity" : "5" }, { "ProductId" : "f2c0ada9-a148-4390-b488-1c94a5960fc9", "Quantity" : "1" } ] } …
32.  Propagation through standardized protocol  W3C Trace Context for HTTP headers 00-84b2b44f79e4434fab19ea9362f6536c-8a381a70bd9d6d42-01 How is a distributed trace created? Version TraceId SpanId Trace flags
33.  Built into .NET  System.Diagnostics API  System.Diagnostics.DiagnosticsSource Nuget package  Start collecting telemetry  Select sources  Set up an exporter Getting started
34. Getting started
35. Exporters & tools
36. OpenTelemetry support HTTP Client NServiceBus Azure SDK gRPC Redis Entity Framework Core ASP.NET Core SQL Client AWS
37. .NET implementation ActivityContext Activity Tracer Span SpanContext ActivitySource
38. What questions should my traces answer?  Can you connect a user action to a trace?  Can you easily group similar traces together?  Can you identify the most load generating operations?  Can you identify which users are stressing the system?  Can you find suspicious events throughout the system?
39. ActivitySource
41. Logging vs tracing  Should we stop logging alltogether?  Keep trace tags high-level  Log the details  Connect traces and logs
42. Additional benefits  Failure investigation  Postmortems  Feature flagging  Chaos engineering
43.  Observability is crucial  Investigate telemetry emitted by frameworks  Connect your existing logs  Enhance & enrich Recap
44. @noctovis @lailabougria @lailabougria