Message processing failed…. But what’s the root cause? LAILA BOUGRIA @noctovis @lailabougria

Message processing failed…. But what’s the root cause? LAILA BOUGRIA @noctovis @lailabougria

 Retail business  Decouple the system  Distributed system  Team Lorem, Ipsum and Dolor  Messaging Once upon a time..

Place order process 1 Store order Charge credit card 2 Package order 3 Ship the order 4 Bill order 5 Adjust stock 6 Verify customer status 8 Order more stock? 7 Team Lorem Team Ipsum Team Dolor

Billing OrderBilled Inventory UpdateStock StockUpdated Shipping OrderPackaged OrderShipped Marketing VerifyCustomerStatus CreateDiscountCode Messages Sales PlaceOrder OrderPlaced Finance ChargeOrder OrderPaid

Interactions Billing Marketing Inventory PlaceOrder Finance OrderCharged When OrderPlaced BillOrder OrderBilled When OrderPlaced StockUpdated Sales OrderPlaced When OrderCharged When OrderPlaced VerifyCustomerStatus Shipping When OrderPlaced OrderPackaged When OrderPackaged OrderShipped UpdateStock

 Autonomous teams  Operate in isolation  Interactions are based on a contract  Evolve independently Autonomous components

Something fails Immediate Retry is going to retry message 'abfbbdb5-15fe-4236-a52c-ae92013ec1c9' because of an exception: System.InvalidOperationException: Sequence contains no matching element at System.Linq.ThrowHelper.ThrowNoMatchException() at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source, Func`2 predicate) at Inventory.UpdateProductStockHandler.Handle(UpdateProductStock message, IMessageHandlerContext context) in \src\Inventory\UpdateProductStockHandler.cs:line 14

Something fails What happened? Where? In what context? Why?

More questions…

Debugging a distributed system… Call stack F11 Start entire solution Start / Attach to process Logging Is this message mine? Forgot a breakpoint

Why debug?

No content

Rethinking the AAA-syntax  Arrange: prepare message  Act: invoke the message handler  Assert:  Desired outcome in any data modifications  Outgoing messages

What about the order?

An example Inventory ChargeOrder Finance OrderCharged When OrderPlaced UpdateStock StockUpdated Sales OrderPlaced When OrderCharged Shipping When OrderPlaced OrderPackaged PlaceOrder ReserveStock

An example ChargeOrder Finance OrderCharged Sales OrderPlaced When OrderCharged UpdateStock PlaceOrder ReserveStock Inventory When OrderPackaged StockUpdated StockReserved InsufficientStock Shipping OrderPackaged When StockReserved When InsufficientStock SendPartiallyRefund OrderPartiallyRefunded RefundPartOfOrder

Recap  Test, test, and then test, again  Assert outgoing messages  Be aware of assumptions  Expect out-of-order messages

A message just failed processing on production…

Example UpdateStock InvalidOperationException Product does not exist OrderPackaged foreach (var orderline in order.orderlines) Send( orderLine.ProductId); OrderPlaced Product Ids were incorrectly mapped

 Looking at a single step in insufficient  Follow business flow through:  All participating & interacting components  Including context propagation We need the big picture

Distributed tracing  Early 2000’s  Independent of architectural styles  Decoupled systems  Root cause of problems

 Many frameworks and tools  Vendor-specific  Need for standardization Distributed tracing = +

 Observability framework  Open source  Tools, APIs and SDKs  Collect and export telemetry data  Cross-platform, cross-runtime OpenTelemetry

 Inner workings  Expected and unexpected system state  Without changing any code  External tools Observability

Observability signals Traces Metrics Logs

 Tracks progression of a request  Tree of spans  So, what’s a span?  A unit of work in the trace  Has a context What is a trace?

TraceId: 84b2b44f79e4434fab19ea9362f6536c Client Sales Finance PlaceOrder 9093d82f33eb41d0 PlaceOrder b7b82293d1356b8a ChargeOrder 8d44929e2cc48fe3 ChargeOrder b8dc83c0dee60327 OrderCharged 921e321d61a5ea0e OrderPlaced a4004dd28ae952af orders/create 8a381a70bd9d6d42 Payment Bc0ea953da2d03c9 OrderPlaced 8b63e7af24719e6c OrderPlaced 9276846f6f5e0506 OrderBilled a74700ae9c1fa359 OrderPackaged 8d133f57cf7a5475 Billing Shipping OrderCharged a4004dd28ae952af PackageOrder 9f94fdf6c6871368

TraceId: 84b2b44f79e4434fab19ea9362f6536c Client Sales Finance PlaceOrder 9093d82f33eb41d0 PlaceOrder b7b82293d1356b8a ChargeOrder 8d44929e2cc48fe3 ChargeOrder b8dc83c0dee60327 OrderCharged 921e321d61a5ea0e OrderPlaced a4004dd28ae952af orders/create 8a381a70bd9d6d42 Payment Bc0ea953da2d03c9 OrderPlaced 8b63e7af24719e6c OrderPlaced 9276846f6f5e0506 OrderBilled a74700ae9c1fa359 OrderPackaged 8d133f57cf7a5475 Billing Shipping OrderCharged a4004dd28ae952af PackageOrder 9f94fdf6c6871368

TraceId: 84b2b44f79e4434fab19ea9362f6536c Client Sales Finance PlaceOrder 9093d82f33eb41d0 PlaceOrder b7b82293d1356b8a ChargeOrder 8d44929e2cc48fe3 ChargeOrder b8dc83c0dee60327 OrderCharged 921e321d61a5ea0e OrderPlaced a4004dd28ae952af orders/create 8a381a70bd9d6d42 Payment Bc0ea953da2d03c9 OrderPlaced 8b63e7af24719e6c OrderPlaced 9276846f6f5e0506 OrderBilled a74700ae9c1fa359 OrderPackaged 8d133f57cf7a5475 Billing Shipping OrderCharged a4004dd28ae952af PackageOrder 9f94fdf6c6871368 Service: Sales | Duration: 1.69s Tags messaging.message_id 848af954-cae1-426a-a101-ae92013e1310 messaging.destination Finance messaging.destination_kind Queue order.id f2c0ada9-a148-4390-b488-1c94a5960fc9 order.amount 35.99 € order.paymentmethod CreditCard …

TraceId: 84b2b44f79e4434fab19ea9362f6536c Sales Finance OrderCharged 921e321d61a5ea0e OrderPlaced a4004dd28ae952af OrderPlaced 8b63e7af24719e6c OrderPlaced 9276846f6f5e0506 OrderBilled a74700ae9c1fa359 OrderPackaged 8d133f57cf7a5475 Billing Shipping OrderCharged a4004dd28ae952af PackageOrder 9f94fdf6c6871368 Inventory OrderPackaged 8ca6599733263101 UpdateProductStock 9201210e48d25d39 UpdateProductStock 9129e854d8597dda UpdateProductStock af46300f5ae8ddad UpdateProductStock 9ede452a93abd3e4 Service: Sales | Duration: 1.69s Tags messaging.conversation_id fb2c6eba-1d59-4dc6-9304-ae92013e1311 messaging.message_id 848af954-cae1-426a-a101-ae92013e1310 messaging.destination Sales order.details {"OrderId" : "f2c0ada9-a148-4390-b488-1c94a5960fc9", "Lines" : [ { "ProductId" : "f2c0ada9-a148-4390-b488-1c94a5960fc9", "Quantity" : "5" }, { "ProductId" : "f2c0ada9-a148-4390-b488-1c94a5960fc9", "Quantity" : "1" } ] } …

 Propagation through standardized protocol  W3C Trace Context for HTTP headers 00-84b2b44f79e4434fab19ea9362f6536c-8a381a70bd9d6d42-01 How is a distributed trace created? Version TraceId SpanId Trace flags

 Built into .NET  System.Diagnostics API  System.Diagnostics.DiagnosticsSource Nuget package  Start collecting telemetry  Select sources  Set up an exporter Getting started

Getting started

Exporters & tools

OpenTelemetry support HTTP Client NServiceBus Azure SDK gRPC Redis Entity Framework Core ASP.NET Core SQL Client AWS

.NET implementation ActivityContext Activity Tracer Span SpanContext ActivitySource

What questions should my traces answer?  Can you connect a user action to a trace?  Can you easily group similar traces together?  Can you identify the most load generating operations?  Can you identify which users are stressing the system?  Can you find suspicious events throughout the system?

ActivitySource

No content

Logging vs tracing  Should we stop logging alltogether?  Keep trace tags high-level  Log the details  Connect traces and logs

Additional benefits  Failure investigation  Postmortems  Feature flagging  Chaos engineering

 Observability is crucial  Investigate telemetry emitted by frameworks  Connect your existing logs  Enhance & enrich Recap

@noctovis @lailabougria @lailabougria