Azure Durable Functions

A quick introduction to Azure Durable Functions.
Sorry for the animation at page 12, I'll see if I can decompose it.

Transcript

2. F1 F2 F3 An extension to build long-running, stateful function orchestrations. Still not impressed? Wait, there is magic!
3. 1. Example 2. Patterns 3. Failure handling, retry and timers 4. Monitoring 5. Constraints 6. Web jobs 7. Saga 8. Caveats 9. Interesting things to look into 10.Benefits Summary 11.References
4. Book seat Book extras Charge credit card
5. 1. Where the chaining is defined? (docs? xml? nowhere) 2. What’s the status of a particular execution? (where to find it?) 3. How to handle failure? Book seat Book extras Charge credit card
6. Where the chaining is defined? There are a number of solution of course. Book seat Book extras Charge credit card Orchestrator 1) Orchestrator 2) Events Choreography 3) Route slip Should we use an orchestrator?
7. Book seat Book extras Charge credit card Sounds good but there are still some problems: 1. What if the orchestrator is not available? 2. How to recover from crash? 3. Should we save his state? Where/how? Orchestrator
8. Book seat Book extras Charge credit card Maybe we need something like this? Complexity of course grows. Orchestrator Orchestration State
9. Orchestrator Book extras Charge cc Book seat Forget complexity and start simple again. This is our orchestration.
10. Orchestrator Book seat HTTP Starter Book extras Charge cc
11. 1. Create a new function project (empty template) 2. Add Microsoft.Azure.WebJobs.Extensions.DurableTask nuget 3. Add to the project a Durable Functions Orchestration 4. Select/create a storage account 5. Or a storage emulator
12. Orchestrator Book extras Charge cc Book seat Execution Started Orchestrator Started Orchestration history Task Scheduled (book seat) Orchestrator Completed Task Completed (book seat) Orchestrator Started Task Scheduled (book extra) Orchestrator Completed Task Completed (book extra) Orchestrator Started Task Scheduled (charge cc) Orchestrator Completed Task Completed (charge cc) Orchestrator Started Orchestrator Completed Execution Completed Result “Booked seat for user” Result “Booked extras for user” Result “Charged credit card”
13. • Orchestrator deterministic • Orchestration/activities wake up thru queues • Did you notice we used 3 lines of code for this? Book seat Book extras Charge credit card Orchestrator Orchestration State
14. Some great patterns become trivial to implement: 1. Function chaining (just saw it) 2. Fan out/Fan in 3. Async HTTP APIs 4. Monitor 5. Human interactions/event integration
15. F1 F2 F3 F2 F2 You don’t need durable functions for Fan Out. That’s easy. But what about Fan In?
16. Same function Parse xml Process Order node Store JSON Process Order node Process Order node
17. Parse xml Process Order node Store JSON Process Order node Process Order node
18. Different functions Process Card Payment Schedule Shipping Close Order Send WH request Send user notification
19. Process Card Payment Schedule Shipping Close Order Send WH request Send user notification
20. Start Do Work Get Status
21. F1 F2 The inverse of the async HTTP APIs. More flexible than time triggered function
22. Events sent thru • HTTP API management • durable orchestration client await : - Single event - All of events - One of events (winner)
24. Timer for delay.
25. Timer for timeout.
26. • AppIication Insights • Azure storage logging Custom status logging 4. Monitoring
27. • DurableOrchestrationClient • HTTP API Management 4. Monitoring
28. Orchestrator functions have some constraints: 1. Deterministic 2. No I/O: orchestrator are executed on single threads, can’t block or be slow. Also they are run many times per instance, use activities if you don’t want to pay more. 3. No DateTime: use context.CurrentUtcDateTime (i.e. now +30min?) 4. No infinite loop: use context.ContinueAsNew (because of event history)
29. It has some differences from the functions: • No HTTP trigger. Need to use queue or cron triggers • No function name attributes. Need to use the name of the function. • No HTTP management api. Need to use the orchestration client that offers the same functionalities It opens up different scenarios for long running, non atomic activities that need a more structured code.
30. Durable functions extension seems to be perfect to implement a SAGA pattern with an orchestrator. In fact ,one of the main difficulties of implementing such a pattern is to build a reliable long running orchestrator: • It needs to be stateful • It needs to be resilient to crashes Luckily these characteristics are provided off the shelf by the extension and there is no need to add code for the infrastructure.
31. The simplest implementation makes use of a stack where each compensation is stored just before scheduling the corresponding activity. The stack will have all the compensations that need to be executed up to that point and in the right order (LIFO).
32. 1. Correlation with orchestrator, activities and calling process in App Insight: It’s a queue world. 1. Introduce your correlation ids, embed it into the payload and extract it when logging 2. No metrics for durations, you need to roll your own and it’s useful to understand how the system behaves 2. Problems I faced: 1. Scalability issues with dynamic hub name configuration. (It’s being tracked as a bug) 2. (azure functions in general) EF core version depends on the azure function underlying infrastructure. EF core 2.2.4 not yet supported at time of writing. 3. It’s async processing. Don’t use it for sync unless you have a dedicated plan and a quick execution.
33. 1. Task hub can be custom. Eventually you could use two different environment with the same storage account but isolate them through a different custom name (don’t do it for prod). 2. Examine the underlying storage infrastructure to understand how it works (but don’t rely on it for functionality): • History table • Queues • Locks on queues blobs 3. Is based on at least once delivery (azure storage queues), so idempotency is required for your steps.
34. 1. Reliable execution 2. Beautiful and clean encapsulation, plumbing is hidden 3. It is designed and implemented following cloud patterns (event sourcing, claim-check, partitioning) 4. Define workflow through code. No JSON, no yaml, no designer 5. Work with tasks -instead of queues and event sourcing tables- which is a more comfortable environment for devs. 6. Many important patterns become trivial to implement: fan in/fan out, http client status polling 7. Serverless advantages: scalability and billing. 8. Seems ready for SAGA (we haven’t seen it yet).
35. • Scott Hanselman: https://channel9.msdn.com/Shows/Azure-Friday/Durable-Functions-in-Azure- Functions • Good context: https://mikhail.io/2018/12/making-sense-of-azure-durable-functions/ • Pluralsight course: https://app.pluralsight.com/library/courses/azure-durable-functions- fundamentals/table-of-contents • Jeff Hollan: https://www.youtube.com/watch?v=UQ4iBl7QMno&t=2513s • Official documentation: https://docs.microsoft.com/en-us/azure/azure- functions/durable/durable-functions-overview