Building Serverless Workflows with Durable Functions

How Azure Durable Functions can help you create reliable and robust serverless workflows

Transcript

1. Order from Chaos Building Serverless Workflows with Durable Functions @mark_heath https://markheath.net
2. @mark_heath Video uploaded Virus Scan File copy Extract metadata Database Audit Transcode Thumbnail Analytics Voice detection Notification icons designed by Smashicons from flaticons
3. “Functions as a Service” (FaaS) A managed platform that runs your code (a function) in response to an event (trigger) @mark_heath
4. Blob Cosmos DB HTTP Request Timer Queue Azure Functions Triggers @mark_heath
5. Serverless @mark_heath Managed Servers Automatic Scale Per-Second Billing icons designed by Smashicons from flaticons
6. Can we build serverless workflows with Azure Functions? @mark_heath
7. @mark_heath Function chaining with queues Video uploaded Extract Metadata Virus Scan File Copy
8. @mark_heath
9. @mark_heath
10. @mark_heath Video uploaded Virus Scan File copy Extract metadata Database Audit Transcode Thumbnail Analytics Voice detection Notification icons designed by Smashicons from flaticons Can’t see the big picture?
11. Image from The Art of Rube Goldberg: (A) Inventive (B) Cartoon (C) Genius, Selected and with commentary by Jennifer George; Introduction by Adam Gopnik, Published by Abrams ComicArts.
12. @mark_heath Video uploaded Virus Scan File copy Extract metadata Database Audit Transcode Thumbnail Analytics Voice detection Notification icons designed by Smashicons from flaticons Retries and backoff
13. @mark_heath Video uploaded Virus Scan File copy Extract metadata Database Audit Transcode Thumbnail Analytics Voice detection Notification icons designed by Smashicons from flaticons Error Handling Stuck Workflows
14. @mark_heath Video uploaded Virus Scan File copy Extract metadata Database Audit Transcode Thumbnail Analytics Voice detection Notification icons designed by Smashicons from flaticons Waiting for external events
15. Fan-out Fan-in Workflow @mark_heath easy! difficult!
16. Durable Functions to the rescue! @mark_heath
17. Durable Functions • An extension to Azure Functions • Microsoft.Azure.WebJobs.Extensions.DurableTask NuGet package • C# supported, JavaScript in preview • Open source • https://github.com/Azure/azure-functions-durable-extension @mark_heath
18. Orchestrator Functions • Define the steps in the workflow • Shows the big picture • Calls “activity functions” • Or other orchestrators (“sub-orchestrations”) • Decides what to do next • Handle exceptions across all activities • Must adhere to strict rules @mark_heath
19. Activity Functions • Perform steps in the workflow • Can receive input data and return output data • Can use “bindings” to connect to external services • (Just like a regular Azure Function) @mark_heath
20. Task Hubs • Stores state associated with workflows • Uses an Azure Storage account behind the scenes • (Tables and Queues) • Uses an event sourcing technique @mark_heath
21. REST API • Query orchestration status (GET) • https://{yourapp}.azurewebsites.net/runtime/webhooks/durabletask/ instances/{orchestration_id}?taskHub=DurableFunctionsHub& connection=Storage&code={code} • Send an event to an orchestration (POST) • https://{yourapp}.azurewebsites.net/runtime/webhooks/durabletask/ instances/{orchestration_id}/raiseEvent/{eventName}? taskHub=DurableFunctionsHub&connection=Storage&code={code} • Terminate an orchestration (POST) • https://{yourapp}.azurewebsites.net/runtime/webhooks/durabletask/ instances/{orchestration_id}/terminate?reason={text}& taskHub=DurableFunctionsHub&connection=Storage&code={code}
22. Demo Time! @mark_heath https://github.com/markheath/durable-functions-ecommerce-sample
23. @mark_heath Payment received Database Generate PDF Await approval Email customer Generate Video
24. [FunctionName("NewPurchaseWebhook")] public static async Task<IActionResult> Run( [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req, [OrchestrationClient] DurableOrchestrationClient client) { string requestBody = await new StreamReader(req.Body).ReadToEndAsync(); var order = JsonConvert.DeserializeObject<Order>(requestBody); var orchestrationId = await client.StartNewAsync("O_ProcessOrder", order); return new OkResult(); } Starter Function @mark_heath
25. Orchestrator V1 [FunctionName("O_ProcessOrder")] public static async Task<object> ProcessOrder( [OrchestrationTrigger] DurableOrchestrationContext ctx) { var order = ctx.GetInput<Order>(); await ctx.CallActivityAsync("A_SaveOrderToDatabase", order); var pdfLocation = await ctx.CallActivityAsync<string>("A_CreatePersonalizedPdf", order); var videoLocation = await ctx.CallActivityAsync<string>("A_CreateWatermarkedVideo", order); await ctx.CallActivityAsync("A_SendEmail", (order, pdfLocation, videoLocation)); return "Order processed successfully"; } @mark_heath
26. Activity Function [FunctionName("A_CreatePersonalizedPdf")] public static string CreatePersonalizedPdf( [ActivityTrigger] Order order, TraceWriter log) { log.Info("Creating PDF"); return $"{order.Id}.pdf"; } @mark_heath
27. Orchestrator V2 – Parallel Tasks var pdfTask = ctx.CallActivityAsync<string>("A_CreatePersonalizedPdf", order); var videoTask = ctx.CallActivityAsync<string>("A_CreateWatermarkedVideo", order); await Task.WhenAll(pdfTask, videoTask); var pdfLocation = pdfTask.Result; var videoLocation = videoTask.Result; @mark_heath
28. Exception Handling and Retry try { await ctx.CallActivityAsync<string> ("A_CreatePersonalizedPdf", order); await ctx.CallActivityWithRetryAsync<string> ("A_CreateWatermarkedVideo", new RetryOptions(TimeSpan.FromSeconds(30), 3), order); } catch (Exception ex) { log.Error($"Problem creating media files", ex); } @mark_heath
29. Waiting for External Events var approvalResult = await ctx.WaitForExternalEvent<string>( "OrderApprovalResult", TimeSpan.FromMinutes(30), "TimedOut"); if (approvalResult == "TimedOut" || approvalResult == "Rejected") { await ctx.CallActivityAsync("A_OrderNotApproved", order); } else { // order was approved await ctx.CallActivityAsync("A_EmailCustomer", order); } @mark_heath
30. Sending an External Event [FunctionName("ApproveOrder")] public static async Task<IActionResult> ApproveOrder( [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req, [OrchestrationClient] DurableOrchestrationClient client) { string requestBody = await new StreamReader(req.Body).ReadToEndAsync(); var approvalResult = JsonConvert.DeserializeObject<ApprovalResult>(requestBody); await client.RaiseEventAsync(approvalResult.OrchestrationId, "OrderApprovalResult", approvalResult); return new OkResult(); } @mark_heath
31. Why Durable Functions? • Define the big picture in code • Handle errors for the workflow as a whole • Retry individual steps with back-off • Easily implement waiting for external events with timeout • Easily implement fan-out fan-in patterns • Track progress of workflows • Cancel workflows • It’s free and open source @mark_heath
32. @mark_heath
33. https://github.com/markheath/ durable-functions-ecommerce-sample https://markheath.net @mark_heath https://skillsmatter.com/conferences/ 10107-prognet-london-2018#skillscasts