Building Serverless Workflows with Durable Functions

Db4b131b1b1519a5a22f84b1e8737f9b?s=47 Mark Heath
September 12, 2018

Building Serverless Workflows with Durable Functions

How Azure Durable Functions can help you create reliable and robust serverless workflows

Db4b131b1b1519a5a22f84b1e8737f9b?s=128

Mark Heath

September 12, 2018
Tweet

Transcript

  1. Order from Chaos Building Serverless Workflows with Durable Functions @mark_heath

    https://markheath.net
  2. @mark_heath Video uploaded Virus Scan File copy Extract metadata Database

    Audit Transcode Thumbnail Analytics Voice detection Notification icons designed by Smashicons from flaticons
  3. “Functions as a Service” (FaaS) A managed platform that runs

    your code (a function) in response to an event (trigger) @mark_heath
  4. Blob Cosmos DB HTTP Request Timer Queue Azure Functions Triggers

    @mark_heath
  5. Serverless @mark_heath Managed Servers Automatic Scale Per-Second Billing icons designed

    by Smashicons from flaticons
  6. Can we build serverless workflows with Azure Functions? @mark_heath

  7. @mark_heath Function chaining with queues Video uploaded Extract Metadata Virus

    Scan File Copy
  8. @mark_heath

  9. @mark_heath

  10. @mark_heath Video uploaded Virus Scan File copy Extract metadata Database

    Audit Transcode Thumbnail Analytics Voice detection Notification icons designed by Smashicons from flaticons Can’t see the big picture?
  11. Image from The Art of Rube Goldberg: (A) Inventive (B)

    Cartoon (C) Genius, Selected and with commentary by Jennifer George; Introduction by Adam Gopnik, Published by Abrams ComicArts.
  12. @mark_heath Video uploaded Virus Scan File copy Extract metadata Database

    Audit Transcode Thumbnail Analytics Voice detection Notification icons designed by Smashicons from flaticons Retries and backoff
  13. @mark_heath Video uploaded Virus Scan File copy Extract metadata Database

    Audit Transcode Thumbnail Analytics Voice detection Notification icons designed by Smashicons from flaticons Error Handling Stuck Workflows
  14. @mark_heath Video uploaded Virus Scan File copy Extract metadata Database

    Audit Transcode Thumbnail Analytics Voice detection Notification icons designed by Smashicons from flaticons Waiting for external events
  15. Fan-out Fan-in Workflow @mark_heath easy! difficult!

  16. Durable Functions to the rescue! @mark_heath

  17. Durable Functions • An extension to Azure Functions • Microsoft.Azure.WebJobs.Extensions.DurableTask

    NuGet package • C# supported, JavaScript in preview • Open source • https://github.com/Azure/azure-functions-durable-extension @mark_heath
  18. Orchestrator Functions • Define the steps in the workflow •

    Shows the big picture • Calls “activity functions” • Or other orchestrators (“sub-orchestrations”) • Decides what to do next • Handle exceptions across all activities • Must adhere to strict rules @mark_heath
  19. Activity Functions • Perform steps in the workflow • Can

    receive input data and return output data • Can use “bindings” to connect to external services • (Just like a regular Azure Function) @mark_heath
  20. Task Hubs • Stores state associated with workflows • Uses

    an Azure Storage account behind the scenes • (Tables and Queues) • Uses an event sourcing technique @mark_heath
  21. REST API • Query orchestration status (GET) • https://{yourapp}.azurewebsites.net/runtime/webhooks/durabletask/ instances/{orchestration_id}?taskHub=DurableFunctionsHub&

    connection=Storage&code={code} • Send an event to an orchestration (POST) • https://{yourapp}.azurewebsites.net/runtime/webhooks/durabletask/ instances/{orchestration_id}/raiseEvent/{eventName}? taskHub=DurableFunctionsHub&connection=Storage&code={code} • Terminate an orchestration (POST) • https://{yourapp}.azurewebsites.net/runtime/webhooks/durabletask/ instances/{orchestration_id}/terminate?reason={text}& taskHub=DurableFunctionsHub&connection=Storage&code={code}
  22. Demo Time! @mark_heath https://github.com/markheath/durable-functions-ecommerce-sample

  23. @mark_heath Payment received Database Generate PDF Await approval Email customer

    Generate Video
  24. [FunctionName("NewPurchaseWebhook")] public static async Task<IActionResult> Run( [HttpTrigger(AuthorizationLevel.Function, "post", Route =

    null)] HttpRequest req, [OrchestrationClient] DurableOrchestrationClient client) { string requestBody = await new StreamReader(req.Body).ReadToEndAsync(); var order = JsonConvert.DeserializeObject<Order>(requestBody); var orchestrationId = await client.StartNewAsync("O_ProcessOrder", order); return new OkResult(); } Starter Function @mark_heath
  25. Orchestrator V1 [FunctionName("O_ProcessOrder")] public static async Task<object> ProcessOrder( [OrchestrationTrigger] DurableOrchestrationContext

    ctx) { var order = ctx.GetInput<Order>(); await ctx.CallActivityAsync("A_SaveOrderToDatabase", order); var pdfLocation = await ctx.CallActivityAsync<string>("A_CreatePersonalizedPdf", order); var videoLocation = await ctx.CallActivityAsync<string>("A_CreateWatermarkedVideo", order); await ctx.CallActivityAsync("A_SendEmail", (order, pdfLocation, videoLocation)); return "Order processed successfully"; } @mark_heath
  26. Activity Function [FunctionName("A_CreatePersonalizedPdf")] public static string CreatePersonalizedPdf( [ActivityTrigger] Order order,

    TraceWriter log) { log.Info("Creating PDF"); return $"{order.Id}.pdf"; } @mark_heath
  27. Orchestrator V2 – Parallel Tasks var pdfTask = ctx.CallActivityAsync<string>("A_CreatePersonalizedPdf", order);

    var videoTask = ctx.CallActivityAsync<string>("A_CreateWatermarkedVideo", order); await Task.WhenAll(pdfTask, videoTask); var pdfLocation = pdfTask.Result; var videoLocation = videoTask.Result; @mark_heath
  28. Exception Handling and Retry try { await ctx.CallActivityAsync<string> ("A_CreatePersonalizedPdf", order);

    await ctx.CallActivityWithRetryAsync<string> ("A_CreateWatermarkedVideo", new RetryOptions(TimeSpan.FromSeconds(30), 3), order); } catch (Exception ex) { log.Error($"Problem creating media files", ex); } @mark_heath
  29. Waiting for External Events var approvalResult = await ctx.WaitForExternalEvent<string>( "OrderApprovalResult",

    TimeSpan.FromMinutes(30), "TimedOut"); if (approvalResult == "TimedOut" || approvalResult == "Rejected") { await ctx.CallActivityAsync("A_OrderNotApproved", order); } else { // order was approved await ctx.CallActivityAsync("A_EmailCustomer", order); } @mark_heath
  30. Sending an External Event [FunctionName("ApproveOrder")] public static async Task<IActionResult> ApproveOrder(

    [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req, [OrchestrationClient] DurableOrchestrationClient client) { string requestBody = await new StreamReader(req.Body).ReadToEndAsync(); var approvalResult = JsonConvert.DeserializeObject<ApprovalResult>(requestBody); await client.RaiseEventAsync(approvalResult.OrchestrationId, "OrderApprovalResult", approvalResult); return new OkResult(); } @mark_heath
  31. Why Durable Functions? • Define the big picture in code

    • Handle errors for the workflow as a whole • Retry individual steps with back-off • Easily implement waiting for external events with timeout • Easily implement fan-out fan-in patterns • Track progress of workflows • Cancel workflows • It’s free and open source @mark_heath
  32. @mark_heath

  33. https://github.com/markheath/ durable-functions-ecommerce-sample https://markheath.net @mark_heath https://skillsmatter.com/conferences/ 10107-prognet-london-2018#skillscasts