apidays Australia 2022 - Spinning Your Drones with Cadence Workflows and Apache Kafka, Paul Brebner, Instaclustr

apidays Australia 2022 - Enabling Business Networks
September 14 & 15, 2022

Spinning Your Drones with Cadence Workflows and Apache Kafka
Paul Brebner, Technology Evangelist at Instaclustr
------------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Deep dive into the API industry with our reports:
https://www.apidays.global/industry-reports/

Subscribe to our global newsletter:
https://apidays.typeform.com/to/i1MPEW

Transcript

1. Spinning your Drones with Cadence Workflows and Apache Kafka® Paul Brebner Instaclustr—Technology Evangelist ©Instaclustr Pty Limited, 2022 APIDAYS Melbourne- September 2022
2. 2022 SERIES OF EVENTS New York JULY (HYBRID) Australia SEPTEMBER (HYBRID) Singapore APRIL (VIRTUAL) Helsinki & North MARCH (VIRTUAL) Paris DECEMBER (HYBRID) London OCTOBER (HYBRID) Hong Kong AUGUST (VIRTUAL) JUNE (VIRTUAL) India MAY (VIRTUAL) APRIL (VIRTUAL) Dubai & Middle East JUNE (VIRTUAL) Check out our API Conferences www.a pida ys .globa l Want to talk at one of our conferences? apidays.typeform.com/to/ILJeAaV8
3. Who Am I? Previously § R&D in distributed systems and performance engineering. Last 5 years § Technology Evangelist for Instaclustr by NetApp § 100+ Blogs, demo applications, talks § Open Source technologies including o Apache Cassandra®, Spark™, Kafka® o OpenSearch®, Redis™ o and now Uber’s Cadence® © Instaclustr Pty Limited, 2022
4. Cloud Platform for Big Data Open Source Technologies Latest addition is Workflow Orchestration with Instaclustr Managed Platform © Instaclustr Pty Limited, 2022
5. Workflow Orchestration – task ordering Start © Instaclustr Pty Limited, 2022
6. Workflow Orchestration – task ordering Start Task 1 © Instaclustr Pty Limited, 2022
7. Workflow Orchestration – task ordering Start Task 1 Task 2 © Instaclustr Pty Limited, 2022
8. Workflow Orchestration – task ordering Start Task 1 Task 2 End © Instaclustr Pty Limited, 2022
9. Pedalling with a high Cadence (pedal revolutions) is called Spinning! Mashing or grinding is slow (and bad.) (Source: Shutterstock) © Instaclustr Pty Limited, 2022 With
10. Cadence is Fault-Tolerant— Workflows can fail! (Source: Shutterstock) © Instaclustr Pty Limited, 2022
11. Cadence is Fault-Tolerant—Workflows continue from point of failure (Source: Shutterstock) © Instaclustr Pty Limited, 2022
12. Cadence is Horizontally Scalable—the number of workflows is unlimited (Source: Shutterstock) © Instaclustr Pty Limited, 2022
13. Cadence Workflows Are Code @Override public void startWorkflow(String a) { String b = activities.task1(a); String c = activities.task2(b); } A workflow implementation with 2 tasks executed sequentially © Instaclustr Pty Limited, 2022 Start Task 1 Task 2 End
14. Cadence Architecture Instaclustr Managed Service Workers Run workflow logic Customer Managed Clients and Workers Workers Workers Clients and Workers Instaclustr Managed Cadence and Database Clusters Cadence Servers Database Cassandra PostgreSQL Kafka OpenSearch (optional) Java and Go Clients are supported APIs © Instaclustr Pty Limited, 2022
15. § com.uber.cadence.activity APIs to implement activity, accessing activity info, or send heartbeat § com.uber.cadence.client APIs for external application code to interact with Cadence workflows, start workflows, send signals or query workflows. § com.uber.cadence.workflow APIs to implement workflows § com.uber.cadence.worker APIs to configure and start workers And more Cadence Client APIs © Instaclustr Pty Limited, 2022
16. © Instaclustr Pty Limited, 2022 Start Task 1 Task 2 End Database Workflow History Written to Database Start Workflow Start Task 1 End Task 1 HERE How Does Cadence Fault-Tolerance Work? Event-Sourcing = History + Replaying
17. Workflow State Recovery by Replaying History © Instaclustr Pty Limited, 2022 Start Task 1 Task 2 End Database Failure causes complete history replay Replay
18. Workflow State Recovery by Replaying History © Instaclustr Pty Limited, 2022 Start Task 1 Task 2 End Database Replay recreates Workflow state and restart point Replay Restart
19. Cadence Activities Activities (including backwards cycling) can fail © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
20. Activity: Task 2 Activity: Task 1 § Activities are core to Cadence § Remote calls can fail, so wrap them in Activities § They can contain any code § Activities are executed at most once and can be automatically retried on failure Cadence Activities © Instaclustr Pty Limited, 2022 Start End Remote Call
21. Workflow Restrictions: Activities @Override public void startWorkflow(String a) { if (a == null) a = “”; String b = activities.task1(a); // executed at most once long t = Workflow.currentTimeMillis(); String c = activities.task2(t, b); // executed at most once } Replaying must not produce new history events! Activities executed at most once—this ensures that once they succeed then the result is immutable (fixed) © Instaclustr Pty Limited, 2022 Start Activity: Task 1 Activity: Task 2 End Workflow code Workflow code <= 1 <= 1
22. @Override public void startWorkflow(String a) { if (a == null) a = “”; // executed multiple times String b = activities.task1(a); long t = Workflow.currentTimeMillis(); // built-in time method not replayed String c = activities.task2(t, b); } Replaying must not produce new history events! Due to replaying, workflow code executed multiple times, so must be deterministic Use built-in methods for time, sleeping, random, and side-effects © Instaclustr Pty Limited, 2022 Start Activity: Task 1 Activity: Task 2 End Workflow code Workflow code >= 1 >= 1 Workflow Restrictions: Workflows
23. § 100s to Millions of running workflows § Long running processes, sleeping & scheduled tasks § Stateful fault-tolerant applications § Complex workflows § Integration with unreliable external systems § Integration with streaming and event-based systems (E.g. Kafka) § Short workflows (100s of steps) – long workflows (1000s of steps) may take longer to replay history For example? Financial, retail, delivery! Good Use Cases? © Instaclustr Pty Limited, 2022
24. Spinning your Drones! A Drone Delivery Application (Source: Shutterstock) © Instaclustr Pty Limited, 2022
25. Drone Workflow Steps § Charged ready to go at base § Get an Order to deliver § Fly to Order location and collect Order § Fly to Delivery location and drop Order § Fly back to base § Recharge § Repeat © Instaclustr Pty Limited, 2022
26. Base to Order to Delivery to Base Example of Drone Delivery Flight © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
27. Drone Way Point Flight Calculations § Drone flight path is computed in an activity § Using location, distance, bearing, speed, and charge § Every 10 seconds § On failure, the drone won’t crash and will continue flying from the last location Returning to Base Example © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
28. The System: Swim-lane diagrams 1st Workflow: Drone Workflow Recharge © Instaclustr Pty Limited, 2022 Start, Ready, Wait for Order, Movement Activities, Recharge, Repeat
29. Orders Are Also Stateful →Workflow 2nd Workflow: Order Workflow © Instaclustr Pty Limited, 2022 (Source: Shutterstock) Start, Generate locations, ready for drone, update locations, End if delivered
30. Orders Are Also Stateful →Workflow 2nd Workflow: Order Workflow So we need coordination between the workflows: (1) Asynchronous Signaling between workflows 1 © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
31. Next, Cadence Meets Kafka (source: https://bahumbug.wordpress.com/2014/08/03/kafka-onna-bike/)
32. Kafka is a distributed pub-sub streams processing system, it allows distributed producers to send messages to distributed consumers via a Kafka cluster. It uses Topics to loosely couple producers and consumers. What is ©Instaclustr Pty Limited 2019, 2021, 2022 Kafka?
33. Integration with Kafka adds A Kafka Cluster with: 3 Kafka Topics 3 Kafka Producers and 2 Consumers 1 © Instaclustr Pty Limited, 2022
34. (2) Starting a workflow from Kafka 1 © Instaclustr Pty Limited, 2022 2 Integration with Kafka adds
35. (3) Using a Kafka microservice to coordinate Drone and Order workflows 1 © Instaclustr Pty Limited, 2022 3 2 Integration with Kafka adds
36. (Source: Shutterstock) Cadence Orchestration + Kafka Choreography = Drone Ballet? (Source: Wikimedia)
37. Step 1: Customer places an order— Order Sent to Kafka New Orders Topic 1 © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
38. 2 Consumer Step 2: Start New Order Workflow— Kafka Consumer Gets Order, Starts New Order Workflow Using Cadence Client © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
39. 3 Activity Step 3: Order Ready for Drone Pickup— Activity: Send Order Ready Message to Orders Ready Topic © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
40. 4 Activity Consumer A Drone+Order Matching Microservice Step 4: Drone Workflow— Activity: Send Drone Ready Message to Drone Ready Topic, Kafka Consumer Gets an Order ID and Sends Signal Back to Drone to start the order pickup © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
41. 5 Step 5: Drone Workflow—Fly To Order Activity: Fly to Order Location © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
42. 5 Step 5: Drone Workflow—Fly To Order And Pickup Order © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
43. 6 Step 6: Drone Workflow—Fly To Delivery Location Activity: Fly to Delivery Location © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
44. Step 6: Drone Workflow—Fly To Delivery Location And Drop Order; Send Location Updates to Order Workflow Every 10s 6 © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
45. 7 Step 7: Drone Workflow—Fly To Base Activity: Fly to Delivery Location; recharge when back at base and start new drone workflow © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
46. 8 Step 8: Order Workflow—Update Location Receive State and Location Signal From Drone Workflow, Update State; If Delivered Then End Workflow © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
47. Sequence… © Instaclustr Pty Limited, 2022
48. Sequence… © Instaclustr Pty Limited, 2022
49. Sequence… © Instaclustr Pty Limited, 2022
50. Sequence… © Instaclustr Pty Limited, 2022
51. Sequence… © Instaclustr Pty Limited, 2022
52. Sequence… © Instaclustr Pty Limited, 2022
53. Sequence… © Instaclustr Pty Limited, 2022
54. Sequence… © Instaclustr Pty Limited, 2022
55. Sequence… © Instaclustr Pty Limited, 2022
56. Sequence… © Instaclustr Pty Limited, 2022
57. Sequence… © Instaclustr Pty Limited, 2022
58. Sequence… © Instaclustr Pty Limited, 2022
59. Sequence… © Instaclustr Pty Limited, 2022
60. Sequence… © Instaclustr Pty Limited, 2022
61. Sequence… © Instaclustr Pty Limited, 2022
62. Sequence… © Instaclustr Pty Limited, 2022
63. Sequence… © Instaclustr Pty Limited, 2022
64. Sequence… © Instaclustr Pty Limited, 2022
65. Sequence… © Instaclustr Pty Limited, 2022
66. Sequence… © Instaclustr Pty Limited, 2022
67. Sequence… © Instaclustr Pty Limited, 2022
68. Sequence… © Instaclustr Pty Limited, 2022
69. Sequence… © Instaclustr Pty Limited, 2022
70. Sequence… © Instaclustr Pty Limited, 2022
71. Sequence… © Instaclustr Pty Limited, 2022
72. Sequence… © Instaclustr Pty Limited, 2022
73. Sequence… © Instaclustr Pty Limited, 2022
74. How Many Drones Can We Fly? © Instaclustr Pty Limited, 2022 (Source: Shutterstock)
75. Cluster Details (VCPUS): Client (8), Cadence (6), Cassandra (18) 6 18 8 VCPUs Per Cluster (32 total) Cadence Cassandra Client (EC2) © Instaclustr Pty Limited, 2022
76. Load Test 1: “Flat-Out” à Max 50 Drones © Instaclustr Pty Limited, 2022
77. Load Test 2: Real-Time à 2,000 Drones & 4,000 Workflows 2000 2000 4000 0 500 1000 1500 2000 2500 3000 3500 4000 4500 Concurrent Workflows Drones Orders Total 4000/32 = 125 workflows/core © Instaclustr Pty Limited, 2022
78. Further Information: Cadence Polling Cookbook https://cadenceworkflow.io/docs/use- cases/polling/ “MegaBurgers” polling example Integration of Cadence with External 3rd party REST APIs for order state updates, using Polling © Instaclustr Pty Limited, 2022
79. Further Information Cadence (and other) Blogs: https://www.instaclustr.com/paul-brebner/ Try us out! © Instaclustr Pty Limited, 2022
80. © Instaclustr Pty Limited, 2022 20 Drones Flying – Q&A
81. www.instaclustr.com info@instaclustr.com @instaclustr THANK YOU! © Instaclustr Pty Limited, 2022 https://www.instaclustr.com/company/policies/terms-conditions/ Except as permitted by the copyright law applicable to you, you may not reproduce, distribute, publish, display, communicate or transmit any of the content of this document, in any form, but any means, without the prior written permission of Instaclustr Pty Limited