Tweet
Tweet

Slide 1

Slide 1 text

Microworkflows for Microservices Complex Workflows in Distributed Applications 1

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

@[email protected] Why me? - Built my first message-driven, asynchronous system for the Bundesbank starting 2001 - Worked on various event-driven systems, e.g. for Zalando, ING, ista, Maersk - Currently building a bank (!) with Event-driven Architecture, Microservices, DDD 3

Slide 4

Slide 4 text

@[email protected] Context 1/3 Business Applications (e.g. applications where people can buy, rent, book something, perform business transactions) (not: IoT, Big Data…) 4

Slide 5

Slide 5 text

@[email protected] Context 2/3 Communication Between Services 5

Slide 6

Slide 6 text

@[email protected] 6

Slide 7

Slide 7 text

@[email protected] Context 3/3 Assuming a Preference for Event-driven (Reminder: all the advantages of microservices, plus runtime decoupling) 7

Slide 8

Slide 8 text

@[email protected] 8

Slide 9

Slide 9 text

@[email protected] 9

Slide 10

Slide 10 text

@[email protected] 10

Slide 11

Slide 11 text

@[email protected] 11

Slide 12

Slide 12 text

@[email protected] So what have we looked at so far? Implementing a workflow by: - Making one service the coordinator, or - Using a workflow engine for coordination, and - Implementing a saga (can be done with either of the above) What do all approaches have in common? 12

Slide 13

Slide 13 text

@[email protected] - Introduce a (complex) piece of software - Introduce a split responsibility - Are not event-driven 13

Slide 14

Slide 14 text

@[email protected] Technical complexity: - Must be able to resume after crash - Must be able to handle timeouts - Must be distributed to avoid single point of failure - Needs distributed coordination (stateful) - (Some of) your business logic is now in a YAML or XML file 14

Slide 15

Slide 15 text

@[email protected] Organizational complexity: - Business logic is split - Error handling is split, leading to reduced clarity of responsibility if something goes wrong 15

Slide 16

Slide 16 text

@[email protected] 16 ❌ ❌

Slide 17

Slide 17 text

@[email protected] Reminder: Events vs. Commands 17 Is.. Command An intention to perform an operation or change a state Event A fact, something that undisputedly happened in the past

Slide 18

Slide 18 text

@[email protected] Reminder: Events vs. Commands 18 Is.. Expected Response Communication Pattern Command An intention to perform an operation or change a state A confirmation that the command has been executed, or an error message Request-Response Event A fact, something that undisputedly happened in the past None Fire-And-Forget

Slide 19

Slide 19 text

@[email protected] 19

Slide 20

Slide 20 text

@[email protected] 20

Slide 21

Slide 21 text

@[email protected] 21

Slide 22

Slide 22 text

@[email protected] Q: How to implement complex workflows over multiple microservices? A: Don’t. Instead, break up the workflow. To match actual event-driven architecture. Monolith -> Microservices Workflow -> Microworkflows 22

Slide 23

Slide 23 text

@[email protected] 23

Slide 24

Slide 24 text

@[email protected] 24

Slide 25

Slide 25 text

@[email protected] Q: But what about visibility of the end-to-end process? A: Don’t mix control and observability! 25

Slide 26

Slide 26 text

@[email protected] 26

Slide 27

Slide 27 text

@[email protected] Digression: What the “end” is, is very arbitrary. Does it “end” when the parcel is sent? When it has arrived? When the goods have not been returned in 2 weeks? Where does one workflow end, and a new one start? It’s designed. 27

Slide 28

Slide 28 text

@[email protected] Q: But what about timeouts? A: - flaky connections, retries - within service - processing times - SLA - service needs to scale, e.g. based on consumer lag - overall process - observer 28

Slide 29

Slide 29 text

@[email protected] Q: But what about error handling? A: Errors must be handled within a service. The service gives a promise. This is a fundamental decoupling in event-driven architecture. 29

Slide 30

Slide 30 text

@[email protected] Events are “fire-and-forget”... … but based on previous agreements, promises 30

Slide 31

Slide 31 text

@[email protected] Q: Surely, this only works for unrealistic, simplistic examples? A: No, it’s just much harder to fit it on slides in a non-confusing way 󰤇. 31

Slide 32

Slide 32 text

@[email protected] Image: https://blog.codecentric.de/wer-microservices-richtig-macht-braucht-keine-workflow-engine-und-kein-bpmn 32

Slide 33

Slide 33 text

@[email protected] Image: https://blog.codecentric.de/wer-microservices-richtig-macht-braucht-keine-workflow-engine-und-kein-bpmn 33

Slide 34

Slide 34 text

@[email protected] Challenge accepted! Send your workflow to: [email protected] 34

Slide 35

Slide 35 text

@[email protected] Technology is not the problem, it’s a mindset issue. Think Promises. 35

Slide 36

Slide 36 text

@[email protected] Some ideas seem outlandish, but they’re nothing new. Prior art: - Unix pipes - Actors - Stream Processing 36

Slide 37

Slide 37 text

@[email protected] Think Unix Philosophy 1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features". 2. Expect the output of every program to become the input to another, as yet unknown, program. ... 37 $ Cat file3.txt | grep “dwx” | tee file4.txt | wc –l

Slide 38

Slide 38 text

@[email protected] Think Actors 38 Supervisor Actor Client

Slide 39

Slide 39 text

@[email protected] Think Stream Processing 39

Slide 40

Slide 40 text

@[email protected] Enjoy the organizational clarity 40 ❌ ❌

Slide 41

Slide 41 text

@[email protected] Enjoy the (load) testing 41 While you’re freed from having e.g. the workflow engine in your end-to-end test, there are other challenges. Also: Testing a microworkflow is still an integration test. But it’s nice to be able to test e.g. throughput in a very isolated way.

Slide 42

Slide 42 text

42

Slide 43

Slide 43 text

@[email protected] Orchestration vs. Choreography While these terms might be fitting to describe the concepts, most comparisons are not useful, but based on some folkloristic concerns. Hence I tried to avoid them. 43

Slide 44

Slide 44 text

@[email protected] Example Heuristic “Complex workflows..” Problem: Taking a workflow as given, not as something that is designed and can be changed. 44

Slide 45

Slide 45 text

@[email protected] Think reducing complexity, not managing it. https://twitter.com/bitfield/status/1411731604335214592?ref_src=twsrc%5Etfw 45

Slide 46

Slide 46 text

@[email protected] Think Agile. And then putting SAFE on top of it. 46

Slide 47

Slide 47 text

@[email protected] Workflows are useful tools. Workflow engines too, probably. But not for implementing complex workflows on top of Microservices. That use case shouldn’t exist. Consider going event-driven instead. Break the workflow down into microworkflows. 47

Slide 48

Slide 48 text

@[email protected] People are quick to jump to sagas and workflow engines! - Is it someone who’s familiar with it, but not with EDA, and is scared to leave their comfort zone? - Is it someone who wants to sell you something? 48

Slide 49

Slide 49 text

@[email protected] Summary: Workflows and Sagas are at odds with event driven architecture. If you really want to get the benefits of event-driven architecture, go all-in, don’t impose an orchestrator on top of it. Keep your event-driven architecture sane. 49

Slide 50

Slide 50 text

Let’s Discuss! Follow me on Mastodon @[email protected] Follow me on LinkedIn https://linkedin.com/in/lutzh Read my blog https://www.reactivesystems.eu/ 50