Tweet
Tweet

Slide 1

Slide 1 text

Microworkflows for Microservices Complex Workflows in Distributed Applications 1

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

@lutzhuehnken@mastodon.social Why me? - Built my first message-driven, asynchronous system for the Bundesbank starting 2001 - Worked on various event-driven systems, e.g. for Zalando, ING, ista, Maersk - Currently building a bank (!) with Event-driven Architecture, Microservices, DDD 3

Slide 4

Slide 4 text

@lutzhuehnken@mastodon.social Context 1/3 Business Applications (e.g. applications where people can buy, rent, book something, perform business transactions) (not: IoT, Big Data…) 4

Slide 5

Slide 5 text

@lutzhuehnken@mastodon.social Context 2/3 Communication Between Services 5

Slide 6

Slide 6 text

@lutzhuehnken@mastodon.social 6

Slide 7

Slide 7 text

@lutzhuehnken@mastodon.social Context 3/3 Assuming a Preference for Event-driven (Reminder: all the advantages of microservices, plus runtime decoupling) 7

Slide 8

Slide 8 text

@lutzhuehnken@mastodon.social 8

Slide 9

Slide 9 text

@lutzhuehnken@mastodon.social 9

Slide 10

Slide 10 text

@lutzhuehnken@mastodon.social 10

Slide 11

Slide 11 text

@lutzhuehnken@mastodon.social 11

Slide 12

Slide 12 text

@lutzhuehnken@mastodon.social So what have we looked at so far? Implementing a workflow by: - Making one service the coordinator, or - Using a workflow engine for coordination, and - Implementing a saga (can be done with either of the above) What do all approaches have in common? 12

Slide 13

Slide 13 text

@lutzhuehnken@mastodon.social - Introduce a (complex) piece of software - Introduce a split responsibility - Are not event-driven 13

Slide 14

Slide 14 text

@lutzhuehnken@mastodon.social Technical complexity: - Must be able to resume after crash - Must be able to handle timeouts - Must be distributed to avoid single point of failure - Needs distributed coordination (stateful) - (Some of) your business logic is now in a YAML or XML file 14

Slide 15

Slide 15 text

@lutzhuehnken@mastodon.social Organizational complexity: - Business logic is split - Error handling is split, leading to reduced clarity of responsibility if something goes wrong 15

Slide 16

Slide 16 text

@lutzhuehnken@mastodon.social 16 ❌ ❌

Slide 17

Slide 17 text

@lutzhuehnken@mastodon.social Reminder: Events vs. Commands 17 Is.. Command An intention to perform an operation or change a state Event A fact, something that undisputedly happened in the past

Slide 18

Slide 18 text

@lutzhuehnken@mastodon.social Reminder: Events vs. Commands 18 Is.. Expected Response Communication Pattern Command An intention to perform an operation or change a state A confirmation that the command has been executed, or an error message Request-Response Event A fact, something that undisputedly happened in the past None Fire-And-Forget

Slide 19

Slide 19 text

@lutzhuehnken@mastodon.social 19

Slide 20

Slide 20 text

@lutzhuehnken@mastodon.social 20

Slide 21

Slide 21 text

@lutzhuehnken@mastodon.social 21

Slide 22

Slide 22 text

@lutzhuehnken@mastodon.social Q: How to implement complex workflows over multiple microservices? A: Don’t. Instead, break up the workflow. To match actual event-driven architecture. Monolith -> Microservices Workflow -> Microworkflows 22

Slide 23

Slide 23 text

@lutzhuehnken@mastodon.social 23

Slide 24

Slide 24 text

@lutzhuehnken@mastodon.social 24

Slide 25

Slide 25 text

@lutzhuehnken@mastodon.social Q: But what about visibility of the end-to-end process? A: Don’t mix control and observability! 25

Slide 26

Slide 26 text

@lutzhuehnken@mastodon.social 26

Slide 27

Slide 27 text

@lutzhuehnken@mastodon.social Digression: What the “end” is, is very arbitrary. Does it “end” when the parcel is sent? When it has arrived? When the goods have not been returned in 2 weeks? Where does one workflow end, and a new one start? It’s designed. 27

Slide 28

Slide 28 text

@lutzhuehnken@mastodon.social Q: But what about timeouts? A: - flaky connections, retries - within service - processing times - SLA - service needs to scale, e.g. based on consumer lag - overall process - observer 28

Slide 29

Slide 29 text

@lutzhuehnken@mastodon.social Q: But what about error handling? A: Errors must be handled within a service. The service gives a promise. This is a fundamental decoupling in event-driven architecture. 29

Slide 30

Slide 30 text

@lutzhuehnken@mastodon.social Events are “fire-and-forget”... … but based on previous agreements, promises 30

Slide 31

Slide 31 text

@lutzhuehnken@mastodon.social Q: Surely, this only works for unrealistic, simplistic examples? A: No, it’s just much harder to fit it on slides in a non-confusing way 󰤇. 31

Slide 32

Slide 32 text

@lutzhuehnken@mastodon.social Image: https://blog.codecentric.de/wer-microservices-richtig-macht-braucht-keine-workflow-engine-und-kein-bpmn 32

Slide 33

Slide 33 text

@lutzhuehnken@mastodon.social Image: https://blog.codecentric.de/wer-microservices-richtig-macht-braucht-keine-workflow-engine-und-kein-bpmn 33

Slide 34

Slide 34 text

@lutzhuehnken@mastodon.social Challenge accepted! Send your workflow to: challenge@microworkflows.org 34

Slide 35

Slide 35 text

@lutzhuehnken@mastodon.social Technology is not the problem, it’s a mindset issue. Think Promises. 35

Slide 36

Slide 36 text

@lutzhuehnken@mastodon.social Some ideas seem outlandish, but they’re nothing new. Prior art: - Unix pipes - Actors - Stream Processing 36

Slide 37

Slide 37 text

@lutzhuehnken@mastodon.social Think Unix Philosophy 1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features". 2. Expect the output of every program to become the input to another, as yet unknown, program. ... 37 $ Cat file3.txt | grep “dwx” | tee file4.txt | wc –l

Slide 38

Slide 38 text

@lutzhuehnken@mastodon.social Think Actors 38 Supervisor Actor Client

Slide 39

Slide 39 text

@lutzhuehnken@mastodon.social Think Stream Processing 39

Slide 40

Slide 40 text

@lutzhuehnken@mastodon.social Enjoy the organizational clarity 40 ❌ ❌

Slide 41

Slide 41 text

@lutzhuehnken@mastodon.social Enjoy the (load) testing 41 While you’re freed from having e.g. the workflow engine in your end-to-end test, there are other challenges. Also: Testing a microworkflow is still an integration test. But it’s nice to be able to test e.g. throughput in a very isolated way.

Slide 42

Slide 42 text

42

Slide 43

Slide 43 text

@lutzhuehnken@mastodon.social Orchestration vs. Choreography While these terms might be fitting to describe the concepts, most comparisons are not useful, but based on some folkloristic concerns. Hence I tried to avoid them. 43

Slide 44

Slide 44 text

@lutzhuehnken@mastodon.social Example Heuristic “Complex workflows..” Problem: Taking a workflow as given, not as something that is designed and can be changed. 44

Slide 45

Slide 45 text

@lutzhuehnken@mastodon.social Think reducing complexity, not managing it. https://twitter.com/bitfield/status/1411731604335214592?ref_src=twsrc%5Etfw 45

Slide 46

Slide 46 text

@lutzhuehnken@mastodon.social Think Agile. And then putting SAFE on top of it. 46

Slide 47

Slide 47 text

@lutzhuehnken@mastodon.social Workflows are useful tools. Workflow engines too, probably. But not for implementing complex workflows on top of Microservices. That use case shouldn’t exist. Consider going event-driven instead. Break the workflow down into microworkflows. 47

Slide 48

Slide 48 text

@lutzhuehnken@mastodon.social People are quick to jump to sagas and workflow engines! - Is it someone who’s familiar with it, but not with EDA, and is scared to leave their comfort zone? - Is it someone who wants to sell you something? 48

Slide 49

Slide 49 text

@lutzhuehnken@mastodon.social Summary: Workflows and Sagas are at odds with event driven architecture. If you really want to get the benefits of event-driven architecture, go all-in, don’t impose an orchestrator on top of it. Keep your event-driven architecture sane. 49

Slide 50

Slide 50 text

Let’s Discuss! Follow me on Mastodon @lutzhuehnken@mastodon.social Follow me on LinkedIn https://linkedin.com/in/lutzh Read my blog https://www.reactivesystems.eu/ 50