Designing process flows with #Baker

Baker is a library that reduces the effort to orchestrate (micro)service-based process flows. Using our intuitive scala/java dsl, developers can declare the orchestration logic in a recipe which is made out of interactions (system calls), ingredients (data) and events. A visual recipe as a graph is also generated so you can easily discuss the logic with other stakeholders, i.e. product owners, testers, etc... The recipe is compiled into a petrinet which is running in an akka actor, and probably in an akka cluster. It is already applied in ING to achieve less spaghetti orchestration code in apis, and more reusability, scalability and resilience.

Transcript

1. Bekir Oguz Merlijn van Ittersum Designing process flows with #Baker Amsterdam.Scala Meetup September 2017
2. How to bake a pizza? 2
3. How to bake a pizza? 3
4. How to bake a pizza? 4
5. Why?
6. It is hard to develop complex orchestration APIs 6
8. We need a cleaner way to allow design changes in complex APIs 8 Example: initial design of the API A B C D E 1 2 3 4 5
9. A B C D E 1 2 3 4 5 We need a cleaner way to allow design changes in complex APIs 9 Example: API after one design change
10. A B C D E 1 2 3 4 5 We need a cleaner way to allow design changes in complex APIs 10 Example: API after ten design changes
11. We need a cleaner way to allow design changes in complex APIs 11 Example: API after ten design changes A B C D E 1 2 3 4 5
12. Open Kinder- / Jongerenrekening Open Groei Groter rekening Another Product Verify customer rules Verify customer rules Verify customer rules Register individual Register individual Register Individual Send a present Send a present Not applicable Send a message to the customer Send a message to the customer Send a message to the customer Open a payments account Open a savings account Open a payments account Enable billing Enable billing Assign package There is more need for reuse between teams 12
13. 13 What?
14. Baker is a Scala library that simplifies the development of complex orchestration logic. What? Baker! github.com/ing-bank/baker
15. Intermediate Language validate compile execute Baker runtime
16. Recipe Event Ingredient Interaction
17. Defining ingredients 17 case class CustomerInfo(name: String, address: String, email: String) // ingredients val customerInfo = Ingredient[CustomerInfo]("customerInfo") val trackingId = Ingredient[String]("trackingId") val order = Ingredient[String]("order")
18. Defining events 18 // events val goodsShipped = Event("GoodsShipped", trackingId) val orderPlaced = Event("OrderPlaced", order) val paymentMade = Event("PaymentMade") val valid = Event("Valid") val sorry = Event("Sorry")
19. Defining interactions 19 // interactions val validateOrder = Interaction( name = "ValidateOrder", inputIngredients = order, output = FiresOneOfEvents(valid, sorry) ) val shipGoods = Interaction( name = "ShipGoods", inputIngredients = Ingredients(goods, customerInfo), output = FiresOneOfEvents(goodsShipped) )
20. Defining a recipe 20 // recipe val webShopRecipe: Recipe = Recipe("WebShop") .withInteractions( validateOrder, manufactureGoods .withRequiredEvents(valid, paymentMade), shipGoods, sendInvoice .withRequiredEvent(goodsShipped) ) .withSensoryEvents( customerInfoReceived, orderPlaced, paymentMade)
21. Compile the recipe 21 // compiles the recipe val compiledRecipe = RecipeCompiler.compileRecipe(webShopRecipe) // list of validation errors val errors: Seq[String] = compiledRecipe.validationErrors // .dot (graphviz notation) representation val visualization: String = compiledRecipe.getRecipeVisualization
22. Visualising the recipe 22
23. A Baker instance can be created from the valid recipe. It takes care of the execution of the recipe. The only thing left is to provide sensory events to Baker! Baker runtime 23
24. Initializing Baker instance 24 You can create a new Baker instance by providing the recipe and implementations of the interactions: val compiledRecipe = RecipeCompiler.compileRecipe(webShopRecipe) val validateOrderImpl = validateOrder implement { (order: String) => { // Some logic here valid.instance() // or maybe invalid event to be returned } } val implementations = Seq(validateOrderImpl, manufactureGoodsImpl, sendInvoiceImpl, shipGoodsImpl) val baker = new Baker(compiledRecipe, implementations)
25. For each client request you do the following: Firing events 25 val processId = UUID.randomUUID().toString baker.bake(processId) baker.handleEvent(processId, orderPlaced.instance(testOrder)) baker.handleEvent(processId, paymentMade.instance()) baker.handleEvent(processId, customerInfoReceived.instance(testCustomerInfoData)) val actualIngredients: Map[String, Any] = baker.getIngredients(processId) val actualEvents: Seq[RuntimeEvent] = baker.events(processId)
42. 1. It is hard to develop complex orchestration APIs 2. We need a cleaner way to allow design changes in complex APIs 3. There is more need for reuse between teams Does Baker help with the Why? 42
43. 1. It is hard to develop complex orchestration APIs With Baker, you think about what you want, not about the order. Baker can validate the recipe for you. Clear insight in the current model by visualising the recipe. 2. We need a cleaner way to allow design changes in complex APIs 3. There is more need for reuse between teams Does Baker help with the Why? 43
44. 1. It is hard to develop complex orchestration APIs With Baker, you think about what you want, not about the order. Baker can validate the recipe for you. Clear insight in the current model by visualising the recipe. 2. We need a cleaner way to allow design changes in complex APIs Baker links the interactions when compiling the recipe. If interactions are added, removed or changed, Baker will update the model accordingly. 3. There is more need for reuse between teams Does Baker help with the Why? 44
45. 1. It is hard to develop complex orchestration APIs With Baker, you think about what you want, not about the order. Baker can validate the recipe for you. Clear insight in the current model by visualising the recipe. 2. We need a cleaner way to allow design changes in complex APIs Baker links the interactions when compiling the recipe. If interactions are added, removed or changed, Baker will update the model accordingly. 3. There is more need for reuse between teams The possibility to share interactions, ingredients and events between recipes allows for much more reuse of code among teams. Does Baker help with the Why? 45
46. How?
47. Baker is powered by 47 PetriNet
48. Process logic is isolated in an Akka actor, this gives us the following benefits: • Event sourcing through persistent actors • Supporting many data stores, i.e. cassandra • Distributing actors with Cluster Sharding Akka 48
49. On creation of the Baker instance, the recipe is compiled into a Petri net. A Petri net is a mathematical model that can be used for reasoning about concurrency. This Petri net is then used to determine when & what interactions to execute. PetriNet 49
50. Execution Semantics: Token passing 50 t1 t2 t3
51. Execution Semantics: Token passing 51 t1 t2 t3
52. Execution Semantics: Token passing 52 t1 t2 t3
53. Execution Semantics: Token passing 53 t1 t2 t3
54. Execution Semantics: Concurrency 54 t1 t3 t4 t5 t2
55. Execution Semantics: Concurrency 55 t1 t3 t4 t5 t2 Choice
56. Execution Semantics: Concurrency 56 t1 t3 t4 t5 t2 Parallel split
57. Execution Semantics: Concurrency 57 t1 t3 t4 t5 t2 Parallel paths
58. Execution Semantics: Concurrency 58 t1 t3 t4 t5 t2 Parallel paths
59. Execution Semantics: Concurrency 59 t1 t3 t4 t5 t2 synchronization
60. • Simple, uses only a few concepts and components • Exact mathematical definition of execution semantics • Intuitive graphical notation • Well-developed mathematical theory for process analysis Advantages 60
61. Reachability 61 ? A B
62. Unreachable code 62 A B
63. Detect cycles 63
64. Path traversal time 64 Average traversal time: 10 seconds A B
65. Identifying bottlenecks 65 Takes 70% of the time A B
66. Path prediction 66 83% 17% A B
67. 67 Features Recipe Runtime
68. 68 Features Recipe Developer friendly Java/Scala dsl Runtime
69. 69 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Runtime
70. 70 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime
71. 71 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime Execution of the recipe, powered by akka and petrinet model
72. 72 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime Execution of the recipe, powered by akka and petrinet model Persisted state through event sourcing
73. 73 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime Execution of the recipe, powered by akka and petrinet model Persisted state through event sourcing Data encryption
74. 74 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime Execution of the recipe, powered by akka and petrinet model Persisted state through event sourcing Data encryption Automatic retries on technical failures
75. 75 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime Execution of the recipe, powered by akka and petrinet model Persisted state through event sourcing Data encryption Automatic retries on technical failures Parallel execution of paths
76. Property Based Testing 76
77. Property Based Testing 77 val recipeGen: Gen[Recipe] = for { name <- nameGen sensoryEvents <- Gen.listOf(eventGen) suchThat (_.nonEmpty) interactions <- interactionsGen(sensoryEvents) suchThat (_.nonEmpty) } yield Recipe(name) //turn the lists into var args .withSensoryEvents(sensoryEvents: _*) .withInteractions(interactions.toList: _*)
78. Property Based Testing 78 forAll(recipeGen) { recipe => val validations = ValidationSettings( allowCycles = false, allowNonExecutableInteractions = false) val compiledRecipe = RecipeCompiler.compileRecipe(recipe, validations) if (compiledRecipe.validationErrors.nonEmpty) { logRecipeStats(recipe) logCompiledRecipeStats(compiledRecipe) } compiledRecipe.validationErrors.isEmpty }
79. Questions?