Introducing The Instruction Pipeline

Slide 1

Slide 1 text

A presentation by @stuherbert  for @GanbaroDigital Introducing The Instruction Pipeline How To Design And Build  Bespoke Middleware

Slide 2

Slide 2 text

@GanbaroDigital I’m going to show you how to build a pipeline that’s built to your specs.

Slide 3

Slide 3 text

@GanbaroDigital We’re going to build a real-life, production-ready bespoke pipeline.

Slide 4

Slide 4 text

@GanbaroDigital Full design process, writing the code, and testing: all in around 45 mins.

Slide 5

Slide 5 text

@GanbaroDigital We’re going to build it using an open-source package: ganbarodigital/ php-mv-instruction-pipeline

Slide 6

Slide 6 text

@GanbaroDigital If you want to read along: https://github.com/ganbarodigital/  php-mv-messaging-pipeline

Slide 7

Slide 7 text

@GanbaroDigital What Is A Pipeline?

Slide 8

Slide 8 text

@GanbaroDigital “ A pipeline is an assembly of isolated modules, for data to ﬂow along.

Slide 9

Slide 9 text

@GanbaroDigital “ A pipeline is an assembly of isolated modules, for data to ﬂow along.

Slide 10

Slide 10 text

@GanbaroDigital Module 1 Module 2 Module 3 Module 4

Slide 11

Slide 11 text

@GanbaroDigital “ A pipeline is an assembly of isolated modules, for data to ﬂow along.

Slide 12

Slide 12 text

@GanbaroDigital Module 1 Module 2 Module 3 Module 4

Slide 13

Slide 13 text

@GanbaroDigital “ A pipeline is an assembly of isolated modules, for data to ﬂow along.

Slide 14

Slide 14 text

@GanbaroDigital Module 1 Module 2 Module 3 Module 4

Slide 15

Slide 15 text

@GanbaroDigital • Pipelines are ordered • We decide the order of the modules on the pipeline

Slide 16

Slide 16 text

@GanbaroDigital • Pipelines are ordered • We decide the order of the modules on the pipeline

Slide 17

Slide 17 text

@GanbaroDigital Pipelines are perfect if you need ﬂexibility and the ability to respond rapidly to change.

Slide 18

Slide 18 text

@GanbaroDigital “ We can chop and change the modules - and their order - without having to touch the modules themselves

Slide 19

Slide 19 text

@GanbaroDigital “ We can chop and change the modules - and their order - without having to touch the modules themselves

Slide 20

Slide 20 text

@GanbaroDigital Module 1 Module 2 Module 3 Module 4

Slide 21

Slide 21 text

@GanbaroDigital Module 1 Module 2 Module 3 Module 4

Slide 22

Slide 22 text

@GanbaroDigital “ We can chop and change the modules - and their order - without having to touch the modules themselves

Slide 23

Slide 23 text

@GanbaroDigital Module 1 Module 2 Module 3 Module 4

Slide 24

Slide 24 text

@GanbaroDigital Module 1 Module 5 Module 3 Module 4

Slide 25

Slide 25 text

@GanbaroDigital Module 1 Module 3 Module 4

Slide 26

Slide 26 text

@GanbaroDigital “ We can chop and change the modules - and their order - without having to touch the modules themselves

Slide 27

Slide 27 text

@GanbaroDigital The pipeline itself is deﬁned in conﬁg, not code.

Slide 28

Slide 28 text

@GanbaroDigital

Slide 29

Slide 29 text

@GanbaroDigital ?? ?? What Happens To The Data?

Slide 30

Slide 30 text

@GanbaroDigital Data In A Pipeline Can Be … 1. modiﬁed 2. transformed 3. or rejected

Slide 31

Slide 31 text

@GanbaroDigital Data In A Pipeline Can Be … 1. modiﬁed 2. transformed 3. or rejected

Slide 32

Slide 32 text

@GanbaroDigital Data In A Pipeline Can Be … 1. modiﬁed 2. transformed 3. or rejected

Slide 33

Slide 33 text

@GanbaroDigital PSR7 Middleware is a data transformation pipeline: It turns HTTP requests into HTTP responses.

Slide 34

Slide 34 text

@GanbaroDigital Building A Pipeline

Slide 35

Slide 35 text

@GanbaroDigital Real-life use case: messaging in the #cloud

Slide 36

Slide 36 text

@GanbaroDigital Back Ofﬁce Service Public  Web App API

Slide 37

Slide 37 text

@GanbaroDigital Back Ofﬁce Service Public  Web App API Status Events

Slide 38

Slide 38 text

@GanbaroDigital Back Ofﬁce Service Public  Web App API Status Events Message Queue

Slide 39

Slide 39 text

@GanbaroDigital Design Constraints 1. We must use the cloud-provider’s message queueing service 2. Different programming languages in use for transmitting and receiving systems

Slide 40

Slide 40 text

@GanbaroDigital Principle: Always treat the cloud-provider’s infrastructure as an untrusted third-party system.

Slide 41

Slide 41 text

@GanbaroDigital Trafﬁc to and from the message queue goes over HTTPS. The messages are encrypted on the wire / in transit.

Slide 42

Slide 42 text

@GanbaroDigital Messages at rest inside the queue? Not guaranteed to be encrypted.

Slide 43

Slide 43 text

@GanbaroDigital Encrypt

Slide 44

Slide 44 text

@GanbaroDigital Encrypt HMAC

Slide 45

Slide 45 text

@GanbaroDigital The message queue does not accept arbitrary binary data. Only 7-bit ASCII. Their castle, their rules.

Slide 46

Slide 46 text

@GanbaroDigital Encrypt HMAC

Slide 47

Slide 47 text

@GanbaroDigital Encrypt HMAC Ascii-Safe

Slide 48

Slide 48 text

@GanbaroDigital Design Constraints 1. We must use the cloud-provider’s message queueing service 2. Different programming languages in use for transmitting and receiving systems

Slide 49

Slide 49 text

@GanbaroDigital Encrypt HMAC Ascii-Safe

Slide 50

Slide 50 text

@GanbaroDigital JSON Encrypt HMAC Ascii-Safe

Slide 51

Slide 51 text

@GanbaroDigital In real life, the requirements to encrypt came late in the day.

Slide 52

Slide 52 text

@GanbaroDigital JSON Ascii-Safe

Slide 53

Slide 53 text

@GanbaroDigital Because We Built A Pipeline • Created new modules • Dropped them in where we needed them • Zero changes to signed-off code

Slide 54

Slide 54 text

@GanbaroDigital JSON Ascii-Safe Encrypt HMAC

Slide 55

Slide 55 text

@GanbaroDigital JSON Encrypt HMAC Ascii-Safe

Slide 56

Slide 56 text

@GanbaroDigital Because We Built A Pipeline • Created new modules • Dropped them in where we needed them • Zero changes to signed-off code • Reduced impact on deployment pipeline

Slide 57

Slide 57 text

@GanbaroDigital Requirements drive the design of your pipelines.

Slide 58

Slide 58 text

@GanbaroDigital Let’s break down how to implement your requirements.

Slide 59

Slide 59 text

@GanbaroDigital JSON Encrypt HMAC Ascii-Safe

Slide 60

Slide 60 text

@GanbaroDigital JSON Encrypt HMAC Ascii-Safe Not the code that runs inside the pipeline!

Slide 61

Slide 61 text

@GanbaroDigital JSON Encrypt HMAC Ascii-Safe Instruction Builders

Slide 62

Slide 62 text

@GanbaroDigital “Instruction Builders” are factories that instantiate pipeline “Instructions”.

Slide 63

Slide 63 text

@GanbaroDigital “Instructions” are the individual modules added to each pipeline.

Slide 64

Slide 64 text

@GanbaroDigital Scope “Instructions” to be small and ﬂexible.

Slide 65

Slide 65 text

@GanbaroDigital JSON Encrypt HMAC Ascii-Safe Instruction Builders

Slide 66

Slide 66 text

@GanbaroDigital Encode To JSON Encrypt Add HMAC Encode To Ascii-Safe Instructions

Slide 67

Slide 67 text

@GanbaroDigital Encode To JSON Encrypt Add HMAC Encode To Ascii-Safe Instructions Decode From Ascii-Safe Verify & Strip HMAC Decrypt Decode From JSON Transmit Receive

Slide 68

Slide 68 text

@GanbaroDigital “Instruction Builders” instantiate Instructions for both transmit and receive pipelines.

Slide 69

Slide 69 text

@GanbaroDigital (Yes, you can build just the one pipeline if you need to)

Slide 70

Slide 70 text

@GanbaroDigital Encrypt Instruction Builder HMAC Ascii-Safe JSON Transmit Pipeline Receive Pipeline

Slide 71

Slide 71 text

@GanbaroDigital Encrypt Instruction Builder HMAC Encode To JSON Ascii-Safe JSON Decode From JSON Transmit Pipeline Receive Pipeline

Slide 72

Slide 72 text

@GanbaroDigital Encrypt Add HMAC Encode To Ascii-Safe Instruction Builder HMAC Verify & Strip HMAC Encode To JSON Decode From Ascii-Safe Ascii-Safe JSON Decode From JSON Encrypt Decrypt Transmit Pipeline Receive Pipeline

Slide 73

Slide 73 text

@GanbaroDigital We have discovered the individual Instructions that we need to code up.

Slide 74

Slide 74 text

@GanbaroDigital ?? ?? So we’re ready to jump into writing the code?!?

Slide 75

Slide 75 text

@GanbaroDigital There’s one more level of decomposition to do ﬁrst. We need to break down our Instructions.

Slide 76

Slide 76 text

@GanbaroDigital There’s one more level of decomposition to do ﬁrst. We need to break down our Instructions.

Slide 77

Slide 77 text

@GanbaroDigital Treat your Instructions like you would treat your MVC controllers.

Slide 78

Slide 78 text

Slide 79

Slide 79 text

Slide 80

Slide 80 text

@GanbaroDigital Instruction Add HMAC Operation(s)

Slide 81

Slide 81 text

@GanbaroDigital Instruction Calc HMAC Add HMAC Operation(s) Add HMAC

Slide 82

Slide 82 text

@GanbaroDigital Instruction Verify & Strip HMAC Calc HMAC Add HMAC Operation(s) Add HMAC

Slide 83

Slide 83 text

@GanbaroDigital Instruction Verify & Strip HMAC Calc HMAC Add HMAC Find HMAC Remove HMAC Operation(s) Verify HMAC Add HMAC Calc HMAC

Slide 84

Slide 84 text

@GanbaroDigital Instruction Verify & Strip HMAC Calc HMAC Add HMAC Find HMAC Remove HMAC Operation(s) Verify HMAC Add HMAC Calc HMAC

Slide 85

Slide 85 text

@GanbaroDigital “Operations” are the smallest units of work that it makes sense to reuse.

Slide 86

Slide 86 text

@GanbaroDigital “Operations” have no dependency whatsoever on the pipeline framework.

Slide 87

Slide 87 text

@GanbaroDigital NOW we are ready to write some code :)

Slide 88

Slide 88 text

@GanbaroDigital

Slide 89

Slide 89 text

@GanbaroDigital

Slide 90

Slide 90 text

@GanbaroDigital

Slide 91

Slide 91 text

@GanbaroDigital

Slide 92

Slide 92 text

@GanbaroDigital

Slide 93

Slide 93 text

@GanbaroDigital

Slide 94

Slide 94 text

@GanbaroDigital

Slide 95

Slide 95 text

@GanbaroDigital

Slide 96

Slide 96 text

@GanbaroDigital Operations are very boring. They’re just plain PHP code. In my experience, boring code is reliable code.

Slide 97

Slide 97 text

@GanbaroDigital Instruction Verify & Strip HMAC Calc HMAC Add HMAC Find HMAC Remove HMAC Operation(s) Verify HMAC Add HMAC Calc HMAC

Slide 98

Slide 98 text

@GanbaroDigital Instruction Verify & Strip HMAC Calc HMAC Add HMAC Find HMAC Remove HMAC Operation(s) Verify HMAC Add HMAC Calc HMAC

Slide 99

Slide 99 text

@GanbaroDigital Our Operations are ready to run. We can go ahead and code up our Instructions.

Slide 100

Slide 100 text

@GanbaroDigital Instruction Verify & Strip HMAC Calc HMAC Add HMAC Find HMAC Remove HMAC Operation(s) Verify HMAC Add HMAC Calc HMAC

Slide 101

Slide 101 text

@GanbaroDigital Instructions aggregate Operations to get the work done.

Slide 102

Slide 102 text

@GanbaroDigital Aggregation vs Inheritance: aggregation for reuse, inheritance for specialisation

Slide 103

Slide 103 text

@GanbaroDigital

Slide 104

Slide 104 text

@GanbaroDigital

Slide 105

Slide 105 text

@GanbaroDigital

Slide 106

Slide 106 text

@GanbaroDigital

Slide 107

Slide 107 text

@GanbaroDigital

Slide 108

Slide 108 text

@GanbaroDigital

Slide 109

Slide 109 text

@GanbaroDigital

Slide 110

Slide 110 text

@GanbaroDigital

Slide 111

Slide 111 text

@GanbaroDigital

Slide 112

Slide 112 text

@GanbaroDigital Thanks to identifying and building the Operations, the Instructions are very lean and very clean.

Slide 113

Slide 113 text

@GanbaroDigital Unlike the Operations, the Instructions won’t execute yet. They depend upon NextInstruction.

Slide 114

Slide 114 text

@GanbaroDigital Let’s talk about (and build) the NextInstruction class.

Slide 115

Slide 115 text

@GanbaroDigital NextInstruction is the secret sauce that powers Instruction pipelines!

Slide 116

Slide 116 text

@GanbaroDigital Encode To JSON Encrypt Add HMAC Encode To Ascii-Safe Instructions

Slide 117

Slide 117 text

@GanbaroDigital NextInstruction Encode To JSON Encrypt Add HMAC Encode To Ascii-Safe

Slide 118

Slide 118 text

@GanbaroDigital $next->process() Encode To JSON Encrypt Add HMAC Encode To Ascii-Safe

Slide 119

Slide 119 text

@GanbaroDigital Encode To JSON Encrypt Add HMAC Encode To Ascii-Safe $next->process() $next->process()

Slide 120

Slide 120 text

@GanbaroDigital $next->process() Encode To JSON Encrypt Add HMAC Encode To Ascii-Safe $next->process() $next->process() $next->process() $next->process()

Slide 121

Slide 121 text

@GanbaroDigital $next->process() Encode To JSON Encrypt Add HMAC Encode To Ascii-Safe $next->process() $next->process() $next->process()

Slide 122

Slide 122 text

@GanbaroDigital $next->process() Encode To JSON Encrypt Add HMAC $next->process() $next->process() $next->process()

Slide 123

Slide 123 text

@GanbaroDigital $next->process() Encode To JSON

Slide 124

Slide 124 text

@GanbaroDigital $next->process()

Slide 125

Slide 125 text

@GanbaroDigital

Slide 126

Slide 126 text

@GanbaroDigital We’re now starting to see how to use the underlying InstructionPipeline framework. ganbarodigital/ php-mv-instruction-pipeline

Slide 127

Slide 127 text

@GanbaroDigital

Slide 128

Slide 128 text

@GanbaroDigital

Slide 129

Slide 129 text

Slide 130

Slide 130 text

Slide 131

Slide 131 text

@GanbaroDigital Now that we have working Instructions, we can build out their factories: the “Instruction Builders”

Slide 132

Slide 132 text

Slide 133

Slide 133 text

@GanbaroDigital

Slide 134

Slide 134 text

@GanbaroDigital

Slide 135

Slide 135 text

@GanbaroDigital

Slide 136

Slide 136 text

@GanbaroDigital GenericInstructionBuilder expects your Instruction to take a conﬁg array as its only parameter.

Slide 137

Slide 137 text

@GanbaroDigital

Slide 138

Slide 138 text

@GanbaroDigital

Slide 139

Slide 139 text

Slide 140

Slide 140 text

@GanbaroDigital Follow the same process (Operations > Instructions > Builders) to complete the pipeline’s code.

Slide 141

Slide 141 text

Slide 142

Slide 142 text

@GanbaroDigital How do we execute the Instruction Builders to assemble the pipelines? We need to build one last helper class.

Slide 143

Slide 143 text

@GanbaroDigital

Slide 144

Slide 144 text

@GanbaroDigital

Slide 145

Slide 145 text

@GanbaroDigital

Slide 146

Slide 146 text

@GanbaroDigital

Slide 147

Slide 147 text

@GanbaroDigital

Slide 148

Slide 148 text

@GanbaroDigital We’re done building :)

Slide 149

Slide 149 text

@GanbaroDigital We Have Built … • 11 x Operations (inc base classes) • 10 x Instructions • 4 x Instruction Builders

Slide 150

Slide 150 text

@GanbaroDigital We Have Built … • 11 x Operations (inc base classes) • 10 x Instructions • 4 x Instruction Builders

Slide 151

Slide 151 text

@GanbaroDigital We Have Built … • 11 x Operations (inc base classes) • 10 x Instructions • 4 x Instruction Builders

Slide 152

Slide 152 text

@GanbaroDigital They Are Supported By … • 8 x Robustness helpers   (Checks & Requirements) • 11 x Exceptions • 2 x Type-safety Interfaces

Slide 153

Slide 153 text

@GanbaroDigital They Are Supported By … • 8 x Robustness helpers   (Checks & Requirements) • 11 x Exceptions • 2 x Type-safety Interfaces

Slide 154

Slide 154 text

@GanbaroDigital They Are Supported By … • 8 x Robustness helpers   (Checks & Requirements) • 11 x Exceptions • 2 x Type-safety Interfaces

Slide 155

Slide 155 text

@GanbaroDigital Let’s see it all together.

Slide 156

Slide 156 text

@GanbaroDigital

Slide 157

Slide 157 text

@GanbaroDigital

Slide 158

Slide 158 text

@GanbaroDigital

Slide 159

Slide 159 text

@GanbaroDigital

Slide 160

Slide 160 text

@GanbaroDigital

Slide 161

Slide 161 text

@GanbaroDigital

Slide 162

Slide 162 text

@GanbaroDigital

Slide 163

Slide 163 text

@GanbaroDigital

Slide 164

Slide 164 text

@GanbaroDigital

Slide 165

Slide 165 text

@GanbaroDigital Build Order 1. Operations 2. NextInstruction & build helper 3. Instructions 4. Instruction Builders

Slide 166

Slide 166 text

@GanbaroDigital Type Safety

Slide 167

Slide 167 text

@GanbaroDigital PSR7 (HTTP Messages) and the associated Middleware have been loudly criticised for lack of type-safety.

Slide 168

Slide 168 text

@GanbaroDigital declare(strict_types=1); Enables enforcement of PHP 7 type declarations.

Slide 169

Slide 169 text

@GanbaroDigital “ Types are policy. Policy belongs in your code, not in underlying libraries.

Slide 170

Slide 170 text

@GanbaroDigital ?? ?? How hard is it to add type declarations to a library like MessagingPipeline?

Slide 171

Slide 171 text

@GanbaroDigital There are no problems adding type declarations to Operations. They’re just plain PHP.

Slide 172

Slide 172 text

@GanbaroDigital

Slide 173

Slide 173 text

@GanbaroDigital Instructions … well, it depends on what your pipeline does to the data it processes.

Slide 174

Slide 174 text

@GanbaroDigital Data In A Pipeline Can Be … 1. modiﬁed 2. transformed 3. or rejected

Slide 175

Slide 175 text

@GanbaroDigital When we transform data, we change its data type.

Slide 176

Slide 176 text

@GanbaroDigital Instructions are isolated from each other. They cannot know what data transformations will happen elsewhere on the pipeline.

Slide 177

Slide 177 text

@GanbaroDigital The return type from your pipeline’s last Instruction determines how type-strict you can be.

Slide 178

Slide 178 text

@GanbaroDigital Encrypt Instruction Builder HMAC Ascii-Safe JSON Transmit Pipeline Receive Pipeline mixed > string string > string string > string string > string string > string string > mixed string > string string > string

Slide 179

Slide 179 text

@GanbaroDigital You can deﬁne interfaces to enforce type safety on your Instructions.

Slide 180

Slide 180 text

Slide 181

Slide 181 text

@GanbaroDigital

Slide 182

Slide 182 text

Slide 183

Slide 183 text

@GanbaroDigital

Slide 184

Slide 184 text

@GanbaroDigital NextInstruction also enforces type safety. It’s unique to your library. Only Instructions written for your library will accept your NextInstruction.

Slide 185

Slide 185 text

@GanbaroDigital

Slide 186

Slide 186 text

@GanbaroDigital ?? ?? Did you notice the runtime checks that we still needed?

Slide 187

Slide 187 text

@GanbaroDigital

Slide 188

Slide 188 text

@GanbaroDigital Strict typing is the equivalent of ‘ﬁlters’ from ‘input ﬁltering and validation’. Strict typing doesn’t cover your data validation responsibilities.

Slide 189

Slide 189 text

@GanbaroDigital You still need appropriate runtime checks. You still need appropriate unit tests.

Slide 190

Slide 190 text

@GanbaroDigital Testing

Slide 191

Slide 191 text

@GanbaroDigital Instruction Pipelines are easy to test.

Slide 192

Slide 192 text

@GanbaroDigital Unit Test Build Order 1. Operations 2. Instructions 3. NextInstruction & build helper

Slide 193

Slide 193 text

@GanbaroDigital Operations are plain PHP, by design. There’s nothing special required to test them.

Slide 194

Slide 194 text

@GanbaroDigital

Slide 195

Slide 195 text

@GanbaroDigital Unit Test Build Order 1. Operations 2. Instructions 3. NextInstruction & build helper

Slide 196

Slide 196 text

@GanbaroDigital

Slide 197

Slide 197 text

@GanbaroDigital Unit Test Build Order 1. Operations 2. Instructions 3. NextInstruction & build helper

Slide 198

Slide 198 text

@GanbaroDigital They have single responsibilities. We don’t need many tests to prove that they work.

Slide 199

Slide 199 text

@GanbaroDigital

Slide 200

Slide 200 text

@GanbaroDigital

Slide 201

Slide 201 text

@GanbaroDigital

Slide 202

Slide 202 text

@GanbaroDigital “With 100% code coverage, you could still ship bugs. Without 100% code coverage, you’re deﬁnitely shipping bugs.

Slide 203

Slide 203 text

@GanbaroDigital Full code available on GitHub & Packagist: ganbarodigital/ php-mv-messaging-pipeline

Slide 204

Slide 204 text

Thank You Any Questions? A presentation by @stuherbert  for @GanbaroDigital