Tweet
Tweet

Slide 1

Slide 1 text

Evolving Schemas Without Schema Evolution

Slide 2

Slide 2 text

Abstract Schema Registry allows us to control how schemas can be evolved through time without breaking compatibility. However, when using high retention, you could end up with a convoluted “dirty” schema depending on the selected compatibility level, as well as many different models in your application logic to support all the schema versions. Upcaster chains allow you to read an old version of a message and bring it to what your logic needs today. The upcasters in the chain describe how to jump from one version to the next. They describe what your logic expects instead of covering all the possible variations that were ever published. We created an upcaster registry to allow you to declare upcasters and their chains in a declarative manner allowing them to be used across all your streaming apps. It is a great combo with Schema Registry; one providing the schemas themselves, the other expressing how to evolve from one schema version to the next.

Slide 3

Slide 3 text

Andreas Evers CTO @ KOR Financial Previously Pivotal / VMware ➢ Staff solution architect consulting for some of the biggest European firms ➢ Focused on application transformation & system design ➢ Worked on Spinnaker, together with the Spring team Who is this guy? @andreasevers

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Mission Avoid another financial crisis by mapping out risk in financial markets

Slide 6

Slide 6 text

Selena Gomez explains “CDOs”, one of the financial products that were the cause of the 2008 crash

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

KOR Solution Real-time insights based on regulation to make financial products like CDOs transparent to the market and regulators.

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

Most companies use Kafka with a low retention setting, such as 7 days This works great for messaging use cases

Slide 12

Slide 12 text

Kafka can mean so much more for your company

Slide 13

Slide 13 text

The typical use case offloads the streamed events into an external database Those databases typically use commit logs or sequential files internally Such commit logs and sequential files are in fact very similar to Kafka topics and segment files 🤔 Why not use Kafka as the commit log for your data, and become the system of record instead of merely a tunnel

Slide 14

Slide 14 text

KSQL: Turning the database inside-out Source of truth stream Continuous query Derived table

Slide 15

Slide 15 text

High retention offers a lot of benefits, but also comes with its own unique challenges You can’t just wait 7 days for old formats to leave your cluster

Slide 16

Slide 16 text

How do we deal with Evolution?

Slide 17

Slide 17 text

Let’s start with short retention

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

Class People { - Rick - Morty }

Slide 20

Slide 20 text

Class People { - Rick - Morty }

Slide 21

Slide 21 text

- Rick - Morty Class People { - Rick - Morty } 0

Slide 22

Slide 22 text

- Rick - Morty Class People { - Rick - Morty } 0 Class People { - Rick - Morty }

Slide 23

Slide 23 text

- Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty - Summer }

Slide 24

Slide 24 text

- Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1

Slide 25

Slide 25 text

- Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1

Slide 26

Slide 26 text

- Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1

Slide 27

Slide 27 text

- Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty } 1

Slide 28

Slide 28 text

Class People { - Rick - Morty } - Rick - Morty 0 1 Class People { - Rick - Morty }

Slide 29

Slide 29 text

Class People { - Rick - Morty - Optional } - Rick - Morty 0 1 Class People { - Rick - Morty }

Slide 30

Slide 30 text

Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional } - Rick - Morty 0 1

Slide 31

Slide 31 text

Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional } - Rick - Morty 0 1 - Rick - Morty - Summer

Slide 32

Slide 32 text

Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional } - Rick - Morty - Rick - Morty - Summer 0 1

Slide 33

Slide 33 text

Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional } - Rick - Morty - Rick - Morty - Summer 0 1

Slide 34

Slide 34 text

Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional } - Rick - Morty - Rick - Morty - Summer 0 1

Slide 35

Slide 35 text

1 Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional } - Rick - Morty - Summer

Slide 36

Slide 36 text

1 Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer

Slide 37

Slide 37 text

Isn’t this easier in HTTP APIs?

Slide 38

Slide 38 text

- Rick - Morty Class People { - Rick - Morty } Class People { - Rick - Morty }

Slide 39

Slide 39 text

- Rick - Morty Class People { - Rick - Morty } Class People { - Rick - Morty }

Slide 40

Slide 40 text

- Rick - Morty - Summer Class People { - Rick - Morty } Class People { - Rick - Morty - Summer }

Slide 41

Slide 41 text

Tolerant Reader Postel’s Law, a.k.a. the Robustness Principle: Be conservative in what you do, be liberal in what you accept from others.

Slide 42

Slide 42 text

Client Functions Client Stub Server Functions Server Stub (skeleton) Network Routines Network Routines kernel client process kernel server process network client server

Slide 43

Slide 43 text

Client Objects Client SOAP Handler Server Objects Server SOAP Handler Network Routines Network Routines network kernel client process kernel server process client server

Slide 44

Slide 44 text

SpecificRecord Avro Serdes SpecificRecord Avro Serdes Network Routines Network Routines network kernel client process kernel server process client server

Slide 45

Slide 45 text

mvn generate-sources Look ma, no programming!

Slide 46

Slide 46 text

- Rick - Morty - Summer Class People { - Rick - Morty } Class People { - Rick - Morty - Summer } Be conservative in what you do, be liberal in what you accept from others. @JsonIgnoreProperties (ignoreUnknown = true)

Slide 47

Slide 47 text

- Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1 Be conservative in what you do, be liberal in what you accept from others. Tolerant GenericRecordToDto Deserializer

Slide 48

Slide 48 text

What about some other changes?

Slide 49

Slide 49 text

- Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick } - Rick 1

Slide 50

Slide 50 text

- Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick } - Rick 1

Slide 51

Slide 51 text

- Rick - Morty 0 Class People { - Rick - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1

Slide 52

Slide 52 text

- Rick - Morty 0 Class People { - Rick - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1

Slide 53

Slide 53 text

- Rick - Morty 0 Class People { - PickleRick - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1

Slide 54

Slide 54 text

- Rick - Morty 0 Class People { - PickleRick - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1

Slide 55

Slide 55 text

- Rick: String - Morty 0 Class People { - Rick: String - Morty } Class People { - Rick: boolean - Morty } - Rick: boolean - Morty 1

Slide 56

Slide 56 text

- Rick: String - Morty 0 Class People { - Rick: String - Morty } Class People { - Rick: boolean - Morty } - Rick: boolean - Morty 1

Slide 57

Slide 57 text

How can we prevent breaking consumers?

Slide 58

Slide 58 text

- Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty } 1

Slide 59

Slide 59 text

Class People { - Rick - Morty } - Rick - Morty 0 1 Class People { - Rick - Morty }

Slide 60

Slide 60 text

Class People { - Rick - Morty - Optional } - Rick - Morty 0 1 Class People { - Rick - Morty }

Slide 61

Slide 61 text

Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional } - Rick - Morty 0 1

Slide 62

Slide 62 text

Consumer-driven Contract Testing Field-by-field compatibility testing

Slide 63

Slide 63 text

Consumer-driven Contract Testing Field-by-field compatibility testing

Slide 64

Slide 64 text

V2

Slide 65

Slide 65 text

V2 Consumer A is on version 2, only uses PickleRick

Slide 66

Slide 66 text

V2 Consumer A is on version 2, only uses PickleRick

Slide 67

Slide 67 text

V2 Consumer A is on version 2, only uses PickleRick

Slide 68

Slide 68 text

V2 Consumer A is on version 2, only uses PickleRick

Slide 69

Slide 69 text

V2 Consumer-Driven Contract Testing Consumer A is on version 2, only uses PickleRick

Slide 70

Slide 70 text

Consumer-Driven Contract Testing ❯ pact-broker can-i-deploy --pacticipant Foo --version 23 --to production

Slide 71

Slide 71 text

Schema Registry Automatic schema management and validation

Slide 72

Slide 72 text

Schema Evolution through Schema Registry Schema Registry • Versions every schema • Every schema has a unique ID • Uses subjects with Topic, Record, or TopicRecord names • For each subject, each schema change gets a version increase Schema ID: foo Schema ID: bar Subject topic-People - Rick - Morty - Rick - Morty - Summer V1 V2

Slide 73

Slide 73 text

Compatibility Types Compatibility Type Changes allowed Check against which schemas Upgrade first BACKWARD Delete fields Add optional fields Last version Consumers BACKWARD_TRANSITIVE Delete fields Add optional fields All previous versions Consumers FORWARD Add fields Delete optional fields Last version Producers FORWARD_TRANSITIVE Add fields Delete optional fields All previous versions Producers FULL Add optional fields Delete optional fields Last version Any order FULL_TRANSITIVE Add optional fields Delete optional fields All previous versions Any order NONE All changes are accepted Compatibility checking disabled Depends

Slide 74

Slide 74 text

Backward

Slide 75

Slide 75 text

Class People { - Rick - Optional } Backward 0 1 - Rick - Morty - Rick - Optional

Slide 76

Slide 76 text

Class People { - Rick - Optional } Class People { - Rick - Optional } Backward - Rick - Morty 0 - Rick - Optional 1

Slide 77

Slide 77 text

Class People { - Rick - Optional } Class People { - Rick - Optional } Backward 0 1 - Rick - Morty - Rick - Optional

Slide 78

Slide 78 text

Class People { - Rick - Optional } Class People { - Rick - Optional } Backward 0 1 - Rick - Morty - Rick - Optional

Slide 79

Slide 79 text

Class People { - Rick - Optional } Class People { - Rick - Optional } - Rick - Optional Backward

Slide 80

Slide 80 text

Class People { - Rick - Optional } Class People { - Rick - Optional } - Rick - Optional Backward

Slide 81

Slide 81 text

Class People { - Rick - Optional } Class People { - Rick - Morty } - Rick - Morty Backward

Slide 82

Slide 82 text

Class People { - Rick - Optional } Class People { - Rick - Morty } - Rick - Morty Backward

Slide 83

Slide 83 text

- Rick - Morty 0 Class People { - Rick - Optional } - Rick - Optional 1 - Rick 2 Backward Transitive

Slide 84

Slide 84 text

- Rick - Morty 0 Class People { - Rick - Optional } Class People { - Rick - Optional } - Rick - Optional 1 - Rick 2 Backward Transitive

Slide 85

Slide 85 text

- Rick - Morty 0 Class People { - Rick - Optional } Class People { - Rick - Optional } - Rick - Optional 1 - Rick 2 Backward Transitive

Slide 86

Slide 86 text

- Rick - Morty 0 Class People { - Rick - Optional } Class People { - Rick - Optional } - Rick - Optional 1 - Rick 2 Backward Transitive

Slide 87

Slide 87 text

- Rick - Morty 0 Class People { - Rick - Optional } Class People { - Rick - Optional } - Rick - Optional 1 - Rick 2 Backward Transitive

Slide 88

Slide 88 text

Forward

Slide 89

Slide 89 text

Class People { - Rick - Optional } Class People { - Rick - Optional } Forward - Rick - Optional

Slide 90

Slide 90 text

Class People { - Rick - Optional } Class People { - Rick - Optional } Forward - Rick - Optional

Slide 91

Slide 91 text

Class People { - Rick - Optional } Class People { - Rick - Summer } Forward - Rick - Optional - Rick - Summer

Slide 92

Slide 92 text

Class People { - Rick - Optional } Class People { - Rick - Optional } - Rick - Optional Forward

Slide 93

Slide 93 text

Class People { - Rick - Optional } Class People { - Rick - Summer } - Rick - Summer Forward

Slide 94

Slide 94 text

- Rick - Optional 0 Class People { - Rick - Optional } Class People { - Rick - Optional } 1 2 Forward Transitive

Slide 95

Slide 95 text

- Rick - Optional 0 Class People { - Rick - Optional } Class People { - Rick - Optional } 1 2 Forward Transitive

Slide 96

Slide 96 text

- Rick - Optional 0 Class People { - Rick - Optional } Class People { - Rick } 1 - Rick 2 Forward Transitive

Slide 97

Slide 97 text

- Rick - Optional 0 Class People { - Rick - Optional } Class People { - Rick - Summer } - Rick - Summer 1 - Rick 2 Forward Transitive

Slide 98

Slide 98 text

Short Retention = Full? Backward when • you want to allow the producer to roll back to a previous version, writing a previous schema, without breaking consumers (provided they are tolerant readers) Forward when • you want to allow consumers to read new versions without needing to be updated first (provided they are tolerant readers and non-compatible upgrades are controlled, e.g. using CDCT)

Slide 99

Slide 99 text

High Retention = Full Transitive? Backward transitive when • you want to allow new consumers which will start at offset 0, to be added at any time in the future • or you want to allow producers to roll back to any previous version, writing a previous schema, without breaking consumers (provided they are tolerant readers) Forward transitive when • you want to allow consumers to read new versions without needing to be updated first (provided they are tolerant readers) regardless from where they start consuming

Slide 100

Slide 100 text

High Retention = Full Transitive forever? Class People { - Optional - Optional - Optional - Optional - Optional }

Slide 101

Slide 101 text

Evolution strategies • Tolerant Reader • Versioned Events • Copy-Transform • Upcasting

Slide 102

Slide 102 text

Copy-Transform A.k.a. Copy & Replace A.k.a. The Parallel Universe

Slide 103

Slide 103 text

Copy-Transform

Slide 104

Slide 104 text

Copy-Transform

Slide 105

Slide 105 text

Copy-Transform

Slide 106

Slide 106 text

Copy-Transform

Slide 107

Slide 107 text

Copy-Transform

Slide 108

Slide 108 text

Copy-Transform

Slide 109

Slide 109 text

Copy-Transform

Slide 110

Slide 110 text

Copy-Transform Benefits Drawbacks Relatively simple to implement Complete freedom to do anything you want Data preservation One-time operation Performance

Slide 111

Slide 111 text

Upcasting A.k.a. Message Translators

Slide 112

Slide 112 text

No content

Slide 113

Slide 113 text

Avro defaults give you wire-compatibility Upcasting gives you business-compatibility

Slide 114

Slide 114 text

- Rick - Morty Class People { - PickleRick - Summer } - PickleRick - Summer - Rick

Slide 115

Slide 115 text

- Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick

Slide 116

Slide 116 text

- Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick

Slide 117

Slide 117 text

- Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3

Slide 118

Slide 118 text

- Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V1 → V2 V2 → V3 V2 → V3

Slide 119

Slide 119 text

- Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3

Slide 120

Slide 120 text

- Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V2 → V3

Slide 121

Slide 121 text

- Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V2 → V3

Slide 122

Slide 122 text

- Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V1 → V2 V2 → V3 V2 → V3

Slide 123

Slide 123 text

- Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V1 → V2 V2 → V3 V2 → V3

Slide 124

Slide 124 text

- Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2

Slide 125

Slide 125 text

- Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2

Slide 126

Slide 126 text

- Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2

Slide 127

Slide 127 text

What if some consumers are missing upcasters?

Slide 128

Slide 128 text

- Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2

Slide 129

Slide 129 text

- Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V3 → V2

Slide 130

Slide 130 text

- Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V3 → V2

Slide 131

Slide 131 text

- Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2

Slide 132

Slide 132 text

- Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2

Slide 133

Slide 133 text

- Rick - Morty Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2

Slide 134

Slide 134 text

- Rick - Morty Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V3 V2 V1 V3 V1 V2 V1

Slide 135

Slide 135 text

Upcaster Registry Centralized repository for up- and downcasters

Slide 136

Slide 136 text

Class People { - Rick - Morty } V1

Slide 137

Slide 137 text

Class People { - Rick - Morty } V1 - Rick - Morty V1

Slide 138

Slide 138 text

- Rick - Morty Class People { - Rick - Morty } Class People { - Rick - Morty } V1 V1 V1

Slide 139

Slide 139 text

- Rick - Morty Class People { - Rick - Morty } Class People { - Rick - Morty } V1 V1 V1

Slide 140

Slide 140 text

- Rick - Morty Class People { - Rick - Morty } Class People { - Rick - Morty } V1 V1 V1

Slide 141

Slide 141 text

- Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2

Slide 142

Slide 142 text

- Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1

Slide 143

Slide 143 text

- Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2

Slide 144

Slide 144 text

- Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2

Slide 145

Slide 145 text

- Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2

Slide 146

Slide 146 text

- Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2 V2 → V1

Slide 147

Slide 147 text

- Rick - Morty Class People { - Rick } Class People { - Rick } V1 V2 V2 V1 → V2 V2 → V1 - Rick V2

Slide 148

Slide 148 text

- Rick - Morty Class People { - Rick } Class People { - Rick } V1 V2 V2 V1 → V2 V2 → V1 - Rick V2

Slide 149

Slide 149 text

- Rick - Morty Class People { - Rick } Class People { - Rick } V1 V2 V2 V1 → V2 V2 → V1 - Rick V2 V1 → V2

Slide 150

Slide 150 text

- Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } V1 V2 V3 V1 → V2 V2 → V1 - Rick V2

Slide 151

Slide 151 text

- Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } V1 V2 V3 V1 → V2 V2 → V1 - Rick V2 V3 → V2 V2 → V3

Slide 152

Slide 152 text

- Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2 V1 → V2 V3 → V2 V2 → V1 V2 → V3 V3

Slide 153

Slide 153 text

https://github.com/KOR-Financial/kopper

Slide 154

Slide 154 text

- Rick - Morty 0 Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer 1 - Rick 2 V1 V2 V3 V2 V1 → V2 V3 → V2 V1 → V2 V3 → V2 V2 → V1 V2 → V3 V3 Consumer A is on V2 Supports V1 and V3

Slide 155

Slide 155 text

❯ pact-broker can-i-deploy --pacticipant Foo --version 23 --to production

Slide 156

Slide 156 text

Can we combine these evolution strategies?

Slide 157

Slide 157 text

Combining strategies for short retention • Versioned Events • Tolerant Reader • Consumer-driven Contract Testing

Slide 158

Slide 158 text

Combining strategies for short retention This allows you to: • Avoid transitive • Avoid the optional dance • Chase consumers to upgrade • Prevent producers from mistakenly releasing incompatible version after releasing compatible version • Keep your consumers tolerant to change • Keep your consumers ignorant of information they don’t need

Slide 159

Slide 159 text

Combining strategies for high retention • Versioned Events • Upcasting • Upcaster registry • Tolerant Reader • Copy-Transform

Slide 160

Slide 160 text

Combining strategies for high retention This allows you to: • Avoid Full Transitive Forever • Avoid every field being optional • Avoid having to write to countless deprecated fields • Keep your consumers tolerant to change • Keep your consumers ignorant of information they don’t need

Slide 161

Slide 161 text

Thank you! Accreditations: Graphics by Arthur Shlain, Creative Stal, Siddharth Dasari, Webtechops LLP, Icons Producer, SBTS from Noun Project Andreas Evers @andreasevers aevers@korfinancial.com KOR Financial www.korfinancial.com