Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Schemas Without Schema Evolution - Current 2022

Evolving Schemas Without Schema Evolution - Current 2022

Schema Registry allows us to control how schemas can be evolved through time without breaking compatibility.

However, when using high retention, you could end up with a convoluted “dirty” schema depending on the selected compatibility level, as well as many different models in your application logic to support all the schema versions.
Upcaster chains allow you to read an old version of a message and bring it to what your logic needs today.

The upcasters in the chain describe how to jump from one version to the next. They describe what your logic expects instead of covering all the possible variations that were ever published.

We created an upcaster registry to allow you to declare upcasters and their chains in a declarative manner allowing them to be used across all your streaming apps. It is a great combo with Schema Registry; one providing the schemas themselves, the other expressing how to evolve from one schema version to the next.

Andreas Evers

October 04, 2022
Tweet

More Decks by Andreas Evers

Other Decks in Technology

Transcript

  1. Abstract Schema Registry allows us to control how schemas can

    be evolved through time without breaking compatibility. However, when using high retention, you could end up with a convoluted “dirty” schema depending on the selected compatibility level, as well as many different models in your application logic to support all the schema versions. Upcaster chains allow you to read an old version of a message and bring it to what your logic needs today. The upcasters in the chain describe how to jump from one version to the next. They describe what your logic expects instead of covering all the possible variations that were ever published. We created an upcaster registry to allow you to declare upcasters and their chains in a declarative manner allowing them to be used across all your streaming apps. It is a great combo with Schema Registry; one providing the schemas themselves, the other expressing how to evolve from one schema version to the next.
  2. Andreas Evers CTO @ KOR Financial Previously Pivotal / VMware

    ➢ Staff solution architect consulting for some of the biggest European firms ➢ Focused on application transformation & system design ➢ Worked on Spinnaker, together with the Spring team Who is this guy? @andreasevers
  3. KOR Solution Real-time insights based on regulation to make financial

    products like CDOs transparent to the market and regulators.
  4. Most companies use Kafka with a low retention setting, such

    as 7 days This works great for messaging use cases
  5. The typical use case offloads the streamed events into an

    external database Those databases typically use commit logs or sequential files internally Such commit logs and sequential files are in fact very similar to Kafka topics and segment files 🤔 Why not use Kafka as the commit log for your data, and become the system of record instead of merely a tunnel
  6. High retention offers a lot of benefits, but also comes

    with its own unique challenges You can’t just wait 7 days for old formats to leave your cluster
  7. - Rick - Morty Class People { - Rick -

    Morty } 0 Class People { - Rick - Morty }
  8. - Rick - Morty 0 Class People { - Rick

    - Morty } Class People { - Rick - Morty - Summer }
  9. - Rick - Morty 0 Class People { - Rick

    - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1
  10. - Rick - Morty 0 Class People { - Rick

    - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1
  11. - Rick - Morty 0 Class People { - Rick

    - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1
  12. - Rick - Morty 0 Class People { - Rick

    - Morty } Class People { - Rick - Morty } 1
  13. Class People { - Rick - Morty } - Rick

    - Morty 0 1 Class People { - Rick - Morty }
  14. Class People { - Rick - Morty - Optional<Summer> }

    - Rick - Morty 0 1 Class People { - Rick - Morty }
  15. Class People { - Rick - Morty - Summer }

    Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty 0 1
  16. Class People { - Rick - Morty - Summer }

    Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty 0 1 - Rick - Morty - Summer
  17. Class People { - Rick - Morty - Summer }

    Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty - Rick - Morty - Summer 0 1
  18. Class People { - Rick - Morty - Summer }

    Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty - Rick - Morty - Summer 0 1
  19. Class People { - Rick - Morty - Summer }

    Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty - Rick - Morty - Summer 0 1
  20. 1 Class People { - Rick - Morty - Summer

    } Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty - Summer
  21. 1 Class People { - Rick - Morty - Summer

    } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer
  22. - Rick - Morty Class People { - Rick -

    Morty } Class People { - Rick - Morty }
  23. - Rick - Morty Class People { - Rick -

    Morty } Class People { - Rick - Morty }
  24. - Rick - Morty - Summer Class People { -

    Rick - Morty } Class People { - Rick - Morty - Summer }
  25. Tolerant Reader Postel’s Law, a.k.a. the Robustness Principle: Be conservative

    in what you do, be liberal in what you accept from others.
  26. Client Functions Client Stub Server Functions Server Stub (skeleton) Network

    Routines Network Routines kernel client process kernel server process network client server
  27. Client Objects Client SOAP Handler Server Objects Server SOAP Handler

    Network Routines Network Routines network kernel client process kernel server process client server
  28. SpecificRecord Avro Serdes SpecificRecord Avro Serdes Network Routines Network Routines

    network kernel client process kernel server process client server
  29. - Rick - Morty - Summer Class People { -

    Rick - Morty } Class People { - Rick - Morty - Summer } Be conservative in what you do, be liberal in what you accept from others. @JsonIgnoreProperties (ignoreUnknown = true)
  30. - Rick - Morty 0 Class People { - Rick

    - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1 Be conservative in what you do, be liberal in what you accept from others. Tolerant GenericRecordToDto Deserializer
  31. - Rick - Morty 0 Class People { - Rick

    - Morty } Class People { - Rick } - Rick 1
  32. - Rick - Morty 0 Class People { - Rick

    - Morty } Class People { - Rick } - Rick 1
  33. - Rick - Morty 0 Class People { - Rick

    - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1
  34. - Rick - Morty 0 Class People { - Rick

    - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1
  35. - Rick - Morty 0 Class People { - PickleRick

    - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1
  36. - Rick - Morty 0 Class People { - PickleRick

    - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1
  37. - Rick: String - Morty 0 Class People { -

    Rick: String - Morty } Class People { - Rick: boolean - Morty } - Rick: boolean - Morty 1
  38. - Rick: String - Morty 0 Class People { -

    Rick: String - Morty } Class People { - Rick: boolean - Morty } - Rick: boolean - Morty 1
  39. - Rick - Morty 0 Class People { - Rick

    - Morty } Class People { - Rick - Morty } 1
  40. Class People { - Rick - Morty } - Rick

    - Morty 0 1 Class People { - Rick - Morty }
  41. Class People { - Rick - Morty - Optional<Summer> }

    - Rick - Morty 0 1 Class People { - Rick - Morty }
  42. Class People { - Rick - Morty - Summer }

    Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty 0 1
  43. V2

  44. Schema Evolution through Schema Registry Schema Registry • Versions every

    schema • Every schema has a unique ID • Uses subjects with Topic, Record, or TopicRecord names • For each subject, each schema change gets a version increase Schema ID: foo Schema ID: bar Subject topic-People - Rick - Morty - Rick - Morty - Summer V1 V2
  45. Compatibility Types Compatibility Type Changes allowed Check against which schemas

    Upgrade first BACKWARD Delete fields Add optional fields Last version Consumers BACKWARD_TRANSITIVE Delete fields Add optional fields All previous versions Consumers FORWARD Add fields Delete optional fields Last version Producers FORWARD_TRANSITIVE Add fields Delete optional fields All previous versions Producers FULL Add optional fields Delete optional fields Last version Any order FULL_TRANSITIVE Add optional fields Delete optional fields All previous versions Any order NONE All changes are accepted Compatibility checking disabled Depends
  46. Class People { - Rick - Optional<Summer> } Backward 0

    1 - Rick - Morty - Rick - Optional<Summer>
  47. Class People { - Rick - Optional<Summer> } Class People

    { - Rick - Optional<Summer> } Backward - Rick - Morty 0 - Rick - Optional<Summer> 1
  48. Class People { - Rick - Optional<Summer> } Class People

    { - Rick - Optional<Summer> } Backward 0 1 - Rick - Morty - Rick - Optional<Summer>
  49. Class People { - Rick - Optional<Summer> } Class People

    { - Rick - Optional<Summer> } Backward 0 1 - Rick - Morty - Rick - Optional<Summer>
  50. Class People { - Rick - Optional<Summer> } Class People

    { - Rick - Optional<Summer> } - Rick - Optional<Summer> Backward
  51. Class People { - Rick - Optional<Summer> } Class People

    { - Rick - Optional<Summer> } - Rick - Optional<Summer> Backward
  52. Class People { - Rick - Optional<Summer> } Class People

    { - Rick - Morty } - Rick - Morty Backward
  53. Class People { - Rick - Optional<Summer> } Class People

    { - Rick - Morty } - Rick - Morty Backward
  54. - Rick - Morty 0 Class People { - Rick

    - Optional<Summer> } - Rick - Optional<Summer> 1 - Rick 2 Backward Transitive
  55. - Rick - Morty 0 Class People { - Rick

    - Optional<Summer> } Class People { - Rick - Optional<Summer> } - Rick - Optional<Summer> 1 - Rick 2 Backward Transitive
  56. - Rick - Morty 0 Class People { - Rick

    - Optional<Summer> } Class People { - Rick - Optional<Summer> } - Rick - Optional<Summer> 1 - Rick 2 Backward Transitive
  57. - Rick - Morty 0 Class People { - Rick

    - Optional<Summer> } Class People { - Rick - Optional<Summer> } - Rick - Optional<Summer> 1 - Rick 2 Backward Transitive
  58. - Rick - Morty 0 Class People { - Rick

    - Optional<Summer> } Class People { - Rick - Optional<Summer> } - Rick - Optional<Summer> 1 - Rick 2 Backward Transitive
  59. Class People { - Rick - Optional<Morty> } Class People

    { - Rick - Optional<Morty> } Forward - Rick - Optional<Morty>
  60. Class People { - Rick - Optional<Morty> } Class People

    { - Rick - Optional<Morty> } Forward - Rick - Optional<Morty>
  61. Class People { - Rick - Optional<Morty> } Class People

    { - Rick - Summer } Forward - Rick - Optional<Morty> - Rick - Summer
  62. Class People { - Rick - Optional<Morty> } Class People

    { - Rick - Optional<Morty> } - Rick - Optional<Morty> Forward
  63. Class People { - Rick - Optional<Morty> } Class People

    { - Rick - Summer } - Rick - Summer Forward
  64. - Rick - Optional<Morty> 0 Class People { - Rick

    - Optional<Morty> } Class People { - Rick - Optional<Morty> } 1 2 Forward Transitive
  65. - Rick - Optional<Morty> 0 Class People { - Rick

    - Optional<Morty> } Class People { - Rick - Optional<Morty> } 1 2 Forward Transitive
  66. - Rick - Optional<Morty> 0 Class People { - Rick

    - Optional<Morty> } Class People { - Rick } 1 - Rick 2 Forward Transitive
  67. - Rick - Optional<Morty> 0 Class People { - Rick

    - Optional<Morty> } Class People { - Rick - Summer } - Rick - Summer 1 - Rick 2 Forward Transitive
  68. Short Retention = Full? Backward when • you want to

    allow the producer to roll back to a previous version, writing a previous schema, without breaking consumers (provided they are tolerant readers) Forward when • you want to allow consumers to read new versions without needing to be updated first (provided they are tolerant readers and non-compatible upgrades are controlled, e.g. using CDCT)
  69. High Retention = Full Transitive? Backward transitive when • you

    want to allow new consumers which will start at offset 0, to be added at any time in the future • or you want to allow producers to roll back to any previous version, writing a previous schema, without breaking consumers (provided they are tolerant readers) Forward transitive when • you want to allow consumers to read new versions without needing to be updated first (provided they are tolerant readers) regardless from where they start consuming
  70. High Retention = Full Transitive forever? Class People { -

    Optional<Rick> - Optional<Morty> - Optional<Summer> - Optional<PickleRick> - Optional<Rick:boolean> }
  71. Copy-Transform Benefits Drawbacks Relatively simple to implement Complete freedom to

    do anything you want Data preservation One-time operation Performance
  72. - Rick - Morty Class People { - PickleRick -

    Summer } - PickleRick - Summer - Rick
  73. - Rick - Morty Class People { - PickleRick -

    Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick
  74. - Rick - Morty Class People { - PickleRick -

    Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick
  75. - Rick - Morty Class People { - PickleRick -

    Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3
  76. - Rick - Morty Class People { - PickleRick -

    Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V1 → V2 V2 → V3 V2 → V3
  77. - Rick - Morty Class People { - PickleRick -

    Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3
  78. - Rick - Morty Class People { - PickleRick -

    Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V2 → V3
  79. - Rick - Morty Class People { - PickleRick -

    Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V2 → V3
  80. - Rick - Morty Class People { - PickleRick -

    Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V1 → V2 V2 → V3 V2 → V3
  81. - Rick - Morty Class People { - PickleRick -

    Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V1 → V2 V2 → V3 V2 → V3
  82. - Rick - Morty Class People { - Rick }

    Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2
  83. - Rick - Morty Class People { - Rick }

    Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2
  84. - Rick - Morty Class People { - Rick }

    Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2
  85. - Rick - Morty Class People { - Rick }

    Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2
  86. - Rick - Morty Class People { - Rick }

    Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V3 → V2
  87. - Rick - Morty Class People { - Rick }

    Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V3 → V2
  88. - Rick - Morty Class People { - Rick }

    Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2
  89. - Rick - Morty Class People { - Rick }

    Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2
  90. - Rick - Morty Class People { - PickleRick -

    Summer } - PickleRick - Summer - Rick V1 V2 V3 V2
  91. - Rick - Morty Class People { - PickleRick -

    Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V3 V2 V1 V3 V1 V2 V1
  92. - Rick - Morty Class People { - Rick -

    Morty } Class People { - Rick - Morty } V1 V1 V1
  93. - Rick - Morty Class People { - Rick -

    Morty } Class People { - Rick - Morty } V1 V1 V1
  94. - Rick - Morty Class People { - Rick -

    Morty } Class People { - Rick - Morty } V1 V1 V1
  95. - Rick - Morty Class People { - Rick -

    Morty } Class People { - Rick } V1 V1 V2
  96. - Rick - Morty Class People { - Rick -

    Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1
  97. - Rick - Morty Class People { - Rick -

    Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2
  98. - Rick - Morty Class People { - Rick -

    Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2
  99. - Rick - Morty Class People { - Rick -

    Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2
  100. - Rick - Morty Class People { - Rick -

    Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2 V2 → V1
  101. - Rick - Morty Class People { - Rick }

    Class People { - Rick } V1 V2 V2 V1 → V2 V2 → V1 - Rick V2
  102. - Rick - Morty Class People { - Rick }

    Class People { - Rick } V1 V2 V2 V1 → V2 V2 → V1 - Rick V2
  103. - Rick - Morty Class People { - Rick }

    Class People { - Rick } V1 V2 V2 V1 → V2 V2 → V1 - Rick V2 V1 → V2
  104. - Rick - Morty Class People { - Rick }

    Class People { - PickleRick - Summer } V1 V2 V3 V1 → V2 V2 → V1 - Rick V2
  105. - Rick - Morty Class People { - Rick }

    Class People { - PickleRick - Summer } V1 V2 V3 V1 → V2 V2 → V1 - Rick V2 V3 → V2 V2 → V3
  106. - Rick - Morty Class People { - Rick }

    Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2 V1 → V2 V3 → V2 V2 → V1 V2 → V3 V3
  107. - Rick - Morty 0 Class People { - Rick

    } Class People { - PickleRick - Summer } - PickleRick - Summer 1 - Rick 2 V1 V2 V3 V2 V1 → V2 V3 → V2 V1 → V2 V3 → V2 V2 → V1 V2 → V3 V3 Consumer A is on V2 Supports V1 and V3
  108. Combining strategies for short retention This allows you to: •

    Avoid transitive • Avoid the optional dance • Chase consumers to upgrade • Prevent producers from mistakenly releasing incompatible version after releasing compatible version • Keep your consumers tolerant to change • Keep your consumers ignorant of information they don’t need
  109. Combining strategies for high retention • Versioned Events • Upcasting

    • Upcaster registry • Tolerant Reader • Copy-Transform
  110. Combining strategies for high retention This allows you to: •

    Avoid Full Transitive Forever • Avoid every field being optional • Avoid having to write to countless deprecated fields • Keep your consumers tolerant to change • Keep your consumers ignorant of information they don’t need
  111. Thank you! Accreditations: Graphics by Arthur Shlain, Creative Stal, Siddharth

    Dasari, Webtechops LLP, Icons Producer, SBTS from Noun Project Andreas Evers @andreasevers aevers@korfinancial.com KOR Financial www.korfinancial.com