Turmbau_zu_Babel.pdf

Transcript

2. Tower of Babel …in message-based Systems Mastering Microservices Kristian Kottke
3. 28.04.2022 Tower of Babel 3 § Lead Software Engineer @ Interests § Reactive Systems § Data Engineering § Stream Processing § Cloud § Software Architecture Kristian Kottke Whoami
4. AGENDA § (Project) Background § Theory § Implementation § Wrap Up
5. 28.04.2022 Tower of Babel 5 § Online Shop § 3 Teams, 9 Developer (initial) – 4 Teams, 14 Developer (now) § Vertical Separation § Domain Services – Self Contained System (SCS) § Communication – Inside Domain => synchronous (REST) – Outside Domain => asynchronous (Kafka) § JVM (Kotlin) Project Team 2 Messages Product Shopping Cart Team 1 Team 3
11. 28.04.2022 Tower of Babel 11 REST § Synchronous § Client-Driven => Request-Response § One Request => One Message Version § Format Change – Backward Compatibility – Breaking Change => Versioned API Message Broker § Asynchronous § Producer-Driven § Multiple Message Versions § Format Change??? – Multi Version Compatibility? – Future Consumer? – Offset Reset? REST vs. Message Broker Format n Format n+1 Format n+2
12. 28.04.2022 Tower of Babel 12 § Contract about Message Format (Syntax) – Contract Enforcement § Support for Format Evolution – Conceptional Evolution Support – Compatibility – Prevent Breaking Changes – Limit Impact of Changes § Consumer Safety § Stability § Decouple Producer and Consumers Requirements for the Solution
13. § Consumer defines & publishes Contract § Consumer & Producer tests compliance with Contract § REST API – Message Formats & Protocol § Message Pact – Message Format § Limited Backward & Forward Compatibility § Consumer-Driven? – Events – Commands 28.04.2022 Tower of Babel 13 Consumer-Driven Contract CDC Pact Broker Consumer Producer 1 2 3 4
14. 28.04.2022 Tower of Babel 14 § Schema-based Serialization System => Contract § Binary Data Format § Compact Encoding (no Field Information) § Optional Code Generation § Schema Evolution – Compatibility § Schema Registry – Central Schema Store – Enforce Compatibility Level – Backward, Forward, Full, *_Transitive, None { "type": "record", "name": "Product", "namespace": "de.company.products", "fields": [ { "name": "id", "type": "string", "doc": "product ID" }, { "name": “shippingCosts", "type": “double", "default": 0.0 }, { "name": "price", "type": “MyPrice" }, { "name": "uvp", "type": [ "null", “MyPrice" ], "doc": "[OPTIONAL]" } ] } Avro
15. § Backward Schema n can read Data n-1 § Forward Schema n can read Data n+1 § Full Backward + Forward § *_Transitive All previous Version § None 28.04.2022 Tower of Babel 15 Compatibility & Schema Evolution Compatibility Type Changes allowed Upgrade first Backward • Delete Fields • Add optional Fields • Convert mandatory Field into optional Consumers Forward • Add Fields • Delete optional Fields • Convert optional Field into mandatory Producers Full • Add optional Fields • Delete optional Fields • Convert mandatory Field into optional • Convert optional Field into mandatory Any Order None • All Changes are accepted Depends source: Confluent
17. 28.04.2022 Tower of Babel 18 § Payload – Avro – Code Generation § Compatibility – Full => Backward + Forward – Breaking changes n – n+2 – No breaking changes in topics – TTL < 2 weeks – Log Compaction / Retention time Messages Kafka Message Header / Meta Data Payload => Avro
18. 28.04.2022 Tower of Babel 19 Consumer Workflow Build Time Generate Code Compile Test Build Artifact Schema Registry
19. 28.04.2022 Tower of Babel 21 Workflow Runtime source: Confluent Generic Record Generated DTO Compatibility
20. 28.04.2022 Tower of Babel 22 Registry § Confluent – De facto Standard – Established Solution § Karapace – Open-Source Implementation of Kafka REST & Schema Registry – Drop in Replacement § Apicurio – API Registry – Broader Approach – Modell Types – Compatibility & Validity Library § Confluent – Serialization / Deserialization – Schema Handling & Caching § Apicurio – Serialization / Deserialization – Schema Handling & Caching – Support Usage of Kafka Headers for Schema Information – Support JSON Serialization Tooling
21. 28.04.2022 Tower of Babel 24 § Header vs. Envelope § Cloud Events – …specification for describing event data in a common way… – Required Fields – id, source, type, specversion – Optional Field – subject, time – Extensions – eventversion – Tool Support Meta Data { "topic": "products", "partition": 0, "offset": 171, "ts": 1617794750706, "headers": [ "specversion", "1.0", "id", "f0e7c0fd-9bd8-4ffb-9ce2-64331f487334", "type", "PRODUCT_UPDATE", "source", "https://company.de/products", "subject", “price_change", "time", "2021-04-07T13:25:50.705Z", "eventversion", "1.0" ], "key": "0815", "payload": “….." }
22. 28.04.2022 Tower of Babel 25 XML Schema Definition (XSD) § XML…Yeah… § Theory vs. Practice § No Contract Enforcement ❌ § No Schema Evolution ❌ § Nasty Processors Json Schema § Human-readalbe Format § Contract Enforcement ✅ § Complex Schema Evolution ❌ – Grammar-based vs. Rule-based – Open & Closed Content Model Protobuf § Comparable to Avro § Contract Enforcement ✅ § Schema Evolution ✅ § Not supported by Karapace Alternatives
24. § Meta Data => Cloud Events § Payload => Schema/Contract-Based Serialization / Deserialization – Contract Enforcement – Full Compatibility => Technical Problems reduced § Problems are still possible – Miss Versions – Semantic Changes => Notice Schema Changes – Notification – Automated Testing 28.04.2022 Tower of Babel 27 Wrap Up
26. Kristian Kottke Lead Software Engineer kristian.kottke@iteratec.com @_kkottke github.com/kkottke xing.to/kkottke speakerdeck.com/kkottke