Current SF 2026 - Your Apache KafkaⓇ Topics Are APIs - Treat Them Like It

Would you build a REST API and make users guess the request and response formats?

What's the domain model?
Name the operation… POST? GET? PUT? PATCH?
What does this error code even mean?
Is the documentation - if it even exists - accurate and relevant?
Yet that's exactly what many data streaming deployments do.

Apache Kafka's flexible, unopinionated serialization is a huge strength, but it's also a potential risk. Even if your event streams carry structured data, enforcing that structure, managing its evolution, and validating its contents aren't hard requirements. This leaves consumer applications preprocessing every event - even ones with no business value.

Designing structured events with a schema (Apache Avro and Google Protobuf) models your business domain and promotes safe evolution practices. Schema-aware serializers enforce event structure before it's written to the stream, while consumers leverage those shared schemas to deserialize on the other end. Build-time validation catches schema problems before code ever reaches review - one last line of defense against poison data. Through all of it, the goal is the same: treat your Kafka topics with the same rigor you'd apply to any API contract - because that's exactly what they are.

Join us to learn how to move validation upstream - transforming your event streams into data consumers can actually trust, with a practical blueprint for schema integrity and automated evolution.

Transcript

1. CURRENT SAN FRANCISCO 2026 · WED, NOV 4 · 1:00 PM Your Kafka topics are APIs. Treat them like it. Schema integrity and automated evolution for event streams that consumers can trust Sandon Jacobs · Sr Developer Advocate · Confluent, an IBM Company
2. ABOUT ME Sandon Jacobs Senior Developer Advocate at Confluent, an IBM Company “Talk data to me…” or Java, Kotlin, Scala, DevOps, Kafka, Flink… @SandonJacobs · linkedin.com/in/sandonjacobs · github.com/sandonjacobs
3. THE REST TEST Would you ship a REST API like this? 01 02 What’s the domain model? Is it POST, GET, PUT or PATCH? 03 04 What does this error code mean? Is the documentation accurate, if it exists? Now imagine doing this in the distributed world of data streaming... Your Kafka topics are APIs · Current SF 2026
4. SAME QUESTIONS, NO CONTRACT What a bare topic gives consumers Question REST API Topic with no contract Domain model OpenAPI or JSON Schema Just bytes... Operation HTTP method and path Maybe there's a "convention" - maybe not... Errors Status codes, problem details Good luck consumers Documentation A published, versioned spec I think there's a wiki page that nobody updates Your Kafka topics are APIs · Current SF 2026
5. FLEXIBILITY CUTS BOTH WAYS Kafka stores bytes. It never asks what they mean. Unopinionated serialization is a strength. Without a contract, it is also a risk.
6. THE COST Every consumer becomes a validator Billing: skips malformed events Producer Ships whatever the team last changed orders Accepts any bytes Each team writes its own guesswork, and each copy drifts. Your Kafka topics are APIs · Current SF 2026 Analytics: patches renamed fields Search: drops events with no business value
7. THE MODEL A topic has an API surface too In REST On a Kafka topic Domain model Avro or Protobuf schema Operation Events are descriptive Error contract Rejected writes, dead letter queue Documentation doc fields inside the schema Versioning Dictates how we update producers and consumers Your Kafka topics are APIs · Current SF 2026
8. SCHEMA REGISTRY The registry is the source of truth Schema Registry orders-value: v1, v2, v3 · BACKWARD 1 Register or look up the schema, get an ID Producer Serializer validates, then adds the ID 2 Record = 5-byte header with the ID, then the payload orders Stores the ID plus payload 3 Fetch the schema by ID, then cache it Consumer Deserializer decodes with that schema Why it matters: every consumer decodes with the exact schema the producer used, and incompatible changes are rejected at registration. Your Kafka topics are APIs · Current SF 2026
9. THE DOMAIN MODEL An event schema documents itself { "type": "record", "name": "OrderPlaced", "doc": "A customer placed an order.", "fields": [ {"name": "orderId", "type": "string"}, {"name": "placedAt", "type": "long"}, {"name": "couponCode", "type": ["null", "string"], "default": null} ] } Your Kafka topics are APIs · Current SF 2026 A named event OrderPlaced says what happened. Built-in docs The doc field travels with the schema. Optional by design A nullable field with a default can evolve safely.
10. SAFE EVOLUTION Compatibility modes set the rules Mode Safe changes Upgrade first BACKWARD Delete fields, add optional fields Consumers FORWARD Add fields, delete optional fields Producers FULL Add or delete optional fields Any order NONE Any change, unchecked No guarantees Pick the mode that matches who upgrades first. That's part of the CONTRACT. Your Kafka topics are APIs · Current SF 2026
11. ENFORCEMENT Validate before the write Serialize Write Read Record is checked against the schema Topic stores schema ID plus payload Deserializer fetches the schema by ID Schema Registry: versioned schemas, compatibility checks, one source of truth Your Kafka topics are APIs · Current SF 2026 Consume Typed objects, no guessing
12. DATA CONTRACTS A contract is schema plus rules 01 02 03 Schema Metadata Rules Structure and types for every field Owner, tags and data classification Validate values, transform data, migrate versions Validation moves out of every consumer and into one shared definition. Your Kafka topics are APIs · Current SF 2026
13. BUILD TIME Catch schema problems before review 01 02 03 04 Change Compile Check Register Edit the schema file in the pull request Generate classes; a broken schema fails the build Test compatibility with the registered version Publish from CI only, never at app startup Contract tests in your CI/CD pipeline give developers a build check to validate schema changes before the code reaches review. Your Kafka topics are APIs · Current SF 2026
14. STREAMHOUSE Contracts mean the Streamhouse is governed Capture Transport Transform Govern Serve Get events out of source systems Move them reliably Shape them in flight Schemas, contracts, lineage Apps, analytics and AI agents Streamhouse is an open definition for architectures that keep the current state of a business continuously available to applications, analytics and AI agents. Schemas and data contracts are where governance starts. Source: streamhouse.com/definition · Streamhouse Working Group, Sep 15, 2026
15. DEMO Break a contract, fail the build 01 02 03 Start green Break it Fix it MemberUpdate is registered and the build passes Refactor address elements to its own type Create migration rules in the schema registry to maintain compatibility Setup: https://bit.ly/4ikTy5c · Schema Registry running · Run the build Your Kafka topics are APIs · Current SF 2026
16. THE BLUEPRINT Five steps from guessing to guarantees 01 Model events for what happened, and document every field 02 Register schemas with a compatibility mode that fits your upgrades 03 Enforce structure with schema-aware serializers on both ends 04 Validate compatibility in CI, before the code reaches review 05 Quarantine anything that still fails in a dead letter queue Your Kafka topics are APIs · Current SF 2026
17. TAKEAWAY Treat the topic like the API it is. Questions? Let’s talk schemas. / / Sandon Jacobs · @SandonJacobs · https: linktr.ee/sandonjacobs