Commander Decoupled, Immutable REST APIs with Kafka Streams 

Hi, I’m Bobby I’m on the Technology Fellow’s team at I dislike accidental complexity [email protected] @bobbycalderwood https://github.com/bobby 

Problem Space • Provide valuable informational and transactional services via Mobile and Web software • To lots of customers, with excellent user experience • Securely and in compliance with regulations • With ability to easily enhance, experiment, monitor, maintain, and operate • By many participants within a large organization 

Big Ideas • Immutability is central to information systems • Data language of system >> Programming language of components • Action and perception are not the same, and immutability facilitates their separation • Businesses services are not databases, they’re event stream reactors • Cross-cutting concerns must be satisfied in the presence of Conway’s Law 

Problematic Architecture /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects 

Immutability is central to information systems 

An Analogy Image by Alan Light CC BY-SA 3.0 

Data Loss by Design /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects 

The (data) language of the System >> The (runtime) language of each component 

What is system language? /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects 

Action != Perception 

Writes != Reads 

Writes Tied to Reads /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects 

Business Services are not Databases 

–John M. Culkin “We shape our tools and thereafter our tools shape us.” 

Database Leaking /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects 

Conway’s Law Yup, totally a thing 

–Melvin Conway “organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations” 

Cross-cutting Concerns? /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects 

We can do better! 

Commander • A better architecture for APIs and services using REST + Immutable Event Log + Reactive Event Stream Processing • The write-handling component of that architecture, my implementation is in Clojure 

Commander Architecture • Several categories of microservices with structured interactions among them • REST + CQRS + Event Sourcing + Reactive Event Stream Processing 

Commander Architecture /commands(/:id) /[query-apis] commands /updates Read- optimized View Command Processing Analytics [events topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation command-results Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit 

Commander Component /commands(/:id) /[query-apis] commands /updates Read- optimized View Command Processing Analytics [events topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation command-results Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit 

Embrace Immutability 

Immutable Data Log /commands(/:id) /[query-apis] commands /updates Read- optimized View Command Processing Analytics [events topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation command-results Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit 

Express actions in domain language (not in database language) 

A Command {"id": "33bb75db-6e13-48ee-8a54-b3976d3d065b", "action": "transfer-money", "data": {"from_account": "12345", "to_account": "54321", "amount": 10000} "timestamp": "2016-05-20T14:33:28.902-00:00"} 

An Event {"id": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58", "parent": "33bb75db-6e13-48ee-8a54-b3976d3d065b", "action": "money-transferred", "data": {"id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938", "from_account": "12345", "to_account": "54321", "amount": 10000}, "timestamp": "2016-05-20T14:33:28.904-00:00"} 

Separate Action from Perception 

Commander Architecture /commands(/:id) /[query-apis] commands /updates Read- optimized View Command Processing Analytics [events topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation command-results Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit 

Exploit Conway’s Law 

Primary Team Provides /commands(/:id) /[query-apis] commands /updates Read- optimized View Command Processing Analytics [events topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation command-results Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit 

Enterprise Provides /commands(/:id) /[query-apis] commands /updates Read- optimized View Command Processing Analytics [events topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation command-results Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit 

Kafka Streams! 

What is Kafka? • Apache Kafka is publish-subscribe messaging rethought as a distributed commit log • But it’s not really about messaging, that’s just the interface • Logs > Messages for my domain • It provides distributed, immutable logs! 

What is Kafka Streams? • A Java library for building streaming applications on top of Kafka, lives in your application • Low-level API for building topologies of processors, streams, and tables • High-level DSL for common patterns like filter, map, aggregations, joins, stateful and stateless processing • Nice operational characteristics (low latency, elastic, fault-tolerant) 

How to use Kafka Streams within Commander • Implement Command Processor • Implement Event consumers and producers • Provide local state management as backend for APIs 

Commander Architecture /commands(/:id) /[query-apis] commands /updates Read- optimized View Command Processing Analytics [events topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation command-results Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit 

KStreamBuilder builder = new KStreamBuilder(); KStream commands = builder.stream(commandsTopic); KStream customerEvents = commands .filter((id, command) -> command.get("action") .equals("create-customer")) .map((id, command) -> { Map userEvent = new HashMap(command); userEvent.put("action", "customer-created"); userEvent.put("parent", id); Map userValue = (Map) userEvent.get("data"); userValue.put("id", UUID.randomUUID()); return new KeyValue<>(UUID.randomUUID(), userEvent); }).through(eventsTopic); KStream customers = customerEvents .map((id, event) -> { Map customer = (Map) event.get("data"); UUID customerId = (UUID) customer.get("id"); return new KeyValue(customerId, customer); }); customers.through(customersTopic); StateStoreSupplier store = Stores.create("Customers") .persistent() .build(); builder.addStateStore(store); customers.process(customerStore, "Customers"); this.kafkaStreams = new KafkaStreams(builder, kafkaStreamsConfig); this.kafkaStreams.start(); 

public class CustomerStore implements Processor { private KeyValueStore store; public List getCustomers() { List customers = new ArrayList<>(); KeyValueIterator iterator = store.all(); while (iterator.hasNext()) { KeyValue entry = iterator.next(); customers.add(new Customer(entry.value)); } iterator.close(); return customers; } public Customer getCustomer(UUID id) { return new Customer(store.get(id)); } @Override public void init(ProcessorContext processorContext) { this.store = (KeyValueStore) context.getStateStore("Customers"); } @Override public void process(UUID uuid, Map map) { store.put(uuid, map); } @Override public void punctuate(long l) {} @Override public void close() {} } 

Demo 

High-level Benefits • Captures customer intent and business events as immutable data • Allow simple evolution of business logic and the addition of new features and integrations • De-coupled cooperation between organizations via a common data lingua franca used by all • Provides many “non-functional” requirements for free! 

Developer Benefits • Not language-specific! Supports existing talent base and/or a polyglot approach • Reduce cognitive load by working with simple, single-purpose pieces and a common data abstraction • Properly divides responsibility between central teams and product teams; provides just enough structure. 

Operational Benefits • Operational monitoring and metrics extensibility built-in! • Better write performance: just write to Kafka • Better read performance: read-optimized databases store exactly what APIs serve • Helps us toward DevOps, microservices, continuous delivery, server-less architectures • Doesn’t require run-time dependency among many services 

Business Benefits • Audit and compliance log built-in! Clear data provenance • Analytics and reporting extensibility built-in! Fast-data first! • Increased agility, velocity from working with small pieces • Ability to A/B test and experiment with new features or offerings with no impact to existing systems • Lower cost of Dev and Ops 

Customer Benefits • Better performing APIs on both read and write ops • More real-time perception and interaction • New products and features quickly • We remember the whole story of our customer interactions with us 

Path • Pursue this idea in parallel with existing systems! 1. Instrument write APIs, either individually or at API gateway, to write commands to Kafka before doing regular processing 2. Build processing infrastructure for each type of command 3. Build read APIs for each event stream 4. Compare results from existing systems to those of new system 5. Cut over to new system when appropriate 

Giant Shoulders • Immutability: Rich Hickey, Stu Halloway • CQRS: Udi Dahan, Martin Fowler, Chris Richardson • Kafka Event Stream Reactors: Neha Narkhede, Jay Kreps, Martin Kleppmann • Organization and Management: Mel Conway, Eliyahu Goldratt, Gene Kim, Michael Nygard 

References • http://www.datomic.com/ • http://blog.cognitect.com/?tag=NewNormal+Series • https://martin.kleppmann.com/2015/03/04/turning-the- database-inside-out.html • https://engineering.linkedin.com/distributed-systems/ log-what-every-software-engineer-should-know-about- real-time-datas-unifying • https://www.infoq.com/presentations/Value-Values