Commander: Decoupled, Immutable REST APIs with Kafka Streams

PLEASE SEE ALSO MY STRANGELOOP TALK, WHICH INCLUDES VIDEO: https://speakerdeck.com/bobbycalderwood/commander-better-distributed-applications-through-cqrs-event-sourcing-and-immutable-logs

Have you ever hit a wall with REST? Does modeling your problem domain into CRUD-able entities feel like fitting a square peg into a round hole? Perhaps instead of modeling our services like little databases, we should instead model them like reactors to event streams. REST APIs are great, but their typical implementation tightly couples various concerns that would be better separated:

* Reads (perception) from writes (action)
* Current state from historical narrative
* Business logic from HTTP design from operational concerns like metrics and monitoring

Commander is a pattern for writing REST APIs that de-couples these concerns, thereby alleviating common frustrations with CRUD-flavored REST. This pattern imposes a clear separation of action from perception, and uses immutable values conveyed by Kafka and the Kafka Streams library to separate business logic from HTTP concerns, all while preserving the historical narrative of the entire event stream. In this talk, I'll discuss the benefits and tradeoffs of this approach, and demonstrate my implementation using Clojure in the HTTP layer, and using Java with the new Kafka Streams library in the event stream processing layer.

Transcript

1. Commander
   Decoupled, Immutable REST APIs with Kafka Streams
   

   View Slide

2. Hi, I’m Bobby
   I’m on the Technology Fellow’s team at
   I dislike accidental complexity
   bobby.calderwood@capitalone.com
   @bobbycalderwood
   https://github.com/bobby
   

   View Slide

3. Problem Space
   • Provide valuable informational and transactional
   services via Mobile and Web software
   • To lots of customers, with excellent user experience
   • Securely and in compliance with regulations
   • With ability to easily enhance, experiment, monitor,
   maintain, and operate
   • By many participants within a large organization
   

   View Slide

4. Big Ideas
   • Immutability is central to information systems
   • Data language of system >> Programming language of
   components
   • Action and perception are not the same, and immutability
   facilitates their separation
   • Businesses services are not databases, they’re event
   stream reactors
   • Cross-cutting concerns must be satisfied in the presence
   of Conway’s Law
   

   View Slide

5. Problematic Architecture
   /[resource](/:id) Database
   Logic Analytics
   Web Services Business Logic Databases and
   Services
   Operations
   Metrics/Monitoring
   Security/Compliance
   Audit
   Afterthoughts
   Other side-effects
   

   View Slide

6. Immutability is central
   to information systems
   

   View Slide

7. An Analogy
   Image by Alan Light CC BY-SA 3.0
   

   View Slide

8. Data Loss by Design
   /[resource](/:id) Database
   Logic Analytics
   Web Services Business Logic Databases and
   Services
   Operations
   Metrics/Monitoring
   Security/Compliance
   Audit
   Afterthoughts
   Other side-effects
   

   View Slide

9. The (data) language of the System
   >>
   The (runtime) language of each
   component
   

   View Slide

10. What is system language?
    /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    

    View Slide

11. Action != Perception
    

    View Slide

12. Writes != Reads
    

    View Slide

13. Writes Tied to Reads
    /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    

    View Slide

14. Business Services
    are not Databases
    

    View Slide

15. –John M. Culkin
    “We shape our tools and thereafter our tools
    shape us.”
    

    View Slide

16. Database Leaking
    /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    

    View Slide

17. Conway’s Law
    Yup, totally a thing
    

    View Slide

18. –Melvin Conway
    “organizations which design systems ... are
    constrained to produce designs which are
    copies of the communication structures of these
    organizations”
    

    View Slide

19. Cross-cutting Concerns?
    /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    

    View Slide

20. We can do better!
    

    View Slide

21. Commander
    • A better architecture for APIs and services using
    REST + Immutable Event Log + Reactive Event
    Stream Processing
    • The write-handling component of that architecture,
    my implementation is in Clojure
    

    View Slide

22. Commander Architecture
    • Several categories of microservices with structured
    interactions among them
    • REST + CQRS + Event Sourcing + Reactive Event
    Stream Processing
    

    View Slide

23. Commander Architecture
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [events topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    command-results
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit
    

    View Slide

24. Commander Component
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [events topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    command-results
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit
    

    View Slide

25. Embrace Immutability
    

    View Slide

26. Immutable Data Log
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [events topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    command-results
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit
    

    View Slide

27. Express actions in
    domain language
    (not in database language)
    

    View Slide

28. A Command
    {"id": "33bb75db-6e13-48ee-8a54-b3976d3d065b",
    "action": "transfer-money",
    "data": {"from_account": "12345",
    "to_account": "54321",
    "amount": 10000}
    "timestamp": "2016-05-20T14:33:28.902-00:00"}
    

    View Slide

29. An Event
    {"id": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58",
    "parent": "33bb75db-6e13-48ee-8a54-b3976d3d065b",
    "action": "money-transferred",
    "data": {"id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938",
    "from_account": "12345",
    "to_account": "54321",
    "amount": 10000},
    "timestamp": "2016-05-20T14:33:28.904-00:00"}
    

    View Slide

30. Separate Action
    from Perception
    

    View Slide

31. Commander Architecture
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [events topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    command-results
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit
    

    View Slide

32. Exploit Conway’s Law
    

    View Slide

33. Primary Team Provides
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [events topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    command-results
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit
    

    View Slide

34. Enterprise Provides
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [events topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    command-results
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit
    

    View Slide

35. Kafka Streams!
    

    View Slide

36. What is Kafka?
    • Apache Kafka is publish-subscribe messaging
    rethought as a distributed commit log
    • But it’s not really about messaging, that’s just the
    interface
    • Logs > Messages for my domain
    • It provides distributed, immutable logs!
    

    View Slide

37. What is Kafka Streams?
    • A Java library for building streaming applications on
    top of Kafka, lives in your application
    • Low-level API for building topologies of processors,
    streams, and tables
    • High-level DSL for common patterns like filter, map,
    aggregations, joins, stateful and stateless processing
    • Nice operational characteristics (low latency, elastic,
    fault-tolerant)
    

    View Slide

38. How to use Kafka Streams
    within Commander
    • Implement Command Processor
    • Implement Event consumers and producers
    • Provide local state management as backend for
    APIs
    

    View Slide

39. Commander Architecture
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [events topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    command-results
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit
    

    View Slide

40. KStreamBuilder builder = new KStreamBuilder();
    KStream commands = builder.stream(commandsTopic);
    KStream customerEvents = commands
    .filter((id, command) -> command.get("action")
    .equals("create-customer"))
    .map((id, command) -> {
    Map userEvent = new HashMap(command);
    userEvent.put("action", "customer-created");
    userEvent.put("parent", id);
    Map userValue = (Map) userEvent.get("data");
    userValue.put("id", UUID.randomUUID());
    return new KeyValue<>(UUID.randomUUID(), userEvent);
    }).through(eventsTopic);
    KStream customers = customerEvents
    .map((id, event) -> {
    Map customer = (Map) event.get("data");
    UUID customerId = (UUID) customer.get("id");
    return new KeyValue(customerId, customer);
    });
    customers.through(customersTopic);
    StateStoreSupplier store = Stores.create("Customers")
    .persistent()
    .build();
    builder.addStateStore(store);
    customers.process(customerStore, "Customers");
    this.kafkaStreams = new KafkaStreams(builder, kafkaStreamsConfig);
    this.kafkaStreams.start();
    

    View Slide

41. public class CustomerStore implements Processor {
    private KeyValueStore store;
    public List getCustomers() {
    List customers = new ArrayList<>();
    KeyValueIterator iterator = store.all();
    while (iterator.hasNext()) {
    KeyValue entry = iterator.next();
    customers.add(new Customer(entry.value));
    }
    iterator.close();
    return customers;
    }
    public Customer getCustomer(UUID id) {
    return new Customer(store.get(id));
    }
    @Override
    public void init(ProcessorContext processorContext) {
    this.store = (KeyValueStore) context.getStateStore("Customers");
    }
    @Override
    public void process(UUID uuid, Map map) {
    store.put(uuid, map);
    }
    @Override
    public void punctuate(long l) {}
    @Override
    public void close() {}
    }
    

    View Slide

42. Demo
    

    View Slide

43. High-level Benefits
    • Captures customer intent and business events as
    immutable data
    • Allow simple evolution of business logic and the
    addition of new features and integrations
    • De-coupled cooperation between organizations via
    a common data lingua franca used by all
    • Provides many “non-functional” requirements for
    free!
    

    View Slide

44. Developer Benefits
    • Not language-specific! Supports existing talent
    base and/or a polyglot approach
    • Reduce cognitive load by working with simple,
    single-purpose pieces and a common data
    abstraction
    • Properly divides responsibility between central
    teams and product teams; provides just enough
    structure.
    

    View Slide

45. Operational Benefits
    • Operational monitoring and metrics extensibility built-in!
    • Better write performance: just write to Kafka
    • Better read performance: read-optimized databases
    store exactly what APIs serve
    • Helps us toward DevOps, microservices, continuous
    delivery, server-less architectures
    • Doesn’t require run-time dependency among many
    services
    

    View Slide

46. Business Benefits
    • Audit and compliance log built-in! Clear data
    provenance
    • Analytics and reporting extensibility built-in! Fast-data
    first!
    • Increased agility, velocity from working with small pieces
    • Ability to A/B test and experiment with new features or
    offerings with no impact to existing systems
    • Lower cost of Dev and Ops
    

    View Slide

47. Customer Benefits
    • Better performing APIs on both read and write ops
    • More real-time perception and interaction
    • New products and features quickly
    • We remember the whole story of our customer
    interactions with us
    

    View Slide

48. Path
    • Pursue this idea in parallel with existing systems!
    1. Instrument write APIs, either individually or at API gateway,
    to write commands to Kafka before doing regular
    processing
    2. Build processing infrastructure for each type of command
    3. Build read APIs for each event stream
    4. Compare results from existing systems to those of new
    system
    5. Cut over to new system when appropriate
    

    View Slide

49. Giant Shoulders
    • Immutability: Rich Hickey, Stu Halloway
    • CQRS: Udi Dahan, Martin Fowler, Chris Richardson
    • Kafka Event Stream Reactors: Neha Narkhede,
    Jay Kreps, Martin Kleppmann
    • Organization and Management: Mel Conway,
    Eliyahu Goldratt, Gene Kim, Michael Nygard
    

    View Slide

50. References
    • http://www.datomic.com/
    • http://blog.cognitect.com/?tag=NewNormal+Series
    • https://martin.kleppmann.com/2015/03/04/turning-the-
    database-inside-out.html
    • https://engineering.linkedin.com/distributed-systems/
    log-what-every-software-engineer-should-know-about-
    real-time-datas-unifying
    • https://www.infoq.com/presentations/Value-Values
    

    View Slide