Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Commander: Better Distributed Applications through CQRS, Event Sourcing, and Immutable Logs

Commander: Better Distributed Applications through CQRS, Event Sourcing, and Immutable Logs

Talk video: https://www.youtube.com/watch?v=B1-gS0oEtYc
Reference implementation on GitHub: https://github.com/capitalone/cqrs-manager-for-distributed-reactive-services

Have you ever hit a wall with REST? Does modeling your problem domain into CRUD-able entities feel like fitting a square peg into a round hole? Perhaps instead of modeling our services like little databases, we should instead model them like reactors over immutable event streams.

REST APIs are great, but their typical implementation tightly couples various concerns that would be better separated:

* Reads (perception) with writes (action)
* Current state with historical narrative
* Business logic with HTTP request processing, and with operational concerns like metrics and monitoring

Commander is:

1. A pattern for writing distributed applications that de-couples these concerns using ideas from CQRS and Event Sourcing, thereby alleviating common frustrations with CRUD-flavored REST
2. An implementation of the core component in this pattern using Clojure in the HTTP layer, using Java with the new Kafka Streams library in the event stream processing layer, and using Datomic for persistence

The Commander pattern imposes a clear separation of action from perception, and uses immutable values conveyed by Kafka and the Kafka Streams library to separate business logic from HTTP request processing, all while preserving the historical narrative of the entire event stream.

In this talk, I'll discuss the benefits and tradeoffs of applying this pattern in an enterprise context, and both demonstrate and open-source my implementation.

Bobby Calderwood

September 17, 2016
Tweet

More Decks by Bobby Calderwood

Other Decks in Programming

Transcript

  1. Commander
    Better Distributed Applications through CQRS, Event
    Sourcing, and Immutable Logs

    View full-size slide

  2. Hi, I’m Bobby
    I’m on the Technology Fellow’s team at
    I dislike accidental complexity
    [email protected]
    @bobbycalderwood
    https://github.com/bobby

    View full-size slide

  3. Problem Space
    • Provide valuable informational and transactional
    services via Mobile and Web software
    • To lots of customers, with excellent user experience
    • Securely and in compliance with regulations
    • With ability to easily enhance, experiment, monitor,
    maintain, and operate
    • By many participants within a large organization

    View full-size slide

  4. Big Ideas
    • Immutability is central to information systems
    • Data language of system >> Programming language of
    components
    • Action and perception are not the same, and immutability
    facilitates their separation
    • Businesses services are not databases, they’re event
    stream reactors
    • Cross-cutting concerns must be satisfied in the presence
    of Conway’s Law

    View full-size slide

  5. Problematic Architecture
    /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects

    View full-size slide

  6. Immutability is central
    to information systems

    View full-size slide

  7. Event Sourcing Analogy
    Image by Alan Light CC BY-SA 3.0

    View full-size slide

  8. Data Loss by Design
    /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects

    View full-size slide

  9. The (data) language of the System
    >>
    The (runtime) language of each
    component

    View full-size slide

  10. What is system language?
    /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects

    View full-size slide

  11. Action != Perception

    View full-size slide

  12. Writes != Reads

    View full-size slide

  13. Writes Tied to Reads
    /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects

    View full-size slide

  14. Business Services
    are not Databases

    View full-size slide

  15. –John M. Culkin
    “We shape our tools and thereafter our tools
    shape us.”

    View full-size slide

  16. Database Leaking
    /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects

    View full-size slide

  17. Conway’s Law
    Yup, totally a thing

    View full-size slide

  18. –Melvin Conway
    “organizations which design systems ... are
    constrained to produce designs which are
    copies of the communication structures of these
    organizations”

    View full-size slide

  19. Cross-cutting Concerns?
    /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects

    View full-size slide

  20. We can do better!

    View full-size slide

  21. Commander
    • A better architecture for APIs and services using
    REST + Immutable Event Log + Reactive Event
    Stream Processing
    • The write-handling component of that architecture,
    my implementation is in Clojure

    View full-size slide

  22. Commander Architecture
    • Several categories of microservices with structured
    interactions among them
    • REST + CQRS + Event Sourcing + Reactive Event
    Stream Processing

    View full-size slide

  23. Commander Architecture
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [other topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    events
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit

    View full-size slide

  24. Commander Component
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [other topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    events
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit

    View full-size slide

  25. Embrace Immutability

    View full-size slide

  26. Immutable Data Log
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [other topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    events
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit

    View full-size slide

  27. Express actions in
    domain language
    (not in database language)

    View full-size slide

  28. A Command
    {"id": "33bb75db-6e13-48ee-8a54-b3976d3d065b",
    "action": "transfer-money",
    "data": {"from_account": "12345",
    "to_account": "54321",
    "amount": 10000}
    "timestamp": "2016-05-20T14:33:28.902-00:00"}

    View full-size slide

  29. An Event
    {"id": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58",
    "parent": "33bb75db-6e13-48ee-8a54-b3976d3d065b",
    "action": "money-transferred",
    "data": {"id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938",
    "from_account": "12345",
    "to_account": "54321",
    "amount": 10000},
    "timestamp": "2016-05-20T14:33:28.904-00:00"}

    View full-size slide

  30. Separate Action
    from Perception

    View full-size slide

  31. 1 Log => n Data Views
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [other topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    events
    Business Logic
    Event Log Datastores and
    Materialized Views
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    HDFS
    Elastic
    Search
    Domain-
    specific

    View full-size slide

  32. Commander Architecture
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [other topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    events
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit

    View full-size slide

  33. Exploit Conway’s Law

    View full-size slide

  34. Primary Team Provides
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [other topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    events
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit

    View full-size slide

  35. Enterprise Provides
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [other topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    events
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit

    View full-size slide

  36. Why Commander
    Component?
    • Single writer to commands topic
    • Ensuring schema conformance
    • Indexing all Commands and Events for reads and
    server-push
    • Provides optional illusion of synchrony to clients

    View full-size slide

  37. How to implement
    Reactive Services?

    View full-size slide

  38. Kafka Streams!

    View full-size slide

  39. What is Kafka?
    • Apache Kafka is publish-subscribe messaging
    rethought as a distributed commit log
    • But it’s not really about messaging, that’s just the
    interface
    • Logs > Messages for my domain
    • It provides distributed, immutable logs!

    View full-size slide

  40. What is Kafka Streams?
    • A Java library for building streaming applications on
    top of Kafka, lives in your application
    • Low-level API for building topologies of processors,
    streams, and tables
    • High-level DSL for common patterns like filter, map,
    aggregations, joins, stateful and stateless processing
    • Nice operational characteristics (low latency, elastic,
    fault-tolerant)

    View full-size slide

  41. How to use Kafka Streams
    within Commander
    • Implement Command Processor
    • Implement Event consumers and producers
    • Provide local state management as backend for
    APIs

    View full-size slide

  42. Commander Architecture
    /commands(/:id)
    /[query-apis]
    commands
    /updates
    Read-
    optimized
    View
    Command
    Processing
    Analytics
    [other topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    Web Services
    Load balancing
    Auth(n|z)
    Input validation
    events
    Business Logic
    Event Log Datastores and
    Materialized Views
    sync
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    User-facing
    APIs
    Third-party
    Partners
    WS/SSE
    audit

    View full-size slide

  43. KStreamBuilder builder = new KStreamBuilder();
    KStream commands = builder.stream(commandsTopic);
    KStream customerEvents = commands
    .filter((id, command) -> command.get("action")
    .equals("create-customer"))
    .map((id, command) -> {
    Map userEvent = new HashMap(command);
    userEvent.put("action", "customer-created");
    userEvent.put("parent", id);
    Map userValue = (Map) userEvent.get("data");
    userValue.put("id", UUID.randomUUID());
    return new KeyValue<>(UUID.randomUUID(), userEvent);
    }).through(eventsTopic);
    KStream customers = customerEvents
    .map((id, event) -> {
    Map customer = (Map) event.get("data");
    UUID customerId = (UUID) customer.get("id");
    return new KeyValue(customerId, customer);
    });
    customers.through(customersTopic);
    StateStoreSupplier store = Stores.create("Customers")
    .persistent()
    .build();
    builder.addStateStore(store);
    customers.process(customerStore, "Customers");
    this.kafkaStreams = new KafkaStreams(builder, kafkaStreamsConfig);
    this.kafkaStreams.start();

    View full-size slide

  44. public class CustomerStore implements Processor {
    private KeyValueStore store;
    public List getCustomers() {
    List customers = new ArrayList<>();
    KeyValueIterator iterator = store.all();
    while (iterator.hasNext()) {
    KeyValue entry = iterator.next();
    customers.add(new Customer(entry.value));
    }
    iterator.close();
    return customers;
    }
    public Customer getCustomer(UUID id) {
    return new Customer(store.get(id));
    }
    @Override
    public void init(ProcessorContext processorContext) {
    this.store = (KeyValueStore) context.getStateStore("Customers");
    }
    @Override
    public void process(UUID uuid, Map map) {
    store.put(uuid, map);
    }
    @Override
    public void punctuate(long l) {}
    @Override
    public void close() {}
    }

    View full-size slide

  45. Summary
    • Capture customer intent and business events as
    immutable data in domain language
    • From these action streams, services implement their
    own functionality in this common lingua franca
    • building many independent data views
    • reactively
    • without temporal or organizational coordination

    View full-size slide

  46. Giant Shoulders
    • Immutability: Rich Hickey, Stu Halloway
    • CQRS: Udi Dahan, Martin Fowler, Chris Richardson
    • Kafka Event Stream Reactors: Neha Narkhede,
    Jay Kreps, Martin Kleppmann
    • Organization and Management: Mel Conway,
    Eliyahu Goldratt, Gene Kim, Michael Nygard

    View full-size slide

  47. References
    • https://tinyurl.com/capital-one-cmdr
    • http://www.datomic.com/
    • http://blog.cognitect.com/?tag=NewNormal+Series
    • http://www.confluent.io/blog/event-sourcing-cqrs-stream-
    processing-apache-kafka-whats-connection/
    • https://engineering.linkedin.com/distributed-systems/log-what-
    every-software-engineer-should-know-about-real-time-datas-
    unifying
    • https://www.infoq.com/presentations/Value-Values

    View full-size slide