Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Decoupled, Immutable REST APIs with Kafka Streams

Decoupled, Immutable REST APIs with Kafka Streams

Have you ever hit a wall with REST? Does modeling your problem domain into CRUD-able entities feel like fitting a square peg into a round hole? Perhaps instead of modeling our services like little databases, we should instead model them like reactors to event streams.

REST APIs are great, but their typical implementation tightly couples various concerns which would be better separated:

Reads (perception) from writes (action)
Current state from historical narrative
Business logic from HTTP design from operational concerns like metrics and monitoring

I'll show a better pattern for writing REST APIs that de-couples these concerns, thereby alleviating common frustrations with CRUD-flavored REST. This pattern imposes a clear separation of action from perception and uses Kafka and Kafka Streams to separate business logic from HTTP concerns, all while preserving the historical narrative of the entire event stream.

In this talk, I’ll discuss the benefits and tradeoffs of this approach, and demonstrate my implementation using Clojure in the HTTP layer, and using Java with the new Kafka Streams library in the event stream processing layer.

Bobby Calderwood

July 14, 2018
Tweet

More Decks by Bobby Calderwood

Other Decks in Programming

Transcript

  1. DEV CON
    ATLANTIC
    MID
    Cloud — Mobile — Web — Dev
    Thanks to
    Our Sponsors

    View Slide

  2. Copyright 2018, Evident Systems LLC
    Decoupled, Immutable Rest APIs
    With Kafka Streams

    View Slide

  3. Hi, I’m Bobby
    I’m the Founder of Evident Systems, LLC
    We help companies re-assert ownership over their core systems and data
    [email protected]
    @bobbycalderwood
    https://github.com/bobby

    View Slide

  4. Problem Space
    • Provide valuable informational and transactional services via Mobile
    and Web software
    • To lots of customers, with excellent user experience
    • Securely and in compliance with regulations
    • With ability to easily enhance, experiment, monitor, maintain, and
    operate
    • By many participants within a large organization

    View Slide

  5. /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    Problematic Architecture

    View Slide

  6. Big Ideas from Functional Programming
    • Immutability is central to information systems
    • Data language of system >> Programming language of components
    • Action and perception are not the same, and immutability facilitates
    their separation
    • Businesses services are not databases, they’re event stream reactors
    • Cross-cutting concerns must be satisfied in the presence of Conway’s
    Law

    View Slide

  7. Immutability is central to information
    systems

    View Slide

  8. /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    Data Loss By Design

    View Slide

  9. The (data) language of the System
    >>
    The (runtime) language of each component

    View Slide

  10. /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    What is the System Language?

    View Slide

  11. Action != Perception

    View Slide

  12. Writes != Reads

    View Slide

  13. /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    Writes Tied to Reads

    View Slide

  14. Business Services are not Databases

    View Slide

  15. –John M. Culkin
    “We shape our tools and thereafter our tools shape us.”

    View Slide

  16. /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    Database as Leaky Abstraction

    View Slide

  17. Conway’s Law
    Yup, totally a thing

    View Slide

  18. –Melvin Conway
    “organizations which design systems ... are constrained to produce
    designs which are copies of the communication structures of
    these organizations.”

    View Slide

  19. /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    Cross-cutting Concerns?

    View Slide

  20. Object Oriented-ish
    • Encapsulated data access via synchronous calls
    • Mutable state change via synchronous calls
    • Dependency web
    • Imperative, sequence-oriented orchestration
    • Referentially opaque

    View Slide

  21. OO thinking doesn’t distribute!

    View Slide

  22. AWS Death Star diagram, circa 2008 as per Werner Vogels tweet https://twitter.com/Werner/status/741673514567143424

    View Slide

  23. We can do better!

    View Slide

  24. Source: NASA, Public Domain

    View Slide

  25. Functional-ish
    • Data access via aggregating immutable values
    • State change via mapping an identity to different immutable values
    over time
    • Data Flow graph
    • Declarative orchestration
    • Referential transparency

    View Slide

  26. In Contrast
    Object Oriented Functional
    Data Access Encapsulated Ubiquitous
    State Change Mutable, in-place Immutable, values over time
    Organization Dependency graph Data-flow graph
    Orchestration Imperative, sequential Declarative, parallelizable
    Referentially… Opaque Transparent

    View Slide

  27. But How?
    Techniques, Rules, and Tools

    View Slide

  28. Event Sourcing
    Image by Alan Light CC BY-SA 3.0

    View Slide

  29. REST + CQRS + Event Sourcing

    View Slide

  30. Embrace Immutability

    View Slide

  31. Capture changes “at the edge” to immutable log
    • A (very) few authorized teams capture all raw observations (Events) and customer
    requests for action (Commands)
    • at the edge of bounded context
    • with minimal processing
    • immutably and durably
    • Causally related events go on same log (Kafka Topic Partition)
    • One (logical) writer per command/event stream
    • No change gets into bounded context via any other means!

    View Slide

  32. Capture changes “at the edge” to immutable log

    View Slide

  33. Express actions in domain language
    (not in database language)

    View Slide

  34. A Command
    {"id": "33bb75db-6e13-48ee-8a54-b3976d3d065b",
    "action": "transfer-money",
    "data": {"from_account": "12345",
    "to_account": "54321",
    "amount": 10000}
    "timestamp": "2016-05-20T14:33:28.902-00:00"}

    View Slide

  35. An Event
    {"id": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58",
    "parent": "33bb75db-6e13-48ee-8a54-b3976d3d065b",
    "action": "money-transferred",
    "data": {"id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938",
    "from_account": "12345",
    "to_account": "54321",
    "amount": 10000},
    "timestamp": "2016-05-20T14:33:28.904-00:00"}

    View Slide

  36. A Domain Entity
    {"account-id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938",
    "account-name": "My Awesome Checking Account",
    "balance": 1000,
    "as-of": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58"}

    View Slide

  37. Separate Action from Perception

    View Slide

  38. Reactively calculate derived state from event log
    • A few authorized teams process Commands into Events (probably using
    aggregate state)
    • A few (more) authorized teams calculate state streams of general interest
    derived from Events
    • Single-entity state changes
    • Regulated or audited calculations
    • Could be recursive, i.e. certain state changes might trigger further events
    downstream

    View Slide

  39. Reactively calculate derived state from event log

    View Slide

  40. Read-
    optimized
    View
    Command
    Processing
    Analytics
    [other topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    events
    Business Logic
    Event Log Datastores and
    Materialized Views
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    HDFS
    Elastic
    Search
    Domain-
    specific
    Aggregate state wherever and however it
    provides value

    View Slide

  41. Aggregate state wherever and however it
    provides value
    • Many teams across orgs aggregate streams into views for their
    respective use-cases
    • Into whatever data store makes sense for required data access pattern
    • Possibly emitting views back onto streams (a la Kafka Streams’ KTable)
    • Facilitates stateful computations: joins, windowed aggregates, command
    processing
    • Everyone traces lineage all the way back to edge Command/Event

    View Slide

  42. Exploit Conway’s Law to Evolve as Needed

    View Slide

  43. Application Team Provides

    View Slide

  44. Cross-cutting Teams Provide

    View Slide

  45. Manage “side-effects” carefully
    • A few authorized teams react to Events by reaching outside of bounded context to cause
    “side-effects”
    • A “side-effect” in this case is any action not associated with reading or writing the
    streams or aggregates, e.g.
    • Sending email, text message
    • Making a call to an outside web service
    • Calling a service to write a command/event to a stream you don’t own
    • Log the results of attempt to a stream, to facilitate retry/Saga-rollback/compensating-
    action

    View Slide

  46. Manage “side-effects” carefully

    View Slide

  47. Summary
    • Capture customer intent and business events at the edge as immutable
    data in domain language
    • From these action streams, services implement their own functionality
    via this common lingua franca
    • building many independent data views
    • reactively
    • without temporal or organizational coordination

    View Slide

  48. Giant Shoulders
    • Immutability: Rich Hickey, Stu Halloway
    • CQRS: Udi Dahan, Martin Fowler, Chris Richardson
    • Kafka and Streaming Data: Neha Narkhede, Jay Kreps, Martin
    Kleppmann, Ben Stopford
    • Organization and Management: Mel Conway, Eliyahu Goldratt, Gene
    Kim, Michael Nygard

    View Slide

  49. References
    • https://www.confluent.io/blog/toward-functional-programming-analogy-
    microservices/
    • https://engineering.linkedin.com/distributed-systems/log-what-every-software-
    engineer-should-know-about-real-time-datas-unifying
    • http://blog.cognitect.com/?tag=NewNormal+Series
    • https://www.youtube.com/watch?v=2V1FtfBDsLU
    • http://www.datomic.com/
    • https://www.infoq.com/presentations/Value-Values

    View Slide

  50. Shameless Plug
    Evident Systems provides:
    • API & Microservices Design Training: https://hubs.ly/H0d0lsN0
    • API & Microservices Implementation Training: https://hubs.ly/H0d0lsT0
    We’re also Hiring!

    View Slide