Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Decoupled, Immutable REST APIs with Kafka Streams

Decoupled, Immutable REST APIs with Kafka Streams

Have you ever hit a wall with REST? Does modeling your problem domain into CRUD-able entities feel like fitting a square peg into a round hole? Perhaps instead of modeling our services like little databases, we should instead model them like reactors to event streams.

REST APIs are great, but their typical implementation tightly couples various concerns which would be better separated:

Reads (perception) from writes (action)
Current state from historical narrative
Business logic from HTTP design from operational concerns like metrics and monitoring

I'll show a better pattern for writing REST APIs that de-couples these concerns, thereby alleviating common frustrations with CRUD-flavored REST. This pattern imposes a clear separation of action from perception and uses Kafka and Kafka Streams to separate business logic from HTTP concerns, all while preserving the historical narrative of the entire event stream.

In this talk, I’ll discuss the benefits and tradeoffs of this approach, and demonstrate my implementation using Clojure in the HTTP layer, and using Java with the new Kafka Streams library in the event stream processing layer.

C4adca5489db5657ba31ca253525147f?s=128

Bobby Calderwood

July 14, 2018
Tweet

More Decks by Bobby Calderwood

Other Decks in Programming

Transcript

  1. DEV CON ATLANTIC MID Cloud — Mobile — Web —

    Dev Thanks to Our Sponsors
  2. Copyright 2018, Evident Systems LLC Decoupled, Immutable Rest APIs With

    Kafka Streams
  3. Hi, I’m Bobby I’m the Founder of Evident Systems, LLC

    We help companies re-assert ownership over their core systems and data bobby@evidentsystems.com @bobbycalderwood https://github.com/bobby
  4. Problem Space • Provide valuable informational and transactional services via

    Mobile and Web software • To lots of customers, with excellent user experience • Securely and in compliance with regulations • With ability to easily enhance, experiment, monitor, maintain, and operate • By many participants within a large organization
  5. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and

    Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Problematic Architecture
  6. Big Ideas from Functional Programming • Immutability is central to

    information systems • Data language of system >> Programming language of components • Action and perception are not the same, and immutability facilitates their separation • Businesses services are not databases, they’re event stream reactors • Cross-cutting concerns must be satisfied in the presence of Conway’s Law
  7. Immutability is central to information systems

  8. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and

    Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Data Loss By Design
  9. The (data) language of the System >> The (runtime) language

    of each component
  10. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and

    Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects What is the System Language?
  11. Action != Perception

  12. Writes != Reads

  13. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and

    Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Writes Tied to Reads
  14. Business Services are not Databases

  15. –John M. Culkin “We shape our tools and thereafter our

    tools shape us.”
  16. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and

    Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Database as Leaky Abstraction
  17. Conway’s Law Yup, totally a thing

  18. –Melvin Conway “organizations which design systems ... are constrained to

    produce designs which are copies of the communication structures of these organizations.”
  19. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and

    Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Cross-cutting Concerns?
  20. Object Oriented-ish • Encapsulated data access via synchronous calls •

    Mutable state change via synchronous calls • Dependency web • Imperative, sequence-oriented orchestration • Referentially opaque
  21. OO thinking doesn’t distribute!

  22. AWS Death Star diagram, circa 2008 as per Werner Vogels

    tweet https://twitter.com/Werner/status/741673514567143424
  23. We can do better!

  24. Source: NASA, Public Domain

  25. Functional-ish • Data access via aggregating immutable values • State

    change via mapping an identity to different immutable values over time • Data Flow graph • Declarative orchestration • Referential transparency
  26. In Contrast Object Oriented Functional Data Access Encapsulated Ubiquitous State

    Change Mutable, in-place Immutable, values over time Organization Dependency graph Data-flow graph Orchestration Imperative, sequential Declarative, parallelizable Referentially… Opaque Transparent
  27. But How? Techniques, Rules, and Tools

  28. Event Sourcing Image by Alan Light CC BY-SA 3.0

  29. REST + CQRS + Event Sourcing

  30. Embrace Immutability

  31. Capture changes “at the edge” to immutable log • A

    (very) few authorized teams capture all raw observations (Events) and customer requests for action (Commands) • at the edge of bounded context • with minimal processing • immutably and durably • Causally related events go on same log (Kafka Topic Partition) • One (logical) writer per command/event stream • No change gets into bounded context via any other means!
  32. Capture changes “at the edge” to immutable log

  33. Express actions in domain language (not in database language)

  34. A Command {"id": "33bb75db-6e13-48ee-8a54-b3976d3d065b", "action": "transfer-money", "data": {"from_account": "12345", "to_account":

    "54321", "amount": 10000} "timestamp": "2016-05-20T14:33:28.902-00:00"}
  35. An Event {"id": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58", "parent": "33bb75db-6e13-48ee-8a54-b3976d3d065b", "action": "money-transferred", "data": {"id":

    "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938", "from_account": "12345", "to_account": "54321", "amount": 10000}, "timestamp": "2016-05-20T14:33:28.904-00:00"}
  36. A Domain Entity {"account-id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938", "account-name": "My Awesome Checking Account",

    "balance": 1000, "as-of": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58"}
  37. Separate Action from Perception

  38. Reactively calculate derived state from event log • A few

    authorized teams process Commands into Events (probably using aggregate state) • A few (more) authorized teams calculate state streams of general interest derived from Events • Single-entity state changes • Regulated or audited calculations • Could be recursive, i.e. certain state changes might trigger further events downstream
  39. Reactively calculate derived state from event log

  40. Read- optimized View Command Processing Analytics [other topics] Arbitrary command

    action Arbitrary command action Microconsume rs events Business Logic Event Log Datastores and Materialized Views “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit HDFS Elastic Search Domain- specific Aggregate state wherever and however it provides value
  41. Aggregate state wherever and however it provides value • Many

    teams across orgs aggregate streams into views for their respective use-cases • Into whatever data store makes sense for required data access pattern • Possibly emitting views back onto streams (a la Kafka Streams’ KTable) • Facilitates stateful computations: joins, windowed aggregates, command processing • Everyone traces lineage all the way back to edge Command/Event
  42. Exploit Conway’s Law to Evolve as Needed

  43. Application Team Provides

  44. Cross-cutting Teams Provide

  45. Manage “side-effects” carefully • A few authorized teams react to

    Events by reaching outside of bounded context to cause “side-effects” • A “side-effect” in this case is any action not associated with reading or writing the streams or aggregates, e.g. • Sending email, text message • Making a call to an outside web service • Calling a service to write a command/event to a stream you don’t own • Log the results of attempt to a stream, to facilitate retry/Saga-rollback/compensating- action
  46. Manage “side-effects” carefully

  47. Summary • Capture customer intent and business events at the

    edge as immutable data in domain language • From these action streams, services implement their own functionality via this common lingua franca • building many independent data views • reactively • without temporal or organizational coordination
  48. Giant Shoulders • Immutability: Rich Hickey, Stu Halloway • CQRS:

    Udi Dahan, Martin Fowler, Chris Richardson • Kafka and Streaming Data: Neha Narkhede, Jay Kreps, Martin Kleppmann, Ben Stopford • Organization and Management: Mel Conway, Eliyahu Goldratt, Gene Kim, Michael Nygard
  49. References • https://www.confluent.io/blog/toward-functional-programming-analogy- microservices/ • https://engineering.linkedin.com/distributed-systems/log-what-every-software- engineer-should-know-about-real-time-datas-unifying • http://blog.cognitect.com/?tag=NewNormal+Series •

    https://www.youtube.com/watch?v=2V1FtfBDsLU • http://www.datomic.com/ • https://www.infoq.com/presentations/Value-Values
  50. Shameless Plug Evident Systems provides: • API & Microservices Design

    Training: https://hubs.ly/H0d0lsN0 • API & Microservices Implementation Training: https://hubs.ly/H0d0lsT0 We’re also Hiring!