Decoupled, Immutable REST APIs with Kafka Streams

Have you ever hit a wall with REST? Does modeling your problem domain into CRUD-able entities feel like fitting a square peg into a round hole? Perhaps instead of modeling our services like little databases, we should instead model them like reactors to event streams.

REST APIs are great, but their typical implementation tightly couples various concerns which would be better separated:

Reads (perception) from writes (action)
Current state from historical narrative
Business logic from HTTP design from operational concerns like metrics and monitoring

I'll show a better pattern for writing REST APIs that de-couples these concerns, thereby alleviating common frustrations with CRUD-flavored REST. This pattern imposes a clear separation of action from perception and uses Kafka and Kafka Streams to separate business logic from HTTP concerns, all while preserving the historical narrative of the entire event stream.

In this talk, I’ll discuss the benefits and tradeoffs of this approach, and demonstrate my implementation using Clojure in the HTTP layer, and using Java with the new Kafka Streams library in the event stream processing layer.

Transcript

1. DEV CON ATLANTIC MID Cloud — Mobile — Web — Dev Thanks to Our Sponsors
2. Copyright 2018, Evident Systems LLC Decoupled, Immutable Rest APIs With Kafka Streams
3. Hi, I’m Bobby I’m the Founder of Evident Systems, LLC We help companies re-assert ownership over their core systems and data bobby@evidentsystems.com @bobbycalderwood https://github.com/bobby
4. Problem Space • Provide valuable informational and transactional services via Mobile and Web software • To lots of customers, with excellent user experience • Securely and in compliance with regulations • With ability to easily enhance, experiment, monitor, maintain, and operate • By many participants within a large organization
5. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Problematic Architecture
6. Big Ideas from Functional Programming • Immutability is central to information systems • Data language of system >> Programming language of components • Action and perception are not the same, and immutability facilitates their separation • Businesses services are not databases, they’re event stream reactors • Cross-cutting concerns must be satisfied in the presence of Conway’s Law
7. Immutability is central to information systems
8. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Data Loss By Design
9. The (data) language of the System >> The (runtime) language of each component
10. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects What is the System Language?
11. Action != Perception
12. Writes != Reads
13. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Writes Tied to Reads
14. Business Services are not Databases
15. –John M. Culkin “We shape our tools and thereafter our tools shape us.”
16. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Database as Leaky Abstraction
17. Conway’s Law Yup, totally a thing
18. –Melvin Conway “organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.”
19. /[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Cross-cutting Concerns?
20. Object Oriented-ish • Encapsulated data access via synchronous calls • Mutable state change via synchronous calls • Dependency web • Imperative, sequence-oriented orchestration • Referentially opaque
21. OO thinking doesn’t distribute!
22. AWS Death Star diagram, circa 2008 as per Werner Vogels tweet https://twitter.com/Werner/status/741673514567143424
23. We can do better!
24. Source: NASA, Public Domain
25. Functional-ish • Data access via aggregating immutable values • State change via mapping an identity to different immutable values over time • Data Flow graph • Declarative orchestration • Referential transparency
26. In Contrast Object Oriented Functional Data Access Encapsulated Ubiquitous State Change Mutable, in-place Immutable, values over time Organization Dependency graph Data-flow graph Orchestration Imperative, sequential Declarative, parallelizable Referentially… Opaque Transparent
27. But How? Techniques, Rules, and Tools
28. Event Sourcing Image by Alan Light CC BY-SA 3.0
29. REST + CQRS + Event Sourcing
30. Embrace Immutability
31. Capture changes “at the edge” to immutable log • A (very) few authorized teams capture all raw observations (Events) and customer requests for action (Commands) • at the edge of bounded context • with minimal processing • immutably and durably • Causally related events go on same log (Kafka Topic Partition) • One (logical) writer per command/event stream • No change gets into bounded context via any other means!
32. Capture changes “at the edge” to immutable log
33. Express actions in domain language (not in database language)
34. A Command {"id": "33bb75db-6e13-48ee-8a54-b3976d3d065b", "action": "transfer-money", "data": {"from_account": "12345", "to_account": "54321", "amount": 10000} "timestamp": "2016-05-20T14:33:28.902-00:00"}
35. An Event {"id": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58", "parent": "33bb75db-6e13-48ee-8a54-b3976d3d065b", "action": "money-transferred", "data": {"id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938", "from_account": "12345", "to_account": "54321", "amount": 10000}, "timestamp": "2016-05-20T14:33:28.904-00:00"}
36. A Domain Entity {"account-id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938", "account-name": "My Awesome Checking Account", "balance": 1000, "as-of": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58"}
37. Separate Action from Perception
38. Reactively calculate derived state from event log • A few authorized teams process Commands into Events (probably using aggregate state) • A few (more) authorized teams calculate state streams of general interest derived from Events • Single-entity state changes • Regulated or audited calculations • Could be recursive, i.e. certain state changes might trigger further events downstream
39. Reactively calculate derived state from event log
40. Read- optimized View Command Processing Analytics [other topics] Arbitrary command action Arbitrary command action Microconsume rs events Business Logic Event Log Datastores and Materialized Views “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit HDFS Elastic Search Domain- specific Aggregate state wherever and however it provides value
41. Aggregate state wherever and however it provides value • Many teams across orgs aggregate streams into views for their respective use-cases • Into whatever data store makes sense for required data access pattern • Possibly emitting views back onto streams (a la Kafka Streams’ KTable) • Facilitates stateful computations: joins, windowed aggregates, command processing • Everyone traces lineage all the way back to edge Command/Event
42. Exploit Conway’s Law to Evolve as Needed
43. Application Team Provides
44. Cross-cutting Teams Provide
45. Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects” • A “side-effect” in this case is any action not associated with reading or writing the streams or aggregates, e.g. • Sending email, text message • Making a call to an outside web service • Calling a service to write a command/event to a stream you don’t own • Log the results of attempt to a stream, to facilitate retry/Saga-rollback/compensating- action
46. Manage “side-effects” carefully
47. Summary • Capture customer intent and business events at the edge as immutable data in domain language • From these action streams, services implement their own functionality via this common lingua franca • building many independent data views • reactively • without temporal or organizational coordination
48. Giant Shoulders • Immutability: Rich Hickey, Stu Halloway • CQRS: Udi Dahan, Martin Fowler, Chris Richardson • Kafka and Streaming Data: Neha Narkhede, Jay Kreps, Martin Kleppmann, Ben Stopford • Organization and Management: Mel Conway, Eliyahu Goldratt, Gene Kim, Michael Nygard
49. References • https://www.confluent.io/blog/toward-functional-programming-analogy- microservices/ • https://engineering.linkedin.com/distributed-systems/log-what-every-software- engineer-should-know-about-real-time-datas-unifying • http://blog.cognitect.com/?tag=NewNormal+Series • https://www.youtube.com/watch?v=2V1FtfBDsLU • http://www.datomic.com/ • https://www.infoq.com/presentations/Value-Values
50. Shameless Plug Evident Systems provides: • API & Microservices Design Training: https://hubs.ly/H0d0lsN0 • API & Microservices Implementation Training: https://hubs.ly/H0d0lsT0 We’re also Hiring!