Decoupled, Immutable REST APIs with Kafka Streams

Have you ever hit a wall with REST? Does modeling your problem domain into CRUD-able entities feel like fitting a square peg into a round hole? Perhaps instead of modeling our services like little databases, we should instead model them like reactors to event streams.

REST APIs are great, but their typical implementation tightly couples various concerns which would be better separated:

Reads (perception) from writes (action)
Current state from historical narrative
Business logic from HTTP design from operational concerns like metrics and monitoring

I'll show a better pattern for writing REST APIs that de-couples these concerns, thereby alleviating common frustrations with CRUD-flavored REST. This pattern imposes a clear separation of action from perception and uses Kafka and Kafka Streams to separate business logic from HTTP concerns, all while preserving the historical narrative of the entire event stream.

In this talk, I’ll discuss the benefits and tradeoffs of this approach, and demonstrate my implementation using Clojure in the HTTP layer, and using Java with the new Kafka Streams library in the event stream processing layer.

Transcript

1. DEV CON
   ATLANTIC
   MID
   Cloud — Mobile — Web — Dev
   Thanks to
   Our Sponsors
   

   View Slide

2. Copyright 2018, Evident Systems LLC
   Decoupled, Immutable Rest APIs
   With Kafka Streams
   

   View Slide

3. Hi, I’m Bobby
   I’m the Founder of Evident Systems, LLC
   We help companies re-assert ownership over their core systems and data
   bobby@evidentsystems.com
   @bobbycalderwood
   https://github.com/bobby
   

   View Slide

4. Problem Space
   • Provide valuable informational and transactional services via Mobile
   and Web software
   • To lots of customers, with excellent user experience
   • Securely and in compliance with regulations
   • With ability to easily enhance, experiment, monitor, maintain, and
   operate
   • By many participants within a large organization
   

   View Slide

5. /[resource](/:id) Database
   Logic Analytics
   Web Services Business Logic Databases and
   Services
   Operations
   Metrics/Monitoring
   Security/Compliance
   Audit
   Afterthoughts
   Other side-effects
   Problematic Architecture
   

   View Slide

6. Big Ideas from Functional Programming
   • Immutability is central to information systems
   • Data language of system >> Programming language of components
   • Action and perception are not the same, and immutability facilitates
   their separation
   • Businesses services are not databases, they’re event stream reactors
   • Cross-cutting concerns must be satisfied in the presence of Conway’s
   Law
   

   View Slide

7. Immutability is central to information
   systems
   

   View Slide

8. /[resource](/:id) Database
   Logic Analytics
   Web Services Business Logic Databases and
   Services
   Operations
   Metrics/Monitoring
   Security/Compliance
   Audit
   Afterthoughts
   Other side-effects
   Data Loss By Design
   

   View Slide

9. The (data) language of the System
   >>
   The (runtime) language of each component
   

   View Slide

10. /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    What is the System Language?
    

    View Slide

11. Action != Perception
    

    View Slide

12. Writes != Reads
    

    View Slide

13. /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    Writes Tied to Reads
    

    View Slide

14. Business Services are not Databases
    

    View Slide

15. –John M. Culkin
    “We shape our tools and thereafter our tools shape us.”
    

    View Slide

16. /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    Database as Leaky Abstraction
    

    View Slide

17. Conway’s Law
    Yup, totally a thing
    

    View Slide

18. –Melvin Conway
    “organizations which design systems ... are constrained to produce
    designs which are copies of the communication structures of
    these organizations.”
    

    View Slide

19. /[resource](/:id) Database
    Logic Analytics
    Web Services Business Logic Databases and
    Services
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    Afterthoughts
    Other side-effects
    Cross-cutting Concerns?
    

    View Slide

20. Object Oriented-ish
    • Encapsulated data access via synchronous calls
    • Mutable state change via synchronous calls
    • Dependency web
    • Imperative, sequence-oriented orchestration
    • Referentially opaque
    

    View Slide

21. OO thinking doesn’t distribute!
    

    View Slide

22. AWS Death Star diagram, circa 2008 as per Werner Vogels tweet https://twitter.com/Werner/status/741673514567143424
    

    View Slide

23. We can do better!
    

    View Slide

24. Source: NASA, Public Domain
    

    View Slide

25. Functional-ish
    • Data access via aggregating immutable values
    • State change via mapping an identity to different immutable values
    over time
    • Data Flow graph
    • Declarative orchestration
    • Referential transparency
    

    View Slide

26. In Contrast
    Object Oriented Functional
    Data Access Encapsulated Ubiquitous
    State Change Mutable, in-place Immutable, values over time
    Organization Dependency graph Data-flow graph
    Orchestration Imperative, sequential Declarative, parallelizable
    Referentially… Opaque Transparent
    

    View Slide

27. But How?
    Techniques, Rules, and Tools
    

    View Slide

28. Event Sourcing
    Image by Alan Light CC BY-SA 3.0
    

    View Slide

29. REST + CQRS + Event Sourcing
    

    View Slide

30. Embrace Immutability
    

    View Slide

31. Capture changes “at the edge” to immutable log
    • A (very) few authorized teams capture all raw observations (Events) and customer
    requests for action (Commands)
    • at the edge of bounded context
    • with minimal processing
    • immutably and durably
    • Causally related events go on same log (Kafka Topic Partition)
    • One (logical) writer per command/event stream
    • No change gets into bounded context via any other means!
    

    View Slide

32. Capture changes “at the edge” to immutable log
    

    View Slide

33. Express actions in domain language
    (not in database language)
    

    View Slide

34. A Command
    {"id": "33bb75db-6e13-48ee-8a54-b3976d3d065b",
    "action": "transfer-money",
    "data": {"from_account": "12345",
    "to_account": "54321",
    "amount": 10000}
    "timestamp": "2016-05-20T14:33:28.902-00:00"}
    

    View Slide

35. An Event
    {"id": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58",
    "parent": "33bb75db-6e13-48ee-8a54-b3976d3d065b",
    "action": "money-transferred",
    "data": {"id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938",
    "from_account": "12345",
    "to_account": "54321",
    "amount": 10000},
    "timestamp": "2016-05-20T14:33:28.904-00:00"}
    

    View Slide

36. A Domain Entity
    {"account-id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938",
    "account-name": "My Awesome Checking Account",
    "balance": 1000,
    "as-of": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58"}
    

    View Slide

37. Separate Action from Perception
    

    View Slide

38. Reactively calculate derived state from event log
    • A few authorized teams process Commands into Events (probably using
    aggregate state)
    • A few (more) authorized teams calculate state streams of general interest
    derived from Events
    • Single-entity state changes
    • Regulated or audited calculations
    • Could be recursive, i.e. certain state changes might trigger further events
    downstream
    

    View Slide

39. Reactively calculate derived state from event log
    

    View Slide

40. Read-
    optimized
    View
    Command
    Processing
    Analytics
    [other topics]
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Microconsume
    rs
    events
    Business Logic
    Event Log Datastores and
    Materialized Views
    “Ledger” of events
    Operations
    Metrics/Monitoring
    Security/Compliance
    Audit
    HDFS
    Elastic
    Search
    Domain-
    specific
    Aggregate state wherever and however it
    provides value
    

    View Slide

41. Aggregate state wherever and however it
    provides value
    • Many teams across orgs aggregate streams into views for their
    respective use-cases
    • Into whatever data store makes sense for required data access pattern
    • Possibly emitting views back onto streams (a la Kafka Streams’ KTable)
    • Facilitates stateful computations: joins, windowed aggregates, command
    processing
    • Everyone traces lineage all the way back to edge Command/Event
    

    View Slide

42. Exploit Conway’s Law to Evolve as Needed
    

    View Slide

43. Application Team Provides
    

    View Slide

44. Cross-cutting Teams Provide
    

    View Slide

45. Manage “side-effects” carefully
    • A few authorized teams react to Events by reaching outside of bounded context to cause
    “side-effects”
    • A “side-effect” in this case is any action not associated with reading or writing the
    streams or aggregates, e.g.
    • Sending email, text message
    • Making a call to an outside web service
    • Calling a service to write a command/event to a stream you don’t own
    • Log the results of attempt to a stream, to facilitate retry/Saga-rollback/compensating-
    action
    

    View Slide

46. Manage “side-effects” carefully
    

    View Slide

47. Summary
    • Capture customer intent and business events at the edge as immutable
    data in domain language
    • From these action streams, services implement their own functionality
    via this common lingua franca
    • building many independent data views
    • reactively
    • without temporal or organizational coordination
    

    View Slide

48. Giant Shoulders
    • Immutability: Rich Hickey, Stu Halloway
    • CQRS: Udi Dahan, Martin Fowler, Chris Richardson
    • Kafka and Streaming Data: Neha Narkhede, Jay Kreps, Martin
    Kleppmann, Ben Stopford
    • Organization and Management: Mel Conway, Eliyahu Goldratt, Gene
    Kim, Michael Nygard
    

    View Slide

49. References
    • https://www.confluent.io/blog/toward-functional-programming-analogy-
    microservices/
    • https://engineering.linkedin.com/distributed-systems/log-what-every-software-
    engineer-should-know-about-real-time-datas-unifying
    • http://blog.cognitect.com/?tag=NewNormal+Series
    • https://www.youtube.com/watch?v=2V1FtfBDsLU
    • http://www.datomic.com/
    • https://www.infoq.com/presentations/Value-Values
    

    View Slide

50. Shameless Plug
    Evident Systems provides:
    • API & Microservices Design Training: https://hubs.ly/H0d0lsN0
    • API & Microservices Implementation Training: https://hubs.ly/H0d0lsT0
    We’re also Hiring!
    

    View Slide