Tweet
Tweet

Slide 1

Slide 1 text

DEV CON ATLANTIC MID Cloud — Mobile — Web — Dev Thanks to Our Sponsors

Slide 2

Slide 2 text

Copyright 2018, Evident Systems LLC Decoupled, Immutable Rest APIs With Kafka Streams

Slide 3

Slide 3 text

Hi, I’m Bobby I’m the Founder of Evident Systems, LLC We help companies re-assert ownership over their core systems and data [email protected] @bobbycalderwood https://github.com/bobby

Slide 4

Slide 4 text

Problem Space • Provide valuable informational and transactional services via Mobile and Web software • To lots of customers, with excellent user experience • Securely and in compliance with regulations • With ability to easily enhance, experiment, monitor, maintain, and operate • By many participants within a large organization

Slide 5

Slide 5 text

/[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Problematic Architecture

Slide 6

Slide 6 text

Big Ideas from Functional Programming • Immutability is central to information systems • Data language of system >> Programming language of components • Action and perception are not the same, and immutability facilitates their separation • Businesses services are not databases, they’re event stream reactors • Cross-cutting concerns must be satisfied in the presence of Conway’s Law

Slide 7

Slide 7 text

Immutability is central to information systems

Slide 8

Slide 8 text

/[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Data Loss By Design

Slide 9

Slide 9 text

The (data) language of the System >> The (runtime) language of each component

Slide 10

Slide 10 text

/[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects What is the System Language?

Slide 11

Slide 11 text

Action != Perception

Slide 12

Slide 12 text

Writes != Reads

Slide 13

Slide 13 text

/[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Writes Tied to Reads

Slide 14

Slide 14 text

Business Services are not Databases

Slide 15

Slide 15 text

–John M. Culkin “We shape our tools and thereafter our tools shape us.”

Slide 16

Slide 16 text

/[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Database as Leaky Abstraction

Slide 17

Slide 17 text

Conway’s Law Yup, totally a thing

Slide 18

Slide 18 text

–Melvin Conway “organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.”

Slide 19

Slide 19 text

/[resource](/:id) Database Logic Analytics Web Services Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects Cross-cutting Concerns?

Slide 20

Slide 20 text

Object Oriented-ish • Encapsulated data access via synchronous calls • Mutable state change via synchronous calls • Dependency web • Imperative, sequence-oriented orchestration • Referentially opaque

Slide 21

Slide 21 text

OO thinking doesn’t distribute!

Slide 22

Slide 22 text

AWS Death Star diagram, circa 2008 as per Werner Vogels tweet https://twitter.com/Werner/status/741673514567143424

Slide 23

Slide 23 text

We can do better!

Slide 24

Slide 24 text

Source: NASA, Public Domain

Slide 25

Slide 25 text

Functional-ish • Data access via aggregating immutable values • State change via mapping an identity to different immutable values over time • Data Flow graph • Declarative orchestration • Referential transparency

Slide 26

Slide 26 text

In Contrast Object Oriented Functional Data Access Encapsulated Ubiquitous State Change Mutable, in-place Immutable, values over time Organization Dependency graph Data-flow graph Orchestration Imperative, sequential Declarative, parallelizable Referentially… Opaque Transparent

Slide 27

Slide 27 text

But How? Techniques, Rules, and Tools

Slide 28

Slide 28 text

Event Sourcing Image by Alan Light CC BY-SA 3.0

Slide 29

Slide 29 text

REST + CQRS + Event Sourcing

Slide 30

Slide 30 text

Embrace Immutability

Slide 31

Slide 31 text

Capture changes “at the edge” to immutable log • A (very) few authorized teams capture all raw observations (Events) and customer requests for action (Commands) • at the edge of bounded context • with minimal processing • immutably and durably • Causally related events go on same log (Kafka Topic Partition) • One (logical) writer per command/event stream • No change gets into bounded context via any other means!

Slide 32

Slide 32 text

Capture changes “at the edge” to immutable log

Slide 33

Slide 33 text

Express actions in domain language (not in database language)

Slide 34

Slide 34 text

A Command {"id": "33bb75db-6e13-48ee-8a54-b3976d3d065b", "action": "transfer-money", "data": {"from_account": "12345", "to_account": "54321", "amount": 10000} "timestamp": "2016-05-20T14:33:28.902-00:00"}

Slide 35

Slide 35 text

An Event {"id": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58", "parent": "33bb75db-6e13-48ee-8a54-b3976d3d065b", "action": "money-transferred", "data": {"id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938", "from_account": "12345", "to_account": "54321", "amount": 10000}, "timestamp": "2016-05-20T14:33:28.904-00:00"}

Slide 36

Slide 36 text

A Domain Entity {"account-id": "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938", "account-name": "My Awesome Checking Account", "balance": 1000, "as-of": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58"}

Slide 37

Slide 37 text

Separate Action from Perception

Slide 38

Slide 38 text

Reactively calculate derived state from event log • A few authorized teams process Commands into Events (probably using aggregate state) • A few (more) authorized teams calculate state streams of general interest derived from Events • Single-entity state changes • Regulated or audited calculations • Could be recursive, i.e. certain state changes might trigger further events downstream

Slide 39

Slide 39 text

Reactively calculate derived state from event log

Slide 40

Slide 40 text

Read- optimized View Command Processing Analytics [other topics] Arbitrary command action Arbitrary command action Microconsume rs events Business Logic Event Log Datastores and Materialized Views “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit HDFS Elastic Search Domain- specific Aggregate state wherever and however it provides value

Slide 41

Slide 41 text

Aggregate state wherever and however it provides value • Many teams across orgs aggregate streams into views for their respective use-cases • Into whatever data store makes sense for required data access pattern • Possibly emitting views back onto streams (a la Kafka Streams’ KTable) • Facilitates stateful computations: joins, windowed aggregates, command processing • Everyone traces lineage all the way back to edge Command/Event

Slide 42

Slide 42 text

Exploit Conway’s Law to Evolve as Needed

Slide 43

Slide 43 text

Application Team Provides

Slide 44

Slide 44 text

Cross-cutting Teams Provide

Slide 45

Slide 45 text

Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects” • A “side-effect” in this case is any action not associated with reading or writing the streams or aggregates, e.g. • Sending email, text message • Making a call to an outside web service • Calling a service to write a command/event to a stream you don’t own • Log the results of attempt to a stream, to facilitate retry/Saga-rollback/compensating- action

Slide 46

Slide 46 text

Manage “side-effects” carefully

Slide 47

Slide 47 text

Summary • Capture customer intent and business events at the edge as immutable data in domain language • From these action streams, services implement their own functionality via this common lingua franca • building many independent data views • reactively • without temporal or organizational coordination

Slide 48

Slide 48 text

Giant Shoulders • Immutability: Rich Hickey, Stu Halloway • CQRS: Udi Dahan, Martin Fowler, Chris Richardson • Kafka and Streaming Data: Neha Narkhede, Jay Kreps, Martin Kleppmann, Ben Stopford • Organization and Management: Mel Conway, Eliyahu Goldratt, Gene Kim, Michael Nygard

Slide 49

Slide 49 text

References • https://www.confluent.io/blog/toward-functional-programming-analogy- microservices/ • https://engineering.linkedin.com/distributed-systems/log-what-every-software- engineer-should-know-about-real-time-datas-unifying • http://blog.cognitect.com/?tag=NewNormal+Series • https://www.youtube.com/watch?v=2V1FtfBDsLU • http://www.datomic.com/ • https://www.infoq.com/presentations/Value-Values

Slide 50

Slide 50 text

Shameless Plug Evident Systems provides: • API & Microservices Design Training: https://hubs.ly/H0d0lsN0 • API & Microservices Implementation Training: https://hubs.ly/H0d0lsT0 We’re also Hiring!