Event-based Architecture and Implementation with Atom and Kafka

Event-based Architecture is a great answer to typical challenge of microservices architectures like resilience. This presentation shows how asynchronous microservices can be implemented with Atom / REST and Kafka.

1. Eberhard Wolff Fellow INNOQ @ewolff http://ewolff.com
2. http://continuous-delivery-buch.de/ http://continuous-delivery-book.com/
3. http://microservices-buch.de/ http://microservices-book.com/
4. http://microservices-buch.de/ ueberblick.html http://microservices-book.com/ primer.html FREE!!!!
5. http://microservices-praxisbuch.de http://practical-microservices.com/
6. http://microservices-praxisbuch.de/ rezepte.html http://practical-microservices.com/ recipes.html FREE!!!!
8. Independent Systems Architecture: ISA
9. Modules Macro / Micro Architecture Independent Continuous Delivery Pipeline Resilience Integration & Communication Authentication & Metadata Standardized Operations Standards: Interface only Container Independent Systems Architecture
10. Microservices = Extremely decoupled modules
11. Most important factor for microservice: Getting the split right!!!
12. Bounded Context: Specialized domain model for some use cases.
13. Order Invoice Delivery Items Customer Credit cards Taxes Parcel service Returns
15. Events • Communication between Bounded Contexts • Event in business domain • Past tense • Immutable
16. Events enable decoupling.
17. Events & microservices:
18. Events & microservices: A match made in heaven.
19. Event: Decoupling • Time: Might handle event later • Availability: Down? Handle event later • Change?
20. Order Accepted Invoice Delivery Asynchronous Communication ???
22. Order Accepted Item prices Billing address Item sizes Item weights Delivery address ??? Invoice Delivery ??? • Lots of data • Which Bounded Context uses what? • Changes can impact all modules • DDD: Published Language • THE order model?
23. Delivery Order Accepted Invoice Order Accepted Item prices Billing address Item sizes Item weights Delivery address • Specialized “events” • Independent evolution • Easy to figure out which Bounded Context uses what • Extensibility = new event • DDD: Customer / Supplier ??? Order Accepted ???
24. Order Accepted Order ID • Just an ID • Provide additional specialized API for data for each consumer • Compromise between extensibility and specialized events
25. Events: Conclusion • Events per se provide limited decoupling. • What data should be included? • Prefer specialized events and just IDs
26. Events: Conclusion • Best option for communication between Bounded Contexts. • Great fit for a Ubiquitous Language. • But: Implicit replication
27. Inconsistencies!!!
29. What Kind of Inconsistencies? • One system for invoices • One for deliveries • One for orders • One source of truth for invoices/deliveries/orders. • No “invoice might be there or not”
30. Inconsistencies • Either delivery without invoice • …or invoice without delivery. • Will eventually be solved. • Delays are not unheard of. • What is the alternative? • Centralized complex database?
32. Inconsistencies • Consequences if data appears after 0.1s / 1s / 1min / 1h / 1d ? • “Invoice has to be there in <1min!!!!!”
34. Event Sourcing • Store the events that lead to a specific state + state (optional) • System of record: Event or state • No clear definition https://martinfowler.com/ eaaDev/EventSourcing.html
35. Microservice Interface Calls, messages, … Event Store Order Cancelled 42 Order Accepted 23 Order Accepted 42 Order 42 Order 23 Order Delivered 23 Order 23 ✔ State
36. State Events Calculate State on Demand State+ Events Event Sourcing
37. Event Sourcing: Advantages • Might be a better model • History log • Audit log • Can rebuild state • Even after the domain model changed
38. Event Sourcing: Advantages • Temporal queries easy (yesterday’s state) • Replay • Reversal • Scalability – might replicate state using events
39. Event Sourcing: Challenges • External systems might have different data • …and behave differently during replay. • Code changes might change response to events • …even if it is just a bug fix.
40. Event Sourcing: Pattern for Persistence
41. Persistence • Responsibility of the microservices • Might be changed • Might be different for each microservice • Event Sourcing should be a local decision • Event sourcing makes it easier to fix inconsistencies.
42. event sourcing
44. Kafka Records • Key • Value • Timestamp • Headers • Stored forever (!) Record Key Value Timestamp
45. Kafka Topics • Topics have a name • Topics have partitions • Order guaranteed per partition • Consumer commits offset per partition • Consumer group: One consumer per partition Topic Partition Record Record Record
46. Topic Partition Record Record Record Consumer Offset Offset committed per consumer Offset Partition Record Record Record Partition Record Record Record Producer Consumer Consumer group
47. Kafka Replication • N Replicas (configurable) • In-sync replicas (configurable): write successful if all written
48. Order Invoice Delivery
49. https://github.com/ ewolff/microservice-kafka
50. Docker-Host Apache httpd Load Balancer Postgres Order Shipping Invoicing Kafka Zookeeper HTTP-Routing 8080 8080 80 8080 8080 9092
51. Events • Events contain all information for invoice and delivery • Alternative: specific events • …more independence • Alternative: just ID in the event • …plus service to get detailed information
52. Data • Data is copied into invoice / delivery database • …and denormalized • Each invoice and delivery contains customer and item data from when the order was created. • Bounded Context • No complex historization
53. Kafka Conclusion • Provides access to old events • Guaranteed order per partition • Consumer groups send records to one consumer • Additional (complex) infrastructure
55. Atom • Data format • …for feeds, blogs, podcasts etc • Access through HTTP • Idea: Provide a feeds of events / orders
56. Atom Feed • Some unneeded information author, summary, title… • Mostly links to details • ...and timestamps • Content negotiation can provide different data formats or details
57. Subscribing to Feed • Poll the feed (HTTP GET) • Client decides when to process new events • …but very inefficient • Lots of unchanged data
58. HTTP Caching • Client GETs feed • Server send data + Last-Modified header • Client sends GET + If-Modified-Since header • Server: 304 (Not modified) …or new data
59. https://github.com/ ewolff/microservice-atom
60. Docker-Host 80 8080 Apache httpd Postgres Order Shipping Invoicing HTTP-Routing HTTP + Atom 8080 8080 8080
61. Do you need Atom?
62. Do you need Atom? No!
63. Alternative • Roll your own format …that has no information only relevant for blogs …with links to details ...use content negotiation for details • Only issue: Atom reader cannot handle format
65. Atom Conclusion • Can provides access to old events if they are stored anyway. • One consumer: idempotency • No additional infrastructure • But: Can’t send event to just one receiver
67. Conclusion • Most important: Getting the split right! • Bounded Context • …communicate via business events • Inconsistencies can be dealt with • What is in the event? • Don’t share events for event sourcing! • Do not overuse event sourcing. • Kafka: Great solution for messaging • …but REST might be enough.
68. Send email to gotober20181101@ewolff.com Slides + Microservices Primer DE / EN + Microservices Recipes DE / EN + Sample Microservices Book DE / EN + Sample Practical Microservices DE/EN + Sample of Continuous Delivery Book DE Powered by Amazon Lambda & Microservices