AxonDB, high-performance event store

AxonDB, a specialized, high-performance database system to support microservices and other applications that use the event sourcing pattern.

Transcript

1. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul AxonDB Product Release Presentation
2. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Axon Framework AxonIQ Axon Framework and AxonIQ • Open source Java framework for event-driven microservices • Core concepts: DDD, CQRS, event sourcing • Mature, 600k+ downloads • Company founded in July 2017, focusing on Axon Framework. • Two lines of business 1. Commercial software products in addition to Axon Framework 2. Axon Framework support, training, consulting
3. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul AxonIQ Software Products Available now Roadmap AxonDB AxonIQ EventStore AxonIQ GDPR Module AxonHub Monitoring & Analytics SaaS-delivery of DB, Hub
4. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Agenda • Event sourcing and related concepts • Event store requirements • Evaluating pre-existing options • AxonDB architecture and features • Getting started with AxonDB
5. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul What's an "Event"? • (Description of) something that has happened. • By definition, in the past, at a definite instant • Conceptually immutable • Cannot "fail" • Not the same as a message! • In a domain-driven design context • "Events" are events in the business sense, • which are explicitly represented in application code.
6. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event-Driven Architecture “a software architecture pattern promoting the production, detection, consumption of, and reaction to events” https://en.wikipedia.org/wiki/Event-driven_architecture
7. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event-Driven Architecture Event Sourcing Event Sourcing • is a specific type of Event- Driven Architecture • in which Events are at the heart of the persistence / data storage architecture
8. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Non-event sourced persistence An e-commerce example What's stored in the database What's happening in the app User starts new order Order 93771261272
9. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Non-event sourced persistence An e-commerce example What's stored in the database What's happening in the app User adds sunglasses to order Order 93771261272 1 sunglasses
10. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Non-event sourced persistence An e-commerce example What's stored in the database What's happening in the app User adds Jimmy Choo shoes to order Order 93771261272 1 pair of Jimmy Choo's 1 sunglasses
11. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Non-event sourced persistence An e-commerce example What's stored in the database What's happening in the app User adds a shawl to the order Order 93771261272 1 pair of Jimmy Choo's 1 sunglasses 1 shawl
12. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Non-event sourced persistence An e-commerce example What's stored in the database What's happening in the app User decides not to buy the Jimmy Choo's and removes them from the order Order 93771261272 1 pair of Jimmy Choo's 1 sunglasses 1 shawl
13. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Non-event sourced persistence An e-commerce example What's stored in the database What's happening in the app User enters address and payment info and confirms order Order 93771261272 1 shawl 1 sunglasses Customer: Jane
14. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event sourced persistence An e-commerce example
15. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event sourced persistence An e-commerce example What's stored in the database What's happening in the app User starts new order OrderCreated (93771261272)
16. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event sourced persistence An e-commerce example What's stored in the database What's happening in the app User adds sunglasses to order OrderCreated (93771261272) ProductAdded (sunglasses)
17. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event sourced persistence An e-commerce example What's stored in the database What's happening in the app User adds Jimmy Choo shoes to order 1 sunglasses OrderCreated (93771261272) ProductAdded (sunglasses) ProductAdded (Jimmy Choo's)
18. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event sourced persistence An e-commerce example What's stored in the database What's happening in the app User adds a shawl to the order 1 sunglasses OrderCreated (93771261272) ProductAdded (sunglasses) ProductAdded (Jimmy Choo's) ProductAdded (shawl)
19. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event sourced persistence An e-commerce example What's stored in the database What's happening in the app User decides not to buy the Jimmy Choo's and removes them from the order 1 sunglasses OrderCreated (93771261272) ProductAdded (sunglasses) ProductAdded (Jimmy Choo's) ProductAdded (shawl) ProductRemoved (Jimmy Choo's)
20. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event sourced persistence An e-commerce example What's stored in the database What's happening in the app User enters address and payment info and confirms order 1 sunglasses OrderCreated (93771261272) ProductAdded (sunglasses) ProductAdded (Jimmy Choo's) ProductAdded (shawl) ProductRemoved (Jimmy Choo's) OrderConfirmed (Jane)
21. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Business reasons Technical reasons Why use event sourcing? • Auditing / compliance / transparency • Data mining, analytics: value from data • Guaranteed completeness of raised events • Single source of truth • Concurrency / conflict resolution • Facilitates debugging • Replay into new read models (CQRS) • Easily capturing intent
22. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul What's an "event store"? • In the architecture of an event-sourced application, the event store is the database system used to store the events. • In terms of implementations, this could be • General purpose RDBMS technology (Oracle, MySQL, Postgres etc.) • General purpose NoSQL technology (Mongo, Cassandra etc.) • Specialized event store technology (AxonDB, Greg Young's EventStore, PumpkinDB)
23. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event store in context Application Event store Past events New events • Works well for processing changes (Commands) • Does not work well for, say, finding all orders with total value > EUR 100
24. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul CQRS Command-Query Responsibility Segregation Command Handler Event store Past events New events Projection database Query Handler New events Projection logic Updates Selection criteria Data
25. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event store requirements
26. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements Read Events All for an aggregate (event sourced repository) All since point in time (replay for read models) Read back in write order Append events Insert events at random point Update events Delete events Append Events (Write Events) Ad-hoc queries (for debug, monitoring, support)
27. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements InvestmentAccountCreated(balance = 0, limit = 0) MoneyDepositedToAccount(amount = 1000) Id=8721 Seq = 0 Id=8721 Seq = 1 MoneyWithdrawnFromAccount(amount = 600) Id=8721 Seq = 2
28. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements InvestmentAccountCreated(balance = 0, limit = 0) MoneyDepositedToAccount(amount = 1000) Id=8721 Seq = 0 Id=8721 Seq = 1 MoneyWithdrawnFromAccount(amount = 600) Id=8721 Seq = 2 MoneyWithdrawnFromAccount(amount = 700) Id=8721 Seq = 2
29. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements Read Events All for an aggregate All since point in time Read back in write order Validate aggregate sequence numbers (consistency) Append Events Ad-hoc queries
30. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements InvestmentAccountCreated(balance = 0, limit = 0) MoneyDepositedToAccount(amount = 1000) Id=8721 Seq = 0 Id=8721 Seq = 1 Command: buy 5 shares of XYZ Corp @ 100 MoneyWithdrawnFromAccount(amount = 500) Id=8721 Seq = 2 SharesAddedToAccount(symbol = ‘XYZ’, n = 5) Id=8721 Seq = 3
31. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements Read Events All for an aggregate All since point in time Read back in write order Validate aggregate sequence numbers (consistency) Append Events Append multiple events at once (atomicity) Only read committed events (isolation) Committed events protected against loss (durability) Ad-hoc queries
32. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements BankAccountCreated(balance = 0, limit = 0) MoneyDepositedToAccount(amount = 1000) Id=8721 Seq = 0 Id=8721 Seq = 1 MoneyDepositedToAccount(amount = 500) Id=8721 Seq = 9103 MoneyWithdrawnFromAccount(amount = 700) Id=8721 Seq = 9102 Using the bank account for 10 years
33. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements BankAccountCreated(balance = 0, limit = 0) MoneyDepositedToAccount(amount = 1000) Id=8721 Seq = 0 Id=8721 Seq = 1 MoneyDepositedToAccount(amount = 500) Id=8721 Seq = 9103 MoneyWithdrawnFromAccount(amount = 700) Id=8721 Seq = 9102 BankAccountSnapshot(balance = 5000) Id=8721 Seq = 9080
34. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul • Latests snapshot + later events (event-sourced repository) • All events (auditing) All since point in time Read back in write order Ad-hoc queries Only read committed events (isolation) Event Storage Requirements All for an aggregate Validate aggregate sequence numbers (consistency) Append multiple events at once (atomicity) Read Events/Snapshots Append Events/Snapshots Append snapshots Committed events protected against loss (durability)
35. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Append Events/Snapshots Read Events/Snapshots All since point in time Read back in write order Ad-hoc queries • Latests snapshot + later events • All events Event Storage Requirements All for an aggregate Validate aggregate sequence numbers Append multiple events at once Append snapshots Committed events protected against loss Only read committed events
36. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements All events for all bank accounts for 10 years Billions of events
37. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Append Events/Snapshots Read Events/Snapshots All since point in time Read back in write order Only read committed events • Latests snapshot + later events • All events Event Storage Requirements All for an aggregate Validate aggregate sequence numbers Append multiple events at once Append snapshots Committed events protected against loss Constant performance as a function of storage size Optimized for recent events Ad-hoc queries
38. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements Node 1 Command handler Node 2 Read model Event store
39. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements Node 1 Command handler Node 2 Read model Event store
40. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements Node 1 Command handler Node 2 Read model Event store Message Bus
41. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Event Storage Requirements Node 1 Command handler Node 2 Read model Event store
42. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Append Events/Snapshots Read Events/Snapshots All since point in time Read back in write order Only read committed events • Latests snapshot + later events • All events Event Storage Requirements All for an aggregate Validate aggregate sequence numbers Append multiple events at once Append snapshots Committed events protected against loss Constant performance as a function of storage size Optimized for recent events Ad-hoc queries , pushing new ones
43. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Append Events/Snapshots Read Events/Snapshots All since point in time Read back in write order Only read committed events • Latests snapshot + later events • All events Event Storage Requirements All for an aggregate Validate aggregate sequence numbers Append multiple events at once Append snapshots Committed events protected against loss Constant performance as a function of storage size Optimized for recent events Ad-hoc queries • pushing new ones
44. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Incumbents Contenders Event store options • RDBMS (any vendor) • MongoDB Generic • Kafka • Cassandra Built-for-purpose • Greg Young's EventStore • PumpkinDB
45. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Pros • Well established tech • Transactionality RDBMS
46. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul RDBMS scalability • Smaller % of db in mem buffer • No mechanism to provide rapid access to recent events • Maintaining B-tree indices becomes more expensive
47. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Pros • Well established tech • Transactionality RDBMS Cons • Scalability problems • No (clean) event push
48. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Pros • Horizontal scalability through sharding • Analysis on events MongoDB
49. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Document Events MongoDB
50. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Document Events MongoDB =
51. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Document Events MongoDB = Document =
52. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Document Events MongoDB =
53. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Pros • Horizontal scalability through sharding • Analysis on events MongoDB Cons • Document transactions • No event push • No global sequence #
54. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Pros • Messaging focussed • Extremely scalable Kafka
55. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul A A C D D B B B E F A Aggregate id
56. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul A A C D D B B B E F A B B B
57. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul A A C D D B B B A Topic A Topic B Topic C Topic D
58. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Topic (logical, spanning multiple machines) Partition (physical on a specific machine) 1 1 … n Has an associated directory Segment 1 1 … n Has a log and index file This doesn’t scale to millions of aggregates!
59. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Pros • Messaging focussed • Extremely scalable Kafka Cons • Not scalable in #aggregates in #total events
60. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Pros • Extremely scalable • Multiple global datacenters • Peer 2 peer • Flexible, tunable consistency Cassandra
61. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul InvestmentAccountCreated(balance = 0, limit = 0) MoneyDepositedToAccount(amount = 1000) Id=8721 Seq = 0 Id=8721 Seq = 1 MoneyWithdrawnFromAccount(amount = 600) Id=8721 Seq = 2 MoneyWithdrawnFromAccount(amount = 700) Id=8721 Seq = 2
62. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Node B Node A 0 1 0 1 X 2 Y 2
63. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul INSERT INTO events(aggId, aggSeqNo, payload) VALUES( ‘a’, 2 , ... ) IF NOT EXISTS
64. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul INSERT INTO events(aggId, aggSeqNo, payload) VALUES( ‘a’, 2 , ... ) IF NOT EXISTS “Behind the scenes, Cassandra is making four round trips between a node proposing a lightweight transaction and any needed replicas in the cluster to ensure proper execution so performance is affected. Consequently, reserve lightweight transactions for those situations where they are absolutely necessary; Cassandra’s normal eventual consistency can be used for everything else.” Source: https://docs.datastax.com - our highlighting
65. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Pros • Extremely scalable • Multiple global datacenters • Peer 2 peer • Flexible, tunable consistency Cassandra Cons • Can’t guarantee event store consistency efficiently
66. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Greg Young PumpkinDB Built-for purpose event stores? • Written in .NET, and generally seen as part of .NET ecosystem • Places heavy emphasis on projection logic (JavaScript) inside the event store. • Separate 'database programming environment' inspired by M/MUMPS • Lots of logic would have to implemented in "PumpkinScript" rather than Java.
67. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul AxonDB architecture and features
68. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul AxonDB • AxonIQ's event store • Built ‘from scratch’ in Java. • Purpose-built for event sourcing • Manages files directly - no underlying database system. • Open interfaces based on HTTP+JSON and gRPC • Drop-in event store implementation for Axon Framework
69. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul time Append-only by design
70. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul time Event-stream split into segments
71. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul time events snapshots Built-in support for snapshots
72. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul time Data Index Bloom- filter In each segment, we can efficiently search on aggregate id + seq no
73. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul time Searching for events goes backwards in time search
74. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul
75. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Rich support for ad-hoc queries, through a GUI and an API
76. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Rich support for ad-hoc queries, through a GUI and an API
77. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Node 1 Node 2 Node 3 Master Client Clustering based on floating single-master with write quorum
78. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Getting started with AxonDB • Go to: https://axoniq.io/products/axondb.html • This page contains information on the 4 available editions and their pricing. • The Developer Edition is free to download, and includes the full user manual.
79. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Application changes
80. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Q&A Product page: https://axoniq.io/products/axondb.html
81. Contact me: frans.vanbuul@axoniq.io Follow me: @Frans_vanBuul Frans van Buul About me frans.vanbuul@axoniq.io +31 6 5068 2984 https://www.linkedin.com/in/frans-van-buul-8030743/ @Frans_vanBuul