Embracing the dual nature of state

The talk is about how we at Luno design our micro services and distributed application logic so that it is robust to failures and easy to maintain. We do this by coupling mutable state with immutable events.

Transcript

1. August 2019 Gophercon UK Corver Roos Staff software engineer www.luno.com/gophercon Embracing the dual nature of state
2. One size does not fit all • Hyper scaling distributed team. • Maintainability and consistency is everything. Customers 1 000 000s Requests 1000s per sec Microservices 100s Teams 10s
3. It was a dark and stormy night... Alerting gopher support! Sender microservice going about its business
4. Embracing the dual nature of state Mutable State Immutable Events
5. Embracing the dual nature of state • Store state in a mutable table. • Use state machines to control updates. • Couple with an immutable events table. • Expose events as gRPC streams. • Build application logic as idempotent consumers. • Explicitly introduce failures in application logic.
6. Mutable state • State which is modified. • Easy to understand for humans. • Mimics the real world. • Querying the current state is trivial. type Send struct { ID int64 UserID int64 Amount int64 Address string Status Status // PENDING, COMPLETED, FAILED TxID string // Non-zero if COMPLETED Error string // Non-zero if FAILED }
7. Mutable state
8. Sends FSM: +------> FAILED | PENDING ---> COMPLETED +-- RESETTING -+ v | CREATED --> PENDING --> FAILED | +-------> COMPLETED State machines • Explicit control of state life cycle. • Defined as rooted directed graph. • Easy for humans to understand. • Decompose complexity in discrete steps. • Transitions are events.
9. Shift github.com/luno/shift • API to define a simple persistence layer finite state machine. • Command shiftgen generates SQL boilerplate. var fsm = shift.NewFSM(events). Insert(PENDING, pending{}, COMPLETED, FAILED). Update(COMPLETED, completed{}). Update(FAILED, failed{}). Build() //go:generate shiftgen \ -inserter=pending \ -updaters=completed,failed \ -table=sends
10. Immutable events • State which cannot be modified. • Martin Fowler: event sourcing vs event notification. • Persistence guarantees are critical. • SQL transactions provide exactly once event persistence. type Event struct { ID int64 Timestamp time.Time ForeignID int64 // Send.ID Type Type // Send.Status }
11. Streams • Ordered append-only log. • Consumers process all events sequentially. • Consumers keep a cursor of their position. • Streams are not queues. • Retry on errors or ignore and continue. • Delivery guarantees are critical. • gRPC supports streaming connections.
12. Reflex github.com/luno/reflex • Reflex is an event streaming library. • Wires event streams and cursor stores to consumers. • Supports p2p streams via gRPC and retries on error. func StartSender(sendStream reflex.Stream, store reflex.CursorStore) { f := func(e reflex.Event) error { if !reflex.IsType(e.Type, PENDING) { return nil } return send(e.ForeignID) } go reflex.Consume(sendStream, store, reflex.NewConsumer("sender", f)) }
13. func send(sendID int64) error { send, err := getSend(sendID) if err != nil { return err } err := broadcast(send) if err != nil { return err } return fsm.Update(sendID, PENDING, COMPLETED) } func send(sendID int64) error { send, err := getSend(sendID) if err != nil { return err } if send.Status != PENDING { return nil } err = broadcast(send) if err != nil && err != ErrAlreadyBroadcasted { return err } return fsm.Update(sendID, PENDING, COMPLETED) } Idempotence • Operations applied multiple times without changing the result.
14. Fate github.com/luno/fate • Introducing errors by tempting fate. • Configurable error probability for tests, staging and prod. func send(sendID int64) error { send, _ := getSend(sendID) // Error handling redacted. _ = broadcast(send) if err := fate.Tempt(); err != nil { // Simulates partial broadcast or failure to update fsm. return err } _ = fsm.Update(sendID, PENDING, COMPLETED) return fate.Tempt() // Simulates duplicate event or cursor update failure. }
15. Embracing the dual nature of state Mutable state sends Immutable events send_events Create send Get send Stream events DB Sender Service Layers State Machine Business Logic gRPC API Create hander Sender consumer Get hander Events Insert Stream Reflex Shift PENDING FAILED COMPLETED
16. It was a dark and stormy night... and gopher support is sleeping peacefully
17. Thank you Corver Roos Staff software engineer www.luno.com/gophercon github.com/luno/shift github.com/luno/reflex github.com/luno/fate