Reconciling Everything

A talk I gave at PyCon US 2023.

Transcript

1. EVERYTHING ANDREW GODWIN // @andrew@aeracode.org RECONCILING
2. Andrew Godwin / @andrew@aeracode.org Hi, I’m Andrew Godwin • Principal Engineer at Astronomer.io (Airflow) • Django Migrations, Channels, Async, others • Building distributed systems since 2008
3. Andrew Godwin / @andrew@aeracode.org Let's start with a bold statement I hear it drives engagement. Maybe also heckling.
4. Andrew Godwin / @andrew@aeracode.org Event-Driven Via Queues Totally separate stores of truth that only talk via queues Reconciliation Loops Stateless components that talk to one store of truth There are only two good* ways to build distributed systems**:
5. Andrew Godwin / @andrew@aeracode.org What is "good"? Low maintenance and high reliability, for me at least.
6. Andrew Godwin / @andrew@aeracode.org What kinds of distributed systems? Remember, friends don't let friends write microservices
7. Andrew Godwin / @andrew@aeracode.org Due to size What companies often claim is the reason Due to team structure What the actual reason often is Due to federation It's what the internet was built on!
8. Andrew Godwin / @andrew@aeracode.org RPC Microservices Email Service Accounting Service Billing Service Reservation Service
9. Andrew Godwin / @andrew@aeracode.org A system is defined by its failure modes Nobody gets paged when it's up and running happily
10. Andrew Godwin / @andrew@aeracode.org Event-Driven (Message Passing) Task Runners Billing System QUEUE QUEUE Log Storage Analytics Aggregator QUEUE
11. Andrew Godwin / @andrew@aeracode.org What kind of queue is it? Hint: It will not deliver messages exactly once
12. Andrew Godwin / @andrew@aeracode.org At-most-once In case of failure, a message will not get delivered At-least-once In case of failure, a messages will get delivered twice or more
13. Andrew Godwin / @andrew@aeracode.org Backlogs Need more queue consumers, can be asymmetric Replays I hope you can cope with lots of duplicates! Overflow Do you grind to a halt? Or drop?
14. Andrew Godwin / @andrew@aeracode.org Neither is "perfect" (But I recommend at-least-once unless you truly do not care about the data)
15. Andrew Godwin / @andrew@aeracode.org Reconciliation (Control) Loop Kubelet Deployment Controller DATABASE Pod Controller Kubectl (User)
16. Andrew Godwin / @andrew@aeracode.org Reconciliation (Control) Loop Kubelet Deployment Controller apiserver Pod Controller Kubectl (User) etcd
17. Andrew Godwin / @andrew@aeracode.org Kubernetes Example • User creates a new Deployment • Deployment controller notices that the deployment has no pods ◦ To reconcile this, it makes one • Pod controller notices that the pod is not assigned ◦ To reconcile this, it assigns it to a node • Kubelet notices it has an assigned pod that is not running ◦ To reconcile this, it creates it locally and starts it
18. Andrew Godwin / @andrew@aeracode.org One "single point of failure" is great Keeping one service up is much simpler
19. Andrew Godwin / @andrew@aeracode.org So how do reconciliation loops fail? They slow down and halt, but with easy restarts and no data loss
20. Andrew Godwin / @andrew@aeracode.org Each controller must be stateless This lets you scale them up, down, restart, upgrade, and recover easily
21. Andrew Godwin / @andrew@aeracode.org Practical example: Takahē Of course I wrote an ActivityPub/Fediverse server!
22. Andrew Godwin / @andrew@aeracode.org
23. Andrew Godwin / @andrew@aeracode.org The Fediverse is an event-driven system Well, mostly. It's also imperfect and still evolving.
24. Andrew Godwin / @andrew@aeracode.org Making a single Fediverse post QUEUE Sending Client Origin API Fanout System Destination Inbox Timeline Builder Receiving Client QUEUE
25. Andrew Godwin / @andrew@aeracode.org A server needs background workers Both to fan out sending of posts, and to process incoming posts
26. Andrew Godwin / @andrew@aeracode.org Stator Takahē's state-machine reconciliation system
27. Andrew Godwin / @andrew@aeracode.org See more inside github.com/jointakahe/takahe
28. Andrew Godwin / @andrew@aeracode.org You can also do this with Airflow We have an Astronomer executor which uses this for running tasks
29. Andrew Godwin / @andrew@aeracode.org Airflow Example • A Task Instance is scheduled to run, and the scheduler passes it over • The Executor makes a new Workload for the Task Instance • Allocator notices that the Workload has no Runner ◦ To reconcile this, it assigns it to a runner • Runner notices it has an assigned Workload that is not running ◦ To reconcile this, it creates it locally and starts it
30. Andrew Godwin / @andrew@aeracode.org So, how should you build them? Carefully! Ha, ha.
31. Andrew Godwin / @andrew@aeracode.org All state in the central store You can maybe get away with caching elsewhere Zero service-to-service comms Communication paths are failure modes Controllers should be simple loops Query database, reconcile, repeat
32. Andrew Godwin / @andrew@aeracode.org They are easy to test and refactor Mocking your datastore is all that's needed; everything else should flow
33. Andrew Godwin / @andrew@aeracode.org They self-heal! Provided your datastore stays up, everything else is flexible
34. Andrew Godwin / @andrew@aeracode.org There are downsides - mostly max scale But I generally design systems with upper bounds on their throughput
35. Andrew Godwin / @andrew@aeracode.org The datastore will be your bottleneck But modern DBs are very capable - and you can do replicas!
36. Andrew Godwin / @andrew@aeracode.org Queues are still fine! If you need the complexity and scale, then great. But they are harder!
37. Andrew Godwin / @andrew@aeracode.org Remember, there's no universal solution Just please, please, avoid building a mess of microservices that do RPC calls
38. Thanks! Andrew Godwin andrew@aeracode.org Want to talk more about Takahē and ActivityPub protocols? Come to the Open Space at 3pm in Room 251AB!