Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reconciling Everything

Reconciling Everything

A talk I gave at PyCon US 2023.

Andrew Godwin

April 22, 2023

More Decks by Andrew Godwin

Other Decks in Programming


  1. Andrew Godwin / @[email protected] Hi, I’m Andrew Godwin • Principal

    Engineer at Astronomer.io (Airflow) • Django Migrations, Channels, Async, others • Building distributed systems since 2008
  2. Andrew Godwin / @[email protected] Let's start with a bold statement

    I hear it drives engagement. Maybe also heckling.
  3. Andrew Godwin / @[email protected] Event-Driven Via Queues Totally separate stores

    of truth that only talk via queues Reconciliation Loops Stateless components that talk to one store of truth There are only two good* ways to build distributed systems**:
  4. Andrew Godwin / @[email protected] Due to size What companies often

    claim is the reason Due to team structure What the actual reason often is Due to federation It's what the internet was built on!
  5. Andrew Godwin / @[email protected] A system is defined by its

    failure modes Nobody gets paged when it's up and running happily
  6. Andrew Godwin / @[email protected] Event-Driven (Message Passing) Task Runners Billing

    System QUEUE QUEUE Log Storage Analytics Aggregator QUEUE
  7. Andrew Godwin / @[email protected] What kind of queue is it?

    Hint: It will not deliver messages exactly once
  8. Andrew Godwin / @[email protected] At-most-once In case of failure, a

    message will not get delivered At-least-once In case of failure, a messages will get delivered twice or more
  9. Andrew Godwin / @[email protected] Backlogs Need more queue consumers, can

    be asymmetric Replays I hope you can cope with lots of duplicates! Overflow Do you grind to a halt? Or drop?
  10. Andrew Godwin / @[email protected] Neither is "perfect" (But I recommend

    at-least-once unless you truly do not care about the data)
  11. Andrew Godwin / @[email protected] Kubernetes Example • User creates a

    new Deployment • Deployment controller notices that the deployment has no pods ◦ To reconcile this, it makes one • Pod controller notices that the pod is not assigned ◦ To reconcile this, it assigns it to a node • Kubelet notices it has an assigned pod that is not running ◦ To reconcile this, it creates it locally and starts it
  12. Andrew Godwin / @[email protected] One "single point of failure" is

    great Keeping one service up is much simpler
  13. Andrew Godwin / @[email protected] So how do reconciliation loops fail?

    They slow down and halt, but with easy restarts and no data loss
  14. Andrew Godwin / @[email protected] Each controller must be stateless This

    lets you scale them up, down, restart, upgrade, and recover easily
  15. Andrew Godwin / @[email protected] The Fediverse is an event-driven system

    Well, mostly. It's also imperfect and still evolving.
  16. Andrew Godwin / @[email protected] Making a single Fediverse post QUEUE

    Sending Client Origin API Fanout System Destination Inbox Timeline Builder Receiving Client QUEUE
  17. Andrew Godwin / @[email protected] A server needs background workers Both

    to fan out sending of posts, and to process incoming posts
  18. Andrew Godwin / @[email protected] You can also do this with

    Airflow We have an Astronomer executor which uses this for running tasks
  19. Andrew Godwin / @[email protected] Airflow Example • A Task Instance

    is scheduled to run, and the scheduler passes it over • The Executor makes a new Workload for the Task Instance • Allocator notices that the Workload has no Runner ◦ To reconcile this, it assigns it to a runner • Runner notices it has an assigned Workload that is not running ◦ To reconcile this, it creates it locally and starts it
  20. Andrew Godwin / @[email protected] All state in the central store

    You can maybe get away with caching elsewhere Zero service-to-service comms Communication paths are failure modes Controllers should be simple loops Query database, reconcile, repeat
  21. Andrew Godwin / @[email protected] They are easy to test and

    refactor Mocking your datastore is all that's needed; everything else should flow
  22. Andrew Godwin / @[email protected] There are downsides - mostly max

    scale But I generally design systems with upper bounds on their throughput
  23. Andrew Godwin / @[email protected] The datastore will be your bottleneck

    But modern DBs are very capable - and you can do replicas!
  24. Andrew Godwin / @[email protected] Queues are still fine! If you

    need the complexity and scale, then great. But they are harder!
  25. Andrew Godwin / @[email protected] Remember, there's no universal solution Just

    please, please, avoid building a mess of microservices that do RPC calls
  26. Thanks! Andrew Godwin [email protected] Want to talk more about Takahē

    and ActivityPub protocols? Come to the Open Space at 3pm in Room 251AB!