Upgrade to Pro — share decks privately, control downloads, hide ads and more …

EuRuKo 2018: Scaling a monolith isn't scaling microservices

EuRuKo 2018: Scaling a monolith isn't scaling microservices

talk at EuRuKo 2018 about background jobs and message oriented middleware

Kerstin Puschke

August 25, 2018
Tweet

More Decks by Kerstin Puschke

Other Decks in Programming

Transcript

  1. @titanoboa42 • “At least” vs. “at most” once delivery •

    Idempotent jobs & at least once delivery No exactly once delivery
  2. @titanoboa42 • If order matters, queue sequentially • First job

    queues follow up jobs Out of order delivery
  3. @titanoboa42 • Aborted and requeued • Job may not finish

    before being aborted again Long running jobs - Sidekiq
  4. @titanoboa42 • Split job into collection and task to be

    done • Checkpoint after iteration & requeue Large collections
  5. @titanoboa42 • Shutdown workers anytime • Disaster prevention • Data

    integrity Interruptible job with automatic resuming
  6. @titanoboa42 • Great features based on task queues • Abstracting

    the complexity of jobs running in the future and / or in parallel Background jobs
  7. @titanoboa42 • Simple concrete jobs • Complex overall system •

    Great for monolithic architecture Background jobs
  8. @titanoboa42 Topic with queues
 provides
 advanced routing App Server Business


    Partners Support
 Contracts Orders Messaging Middleware
  9. @titanoboa42 Topic with queues
 provides
 advanced routing App Server Message

    Queue Business
 Partners Support
 Contracts Orders Message Queue Messaging Middleware
  10. @titanoboa42 • Avoid n:m routing • Better representation of domain:


    multiple messages routed 1:n, n:1 instead Keep breaking changes manageable
  11. @titanoboa42 • Messages are kept if not consumed • Growing

    queue becomes problematic Remove queues if consumer is gone
  12. @titanoboa42 • “At least” vs. “at most” once delivery •

    Idempotent consumer & at least once delivery No exactly once delivery
  13. @titanoboa42 • Payload contains updated data • Avoids requests to

    service that owns data Event-carried state transfer
 requires order information
  14. @titanoboa42 • Payload contains updated data • Avoids requests to

    service that owns data • Provide order information (e.g. updated_at) Event-carried state transfer
 requires order information
  15. @titanoboa42 • Payload has id only • Consumer requests data

    separately Event notifications are commutative
  16. @titanoboa42 • Payload has id only • Consumer requests data

    separately • Consumer doesn’t learn every state Event notifications are commutative
  17. @titanoboa42 • Great features based on queues and topics •

    Abstracting the complexity of pub sub Message oriented middleware
  18. @titanoboa42 • Abstracting the complexity of messages being consumed in

    the future / in parallel Message oriented middleware
  19. @titanoboa42 • Abstracting the complexity of messages being consumed in

    the future / in parallel • Simple concrete message consumers Message oriented middleware
  20. @titanoboa42 • Events persisted into append-only log • Consumers read

    shared log • Stateless broker (no queues) Event log
  21. @titanoboa42 • Topics and Queues • Simple, stateless message consumers

    • For distributed, stateful architecture Message oriented middleware
  22. @titanoboa42 • Shared log, no queues • Stateful consumers •

    For event sourcing & real time applications Event logs