$30 off During Our Annual Pro Sale. View Details »

Sidekiq on the surface and under the hood

Sidekiq on the surface and under the hood

A presentation about Sidekiq on Wroclove.rb 2022. Good practices, design patterns to follow, and habits to develop to build a well-working background job processing system. Sidekiq internals, the way it communicates with Redis, and how the middleware is working.

Paweł Dąbrowski

September 18, 2022
Tweet

Other Decks in Programming

Transcript

  1. Hello

  2. SIDEKIQ On the surface Under the hood

  3. ABOUT ME briefly Hello, I'm Paweł Dąbrowski. I write for

    both human beings and computers, a lot. With Ruby since 2010 IT Samurai, CTO @ iRonin https://paweldabrowski.com https://ironin.it/blog https://twitter.com/pdabrowski6 https://github.com/pdabrowski6
  4. SIDEKIQ? What is

  5. SIDEKIQ? What is Simple, efficient background processing for Ruby OPEN

    Source SIMPLE and EFFICIENT WORK OUTSIDE RAILS
  6. ON THE SURFACE UNDER THE HOOD Good practices, useful patterns,

    mistakes to avoid The connection with Redis, paid versions, plugins and middleware m o r e m o r e
  7. GOOD PRACTICES - do not annoy customers - do not

    waste the money - do not waste time Why should you care about
  8. Sidekiq on the surface - good practices PROPERLY name things

  9. LONG LIVE JOBS Workers are dead

  10. NAMING use meaningful Show the job's responsibility DELETE_USERS remove_OUTDATED_users_JOB RESUME_PROCESSOR

    extract_TEXT_FROM_RESUME_JOB ??? ???
  11. JOBS group related UPDATE_STRIPE_BALANCE Make_PAYOUT_FROM_STRIPE STRIPE_DELETE_PRODUCT STRIPE::UPDATE_BALANCE_job STRIPE::MAKE_PAYOUT_job STRIPE::DELETE_PRODUCT_JOB

  12. Sidekiq on the surface - good practices SIMPLE keep parameters

  13. YOU CAN YES USE MANY ARGUMENTS PASS OBJECTS PASS HASHES

  14. YOU SHOULD it does not mean

  15. IDEA OF SIMPLE PARAMS 01 02 03 EASIER TO QUEUE

    EASIER TO FIND BETTER ISOLATION EASY TESTING 04 HAPPIER REDIS 05
  16. None
  17. Sidekiq on the surface - good practices for granted don't

    take data
  18. and data has changed imagine you queue a job

  19. When job is not executed immediately, the data passed as

    arguments can change overtime.
  20. Pass reference instead value if possible, to always have access

    to fresh information.
  21. Don't queue the job if you are not sure about

    the state of the data.
  22. Sidekiq on the surface - good practices SIMPLE keep logic

  23. Sidekiq on the surface - good practices JOBS use smaller

  24. Avoid jobs that are not possible or difficult to retry

    at some point
  25. Run multiple smaller jobs if possible.

  26. Faster processing with concurrency Possible to queue manually only one

    job Easy to retry only one job and it won't affect the whole process Progress tracking
  27. Sidekiq on the surface - good practices TO REDIS share

    connections
  28. YOUR APPLICATION SIDEKIQ Redis

  29. YOUR APPLICATION SIDEKIQ Connection POOL

  30. None
  31. Sidekiq on the surface - good practices INHERITANCE don't use

  32. ONE PARENT the inheritance problem:

  33. JOB A has handling for error a handling for error

    B handling for error C JOB B has JOB C has
  34. PREPEND welcome

  35. None
  36. None
  37. Sidekiq on the surface - good practices PROPERLY retry errors

  38. If you want to retry, make sure you are able

    to rollback to initial state.
  39. Don't pollute your error tracking service and waste API calls.

  40. Create error wrapper Or report error when you retried multiple

    times and the game is over for the job. 1.
  41. Or report error when you retried multiple times and the

    game is over for the job. 2. Re-raise error with the wrapper
  42. Or report error when you retried multiple times and the

    game is over for the job. 3. Do not report the error wrapper
  43. Or report error when you retried multiple times and the

    game is over for the job. 4. Report the original error after all retries
  44. Sidekiq on the surface - good practices LOGGING adjust

  45. MEAN? What do you Take care of the configuration of

    the log, thank me later. SAVE LOGS LOG JOB EXECUTION LOG INSIDE THE JOB
  46. As of Sidekiq 6.0, you have to take care of

    log redirection. bundle exec sidekiq 2>&1 >> ./log/sidekiq.log
  47. The default job execution output

  48. MIDDLEWARE Include arguments in the logs.

  49. Job execution output with arguments

  50. LOGGING Use logger from Sidekiq

  51. Job execution output with logs

  52. SIDEKIQ INTERNALS - learn how to extend Sidekiq - debug

    more effectively - become better developer Why should you care about
  53. MY PHILOSOPHY FOR LEARING Learn how to use BUILD SOMETHING

    RUN IN PRODUCTION Learn good practices Look under the hood
  54. MY PHILOSOPHY FOR LEARING Learn how to use BUILD SOMETHING

    RUN IN PRODUCTION Learn good practices Look under the hood LOOK INTO DOCUMENTATION
  55. + SIDEKIQ REDIS

  56. - in-memory data store - open source - key value

    pairs - data types: sets, sorted sets, lists, hashes
  57. FLOW WITH REDIS Add job to queue pick JOB FROM

    QUEUE
  58. ADD JOB TO QUEUE PASS PARAMS ASSIGN DEFAULT PARAMS validation

    MIDDLEWARE PUSH QUEUE PUSH PAYLOAD VERIFY JSON
  59. PASS PARAMS

  60. VALIDATION JOB IS A HASH AT IS NUMERIC IF GIVEN

    ARGS ARE IN ARRAY TAGS ARE IN ARRAY JOB CLASS IS STRING / CLASS
  61. ASSIGN DEFAULT PARAMS Page 03 of 15 RETRY: TRUE QUEUE:

    DEFAULT CREATED_AT: TIME.now.TO_F JID: SecureRandom.hex(12)
  62. CLIENT MIDDLEWARE

  63. VERIFY JSON

  64. LATER? perform NO YES ZADD Add member with score to

    sorted set. SADD LPUSH Add member to set, ignore if exists Add to the head of the list
  65. PICK JOB FROM QUEUE MANAGER POLLER

  66. poller ZRANGEBYSCORE Elements in sorted set with score between min

    and max
  67. manager BRPOP Blocking list pop

  68. MANAGER FLOW If format is wrong, job is added to

    the dead queue. Simply create instance of job and call #perform on it. DECODE PAYLOAD INVOKE MIDDLEWARE EXECUTE JOB
  69. DASHBOARD Simple Rack application + views in .erb format. Sidekiq

  70. Thank you!