Sidekiq on the surface and under the hood

A presentation about Sidekiq on Wroclove.rb 2022. Good practices, design patterns to follow, and habits to develop to build a well-working background job processing system. Sidekiq internals, the way it communicates with Redis, and how the middleware is working.

Transcript

1. Hello
2. SIDEKIQ On the surface Under the hood
3. ABOUT ME briefly Hello, I'm Paweł Dąbrowski. I write for both human beings and computers, a lot. With Ruby since 2010 IT Samurai, CTO @ iRonin https://paweldabrowski.com https://ironin.it/blog https://twitter.com/pdabrowski6 https://github.com/pdabrowski6
4. SIDEKIQ? What is
5. SIDEKIQ? What is Simple, efficient background processing for Ruby OPEN Source SIMPLE and EFFICIENT WORK OUTSIDE RAILS
6. ON THE SURFACE UNDER THE HOOD Good practices, useful patterns, mistakes to avoid The connection with Redis, paid versions, plugins and middleware m o r e m o r e
7. GOOD PRACTICES - do not annoy customers - do not waste the money - do not waste time Why should you care about
8. Sidekiq on the surface - good practices PROPERLY name things
9. LONG LIVE JOBS Workers are dead
10. NAMING use meaningful Show the job's responsibility DELETE_USERS remove_OUTDATED_users_JOB RESUME_PROCESSOR extract_TEXT_FROM_RESUME_JOB ??? ???
11. JOBS group related UPDATE_STRIPE_BALANCE Make_PAYOUT_FROM_STRIPE STRIPE_DELETE_PRODUCT STRIPE::UPDATE_BALANCE_job STRIPE::MAKE_PAYOUT_job STRIPE::DELETE_PRODUCT_JOB
12. Sidekiq on the surface - good practices SIMPLE keep parameters
13. YOU CAN YES USE MANY ARGUMENTS PASS OBJECTS PASS HASHES
14. YOU SHOULD it does not mean
15. IDEA OF SIMPLE PARAMS 01 02 03 EASIER TO QUEUE EASIER TO FIND BETTER ISOLATION EASY TESTING 04 HAPPIER REDIS 05
17. Sidekiq on the surface - good practices for granted don't take data
18. and data has changed imagine you queue a job
19. When job is not executed immediately, the data passed as arguments can change overtime.
20. Pass reference instead value if possible, to always have access to fresh information.
21. Don't queue the job if you are not sure about the state of the data.
22. Sidekiq on the surface - good practices SIMPLE keep logic
23. Sidekiq on the surface - good practices JOBS use smaller
24. Avoid jobs that are not possible or difficult to retry at some point
25. Run multiple smaller jobs if possible.
26. Faster processing with concurrency Possible to queue manually only one job Easy to retry only one job and it won't affect the whole process Progress tracking
27. Sidekiq on the surface - good practices TO REDIS share connections
28. YOUR APPLICATION SIDEKIQ Redis
29. YOUR APPLICATION SIDEKIQ Connection POOL
31. Sidekiq on the surface - good practices INHERITANCE don't use
32. ONE PARENT the inheritance problem:
33. JOB A has handling for error a handling for error B handling for error C JOB B has JOB C has
34. PREPEND welcome
37. Sidekiq on the surface - good practices PROPERLY retry errors
38. If you want to retry, make sure you are able to rollback to initial state.
39. Don't pollute your error tracking service and waste API calls.
40. Create error wrapper Or report error when you retried multiple times and the game is over for the job. 1.
41. Or report error when you retried multiple times and the game is over for the job. 2. Re-raise error with the wrapper
42. Or report error when you retried multiple times and the game is over for the job. 3. Do not report the error wrapper
43. Or report error when you retried multiple times and the game is over for the job. 4. Report the original error after all retries
44. Sidekiq on the surface - good practices LOGGING adjust
45. MEAN? What do you Take care of the configuration of the log, thank me later. SAVE LOGS LOG JOB EXECUTION LOG INSIDE THE JOB
46. As of Sidekiq 6.0, you have to take care of log redirection. bundle exec sidekiq 2>&1 >> ./log/sidekiq.log
47. The default job execution output
48. MIDDLEWARE Include arguments in the logs.
49. Job execution output with arguments
50. LOGGING Use logger from Sidekiq
51. Job execution output with logs
52. SIDEKIQ INTERNALS - learn how to extend Sidekiq - debug more effectively - become better developer Why should you care about
53. MY PHILOSOPHY FOR LEARING Learn how to use BUILD SOMETHING RUN IN PRODUCTION Learn good practices Look under the hood
54. MY PHILOSOPHY FOR LEARING Learn how to use BUILD SOMETHING RUN IN PRODUCTION Learn good practices Look under the hood LOOK INTO DOCUMENTATION
55. + SIDEKIQ REDIS
56. - in-memory data store - open source - key value pairs - data types: sets, sorted sets, lists, hashes
57. FLOW WITH REDIS Add job to queue pick JOB FROM QUEUE
58. ADD JOB TO QUEUE PASS PARAMS ASSIGN DEFAULT PARAMS validation MIDDLEWARE PUSH QUEUE PUSH PAYLOAD VERIFY JSON
59. PASS PARAMS
60. VALIDATION JOB IS A HASH AT IS NUMERIC IF GIVEN ARGS ARE IN ARRAY TAGS ARE IN ARRAY JOB CLASS IS STRING / CLASS
61. ASSIGN DEFAULT PARAMS Page 03 of 15 RETRY: TRUE QUEUE: DEFAULT CREATED_AT: TIME.now.TO_F JID: SecureRandom.hex(12)
62. CLIENT MIDDLEWARE
63. VERIFY JSON
64. LATER? perform NO YES ZADD Add member with score to sorted set. SADD LPUSH Add member to set, ignore if exists Add to the head of the list
65. PICK JOB FROM QUEUE MANAGER POLLER
66. poller ZRANGEBYSCORE Elements in sorted set with score between min and max
67. manager BRPOP Blocking list pop
68. MANAGER FLOW If format is wrong, job is added to the dead queue. Simply create instance of job and call #perform on it. DECODE PAYLOAD INVOKE MIDDLEWARE EXECUTE JOB
69. DASHBOARD Simple Rack application + views in .erb format. Sidekiq
70. Thank you!