Sidekiq on the surface and under the hood

A presentation about Sidekiq on Wroclove.rb 2022. Good practices, design patterns to follow, and habits to develop to build a well-working background job processing system. Sidekiq internals, the way it communicates with Redis, and how the middleware is working.

Transcript

1. Hello
   

   View Slide

2. SIDEKIQ
   On the surface
   Under the hood
   

   View Slide

3. ABOUT ME
   briefly
   Hello, I'm Paweł Dąbrowski. I write for
   both human beings and computers,
   a lot.
   With Ruby since 2010
   IT Samurai, CTO @ iRonin
   https://paweldabrowski.com
   https://ironin.it/blog
   https://twitter.com/pdabrowski6
   https://github.com/pdabrowski6
   

   View Slide

4. SIDEKIQ?
   What is
   

   View Slide

5. SIDEKIQ?
   What is
   Simple, efficient background processing for Ruby
   OPEN Source SIMPLE and EFFICIENT WORK OUTSIDE RAILS
   

   View Slide

6. ON THE SURFACE UNDER THE HOOD
   Good practices, useful patterns, mistakes to
   avoid
   The connection with Redis, paid versions,
   plugins and middleware
   m o r e m o r e
   

   View Slide

7. GOOD PRACTICES
   - do not annoy customers
   - do not waste the money
   - do not waste time
   Why should you
   care about
   

   View Slide

8. Sidekiq on the surface - good practices
   PROPERLY
   name things
   

   View Slide

9. LONG LIVE JOBS
   Workers are
   dead
   

   View Slide

10. NAMING
    use meaningful
    Show the job's responsibility
    DELETE_USERS
    remove_OUTDATED_users_JOB
    RESUME_PROCESSOR
    extract_TEXT_FROM_RESUME_JOB
    ???
    ???
    

    View Slide

11. JOBS
    group related
    UPDATE_STRIPE_BALANCE
    Make_PAYOUT_FROM_STRIPE
    STRIPE_DELETE_PRODUCT
    STRIPE::UPDATE_BALANCE_job
    STRIPE::MAKE_PAYOUT_job
    STRIPE::DELETE_PRODUCT_JOB
    

    View Slide

12. Sidekiq on the surface - good practices
    SIMPLE
    keep parameters
    

    View Slide

13. YOU CAN
    YES
    USE MANY ARGUMENTS PASS OBJECTS
    PASS HASHES
    

    View Slide

14. YOU SHOULD
    it does not mean
    

    View Slide

15. IDEA OF SIMPLE PARAMS
    01 02 03
    EASIER TO QUEUE EASIER TO FIND BETTER ISOLATION EASY TESTING
    04
    HAPPIER REDIS
    05
    

    View Slide

16. View Slide

17. Sidekiq on the surface - good practices
    for granted
    don't take data
    

    View Slide

18. and data has changed
    imagine you queue a job
    

    View Slide

19. When job is not executed
    immediately, the data passed as
    arguments can change overtime.
    

    View Slide

20. Pass reference instead value if
    possible, to always have access
    to fresh information.
    

    View Slide

21. Don't queue the job if you are not
    sure about the state of the data.
    

    View Slide

22. Sidekiq on the surface - good practices
    SIMPLE
    keep logic
    

    View Slide

23. Sidekiq on the surface - good practices
    JOBS
    use smaller
    

    View Slide

24. Avoid jobs that are not possible
    or difficult to retry at some point
    

    View Slide

25. Run multiple smaller jobs if
    possible.
    

    View Slide

26. Faster processing with
    concurrency
    Possible to queue manually
    only one job
    Easy to retry only one job and
    it won't affect the whole
    process
    Progress tracking
    

    View Slide

27. Sidekiq on the surface - good practices
    TO REDIS
    share connections
    

    View Slide

28. YOUR APPLICATION SIDEKIQ
    Redis
    

    View Slide

29. YOUR APPLICATION SIDEKIQ
    Connection POOL
    

    View Slide

30. View Slide

31. Sidekiq on the surface - good practices
    INHERITANCE
    don't use
    

    View Slide

32. ONE PARENT
    the inheritance problem:
    

    View Slide

33. JOB A
    has
    handling for error a handling for error B handling for error C
    JOB B
    has
    JOB C
    has
    

    View Slide

34. PREPEND
    welcome
    

    View Slide

35. View Slide

36. View Slide

37. Sidekiq on the surface - good practices
    PROPERLY
    retry errors
    

    View Slide

38. If you want to retry, make sure
    you are able to rollback to initial
    state.
    

    View Slide

39. Don't pollute your error tracking
    service and waste API calls.
    

    View Slide

40. Create error wrapper
    Or report error when you retried
    multiple times and the game is
    over for the job.
    1.
    

    View Slide

41. Or report error when you retried
    multiple times and the game is
    over for the job.
    2. Re-raise error with the wrapper
    

    View Slide

42. Or report error when you retried
    multiple times and the game is
    over for the job.
    3. Do not report the error wrapper
    

    View Slide

43. Or report error when you retried
    multiple times and the game is
    over for the job.
    4. Report the original error after
    all retries
    

    View Slide

44. Sidekiq on the surface - good practices
    LOGGING
    adjust
    

    View Slide

45. MEAN?
    What do you
    Take care of the configuration of the log, thank me later.
    SAVE LOGS LOG JOB EXECUTION LOG INSIDE THE JOB
    

    View Slide

46. As of Sidekiq 6.0, you have to take care
    of log redirection.
    bundle exec sidekiq 2>&1 >> ./log/sidekiq.log
    
    
    

    View Slide

47. The default job execution output
    

    View Slide

48. MIDDLEWARE
    Include arguments in the logs.
    

    View Slide

49. Job execution output with arguments
    

    View Slide

50. LOGGING
    Use logger from Sidekiq
    

    View Slide

51. Job execution output with logs
    

    View Slide

52. SIDEKIQ INTERNALS
    - learn how to extend Sidekiq
    - debug more effectively
    - become better developer
    Why should you
    care about
    

    View Slide

53. MY PHILOSOPHY FOR LEARING
    Learn how to use BUILD SOMETHING RUN IN PRODUCTION Learn good practices Look under the hood
    

    View Slide

54. MY PHILOSOPHY FOR LEARING
    Learn how to use BUILD SOMETHING RUN IN PRODUCTION Learn good practices Look under the hood
    LOOK INTO DOCUMENTATION
    

    View Slide

55. +
    SIDEKIQ REDIS
    

    View Slide

56. - in-memory data store
    - open source
    - key value pairs
    - data types: sets, sorted sets,
    lists, hashes
    

    View Slide

57. FLOW WITH REDIS
    Add job to queue pick JOB FROM QUEUE
    

    View Slide

58. ADD JOB TO QUEUE
    PASS PARAMS ASSIGN DEFAULT PARAMS
    validation MIDDLEWARE PUSH QUEUE PUSH PAYLOAD
    VERIFY JSON
    

    View Slide

59. PASS PARAMS
    

    View Slide

60. VALIDATION
    JOB IS A HASH
    AT IS NUMERIC IF GIVEN
    ARGS ARE IN ARRAY
    TAGS ARE IN ARRAY
    JOB CLASS IS STRING / CLASS
    

    View Slide

61. ASSIGN
    DEFAULT
    PARAMS
    Page 03 of 15
    RETRY: TRUE
    QUEUE: DEFAULT
    CREATED_AT: TIME.now.TO_F
    JID: SecureRandom.hex(12)
    

    View Slide

62. CLIENT MIDDLEWARE
    

    View Slide

63. VERIFY JSON
    

    View Slide

64. LATER?
    perform
    NO
    YES
    ZADD
    Add member with score
    to sorted set.
    SADD LPUSH
    Add member to
    set, ignore if exists
    Add to the head
    of the list
    

    View Slide

65. PICK JOB FROM QUEUE
    MANAGER
    POLLER
    

    View Slide

66. poller
    ZRANGEBYSCORE
    Elements in sorted set
    with score between
    min and max
    

    View Slide

67. manager
    BRPOP
    Blocking list pop
    

    View Slide

68. MANAGER FLOW
    If format is wrong, job is added to
    the dead queue.
    Simply create instance of job and
    call #perform on it.
    DECODE PAYLOAD INVOKE MIDDLEWARE EXECUTE JOB
    

    View Slide

69. DASHBOARD
    Simple Rack application + views
    in .erb format.
    Sidekiq
    

    View Slide

70. Thank you!
    

    View Slide