Sidekiq and background processing

What is and when do we need background processing. Example using Ruby Sidekiq

Transcript

1. Sidekiq Background processes Why they’re great and Why they’re tricky
2. Overview: Background processing
3. 408 Request Timeout And even when it works, user feedback within 1s1 Latency affects user experience I was told in games class that any interaction slower than 50s would be noticeable for a user. Google states that a website should load within 2 seconds2. And, there is a maximum Almost all servers have a predefined maximum request timeout set to 30/60s. If a process runs for longer the server will cut off and return an error to the client. 1https://github.com/Tendrl/documentation/wiki/B est-Practices-for-Response-Times-and-Latency 2https://www.semrush.com/blog/how-fast-is-fast-enough-page-lo ad-time-and-your-bottom-line/
4. Common tasks • Imports • Exports • Reports • Non-urgent tasks ◦ Sending email ◦ Collecting data (external sources) ◦ Combining data (multiple sources) ◦ Sending data (shipping)
5. Simple. Dump it on someone else.
6. It’s not that simple Fire and forget? Then what? • Errors? ◦ Interrupts? ◦ Timeouts? ◦ Retries? ◦ Recovery? • Interact ◦ Abort? ◦ Pause? ◦ Rerun? • Results? • Status?
7. Asynchronicity Async tasks requires async thinking. It also requires certain patterns of communication. • Callback argument • Web hooks • Queue listener • Polling
8. What is sidekiq? Sidekiq is a gem (plugin) for Simple, efficient background processing for Ruby https://github.com/mperham/sidekiq
9. Example Using sidekiq
10. Example: Generate a report Requirements: • Trigger with arguments • Long running ◦ Calls external service to get data • Status reporting (is it alive?) • Error reporting (did it fail?) • Getting results
11. Architecture • One client application • One background process • Communication through datasource • Results written to datasource (could be same, could be different)
12. Running sidekiq Client side: Background process:
13. That’s it! Questions on that?