background job patterns

Transcript

1. Approaches to Background Jobs Wednesday, May 23, 12
2. Architectures Wednesday, May 23, 12
3. “one soul, two bodies” Wednesday, May 23, 12
4. Wednesday, May 23, 12
5. prenuptial agreement Wednesday, May 23, 12
6. Wednesday, May 23, 12
7. Types of Jobs Wednesday, May 23, 12
8. Run-once (Resque, DelayedJob) Wednesday, May 23, 12
9. Event occurs, job enqueued Worker instantiate job model Load AR object(s) model Create AR object(s) Hit API job {id, token} job goes away Wednesday, May 23, 12
10. Recurring jobs (Recursive queuing) Wednesday, May 23, 12
11. Event occurs, job enqueued Worker instantiate job model Load AR object(s) model Create AR object(s) Hit API job {id, token} Wait 90s re-enqueue self Wednesday, May 23, 12
12. Scheduled jobs (resque-scheduler, cron) Wednesday, May 23, 12
13. Event occurs, job scheduled Worker instantiate job model Load AR object(s) model Create AR object(s) Hit API job {id, token} job goes away 90s enqueue job Wednesday, May 23, 12
14. CancelAbandonedOrders: cron: "*/5 * * * *" clear_leaderboards_contributors: cron: "30 6 * * 1" class: ClearLeaderboards queue: low args: contributors description: "Reset the weekly leaders" Wednesday, May 23, 12
15. Approach can work, but is a bit kludgy Wednesday, May 23, 12
16. Difficult or impossible to adjust intervals programmatically (dynamically) Wednesday, May 23, 12
17. May want to respond to traffic, load, rate-limiting, by adding or subtracting workers, changing interval Wednesday, May 23, 12
18. Are we meeting the SLA? Wednesday, May 23, 12
19. Service Level Agreement Wednesday, May 23, 12
20. Service Level Agreement “At least every 5 minutes” Wednesday, May 23, 12
21. Hypothetical: SLA: 30 seconds Mean job time: 5 seconds Wednesday, May 23, 12
22. Increased complexity and decision making around when to run “jobs” Wednesday, May 23, 12
23. Sounds like an app Wednesday, May 23, 12
24. models, views, controllers, jobs, client libs Wednesday, May 23, 12
25. models, views, controllers, jobs, client libs jobs, client libraries Wednesday, May 23, 12
26. models, views, controllers jobs, client libraries Wednesday, May 23, 12
27. models, views, controllers jobs, client libraries Wednesday, May 23, 12
28. models, views, controllers jobs, client libraries message queue Wednesday, May 23, 12
29. models, views, controllers jobs, client libraries message queue Emits event messages Wednesday, May 23, 12
30. models, views, controllers jobs, client libraries message queue m “User 123 linked twitter account with token ‘abc’” Wednesday, May 23, 12
31. models, views, controllers jobs, client libraries message queue m “User 123 linked twitter account with token ‘abc’” Wednesday, May 23, 12
32. models, views, controllers jobs, client libraries message queue m Consumes messages, acts Wednesday, May 23, 12
33. models, views, controllers jobs, client libraries message queue m Consumes messages, acts Wednesday, May 23, 12
34. models, views, controllers jobs, client libraries message queue m “Every 30 seconds, I’ll hit the twitter api with token ‘abc’ and then the feed engine api for user 123” Wednesday, May 23, 12
35. Wednesday, May 23, 12
36. pros: increased flexibility & control horizontal scaling smaller apps fewer specs per app cons: burden of complexity on you Wednesday, May 23, 12