Tweet
Tweet

Slide 1

Slide 1 text

Approaches to Background Jobs Wednesday, May 23, 12

Slide 2

Slide 2 text

Architectures Wednesday, May 23, 12

Slide 3

Slide 3 text

“one soul, two bodies” Wednesday, May 23, 12

Slide 4

Slide 4 text

Wednesday, May 23, 12

Slide 5

Slide 5 text

prenuptial agreement Wednesday, May 23, 12

Slide 6

Slide 6 text

Wednesday, May 23, 12

Slide 7

Slide 7 text

Types of Jobs Wednesday, May 23, 12

Slide 8

Slide 8 text

Run-once (Resque, DelayedJob) Wednesday, May 23, 12

Slide 9

Slide 9 text

Event occurs, job enqueued Worker instantiate job model Load AR object(s) model Create AR object(s) Hit API job {id, token} job goes away Wednesday, May 23, 12

Slide 10

Slide 10 text

Recurring jobs (Recursive queuing) Wednesday, May 23, 12

Slide 11

Slide 11 text

Event occurs, job enqueued Worker instantiate job model Load AR object(s) model Create AR object(s) Hit API job {id, token} Wait 90s re-enqueue self Wednesday, May 23, 12

Slide 12

Slide 12 text

Scheduled jobs (resque-scheduler, cron) Wednesday, May 23, 12

Slide 13

Slide 13 text

Event occurs, job scheduled Worker instantiate job model Load AR object(s) model Create AR object(s) Hit API job {id, token} job goes away 90s enqueue job Wednesday, May 23, 12

Slide 14

Slide 14 text

CancelAbandonedOrders: cron: "*/5 * * * *" clear_leaderboards_contributors: cron: "30 6 * * 1" class: ClearLeaderboards queue: low args: contributors description: "Reset the weekly leaders" Wednesday, May 23, 12

Slide 15

Slide 15 text

Approach can work, but is a bit kludgy Wednesday, May 23, 12

Slide 16

Slide 16 text

Difficult or impossible to adjust intervals programmatically (dynamically) Wednesday, May 23, 12

Slide 17

Slide 17 text

May want to respond to traffic, load, rate-limiting, by adding or subtracting workers, changing interval Wednesday, May 23, 12

Slide 18

Slide 18 text

Are we meeting the SLA? Wednesday, May 23, 12

Slide 19

Slide 19 text

Service Level Agreement Wednesday, May 23, 12

Slide 20

Slide 20 text

Service Level Agreement “At least every 5 minutes” Wednesday, May 23, 12

Slide 21

Slide 21 text

Hypothetical: SLA: 30 seconds Mean job time: 5 seconds Wednesday, May 23, 12

Slide 22

Slide 22 text

Increased complexity and decision making around when to run “jobs” Wednesday, May 23, 12

Slide 23

Slide 23 text

Sounds like an app Wednesday, May 23, 12

Slide 24

Slide 24 text

models, views, controllers, jobs, client libs Wednesday, May 23, 12

Slide 25

Slide 25 text

models, views, controllers, jobs, client libs jobs, client libraries Wednesday, May 23, 12

Slide 26

Slide 26 text

models, views, controllers jobs, client libraries Wednesday, May 23, 12

Slide 27

Slide 27 text

models, views, controllers jobs, client libraries Wednesday, May 23, 12

Slide 28

Slide 28 text

models, views, controllers jobs, client libraries message queue Wednesday, May 23, 12

Slide 29

Slide 29 text

models, views, controllers jobs, client libraries message queue Emits event messages Wednesday, May 23, 12

Slide 30

Slide 30 text

models, views, controllers jobs, client libraries message queue m “User 123 linked twitter account with token ‘abc’” Wednesday, May 23, 12

Slide 31

Slide 31 text

models, views, controllers jobs, client libraries message queue m “User 123 linked twitter account with token ‘abc’” Wednesday, May 23, 12

Slide 32

Slide 32 text

models, views, controllers jobs, client libraries message queue m Consumes messages, acts Wednesday, May 23, 12

Slide 33

Slide 33 text

models, views, controllers jobs, client libraries message queue m Consumes messages, acts Wednesday, May 23, 12

Slide 34

Slide 34 text

models, views, controllers jobs, client libraries message queue m “Every 30 seconds, I’ll hit the twitter api with token ‘abc’ and then the feed engine api for user 123” Wednesday, May 23, 12

Slide 35

Slide 35 text

Wednesday, May 23, 12

Slide 36

Slide 36 text

pros: increased flexibility & control horizontal scaling smaller apps fewer specs per app cons: burden of complexity on you Wednesday, May 23, 12