Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
background job patterns
Search
Matt Yoho
May 23, 2012
Programming
2
500
background job patterns
Matt Yoho
May 23, 2012
Tweet
Share
More Decks by Matt Yoho
See All by Matt Yoho
Working with Git and GitHub
mattyoho
4
300
Qualitative Analysis of Algorithms
mattyoho
5
420
Metaprogramming Ruby
mattyoho
19
680
Exploiting The Resource Idiom
mattyoho
2
390
testing revisited - outside-in
mattyoho
1
120
shared_auth.pdf
mattyoho
0
2.3k
Request-Response cycle
mattyoho
5
180
Intro to Rest
mattyoho
6
580
Other Decks in Programming
See All in Programming
CSC305 Lecture 04
javiergs
PRO
0
230
メモリ不足との戦い〜大量データを扱うアプリでの実践例〜
kwzr
1
770
Model Pollution
hschwentner
1
180
複雑化したリポジトリをなんとかした話 pipenvからuvによるモノレポ構成への移行
satoshi256kbyte
1
740
ИИ-Агенты в каждый дом – Алексей Порядин, PythoNN
sobolevn
0
150
CSC305 Lecture 01
javiergs
PRO
1
390
『毎日の移動』を支えるGoバックエンド内製開発
yutautsugi
2
120
CSC509 Lecture 02
javiergs
PRO
0
400
パフォーマンスチューニングで Web 技術を深掘り直す
progfay
18
4.9k
大規模アプリのDIフレームワーク刷新戦略 ~過去最大規模の並行開発を止めずにアプリ全体に導入するまで~
mot_techtalk
0
360
プログラマのための作曲入門
cheebow
0
530
Build your own WebP codec in Swift
kishikawakatsumi
2
890
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
303
21k
Fireside Chat
paigeccino
40
3.7k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
162
15k
How STYLIGHT went responsive
nonsquared
100
5.8k
Bash Introduction
62gerente
615
210k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
54
3k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
For a Future-Friendly Web
brad_frost
180
9.9k
Visualization
eitanlees
148
16k
Thoughts on Productivity
jonyablonski
70
4.9k
Scaling GitHub
holman
463
140k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
Transcript
Approaches to Background Jobs Wednesday, May 23, 12
Architectures Wednesday, May 23, 12
“one soul, two bodies” Wednesday, May 23, 12
Wednesday, May 23, 12
prenuptial agreement Wednesday, May 23, 12
Wednesday, May 23, 12
Types of Jobs Wednesday, May 23, 12
Run-once (Resque, DelayedJob) Wednesday, May 23, 12
Event occurs, job enqueued Worker instantiate job model Load AR
object(s) model Create AR object(s) Hit API job {id, token} job goes away Wednesday, May 23, 12
Recurring jobs (Recursive queuing) Wednesday, May 23, 12
Event occurs, job enqueued Worker instantiate job model Load AR
object(s) model Create AR object(s) Hit API job {id, token} Wait 90s re-enqueue self Wednesday, May 23, 12
Scheduled jobs (resque-scheduler, cron) Wednesday, May 23, 12
Event occurs, job scheduled Worker instantiate job model Load AR
object(s) model Create AR object(s) Hit API job {id, token} job goes away 90s enqueue job Wednesday, May 23, 12
CancelAbandonedOrders: cron: "*/5 * * * *" clear_leaderboards_contributors: cron: "30
6 * * 1" class: ClearLeaderboards queue: low args: contributors description: "Reset the weekly leaders" Wednesday, May 23, 12
Approach can work, but is a bit kludgy Wednesday, May
23, 12
Difficult or impossible to adjust intervals programmatically (dynamically) Wednesday, May
23, 12
May want to respond to traffic, load, rate-limiting, by adding
or subtracting workers, changing interval Wednesday, May 23, 12
Are we meeting the SLA? Wednesday, May 23, 12
Service Level Agreement Wednesday, May 23, 12
Service Level Agreement “At least every 5 minutes” Wednesday, May
23, 12
Hypothetical: SLA: 30 seconds Mean job time: 5 seconds Wednesday,
May 23, 12
Increased complexity and decision making around when to run “jobs”
Wednesday, May 23, 12
Sounds like an app Wednesday, May 23, 12
models, views, controllers, jobs, client libs Wednesday, May 23, 12
models, views, controllers, jobs, client libs jobs, client libraries Wednesday,
May 23, 12
models, views, controllers jobs, client libraries Wednesday, May 23, 12
models, views, controllers jobs, client libraries Wednesday, May 23, 12
models, views, controllers jobs, client libraries message queue Wednesday, May
23, 12
models, views, controllers jobs, client libraries message queue Emits event
messages Wednesday, May 23, 12
models, views, controllers jobs, client libraries message queue m “User
123 linked twitter account with token ‘abc’” Wednesday, May 23, 12
models, views, controllers jobs, client libraries message queue m “User
123 linked twitter account with token ‘abc’” Wednesday, May 23, 12
models, views, controllers jobs, client libraries message queue m Consumes
messages, acts Wednesday, May 23, 12
models, views, controllers jobs, client libraries message queue m Consumes
messages, acts Wednesday, May 23, 12
models, views, controllers jobs, client libraries message queue m “Every
30 seconds, I’ll hit the twitter api with token ‘abc’ and then the feed engine api for user 123” Wednesday, May 23, 12
Wednesday, May 23, 12
pros: increased flexibility & control horizontal scaling smaller apps fewer
specs per app cons: burden of complexity on you Wednesday, May 23, 12