Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Очереди задач без купюр, Артем Малышев, Positiv...
Search
IT-People
July 25, 2016
Programming
1
110
Очереди задач без купюр, Артем Малышев, Positive Technologies
Выступление на конференции PyCon Russia 2016
IT-People
July 25, 2016
Tweet
Share
More Decks by IT-People
See All by IT-People
Александр Кошкин, Positive Technologies: Знай и люби свой CPython во имя луны и великой справедливости
itpeople123
1
200
How Python was Shaped by leaky Internals, Armin Ronacher, Flask framework
itpeople123
1
230
Building social network with Neo4j and Python, Андрей Солдатенко, Toptal
itpeople123
1
230
Осторожно, DSL! Иван Цыганов, Positive Technologies
itpeople123
0
120
Докеризация веб-приложения на Python Антон Егоров, Ostrovok.ru
itpeople123
0
190
Python Robot Operating System вместо Middleware framework Дмитрий Киселев, Endpoint.com
itpeople123
0
100
Falcon — очередная WSGI библиотека? Дмитрий Кукушкин, Wargaming
itpeople123
0
260
Flame graph: новый взгляд на привычное профилирование, Кирилл Борисов, Яндекс
itpeople123
1
250
Как добавить scripting в веб приложение, Константин Лопухин, Scrapinghub
itpeople123
0
150
Other Decks in Programming
See All in Programming
オープンソースソフトウェアへの解像度🔬
utam0k
15
2.8k
Building, Deploying, and Monitoring Ruby Web Applications with Falcon (Kaigi on Rails 2025)
ioquatix
4
2.1k
はじめてのDSPy - 言語モデルを『プロンプト』ではなく『プログラミング』するための仕組み
masahiro_nishimi
2
350
(Extension DC 2025) Actor境界を越える技術
teamhimeh
1
250
dynamic!
moro
10
7.8k
CSC305 Lecture 03
javiergs
PRO
0
240
ソフトウェア設計の実践的な考え方
masuda220
PRO
4
580
階層構造を表現するデータ構造とリファクタリング 〜1年で10倍成長したプロダクトの変化と課題〜
yuhisatoxxx
3
1k
登壇は dynamic! な営みである / speech is dynamic
da1chi
0
320
Devoxx BE - Local Development in the AI Era
kdubois
0
130
そのpreloadは必要?見過ごされたpreloadが技術的負債として爆発した日
mugitti9
2
3.4k
kiroとCodexで最高のSpec駆動開発を!!数時間で web3ネイティブなミニゲームを作ってみたよ!
mashharuki
0
180
Featured
See All Featured
Intergalactic Javascript Robots from Outer Space
tanoku
273
27k
How To Stay Up To Date on Web Technology
chriscoyier
791
250k
Designing Experiences People Love
moore
142
24k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.7k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
The Illustrated Children's Guide to Kubernetes
chrisshort
49
51k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
It's Worth the Effort
3n
187
28k
Designing for humans not robots
tammielis
254
26k
Optimizing for Happiness
mojombo
379
70k
Writing Fast Ruby
sferik
629
62k
Transcript
TASK QUEUES UNCUT МАЛЫШЕВ АРТЁМ
None
None
None
None
CELERY app = Celery(__name__) @app.task def add(x, y): return x
+ y add.delay(1, 2)
RQ def add(x, y): return x + y queue =
rq.Queue('default') queue.enqueue(add, 1, 2)
RABBITMQ QUEUES Name Ready Unacked Total celery 1 0 1
PAYLOAD { "task": "app.add", "args": [1, 2], "kwargs": {}, "id":
"27ebcaee-444f-4379-a438-04c3b5bb0fc3", "callbacks": null, "errbacks": null, "eta": null, "expires": null, "timelimit": [null, null], "taskset": null, "chord": null, "retries": 0 }
REDIS QUEUES >>> KEYS * 1) "celery" >>> LRANGE celery
0 -1 1) { "properties": { "body_encoding": "base64", "delivery_info": { "exchange": "celery", "routing_key": "celery" } }, "body": "eyJ0aW1lbGltaXQiOiBbbnVsb...", "content-type": "application/json" }
RQ >>> SMEMBERS rq:queues 1) "rq:queue:default" >>> LRANGE rq:queue:default 0
-1 1) "be58515f-9f330470995d" >>> HGETALL rq:job:be58515f-9f330470995d 1) "status" "queued" 2) "enqueued_at" "2016-06-14T19:45:39Z" 3) "created_at" "2016-06-14T19:45:39Z" 4) "origin" "default" 5) "description" "lib.add(1, 2)" 6) "data" "\x80\x04\x95\x17\x00\x00..." 7) "timeout" "180"
ACCEPT TASK # celery $ celery -A myapp worker -Q
queue1,queue2 # rq $ rq worker queue1 queue2
RABBITMQ QUEUES Name Ready Unacked Total celery 0 1 1
REDIS QUEUES >>> KEYS * 1) "celery" 2) "unacked" 3)
"unacked_index" 4) "unacked_mutex" >>> HGETALL unacked 1) "13ce9ff2-c596-4471" "[{...}, \"celery\", \"celery\"]"
RQ >>> HGETALL rq:job:be58515f-9f330470995d 1) "status" "started" 2) "enqueued_at" "2016-06-14T19:45:39Z"
3) "created_at" "2016-06-14T19:45:39Z" 4) "origin" "default" 5) "description" "lib.add(1, 2)" 6) "data" "\x80\x04\x95\x17\x00..." 7) "timeout" "180" 8) "started_at" "2016-06-14T21:21:14Z" >>> ZRANGE rq:wip:default 0 -1 WITHSCORES 1) "be58515f-9f330470995d" "1465939514"
FRAGILE ACKNOWLEDGMENT
RPOPLPUSH LUA SCRIPTS
None
None
None
TASK EXECUTION # celery def trace_task(uuid, args, kwargs, request=None): R
= retval = fun(*args, **kwargs) # rq class Job: def perform(self): self._result = self.func( *self.args, **self.kwargs)
CONCURRENCY ✓ gevent ✗ twisted ✗ tornado ✗ asyncio
None
None
None
None
None
CALLBACK app.add.apply_async((1, 2), link=app.mul.s(3)) PAYLOAD "callbacks": [ { "task": "app.mul",
"args": [3], "kwargs": {}, "options": {}, "chord_size": null, "immutable": false, "subtask_type": null } ]
CHAINS c = chain(app.add.s(1, 2), app.add.s(3), app.mul.s(6)) c()
RQ DEPENDENTS job = q.enqueue(lib.add, 1, 2) delayed = q.enqueue(lib.add,
1, 2, depends_on=job) REDIS >>> SMEMBERS rq:job:1183256d-2cf952c76:dependents 1) "00b70a47-bba9051bd7da" >>> HGETALL rq:job:00b70a47-bba9051bd7da 1) "status" "deferred" 2) "dependency_id" "1183256d-2cf952c76"
GROUPS g = group(app.add.s(1, 2), app.mul.s(3, 4)) g()
CHORD ch = chord([app.add.s(1, 2), app.add.s(3, 4)]) ch(app.mul.s(5)) PAYLOAD "chord":
{ "task": "app.mul", "chord_size": 3, "args": [7], "kwargs": null, "options": { "task_id": "2f0f2a7f-e249828b3fc9", "reply_to": "a53cb689-85a86c447428" } }
None
None
Celery: Redis as a broker deprecated in 4.0 Ask Solem
(@asksol) June 28, 2016 So I’m late for work, and seems like Redis as a broker is no longer deprecated :) Ask Solem (@asksol) June 28, 2016
None
THANKS
[email protected]