Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Очереди задач без купюр, Артем Малышев, Positiv...
Search
IT-People
July 25, 2016
Programming
1
110
Очереди задач без купюр, Артем Малышев, Positive Technologies
Выступление на конференции PyCon Russia 2016
IT-People
July 25, 2016
Tweet
Share
More Decks by IT-People
See All by IT-People
Александр Кошкин, Positive Technologies: Знай и люби свой CPython во имя луны и великой справедливости
itpeople123
1
200
How Python was Shaped by leaky Internals, Armin Ronacher, Flask framework
itpeople123
1
230
Building social network with Neo4j and Python, Андрей Солдатенко, Toptal
itpeople123
1
230
Осторожно, DSL! Иван Цыганов, Positive Technologies
itpeople123
0
120
Докеризация веб-приложения на Python Антон Егоров, Ostrovok.ru
itpeople123
0
190
Python Robot Operating System вместо Middleware framework Дмитрий Киселев, Endpoint.com
itpeople123
0
100
Falcon — очередная WSGI библиотека? Дмитрий Кукушкин, Wargaming
itpeople123
0
260
Flame graph: новый взгляд на привычное профилирование, Кирилл Борисов, Яндекс
itpeople123
1
260
Как добавить scripting в веб приложение, Константин Лопухин, Scrapinghub
itpeople123
0
150
Other Decks in Programming
See All in Programming
なぜあの開発者はDevRelに伴走し続けるのか / Why Does That Developer Keep Running Alongside DevRel?
nrslib
3
410
Flutterで分数(Fraction)を表示する方法
koukimiura
0
130
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
220
Things You Thought You Didn’t Need To Care About That Have a Big Impact On Your Job
hollycummins
0
230
Android16 Migration Stories ~Building a Pattern for Android OS upgrades~
reoandroider
0
120
あなたとKaigi on Rails / Kaigi on Rails + You
shimoju
0
170
組込みだけじゃない!TinyGo で始める無料クラウド開発入門
otakakot
1
310
Railsだからできる 例外業務に禍根を残さない 設定設計パターン
ei_ei_eiichi
0
940
TFLintカスタムプラグインで始める Terraformコード品質管理
bells17
2
210
CSC509 Lecture 05
javiergs
PRO
0
300
20251016_Rails News ~Rails 8.1の足音を聴く~
morimorihoge
2
490
monorepo の Go テストをはやくした〜い!~最小の依存解決への道のり~ / faster-testing-of-monorepos
convto
2
500
Featured
See All Featured
GraphQLとの向き合い方2022年版
quramy
49
14k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
61k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
21
1.2k
Build your cross-platform service in a week with App Engine
jlugia
232
18k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
31
9.7k
Mobile First: as difficult as doing things right
swwweet
225
10k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
508
140k
Into the Great Unknown - MozCon
thekraken
40
2.1k
Being A Developer After 40
akosma
91
590k
Building Flexible Design Systems
yeseniaperezcruz
329
39k
How GitHub (no longer) Works
holman
315
140k
Transcript
TASK QUEUES UNCUT МАЛЫШЕВ АРТЁМ
None
None
None
None
CELERY app = Celery(__name__) @app.task def add(x, y): return x
+ y add.delay(1, 2)
RQ def add(x, y): return x + y queue =
rq.Queue('default') queue.enqueue(add, 1, 2)
RABBITMQ QUEUES Name Ready Unacked Total celery 1 0 1
PAYLOAD { "task": "app.add", "args": [1, 2], "kwargs": {}, "id":
"27ebcaee-444f-4379-a438-04c3b5bb0fc3", "callbacks": null, "errbacks": null, "eta": null, "expires": null, "timelimit": [null, null], "taskset": null, "chord": null, "retries": 0 }
REDIS QUEUES >>> KEYS * 1) "celery" >>> LRANGE celery
0 -1 1) { "properties": { "body_encoding": "base64", "delivery_info": { "exchange": "celery", "routing_key": "celery" } }, "body": "eyJ0aW1lbGltaXQiOiBbbnVsb...", "content-type": "application/json" }
RQ >>> SMEMBERS rq:queues 1) "rq:queue:default" >>> LRANGE rq:queue:default 0
-1 1) "be58515f-9f330470995d" >>> HGETALL rq:job:be58515f-9f330470995d 1) "status" "queued" 2) "enqueued_at" "2016-06-14T19:45:39Z" 3) "created_at" "2016-06-14T19:45:39Z" 4) "origin" "default" 5) "description" "lib.add(1, 2)" 6) "data" "\x80\x04\x95\x17\x00\x00..." 7) "timeout" "180"
ACCEPT TASK # celery $ celery -A myapp worker -Q
queue1,queue2 # rq $ rq worker queue1 queue2
RABBITMQ QUEUES Name Ready Unacked Total celery 0 1 1
REDIS QUEUES >>> KEYS * 1) "celery" 2) "unacked" 3)
"unacked_index" 4) "unacked_mutex" >>> HGETALL unacked 1) "13ce9ff2-c596-4471" "[{...}, \"celery\", \"celery\"]"
RQ >>> HGETALL rq:job:be58515f-9f330470995d 1) "status" "started" 2) "enqueued_at" "2016-06-14T19:45:39Z"
3) "created_at" "2016-06-14T19:45:39Z" 4) "origin" "default" 5) "description" "lib.add(1, 2)" 6) "data" "\x80\x04\x95\x17\x00..." 7) "timeout" "180" 8) "started_at" "2016-06-14T21:21:14Z" >>> ZRANGE rq:wip:default 0 -1 WITHSCORES 1) "be58515f-9f330470995d" "1465939514"
FRAGILE ACKNOWLEDGMENT
RPOPLPUSH LUA SCRIPTS
None
None
None
TASK EXECUTION # celery def trace_task(uuid, args, kwargs, request=None): R
= retval = fun(*args, **kwargs) # rq class Job: def perform(self): self._result = self.func( *self.args, **self.kwargs)
CONCURRENCY ✓ gevent ✗ twisted ✗ tornado ✗ asyncio
None
None
None
None
None
CALLBACK app.add.apply_async((1, 2), link=app.mul.s(3)) PAYLOAD "callbacks": [ { "task": "app.mul",
"args": [3], "kwargs": {}, "options": {}, "chord_size": null, "immutable": false, "subtask_type": null } ]
CHAINS c = chain(app.add.s(1, 2), app.add.s(3), app.mul.s(6)) c()
RQ DEPENDENTS job = q.enqueue(lib.add, 1, 2) delayed = q.enqueue(lib.add,
1, 2, depends_on=job) REDIS >>> SMEMBERS rq:job:1183256d-2cf952c76:dependents 1) "00b70a47-bba9051bd7da" >>> HGETALL rq:job:00b70a47-bba9051bd7da 1) "status" "deferred" 2) "dependency_id" "1183256d-2cf952c76"
GROUPS g = group(app.add.s(1, 2), app.mul.s(3, 4)) g()
CHORD ch = chord([app.add.s(1, 2), app.add.s(3, 4)]) ch(app.mul.s(5)) PAYLOAD "chord":
{ "task": "app.mul", "chord_size": 3, "args": [7], "kwargs": null, "options": { "task_id": "2f0f2a7f-e249828b3fc9", "reply_to": "a53cb689-85a86c447428" } }
None
None
Celery: Redis as a broker deprecated in 4.0 Ask Solem
(@asksol) June 28, 2016 So I’m late for work, and seems like Redis as a broker is no longer deprecated :) Ask Solem (@asksol) June 28, 2016
None
THANKS
[email protected]