Share
Embed
Transcript
34 slides
-
Outline 1. About 2. What is Celery? 3. Celery Architecture
4. Broker, Task, Worker 5. Monitoring 6. Coding 7. Q & A 2 -
About A father, a husband and a software engineer Passionate
in distributed systems, real-time data processing, search engine Work @sentifi as a backend engineer Follow me @duydo 3 -
What is Celery? Distributed Task Queue Simple, fast, flexible, highly
available, scalable Mature, feature rich Open source, BSD License Large community 4 -
What is Task Queue? Task Queue is a system for
parallel execution of tasks 5 Client Worker Broker send tasks distribute tasks Worker distribute tasks -
Celery Architecture 6 Client 1 Task Queue 2 … Task
Queue N Task Queue 1 Broker Client 2 Worker 1 Worker 2 Task Result Storage distribute tasks distribute tasks send tasks send tasks store task results store task results get task result get task result -
Broker The middle man holds the tasks (messages) Celery supports:
• RabbitMQ, Redis • MongoDB, CouchDB • ZeroMQ, Amazon SQS, IronMQ 7 -
Task Task is a unit of work, building blocks in
Celery apps Exists until it has been acknowledged Result of the tasks can be stored or ignored States: PENDING, STARTED, SUCCESS, FAILURE, RETRY, REVOKED Periodic task (cron jobs) 8 -
Define Tasks # function style @app.task def add(x, y):
return x * y # class style class AddTask(app.Task): def run(self, x, y): return x + y 9 -
Calling Tasks apply_async(args[, kwargs[, …]]) delay(*args, **kwargs) calling(__call__) e.g:
• result = add.delay(1, 2) • result = add.apply_async((1, 2), countdown=10) 10 -
Calling Task Options eta a specific date time that is
the earliest time at which task will be executed countdown set eta by seconds into the future expires set task’s expire time serializer pickle (default), json, yaml and msgpack compression compress the messages using gzip or bzip2 queue route the tasks to different queues 11 -
Task Result result.ready() true if the task has been executed
result.successful() true if the task executed successfully result.result the return value of the task or exception result.get() blocks until the task is complete, return result or exception 12 -
Tasks Workflows Signatures: Partials, Immutability, Callbacks The Primitives: Chains, Groups,
Chords, Map & Starmap, Chunks 13 -
Signatures signature() wraps args, kwargs, options of a single task
invocation in a way such that it can be: • passed to functions • serialized and sent across the wire like subtasks 14 -
Create Signatures # ws.tasks.add(1, 2) s = signature('ws.tasks.add', args=(1,
2), countdown=10) s = add.subtask((1, 2), countdown=10) s = add.s(1, 2) s = add.s(1, 2, debug=True) # inspect fields s.args # (1, 2) s.kwargs # {'debug': True') s.options # {countdown=10} # execute as task s.delay() s.apply_async() s() 15 -
Partial Signatures 16 Specifying additional args, kwargs or options to
apply_async/delay to create partial • partial = add.s(1) • partial.delay(2) # 1 + 2 • partial.apply_async((2,)) # 1 + 2 -
Immutable Signatures 17 A signature can only be set with
options Using si() to create immutable signature • add.si(1, 2) -
Callbacks Signatures 18 Use the link arg of apply_sync to
add callbacks add.apply_async((1, 2), link=add.s(3)) -
Group 19 A signature takes a list of tasks should
be applied in parallel s = group(add.s(i, i) for i in xrange(5)) s().get() => [0, 2, 4, 6, 8] -
Chain 20 Chain of callbacks, think pipeline c = chain(add.s(1,
2), add.s(3), add.s(4)) c = chain(add.s(1, 2) | add.s(3) | add.s(4)) c().get() => ((1 + 2) + 3) + 4 -
Chord 21 Like a group but with a callback c
= chord((add.s(i, i) for i in xrange(5)), xsum.s()) c = chord(add.s(i, i) for i in xrange(5)) (xsum.s()) c().get() => 20 -
Map 22 Like built-in map function c = task.map([1, 2,
3]) c() => [task(1), task(2), task(3)] -
Starmap 23 Same map except the args are applied as
*args c = add.map([(1, 2), (3, 4)]) c() => [add(1, 2), add(3, 4)] -
Chunks 24 Chunking splits a long list of args to
parts items = zip(xrange(10), xrange(10)) c = add.chunks(items, 5) c() => [0, 2, 4, 6, 8], [10, 12, 14, 16, 18] -
Worker Auto reloading Auto scaling Time & Rate Limits Resource
Leak Protection Scheduling User Components 25 -
Autoloading Automatically reloading the worker source code as it changes
celery worker —autoreload 26 -
Autoscaling Dynamically resizing the worker pool depending on load or
custom metrics defined by user celery worker —autoscale=8,2 => min processes: 2, max processes:8 27 -
Time & Rate Limits number of tasks per second/minute/hour how
long a task can be allowed to run 28 -
Resource Leak Protection Limit number of tasks a pool worker
process can execute before it’s replaced by a new one celery worker —maxtaskperchild=10 29 -
Scheduling Specify the time to run a task in seconds,
date time periodic tasks (interval, crontab expressions) 30 -
User Components Celery uses a dependency graph enabling fire grained
control of the workers internally, called “bootsteps” Customize the worker components, e.g: ConsumerStep Add new components Bootsteps http://celery.readthedocs.org/en/latest/ userguide/extending.html 31 -
Monitoring Flower - Real-time Celery web monitor • Task progress
and history • Show task details (arguments, start time, runtime, and more) • Graphs and statistics • Shutdown, restart worker instances • Control worker pool size, autoscaling settings • … 32