Slide 1

Slide 1 text

Python Celery PyPOA 2014

Slide 2

Slide 2 text

Quem sou eu

Slide 3

Slide 3 text

Quem sou eu • Leonardo Korndörfer • C, Ruby e Python • http://leok.me • http://github.com/leonardok 3

Slide 4

Slide 4 text

Celery • Tarefas (tasks) assíncronas ‣ Processamento de imagens ‣ Mandar emails ‣ Sincronização ‣ Forks in GitHub • Notificação do que seus amigos fizeram no fim de semana 4

Slide 5

Slide 5 text

Como Funciona

Slide 6

Slide 6 text

Como Funciona 6 Web Requests Workers Brokers Grab Tasks to Execute Create Tasks Job Queues

Slide 7

Slide 7 text

Como Funciona 7 Web Requests Brokers Grab Tasks to Execute Create Tasks Job Queues Workers

Slide 8

Slide 8 text

Broker • Transporte das mensagens • RabbitMQ • Redis • Muitos outros suportados em caráter experimental ‣ http://docs.celeryproject.org/en/latest/getting-started/brokers/ ‣ Mongo DB, Couch DB, Zookeeper, SQLAlchemy… 8

Slide 9

Slide 9 text

AMQP • Advanced Message Queuing Protocol ‣ open standard ‣ asynchronous messages between applications ‣ na internet 9

Slide 10

Slide 10 text

Setup Inicial • Virtualenv ‣ sudo apt-get install python-virtualenv • RabbitMQ / Redis ‣ sudo apt-get install rabbitmq-server • Celery ‣ pip install celery 10

Slide 11

Slide 11 text

Tasks.py from celery import Celery
 
 app = Celery('tasks', broker='amqp://guest@localhost//')
 
 @app.task
 def add(x, y):
 return x + y 11

Slide 12

Slide 12 text

Executando Tasks em Background $ python ! >>> from tasks import add
 >>> add.delay(4, 4) ! ! $ celery -A tasks worker 12

Slide 13

Slide 13 text

Little Demo 13

Slide 14

Slide 14 text

Retries • Configurações • Primitivas adicionadas à chamada ! ! ! ! In [1]: from tasks import retry_and_add ! In [2]: retry_and_add.delay(4, 4) Out[2]: ! ! $ celery -A tasks worker --loglevel=info ! [2014-06-06 22:37:20,786: WARNING/Worker-1] Is executing... Retry: 0 [2014-06-06 22:37:31,803: WARNING/Worker-1] Is executing... Retry: 1 [2014-06-06 22:37:43,809: WARNING/Worker-1] Is executing... Retry: 2 [2014-06-06 22:37:54,816: WARNING/Worker-1] Is executing... Retry: 3 14

Slide 15

Slide 15 text

Countdown • Faz com que a task espere por um período antes de tentar novamente ! ! @app.task(bind=True, max_retries=3, default_retry_delay=1) ! raise self.retry(exc=exc, countdown=10) 15

Slide 16

Slide 16 text

16 Controlando Melhor

Slide 17

Slide 17 text

Signatures • Assinatura de invocação da tarefa • Pode ser passada para outra função • Empacota ‣ argumentos, keyword args ("keyword = value”), opções da task ! ! In [1]: from celery import signature In [2]: s = signature('tasks.add_and_retry', args=(2, 2), countdown=10) In [3]: s Out[3]: tasks.add_and_retry(2, 2) ! ! 17

Slide 18

Slide 18 text

Signatures ! In [1]: from tasks import add_and_retry ! In [2]: add_and_retry.subtask((2, 2), countdown=10) Out[2]: tasks.add_and_retry(2, 2) ! In [3]: add_and_retry.s(2, 2) Out[3]: tasks.add_and_retry(2, 2) ! In [4]: add_and_retry.s(2, 2, debug=True) Out[4]: tasks.add_and_retry(2, 2, debug=True) ! ! 18

Slide 19

Slide 19 text

Signatures ! In [4]: s.args Out[4]: (2, 2) ! In [5]: s.kwargs Out[5]: {} ! In [6]: s.options Out[6]: {'countdown': 10} ! ! 19

Slide 20

Slide 20 text

Partials ! In [1]: from tasks import add ! In [2]: partial = add.s(2) ! In [3]: partial.delay(4) Out[3]: 20

Slide 21

Slide 21 text

Callbacks ! In [1]: from tasks import add ! In [2]: sig = add.s(10) ! In [3]: add.apply_async((2, 2), link=add.s(8)) Out[3]: ! ! Received task: tasks.add[1] Received task: tasks.add[2] Task tasks.add[1] succeeded in 0.1s: 4 Task tasks.add[2] succeeded in 0.1s: 12 21

Slide 22

Slide 22 text

Groups • “group” é uma primitiva que pega uma lista de tarefas que devem ser executadas em paralelo • Não tem overhead do envio da mensagem ! ! ! ! In [1]: from celery import group ! In [2]: from tasks import add ! In [3]: res = group(add.s(i, i) for i in xrange(4))() ! In [5]: res Out[5]: 22

Slide 23

Slide 23 text

Chain • Esta primitiva nos permite ligar assinaturas, permitindo que uma seja chamada após a outra, essencialmente nos permitindo criar uma corrente de callbacks ! ! In [3]: from celery import chain ! In [4]: res = chain(add.s(2, 2), add.s(4), add.s(8))() ! In [5]: res Out[5]: ! Received task: tasks.add[——————] Received task: tasks.add[——————] Task tasks.add[——————] succeeded in 0.014s: 4 Received task: tasks.add[——————] Task tasks.add[——————] succeeded in 0.003s: 8 Task tasks.add[——————] succeeded in 0.006s: 16 23

Slide 24

Slide 24 text

Chord • É um grupo com uma callback • Tem um header e um corpo ‣ header: Grupo de tasks ‣ body: Task callback 24

Slide 25

Slide 25 text

Maps • Funciona como “map” do python ! ! ! In [1]: foo = [1, 2, 3] In [2]: bar = [4, 5, 6] In [3]: map(add, foo, bar) Out[3]: [5, 7, 9] ! ! ! In [1]: from tasks import add In [2]: add.map([(1, 2)]) Out[2]: [tasks.add(x) for x in [(1, 2)]] 25

Slide 26

Slide 26 text

Starmaps • Funciona como map mas aplica os argumentos como *args ! ! ! In [6]: add.starmap(zip(range(10), range(10))) Out[6]: [tasks.add(*x) for x in [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 9)]] 26

Slide 27

Slide 27 text

Maps e Starmaps ! In [1]: add.map([range(10), range(10)]) Out[1]: [tasks.add(x) for x in [ [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]] ! In [2]: add.starmap(zip(range(10), range(10))) Out[2]: [tasks.add(*x) for x in [ (0, 0), (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 9)]] 27

Slide 28

Slide 28 text

Chunks • Divide uma lista de argumentos em partes ! ! ! In [1]: items = zip(xrange(10), xrange(10)) In [2]: add.chunks(items, 5) Out[2]: celery.chunks(task=tasks.add(), it=[(0, 0), (1, 1), (2, 2), (3, 3),(4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 9)], n=5) ! In [3]: add.chunks(items, 5)() Out[3]: ! • Vai resultar em 2 tasks com 5 ! Task celery.starmap[2] succeeded in 0.1s: [0, 2, 4, 6, 8] Task celery.starmap[3] succeeded in 0.1s: [10, 12, 14, 16, 18] 28

Slide 29

Slide 29 text

29 Ferramentas

Slide 30

Slide 30 text

Linha de Comando ! $ celery help $ celery —help $ celery status $ celery result -t tasks.add $ celery purge 30

Slide 31

Slide 31 text

Curses Monitor 31

Slide 32

Slide 32 text

Celery Flower https://github.com/mher/flower

Slide 33

Slide 33 text

Flower 33

Slide 34

Slide 34 text

Debugging ! from celery.contrib import rdb ! rdb = rdb.Rdb(port=6900, port_search_limit=100, port_skew=0) ! rdb.set_trace() # set breakpoint ! ! ! $ telnet localhost 6900 Connected to localhost. Escape character is '^]'. > /opt/devel/demoapp/tasks.py(128)add() -> return result (Pdb) 34

Slide 35

Slide 35 text

Perguntas

Slide 36

Slide 36 text

Muito Obrigado PyPOA 2014

Slide 37

Slide 37 text

Referências! • http://in.pycon.org/2010/talks/50-python-celery • https://speakerdeck.com/pyconslides/messaging-at-scale-at-instagram-by-rick-branson • http://www.minvolai.com/blog/2013/10/rabbitmq-vs-redis-message-broker/ • http://blog.x-aeon.com/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/ • https://www.youtube.com/watch?v=gpKMwPoldak ! ! Imagens! • http://www.healthyfoodhouse.com/celery-natural-healing-food/ • http://en.wikipedia.org/wiki/File:Celery_cross_section.jpg • http://vmfarms.com/static/img/logos/celery-logo.png • http://www.softicons.com/web-icons/vector-stylish-weather-icons-by-bartosz-kaszubowski/cloud-dark-icon 37