Python and Celery

This is a portuguese presentation about celery features with some examples.

It was presented during the first PyPOA (http://pypoa.org)

Transcript

1. Python Celery PyPOA 2014
2. Quem sou eu
3. Quem sou eu • Leonardo Korndörfer • C, Ruby e Python • http://leok.me • http://github.com/leonardok 3
4. Celery • Tarefas (tasks) assíncronas ‣ Processamento de imagens ‣ Mandar emails ‣ Sincronização ‣ Forks in GitHub • Notificação do que seus amigos fizeram no fim de semana 4
5. Como Funciona
6. Como Funciona 6 Web Requests Workers Brokers Grab Tasks to Execute Create Tasks Job Queues
7. Como Funciona 7 Web Requests Brokers Grab Tasks to Execute Create Tasks Job Queues Workers
8. Broker • Transporte das mensagens • RabbitMQ • Redis • Muitos outros suportados em caráter experimental ‣ http://docs.celeryproject.org/en/latest/getting-started/brokers/ ‣ Mongo DB, Couch DB, Zookeeper, SQLAlchemy… 8
9. AMQP • Advanced Message Queuing Protocol ‣ open standard ‣ asynchronous messages between applications ‣ na internet 9
10. Setup Inicial • Virtualenv ‣ sudo apt-get install python-virtualenv • RabbitMQ / Redis ‣ sudo apt-get install rabbitmq-server • Celery ‣ pip install celery 10
11. Tasks.py from celery import Celery
 
 app = Celery('tasks', broker='amqp://guest@localhost//')
 
 @app.task
 def add(x, y):
 return x + y 11
12. Executando Tasks em Background $ python ! >>> from tasks import add
 >>> add.delay(4, 4) ! ! $ celery -A tasks worker 12
13. Little Demo 13
14. Retries • Configurações • Primitivas adicionadas à chamada ! ! ! ! In [1]: from tasks import retry_and_add ! In [2]: retry_and_add.delay(4, 4) Out[2]: <AsyncResult: 74f37f17-7f19-4139-b680-ce920316665e> ! ! $ celery -A tasks worker --loglevel=info ! [2014-06-06 22:37:20,786: WARNING/Worker-1] Is executing... Retry: 0 [2014-06-06 22:37:31,803: WARNING/Worker-1] Is executing... Retry: 1 [2014-06-06 22:37:43,809: WARNING/Worker-1] Is executing... Retry: 2 [2014-06-06 22:37:54,816: WARNING/Worker-1] Is executing... Retry: 3 14
15. Countdown • Faz com que a task espere por um período antes de tentar novamente ! ! @app.task(bind=True, max_retries=3, default_retry_delay=1) ! raise self.retry(exc=exc, countdown=10) 15
16. 16 Controlando Melhor
17. Signatures • Assinatura de invocação da tarefa • Pode ser passada para outra função • Empacota ‣ argumentos, keyword args ("keyword = value”), opções da task ! ! In [1]: from celery import signature In [2]: s = signature('tasks.add_and_retry', args=(2, 2), countdown=10) In [3]: s Out[3]: tasks.add_and_retry(2, 2) ! ! 17
18. Signatures ! In [1]: from tasks import add_and_retry ! In [2]: add_and_retry.subtask((2, 2), countdown=10) Out[2]: tasks.add_and_retry(2, 2) ! In [3]: add_and_retry.s(2, 2) Out[3]: tasks.add_and_retry(2, 2) ! In [4]: add_and_retry.s(2, 2, debug=True) Out[4]: tasks.add_and_retry(2, 2, debug=True) ! ! 18
19. Signatures ! In [4]: s.args Out[4]: (2, 2) ! In [5]: s.kwargs Out[5]: {} ! In [6]: s.options Out[6]: {'countdown': 10} ! ! 19
20. Partials ! In [1]: from tasks import add ! In [2]: partial = add.s(2) ! In [3]: partial.delay(4) Out[3]: <AsyncResult: a173781a-3791-4984-952b-f1db650a3fa4> 20
21. Callbacks ! In [1]: from tasks import add ! In [2]: sig = add.s(10) ! In [3]: add.apply_async((2, 2), link=add.s(8)) Out[3]: <AsyncResult: 1> ! ! Received task: tasks.add[1] Received task: tasks.add[2] Task tasks.add[1] succeeded in 0.1s: 4 Task tasks.add[2] succeeded in 0.1s: 12 21
22. Groups • “group” é uma primitiva que pega uma lista de tarefas que devem ser executadas em paralelo • Não tem overhead do envio da mensagem ! ! ! ! In [1]: from celery import group ! In [2]: from tasks import add ! In [3]: res = group(add.s(i, i) for i in xrange(4))() ! In [5]: res Out[5]: <GroupResult: 00bfbf2b-0de9-4d18-8243-c4abbf4c0d2d [ 2d1022a5-8edf-44b7-a90e-906af3551138, 5fb855b2-2e86-4dca-b680-05a4a0a24355, 82ec7596-e7dc-4030-ba9e-c2f070e3890d, ac844772-7c25-4e75-a03a-29fa08837ae7]> 22
23. Chain • Esta primitiva nos permite ligar assinaturas, permitindo que uma seja chamada após a outra, essencialmente nos permitindo criar uma corrente de callbacks ! ! In [3]: from celery import chain ! In [4]: res = chain(add.s(2, 2), add.s(4), add.s(8))() ! In [5]: res Out[5]: <AsyncResult: daf50bbd-175f-4ab2-8031-efaeb184811e> ! Received task: tasks.add[——————] Received task: tasks.add[——————] Task tasks.add[——————] succeeded in 0.014s: 4 Received task: tasks.add[——————] Task tasks.add[——————] succeeded in 0.003s: 8 Task tasks.add[——————] succeeded in 0.006s: 16 23
24. Chord • É um grupo com uma callback • Tem um header e um corpo ‣ header: Grupo de tasks ‣ body: Task callback 24
25. Maps • Funciona como “map” do python ! ! ! In [1]: foo = [1, 2, 3] In [2]: bar = [4, 5, 6] In [3]: map(add, foo, bar) Out[3]: [5, 7, 9] ! ! ! In [1]: from tasks import add In [2]: add.map([(1, 2)]) Out[2]: [tasks.add(x) for x in [(1, 2)]] 25
26. Starmaps • Funciona como map mas aplica os argumentos como *args ! ! ! In [6]: add.starmap(zip(range(10), range(10))) Out[6]: [tasks.add(*x) for x in [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 9)]] 26
27. Maps e Starmaps ! In [1]: add.map([range(10), range(10)]) Out[1]: [tasks.add(x) for x in [ [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]] ! In [2]: add.starmap(zip(range(10), range(10))) Out[2]: [tasks.add(*x) for x in [ (0, 0), (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 9)]] 27
28. Chunks • Divide uma lista de argumentos em partes ! ! ! In [1]: items = zip(xrange(10), xrange(10)) In [2]: add.chunks(items, 5) Out[2]: celery.chunks(task=tasks.add(), it=[(0, 0), (1, 1), (2, 2), (3, 3),(4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 9)], n=5) ! In [3]: add.chunks(items, 5)() Out[3]: <GroupResult: 1 [2, 3]> ! • Vai resultar em 2 tasks com 5 ! Task celery.starmap[2] succeeded in 0.1s: [0, 2, 4, 6, 8] Task celery.starmap[3] succeeded in 0.1s: [10, 12, 14, 16, 18] 28
29. 29 Ferramentas
30. Linha de Comando ! $ celery help $ celery <command> —help $ celery status $ celery result -t tasks.add <id> $ celery purge 30
31. Curses Monitor 31
32. Celery Flower https://github.com/mher/flower
33. Flower 33
34. Debugging ! from celery.contrib import rdb ! rdb = rdb.Rdb(port=6900, port_search_limit=100, port_skew=0) ! rdb.set_trace() # set breakpoint ! ! ! $ telnet localhost 6900 Connected to localhost. Escape character is '^]'. > /opt/devel/demoapp/tasks.py(128)add() -> return result (Pdb) 34
35. Perguntas
36. Muito Obrigado PyPOA 2014
37. Referências! • http://in.pycon.org/2010/talks/50-python-celery • https://speakerdeck.com/pyconslides/messaging-at-scale-at-instagram-by-rick-branson • http://www.minvolai.com/blog/2013/10/rabbitmq-vs-redis-message-broker/ • http://blog.x-aeon.com/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/ • https://www.youtube.com/watch?v=gpKMwPoldak ! ! Imagens! • http://www.healthyfoodhouse.com/celery-natural-healing-food/ • http://en.wikipedia.org/wiki/File:Celery_cross_section.jpg • http://vmfarms.com/static/img/logos/celery-logo.png • http://www.softicons.com/web-icons/vector-stylish-weather-icons-by-bartosz-kaszubowski/cloud-dark-icon 37