Python Celery PyPOA 2014 

Quem sou eu 

Quem sou eu • Leonardo Korndörfer • C, Ruby e Python • http://leok.me • http://github.com/leonardok 3 

Celery • Tarefas (tasks) assíncronas ‣ Processamento de imagens ‣ Mandar emails ‣ Sincronização ‣ Forks in GitHub • Notificação do que seus amigos fizeram no fim de semana 4 

Como Funciona 

Como Funciona 6 Web Requests Workers Brokers Grab Tasks to Execute Create Tasks Job Queues 

Como Funciona 7 Web Requests Brokers Grab Tasks to Execute Create Tasks Job Queues Workers 

Broker • Transporte das mensagens • RabbitMQ • Redis • Muitos outros suportados em caráter experimental ‣ http://docs.celeryproject.org/en/latest/getting-started/brokers/ ‣ Mongo DB, Couch DB, Zookeeper, SQLAlchemy… 8 

AMQP • Advanced Message Queuing Protocol ‣ open standard ‣ asynchronous messages between applications ‣ na internet 9 

Setup Inicial • Virtualenv ‣ sudo apt-get install python-virtualenv • RabbitMQ / Redis ‣ sudo apt-get install rabbitmq-server • Celery ‣ pip install celery 10 

Tasks.py from celery import Celery
 
 app = Celery('tasks', broker='amqp://guest@localhost//')
 
 @app.task
 def add(x, y):
 return x + y 11 

Executando Tasks em Background $ python ! >>> from tasks import add
 >>> add.delay(4, 4) ! ! $ celery -A tasks worker 12 

Little Demo 13 

Retries • Configurações • Primitivas adicionadas à chamada ! ! ! ! In [1]: from tasks import retry_and_add ! In [2]: retry_and_add.delay(4, 4) Out[2]: ! ! $ celery -A tasks worker --loglevel=info ! [2014-06-06 22:37:20,786: WARNING/Worker-1] Is executing... Retry: 0 [2014-06-06 22:37:31,803: WARNING/Worker-1] Is executing... Retry: 1 [2014-06-06 22:37:43,809: WARNING/Worker-1] Is executing... Retry: 2 [2014-06-06 22:37:54,816: WARNING/Worker-1] Is executing... Retry: 3 14 

Countdown • Faz com que a task espere por um período antes de tentar novamente ! ! @app.task(bind=True, max_retries=3, default_retry_delay=1) ! raise self.retry(exc=exc, countdown=10) 15 

16 Controlando Melhor 

Signatures • Assinatura de invocação da tarefa • Pode ser passada para outra função • Empacota ‣ argumentos, keyword args ("keyword = value”), opções da task ! ! In [1]: from celery import signature In [2]: s = signature('tasks.add_and_retry', args=(2, 2), countdown=10) In [3]: s Out[3]: tasks.add_and_retry(2, 2) ! ! 17 

Signatures ! In [1]: from tasks import add_and_retry ! In [2]: add_and_retry.subtask((2, 2), countdown=10) Out[2]: tasks.add_and_retry(2, 2) ! In [3]: add_and_retry.s(2, 2) Out[3]: tasks.add_and_retry(2, 2) ! In [4]: add_and_retry.s(2, 2, debug=True) Out[4]: tasks.add_and_retry(2, 2, debug=True) ! ! 18 

Signatures ! In [4]: s.args Out[4]: (2, 2) ! In [5]: s.kwargs Out[5]: {} ! In [6]: s.options Out[6]: {'countdown': 10} ! ! 19 

Partials ! In [1]: from tasks import add ! In [2]: partial = add.s(2) ! In [3]: partial.delay(4) Out[3]: 20 

Callbacks ! In [1]: from tasks import add ! In [2]: sig = add.s(10) ! In [3]: add.apply_async((2, 2), link=add.s(8)) Out[3]: ! ! Received task: tasks.add[1] Received task: tasks.add[2] Task tasks.add[1] succeeded in 0.1s: 4 Task tasks.add[2] succeeded in 0.1s: 12 21 

Groups • “group” é uma primitiva que pega uma lista de tarefas que devem ser executadas em paralelo • Não tem overhead do envio da mensagem ! ! ! ! In [1]: from celery import group ! In [2]: from tasks import add ! In [3]: res = group(add.s(i, i) for i in xrange(4))() ! In [5]: res Out[5]: 22 

Chain • Esta primitiva nos permite ligar assinaturas, permitindo que uma seja chamada após a outra, essencialmente nos permitindo criar uma corrente de callbacks ! ! In [3]: from celery import chain ! In [4]: res = chain(add.s(2, 2), add.s(4), add.s(8))() ! In [5]: res Out[5]: ! Received task: tasks.add[——————] Received task: tasks.add[——————] Task tasks.add[——————] succeeded in 0.014s: 4 Received task: tasks.add[——————] Task tasks.add[——————] succeeded in 0.003s: 8 Task tasks.add[——————] succeeded in 0.006s: 16 23 

Chord • É um grupo com uma callback • Tem um header e um corpo ‣ header: Grupo de tasks ‣ body: Task callback 24 

Maps • Funciona como “map” do python ! ! ! In [1]: foo = [1, 2, 3] In [2]: bar = [4, 5, 6] In [3]: map(add, foo, bar) Out[3]: [5, 7, 9] ! ! ! In [1]: from tasks import add In [2]: add.map([(1, 2)]) Out[2]: [tasks.add(x) for x in [(1, 2)]] 25 

Starmaps • Funciona como map mas aplica os argumentos como *args ! ! ! In [6]: add.starmap(zip(range(10), range(10))) Out[6]: [tasks.add(*x) for x in [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 9)]] 26 

Maps e Starmaps ! In [1]: add.map([range(10), range(10)]) Out[1]: [tasks.add(x) for x in [ [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]] ! In [2]: add.starmap(zip(range(10), range(10))) Out[2]: [tasks.add(*x) for x in [ (0, 0), (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 9)]] 27 

Chunks • Divide uma lista de argumentos em partes ! ! ! In [1]: items = zip(xrange(10), xrange(10)) In [2]: add.chunks(items, 5) Out[2]: celery.chunks(task=tasks.add(), it=[(0, 0), (1, 1), (2, 2), (3, 3),(4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 9)], n=5) ! In [3]: add.chunks(items, 5)() Out[3]: ! • Vai resultar em 2 tasks com 5 ! Task celery.starmap[2] succeeded in 0.1s: [0, 2, 4, 6, 8] Task celery.starmap[3] succeeded in 0.1s: [10, 12, 14, 16, 18] 28 

29 Ferramentas 

Linha de Comando ! $ celery help $ celery —help $ celery status $ celery result -t tasks.add $ celery purge 30 

Curses Monitor 31 

Celery Flower https://github.com/mher/flower 

Flower 33 

Debugging ! from celery.contrib import rdb ! rdb = rdb.Rdb(port=6900, port_search_limit=100, port_skew=0) ! rdb.set_trace() # set breakpoint ! ! ! $ telnet localhost 6900 Connected to localhost. Escape character is '^]'. > /opt/devel/demoapp/tasks.py(128)add() -> return result (Pdb) 34 

Perguntas 

Muito Obrigado PyPOA 2014 

Referências! • http://in.pycon.org/2010/talks/50-python-celery • https://speakerdeck.com/pyconslides/messaging-at-scale-at-instagram-by-rick-branson • http://www.minvolai.com/blog/2013/10/rabbitmq-vs-redis-message-broker/ • http://blog.x-aeon.com/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/ • https://www.youtube.com/watch?v=gpKMwPoldak ! ! Imagens! • http://www.healthyfoodhouse.com/celery-natural-healing-food/ • http://en.wikipedia.org/wiki/File:Celery_cross_section.jpg • http://vmfarms.com/static/img/logos/celery-logo.png • http://www.softicons.com/web-icons/vector-stylish-weather-icons-by-bartosz-kaszubowski/cloud-dark-icon 37