Introducción a Celery

Plática para Chilango Django 20 de Julio de 2016, Ciudad de México

Transcript

1. Celery bit.ly/chilango-celery @henocdz
2. ¿Celery? Es un sistema distribuido para procesar mensajes en una cola de tareas, con soporte para soporte real-0me o programado (scheduling) @henocdz
3. ¿En qué casos podría ser ú2l? • Tareas secundarias: Enviar emails • Tareas que consuman una gran can1dad de 1empo: Procesar mul1media, crear XLSX, PDFs, etc • Tareas repe11vas: Scrapping, traer data de un API, feed, etc. @henocdz
4. Compa&bilidad 1. Python 2.7, 3.4, 3.5 2. Django, Pyramid, Tornado, Flask, otros. @henocdz
5. Elementos Clave • Producer • Broker (AMPQ) • Queues • Task • Consumer (Worker) • Result Backend @henocdz
6. ¿Cómo funciona Celery? @henocdz
7. Tasks El mero mole ! @henocdz
8. from celery.task import task from users.models import CustomUser from annoying.functions import get_object_or_None @task(name='some_django_app.notify_user', ignore_result=True) def notify_user(user_pk): user = get_object_or_None(CustomUser, pk=user_pk) if user is not None: user.notify_something() @henocdz
9. Reintentando @task(bind=True) def send_twitter_status(self, oauth, tweet): try: twitter = Twitter(oauth) twitter.update_status(tweet) except (Twitter.FailWhaleError, Twitter.LoginError) as exc: raise self.retry(exc=exc) @henocdz
10. Ejecutar tareas @henocdz
11. task.delay Fácil y prác*co, e.g: some_task.delay(arg1, arg2, kwarg1='some', kwarg2='value') @henocdz
12. task.apply_async Nos da mayor control sobre el task, e.g: # Ejecutar después de N segundos some_task.apply_async(args=argv, kwargs=kwargs, countdown=10) # Cancelar la ejecución si no sucede después de cierta fecha tomorrow = today + timedelta(hours=1) some_task.apply_async(args=argv, kwargs=kwargs, expires=tomorrow) @henocdz
13. @henocdz
14. Setup con Django @henocdz
15. settings.py from __future__ import absolute_import from datetime import timedelta CELERY_RESULT_BACKEND = 'redis://{user}:{password}@{host}:{port}/{db_num}' BROKER_URL = 'ampq://{user}:{password}@{host}:{port}/' @henocdz
16. project/project/celery_app.py from __future__ import absolute_import from celery import Celery from django.conf import settings # noqa celery_app = Celery('project') celery_app.config_from_object('django.conf:settings') celery_app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) @henocdz
17. project/project/__init__.py from .celery_app import celery_app @henocdz
18. Celery Beat Administrador de tareas periódicas @henocdz
19. Programar tareas periódicas # settings.py from celery.schedules import crontab CELERYBEAT_SCHEDULE = { 'generate_monthly_report': { 'task': 'reports.tasks.generate_monthly_report', 'schedule': crontab(minute='0,20,40', hour='12', day_of_week='mon,thu'), }, 'delete_temp_files': { 'task': 'reports.tasks.delete_temp_files', 'schedule': timedelta(seconds=180), }, } $ celery beat -A project @henocdz
20. Flower Web Monitor & Admin para Celery @henocdz
21. @henocdz
22. @henocdz
23. @henocdz
24. Ejecutar flower $ celery flower --address=0.0.0.0 --port=9000 \ --basic_auth=admin@site.com:sitepwd69 \ --broker=ampq://{user}:{password}@{host}:{port}/ \ --broker_api=ampq://{user}:{password}@{host}:{port}/ @henocdz
25. bit.ly/celery-bp @henocdz
26. Tareas pequeñas from myproject.tasks import app # app is your celery application from myproject.exceptions import InvalidUserInput from utils.mail import api_send_mail @task(max_retries=3) def send_mail(self, recipients, sender_email, subject, body): try: data = api_send_mail(recipients, sender_email, subject, body) except InvalidUserInput: raise except Exception as exc: sentrycli.captureException() send_mail.retry(countdown=10, exc=exc) return data @henocdz
27. Falla rápido y sin bloquear el queue @task(max_retries=3, soft_time_limit=5) def send_mail(self, recipients, sender_email, subject, body): ... @henocdz
28. Aumenta gradualmente el -empo para reintentar def backoff(attempts): return 2 ** attempts @task(max_retries=3) def send_mail(self, recipients, sender_email, subject, body): try: data = api_send_mail(recipients, sender_email, subject, body) except InvalidUserInput: raise except Exception as exc: send_mail.retry(countdown=backoff(self.request.retries), exc=exc) ... @henocdz
29. Comparte funcionalidad entre tareas usando clases from celery.tasks import Task class BaseTask(Task): """Abstract base class for all tasks in my app.""" abstract = True def on_retry(self, exc, task_id, args, kwargs, einfo): """Log the exceptions to sentry at retry.""" sentrycli.captureException(exc) super(BaseTask, self).on_retry(exc, task_id, args, kwargs, einfo) def on_failure(self, exc, task_id, args, kwargs, einfo): """Log the exceptions to sentry.""" sentrycli.captureException(exc) super(BaseTask, self).on_failure(exc, task_id, args, kwargs, einfo) @henocdz
30. base=BaseTask @task(max_retries=3, soft_time_limit=5, base=BaseTask) def send_mail(self, recipients, sender_email, subject, body): try: data = api_send_mail(recipients, sender_email, subject, body) except InvalidUserInput: raise except Exception as exc: self.retry(countdown=backoff(self.request.retries), exc=exc) return data @henocdz
31. Mul$ples Workers1 y Queues # settings.py CELERY_ROUTES = { 'reports.tasks.generate_monthly_report': {'queue': 'reports'}, 'reports.tasks.delete_temp_files': {'queue': 'cleanup'}, } $ celery worker -n celery@ServerName -A project -Q default,cleanup $ celery worker -n celery2@ServerName -A project -Q reports 1 Número recomendado: n + 1, n = número de cores @henocdz
32. NUNCA enviar objetos del ORM from clients.models import ClientUser from clients.tasks import send_report # Esto está mal. def send_report(client) send_email(...client.email...) send_report.delay(client) @henocdz
33. # Solución def send_report(client_pk) client = ClientUser.objects.get(pk=client_pk) send_report.delay(client.pk) @henocdz
34. y ya... @henocdz
35. ¡Gracias! ! @henocdz