Introducción a Celery

Introducción a Celery

Plática para Chilango Django 20 de Julio de 2016, Ciudad de México

A04788bd45feba612272eaef0f4a01ee?s=128

Henoc Díaz

July 20, 2016
Tweet

Transcript

  1. Celery bit.ly/chilango-celery @henocdz

  2. ¿Celery? Es un sistema distribuido para procesar mensajes en una

    cola de tareas, con soporte para soporte real-0me o programado (scheduling) @henocdz
  3. ¿En qué casos podría ser ú2l? • Tareas secundarias: Enviar

    emails • Tareas que consuman una gran can1dad de 1empo: Procesar mul1media, crear XLSX, PDFs, etc • Tareas repe11vas: Scrapping, traer data de un API, feed, etc. @henocdz
  4. Compa&bilidad 1. Python 2.7, 3.4, 3.5 2. Django, Pyramid, Tornado,

    Flask, otros. @henocdz
  5. Elementos Clave • Producer • Broker (AMPQ) • Queues •

    Task • Consumer (Worker) • Result Backend @henocdz
  6. ¿Cómo funciona Celery? @henocdz

  7. Tasks El mero mole ! @henocdz

  8. from celery.task import task from users.models import CustomUser from annoying.functions

    import get_object_or_None @task(name='some_django_app.notify_user', ignore_result=True) def notify_user(user_pk): user = get_object_or_None(CustomUser, pk=user_pk) if user is not None: user.notify_something() @henocdz
  9. Reintentando @task(bind=True) def send_twitter_status(self, oauth, tweet): try: twitter = Twitter(oauth)

    twitter.update_status(tweet) except (Twitter.FailWhaleError, Twitter.LoginError) as exc: raise self.retry(exc=exc) @henocdz
  10. Ejecutar tareas @henocdz

  11. task.delay Fácil y prác*co, e.g: some_task.delay(arg1, arg2, kwarg1='some', kwarg2='value') @henocdz

  12. task.apply_async Nos da mayor control sobre el task, e.g: #

    Ejecutar después de N segundos some_task.apply_async(args=argv, kwargs=kwargs, countdown=10) # Cancelar la ejecución si no sucede después de cierta fecha tomorrow = today + timedelta(hours=1) some_task.apply_async(args=argv, kwargs=kwargs, expires=tomorrow) @henocdz
  13. @henocdz

  14. Setup con Django @henocdz

  15. settings.py from __future__ import absolute_import from datetime import timedelta CELERY_RESULT_BACKEND

    = 'redis://{user}:{password}@{host}:{port}/{db_num}' BROKER_URL = 'ampq://{user}:{password}@{host}:{port}/' @henocdz
  16. project/project/celery_app.py from __future__ import absolute_import from celery import Celery from

    django.conf import settings # noqa celery_app = Celery('project') celery_app.config_from_object('django.conf:settings') celery_app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) @henocdz
  17. project/project/__init__.py from .celery_app import celery_app @henocdz

  18. Celery Beat Administrador de tareas periódicas @henocdz

  19. Programar tareas periódicas # settings.py from celery.schedules import crontab CELERYBEAT_SCHEDULE

    = { 'generate_monthly_report': { 'task': 'reports.tasks.generate_monthly_report', 'schedule': crontab(minute='0,20,40', hour='12', day_of_week='mon,thu'), }, 'delete_temp_files': { 'task': 'reports.tasks.delete_temp_files', 'schedule': timedelta(seconds=180), }, } $ celery beat -A project @henocdz
  20. Flower Web Monitor & Admin para Celery @henocdz

  21. @henocdz

  22. @henocdz

  23. @henocdz

  24. Ejecutar flower $ celery flower --address=0.0.0.0 --port=9000 \ --basic_auth=admin@site.com:sitepwd69 \

    --broker=ampq://{user}:{password}@{host}:{port}/ \ --broker_api=ampq://{user}:{password}@{host}:{port}/ @henocdz
  25. bit.ly/celery-bp @henocdz

  26. Tareas pequeñas from myproject.tasks import app # app is your

    celery application from myproject.exceptions import InvalidUserInput from utils.mail import api_send_mail @task(max_retries=3) def send_mail(self, recipients, sender_email, subject, body): try: data = api_send_mail(recipients, sender_email, subject, body) except InvalidUserInput: raise except Exception as exc: sentrycli.captureException() send_mail.retry(countdown=10, exc=exc) return data @henocdz
  27. Falla rápido y sin bloquear el queue @task(max_retries=3, soft_time_limit=5) def

    send_mail(self, recipients, sender_email, subject, body): ... @henocdz
  28. Aumenta gradualmente el -empo para reintentar def backoff(attempts): return 2

    ** attempts @task(max_retries=3) def send_mail(self, recipients, sender_email, subject, body): try: data = api_send_mail(recipients, sender_email, subject, body) except InvalidUserInput: raise except Exception as exc: send_mail.retry(countdown=backoff(self.request.retries), exc=exc) ... @henocdz
  29. Comparte funcionalidad entre tareas usando clases from celery.tasks import Task

    class BaseTask(Task): """Abstract base class for all tasks in my app.""" abstract = True def on_retry(self, exc, task_id, args, kwargs, einfo): """Log the exceptions to sentry at retry.""" sentrycli.captureException(exc) super(BaseTask, self).on_retry(exc, task_id, args, kwargs, einfo) def on_failure(self, exc, task_id, args, kwargs, einfo): """Log the exceptions to sentry.""" sentrycli.captureException(exc) super(BaseTask, self).on_failure(exc, task_id, args, kwargs, einfo) @henocdz
  30. base=BaseTask @task(max_retries=3, soft_time_limit=5, base=BaseTask) def send_mail(self, recipients, sender_email, subject, body):

    try: data = api_send_mail(recipients, sender_email, subject, body) except InvalidUserInput: raise except Exception as exc: self.retry(countdown=backoff(self.request.retries), exc=exc) return data @henocdz
  31. Mul$ples Workers1 y Queues # settings.py CELERY_ROUTES = { 'reports.tasks.generate_monthly_report':

    {'queue': 'reports'}, 'reports.tasks.delete_temp_files': {'queue': 'cleanup'}, } $ celery worker -n celery@ServerName -A project -Q default,cleanup $ celery worker -n celery2@ServerName -A project -Q reports 1 Número recomendado: n + 1, n = número de cores @henocdz
  32. NUNCA enviar objetos del ORM from clients.models import ClientUser from

    clients.tasks import send_report # Esto está mal. def send_report(client) send_email(...client.email...) send_report.delay(client) @henocdz
  33. # Solución def send_report(client_pk) client = ClientUser.objects.get(pk=client_pk) send_report.delay(client.pk) @henocdz

  34. y ya... @henocdz

  35. ¡Gracias! ! @henocdz