Share
Embed
Introducción a Celery
Plática para Chilango Django 20 de Julio de 2016, Ciudad de México
Henoc Díaz
July 20, 2016
Transcript
35 slides
-
¿Celery? Es un sistema distribuido para procesar mensajes en una
cola de tareas, con soporte para soporte real-0me o programado (scheduling) @henocdz -
¿En qué casos podría ser ú2l? • Tareas secundarias: Enviar
emails • Tareas que consuman una gran can1dad de 1empo: Procesar mul1media, crear XLSX, PDFs, etc • Tareas repe11vas: Scrapping, traer data de un API, feed, etc. @henocdz -
Compa&bilidad 1. Python 2.7, 3.4, 3.5 2. Django, Pyramid, Tornado,
Flask, otros. @henocdz -
Elementos Clave • Producer • Broker (AMPQ) • Queues •
Task • Consumer (Worker) • Result Backend @henocdz -
from celery.task import task from users.models import CustomUser from annoying.functions
import get_object_or_None @task(name='some_django_app.notify_user', ignore_result=True) def notify_user(user_pk): user = get_object_or_None(CustomUser, pk=user_pk) if user is not None: user.notify_something() @henocdz -
Reintentando @task(bind=True) def send_twitter_status(self, oauth, tweet): try: twitter = Twitter(oauth)
twitter.update_status(tweet) except (Twitter.FailWhaleError, Twitter.LoginError) as exc: raise self.retry(exc=exc) @henocdz -
task.apply_async Nos da mayor control sobre el task, e.g: #
Ejecutar después de N segundos some_task.apply_async(args=argv, kwargs=kwargs, countdown=10) # Cancelar la ejecución si no sucede después de cierta fecha tomorrow = today + timedelta(hours=1) some_task.apply_async(args=argv, kwargs=kwargs, expires=tomorrow) @henocdz -
settings.py from __future__ import absolute_import from datetime import timedelta CELERY_RESULT_BACKEND
= 'redis://{user}:{password}@{host}:{port}/{db_num}' BROKER_URL = 'ampq://{user}:{password}@{host}:{port}/' @henocdz -
project/project/celery_app.py from __future__ import absolute_import from celery import Celery from
django.conf import settings # noqa celery_app = Celery('project') celery_app.config_from_object('django.conf:settings') celery_app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) @henocdz -
Programar tareas periódicas # settings.py from celery.schedules import crontab CELERYBEAT_SCHEDULE
= { 'generate_monthly_report': { 'task': 'reports.tasks.generate_monthly_report', 'schedule': crontab(minute='0,20,40', hour='12', day_of_week='mon,thu'), }, 'delete_temp_files': { 'task': 'reports.tasks.delete_temp_files', 'schedule': timedelta(seconds=180), }, } $ celery beat -A project @henocdz -
Ejecutar flower $ celery flower --address=0.0.0.0 --port=9000 \ [email protected]:sitepwd69 \
--broker=ampq://{user}:{password}@{host}:{port}/ \ --broker_api=ampq://{user}:{password}@{host}:{port}/ @henocdz -
Tareas pequeñas from myproject.tasks import app # app is your
celery application from myproject.exceptions import InvalidUserInput from utils.mail import api_send_mail @task(max_retries=3) def send_mail(self, recipients, sender_email, subject, body): try: data = api_send_mail(recipients, sender_email, subject, body) except InvalidUserInput: raise except Exception as exc: sentrycli.captureException() send_mail.retry(countdown=10, exc=exc) return data @henocdz -
Falla rápido y sin bloquear el queue @task(max_retries=3, soft_time_limit=5) def
send_mail(self, recipients, sender_email, subject, body): ... @henocdz -
Aumenta gradualmente el -empo para reintentar def backoff(attempts): return 2
** attempts @task(max_retries=3) def send_mail(self, recipients, sender_email, subject, body): try: data = api_send_mail(recipients, sender_email, subject, body) except InvalidUserInput: raise except Exception as exc: send_mail.retry(countdown=backoff(self.request.retries), exc=exc) ... @henocdz -
Comparte funcionalidad entre tareas usando clases from celery.tasks import Task
class BaseTask(Task): """Abstract base class for all tasks in my app.""" abstract = True def on_retry(self, exc, task_id, args, kwargs, einfo): """Log the exceptions to sentry at retry.""" sentrycli.captureException(exc) super(BaseTask, self).on_retry(exc, task_id, args, kwargs, einfo) def on_failure(self, exc, task_id, args, kwargs, einfo): """Log the exceptions to sentry.""" sentrycli.captureException(exc) super(BaseTask, self).on_failure(exc, task_id, args, kwargs, einfo) @henocdz -
base=BaseTask @task(max_retries=3, soft_time_limit=5, base=BaseTask) def send_mail(self, recipients, sender_email, subject, body):
try: data = api_send_mail(recipients, sender_email, subject, body) except InvalidUserInput: raise except Exception as exc: self.retry(countdown=backoff(self.request.retries), exc=exc) return data @henocdz -
Mul$ples Workers1 y Queues # settings.py CELERY_ROUTES = { 'reports.tasks.generate_monthly_report':
{'queue': 'reports'}, 'reports.tasks.delete_temp_files': {'queue': 'cleanup'}, } $ celery worker -n celery@ServerName -A project -Q default,cleanup $ celery worker -n celery2@ServerName -A project -Q reports 1 Número recomendado: n + 1, n = número de cores @henocdz -
NUNCA enviar objetos del ORM from clients.models import ClientUser from
clients.tasks import send_report # Esto está mal. def send_report(client) send_email(...client.email...) send_report.delay(client) @henocdz