Case Study: Внедрение Celery 4

Александр Полищук (ООО "Код безопасности") @ Moscow Python Conf 2017
"Необходимо было внедрить (обновить) celery 4 версии для выполнения асинхронных тасков в фоновом режиме. Если необходим броадкаст - выбирайте RabbitMQ в качестве брокера".

Transcript

1. Внедрение Celery 4
   Немного садоводства
   

   View Slide

2. Что такое Celery?
   

   View Slide

3. Дополняем огород
   

   View Slide

4. Наши овощи и Django
   from __future__ import absolute_import
   import os
   from celery import Celery
   

   View Slide

5. Наши овощи и Django
   from __future__ import absolute_import
   import os
   from celery import Celery
   # set the default Django settings module for the 'celery' program.
   os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' myproject.settings‘)
   

   View Slide

6. Наши овощи и Django
   from __future__ import absolute_import
   import os
   from celery import Celery
   # set the default Django settings module for the 'celery' program.
   os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' myproject.settings')
   app = Celery(‘myproject')
   

   View Slide

7. Наши овощи и Django
   from __future__ import absolute_import
   import os
   from celery import Celery
   # set the default Django settings module for the 'celery' program.
   os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' myproject.settings')
   app = Celery(‘myproject')
   # Using a string here means the worker will not have to
   # pickle the object when using Windows.
   app.config_from_object('django.conf:settings')
   

   View Slide

8. Наши овощи и Django
   from __future__ import absolute_import
   import os
   from celery import Celery
   # set the default Django settings module for the 'celery' program.
   os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' myproject.settings')
   app = Celery(‘myproject')
   # Using a string here means the worker will not have to
   # pickle the object when using Windows.
   app.config_from_object('django.conf:settings')
   app.autodiscover_tasks()
   

   View Slide

9. Настройки в settings
   BROKER_URL = 'redis://localhost:6379/0'
   

   View Slide

10. Настройки в settings
    BROKER_URL = 'redis://localhost:6379/0'
    CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
    

    View Slide

11. Настройки в settings
    BROKER_URL = 'redis://localhost:6379/0'
    CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
    CELERY_ACCEPT_CONTENT = ['json']
    

    View Slide

12. Настройки в settings
    BROKER_URL = 'redis://localhost:6379/0'
    CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
    CELERY_ACCEPT_CONTENT = ['json']
    CELERY_TASK_SERIALIZER = 'json'
    

    View Slide

13. Настройки в settings
    BROKER_URL = 'redis://localhost:6379/0'
    CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
    CELERY_ACCEPT_CONTENT = ['json']
    CELERY_TASK_SERIALIZER = 'json'
    CELERY_RESULT_SERIALIZER = 'json'
    

    View Slide

14. Настройки в slave_settings
    from settings import *
    

    View Slide

15. Настройки в slave_settings
    from settings import *
    BROKER_URL = 'redis://localhost:16379/0'
    

    View Slide

16. Настройки в slave_settings
    from settings import *
    BROKER_URL = 'redis://localhost:16379/0'
    CELERY_RESULT_BACKEND = 'redis://localhost:16379/0'
    

    View Slide

17. Демонизируем
    export DJANGO_SETTINGS_MODULE="settings“
    

    View Slide

18. Демонизируем
    export DJANGO_SETTINGS_MODULE="settings“
    Для slave:
    export DJANGO_SETTINGS_MODULE=“slave_settings“
    

    View Slide

19. Демонизируем
    export DJANGO_SETTINGS_MODULE="settings“
    CELERYD_NODES=“myworker"
    CELERY_BIN="/usr/bin/celery"
    CELERY_APP=“myproject"
    

    View Slide

20. Демонизируем
    export DJANGO_SETTINGS_MODULE="settings“
    CELERYD_NODES=“myworker"
    CELERY_BIN="/usr/bin/celery"
    CELERY_APP=“myproject"
    CELERYD_CHDIR="/path/to/project"
    

    View Slide

21. Демонизируем
    export DJANGO_SETTINGS_MODULE="settings“
    CELERYD_NODES=“myworker"
    CELERY_BIN="/usr/bin/celery"
    CELERY_APP=“myproject"
    CELERYD_CHDIR="/path/to/project"
    CELERYD_USER="root"
    CELERYD_GROUP="root"
    

    View Slide

22. Демонизируем
    export DJANGO_SETTINGS_MODULE="settings“
    CELERYD_NODES=“myworker"
    CELERY_BIN="/usr/bin/celery"
    CELERY_APP=“myproject"
    CELERYD_CHDIR="/path/to/project"
    CELERYD_USER="root"
    CELERYD_GROUP="root"
    CELERY_CREATE_DIRS=1
    

    View Slide

23. Демонизируем
    export DJANGO_SETTINGS_MODULE="settings“
    CELERYD_NODES=“myworker"
    CELERY_BIN="/usr/bin/celery"
    CELERY_APP=“myproject"
    CELERYD_CHDIR="/path/to/project"
    CELERYD_USER="root"
    CELERYD_GROUP="root"
    CELERY_CREATE_DIRS=1
    CELERYD_OPTS="--time-limit=3600 --concurrency=1"
    

    View Slide

24. Демонизируем
    export DJANGO_SETTINGS_MODULE="settings“
    CELERYD_NODES=“myworker"
    CELERY_BIN="/usr/bin/celery"
    CELERY_APP=“myproject"
    CELERYD_CHDIR="/path/to/project"
    CELERYD_USER="root"
    CELERYD_GROUP="root"
    CELERY_CREATE_DIRS=1
    CELERYD_OPTS="--time-limit=3600 --concurrency=1"
    Для slave:
    CELERYD_OPTS="--time-limit=3600 --concurrency=1 –Q myworker@salve_name.dq"
    

    View Slide

25. Демонизируем
    export DJANGO_SETTINGS_MODULE="settings“
    CELERYD_NODES=“myworker"
    CELERY_BIN="/usr/bin/celery"
    CELERY_APP=“myproject"
    CELERYD_CHDIR="/path/to/project"
    CELERYD_USER="root"
    CELERYD_GROUP="root"
    CELERY_CREATE_DIRS=1
    CELERYD_OPTS="--time-limit=3600 --concurrency=1“
    https://github.com/celery/celery/blob/4.0/extra/generic-init.d/celeryd
    

    View Slide

26. Демонизируем
    export DJANGO_SETTINGS_MODULE="settings“
    CELERYD_NODES=“myworker"
    CELERY_BIN="/usr/bin/celery"
    CELERY_APP=“myproject"
    CELERYD_CHDIR="/path/to/project"
    CELERYD_USER="root"
    CELERYD_GROUP="root"
    CELERY_CREATE_DIRS=1
    CELERYD_OPTS="--time-limit=3600 --concurrency=1"
    https://github.com/celery/celery/blob/4.0/extra/generic-init.d/celeryd
    Service celeryd start
    

    View Slide

27. Салат готов!
    Спасибо за внимание
    

    View Slide