Packaging Django Projects for PyPI

How to package a Django project to behave and install like a Python package. Lessons learned packaging Mayan EDMS the free open source document manager based in Django.

Transcript

1. Packaging Django Projects for PyPI @siloraptor http://robertorosario.com https://github.com/rosarior
2. Who am I?
3. http://rosarior.github.io/awesome-django/
10. http://pythonlatino.com
13. $ source venv/bin/activate $ pip install python-package
14. virtualenv venv source venv/bin/activate git clone <project> pip install -r requirements.txt vi <project>/settings.py # Setup database # Setup media files # Set secret key # Customize other settings $ ./manage.py syncdb You have installed Django's auth system, and don't have any superusers defined. Would you like to create one now? (yes/no): yes Username (leave blank to use 'rosarior'): admin Email address: admin@admin.com Password: Password (again): Superuser created successfully. $ ./manage.py migrate $ ./manage.py runserver
16. One truth...
17. Using Django projects still require Django proficiency.
18. https://mayan.readthedocs.org/en/latest/topics/installation.html
19. https://mayan.readthedocs.org/en/latest/topics/installation.html
21. • Our users are rarely Django/Python programmers • Hard to install • Hard to override settings • One settings.py file per instance • Manual SECRET_KEY creation • Flat structure: project, media files and apps at the same level • Media files? Where? • Initial setup is not configurable • Insecure initial admin user creation • Hard to manage after install • Least amount of installation steps
22. Python package $ source venv/bin/activate $ pip install python-package Mayan EDMS $ virtualenv venv $ source venv/bin/activate $ pip install mayan-edms $ mayan-edms.py initialsetup
23. Project structure
24. mysite/ manage.py project/ __init__.py settings.py urls.py wsgi.py polls/ __init__.py admin.py migrations/ __init__.py models.py tests.py views.py
25. Namespaces are one honking great idea -- let's do more of those! https://www.python.org/dev/peps/pep-0020/
26. mysite/ manage.py project/ __init__.py settings.py urls.py wsgi.py polls/ __init__.py admin.py migrations/ __init__.py models.py tests.py views.py
27. mysite/ manage.py project/ __init__.py settings.py urls.py wsgi.py apps/ __init__.py polls/ __init__.py admin.py migrations/ __init__.py models.py tests.py views.py
28. # setup.py #!/usr/bin/env python import os import sys try: from setuptools import setup except ImportError: from distutils.core import setup Import mayan PACKAGE_NAME = 'mayan-edms' PACKAGE_DIR = 'mayan'
30. Settings
31. mysite/ project/ __init__.py settings.py
32. settings.py development.py production.py
35. Solution?
37. try: from settings_local import * except ImportError: Pass if DEVELOPMENT: INTERNAL_IPS = ('127.0.0.1',) TEMPLATE_LOADERS = ( 'django.template.loaders.filesystem.Loader', 'django.template.loaders.app_directories.Loader', ) try: import rosetta INSTALLED_APPS += ('rosetta',) except ImportError: pass try: import django_extensions INSTALLED_APPS += ('django_extensions',) except ImportError: pass
39. How to fix settings spaghetti?
41. mysite/ project/ __init__.py settings.py
42. mysite/ project/ __init__.py settings/ __init__.py base.py
43. settings/base.py settings/development.py settings/production.py from .base import * INSTALLED_APPS += ( 'rosetta', 'django_extensions', 'debug_toolbar' ) from .base import * DEBUG = False
44. mysite/ project/ __init__.py settings/ __init__.py base.py
45. mysite/ project/ __init__.py settings/ __init__.py base.py development.py production.py
46. # wsgi.py import os os.environ.setdefault( "DJANGO_SETTINGS_MODULE", "mayan.settings.production" ) from django.core.wsgi import get_wsgi_application application = get_wsgi_application()
47. Explicit is better than implicit. Readability counts. https://www.python.org/dev/peps/pep-0020/
48. Instead of: ./manage.py runserver –settings = mayan.robert We get intuitive: ./manage.py runserver –settings = mayan.settings.development ./manage.py runserver –settings = mayan.settings.testing DJANGO_SETTINGS_MODULE = 'mayan.settings.celery_redis' ./manage.py runserver
49. SECRET_KEY
53. SECRET_KEY = “-- secret key --”
55. $ vi settings.py SECRET_KEY = ”qwertyuias m,1389vbqaf-0/qax/xqweqi3”
56. SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 92-93: truncated \xXX escape SyntaxError: EOL while scanning string literal
57. “I don’t know what the SECRET_KEY is used for. I fixed it by commenting the line and restarting nginx.”
58. “I don’t know what the SECRET_KEY is used for. I fixed it by commenting the line and restarting nginx.”
60. # /django/core/management/commands/startproject.py # Create a random SECRET_KEY to put it in the main settings. chars = 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)' options['secret_key'] = get_random_string(50, chars)
61. Single Step Setup
62. def _generate_secret_key(self): chars = 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)' return get_random_string(50, chars) path = os.path.join( settings.BASE_DIR, 'mayan', 'settings', 'local.py' ) with open(path, 'w+') as file_object: file_object.write('\n'.join([ 'from __future__ import absolute_import', '', 'from .base import *', '', "SECRET_KEY='{0}'".format(self._generate_secret_key()), '', ]))
63. path = os.path.join( settings.BASE_DIR, 'mayan', 'settings', 'local.py' ) ... if os.path.exists(path): print 'Existing file at: {0}. Backup, remove this file and try again.'.format(path) exit(1)
65. Keeping SECRET_KEY secret
68. git commit ­a <tab> <enter>
69. git commit ­­amend <enter>
70. git commit ­a <enter>
72. # .gitignore mayan/settings/local.py
73. # MANIFEST.in ... global-exclude mayan/settings/local.py
74. #!/usr/bin/env python import os import sys if __name__ == "__main__": os.environ.setdefault( "DJANGO_SETTINGS_MODULE", "mayan.settings.local" ) from django.core.management import execute_from_command_line execute_from_command_line(sys.argv)
75. $ ./manage.py initialsetup Won’t execute. Why?
77. # mayan/settings/__init__.py try: from .local import * # NOQA except ImportError: from .base import * # NOQA from mayan import settings # always works now
78. ./manage.py initialsetup # tries to imports from local.py and fallsback to base.py ./manage.py runserver # imports from local.py which imports from base.py wsgi.py # imports from production.py which imports from local.py which import from base.py
79. settings/base.py settings/development.py settings/production.py From .base import * SECRET_KEY = “qwerty12345” #CELERY_ALWAYS_EAGER = False BROKER_URL = 'redis: //127.0.0.1:6379/0' CELERY_RESULT_BACKEND = 'redis://127.0.0.1: 6379/0' from . import * INSTALLED_APPS += ( 'rosetta', 'django_extensions', 'debug_toolbar' ) from . import * DEBUG = False settings/local.py
80. Single Step Setup
81. $ ./manage.py syncdb # Don’t create admin user $ ./manage.py migrate
83. ./manage.py syncdb --migrate --noinput
84. from django.core import management management.call_command('syncdb', migrate=True, interactive=False) # Don’t ask to create admin user
85. class Command(management.BaseCommand): help = 'Gets Mayan EDMS ready to be used (initializes database,creates a secret key,etc).' def _generate_secret_key(self): chars = 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)' return get_random_string(50, chars) def handle(self, *args, **options): path = os.path.join(settings.BASE_DIR, 'mayan', 'settings','local.py') if os.path.exists(path): print 'Existing file at: {0}. Backup, remove this file and try again.'.format(path) Exit(1) with open(path, 'w+') as file_object: file_object.write('\n'.join([ 'from __future__ import absolute_import', '', 'from .base import *', '', "SECRET_KEY = '{0}'".format(self._generate_secret_key()),'', ])) management.call_command('syncdb', migrate=True, interactive=False)
86. $ ./manage.py initialsetup • Secret key • Create settings/local.py file • Generate database w/out creating admin • Don’t overwrite existing settings file https://github.com/mayan-edms/mayan-edms/blob/master/mayan/apps/main/management/c ommands/initialsetup.py
87. Initial Admin Creation
88. Most common admin passwords?
89. 123456 admin password
90. Worst than common admin passwords?
91. Default admin passwords!
93. http://www.defaultpassword.com/
94. # models.py from solo.models import SingletonModel class AutoAdminSingleton(SingletonModel): account = models.ForeignKey( User, null=True, blank=True, related_name='auto_admin_account' ) password = models.CharField( null=True, blank=True, max_length=128 ) password_hash = models.CharField( null=True, blank=True, max_length=128 ) class Meta: verbose_name = verbose_name_plural = _('Auto admin properties')
95. @receiver(post_migrate, dispatch_uid='create_superuser_and_anonymous_user') def create_superuser_and_anonymous_user(sender, **kwargs): ... assert auth_models.User.objects.create_superuser( AUTO_ADMIN_USERNAME,'autoadmin@autoadmin.com', AUTO_ADMIN_PASSWORD ) admin = auth_models.User.objects.get(username=AUTO_ADMIN_USERNAME) auto_admin_properties, created = AutoAdminSingleton.objects. get_or_create() auto_admin_properties.account = admin auto_admin_properties.password = AUTO_ADMIN_PASSWORD auto_admin_properties.password_hash = admin.password auto_admin_properties.save()
96. # settings.py register_setting( namespace='common', module='common.settings', name='AUTO_ADMIN_PASSWORD', global_name='COMMON_AUTO_ADMIN_PASSWORD', default=User.objects.make_random_password(), )
98. # templatetags/autoadmin_tags.py @register.simple_tag(takes_context=True) def auto_admin_properties(context): context['auto_admin_properties'] = AutoAdminSingleton.objects.get() return ''
99. # login.html {% auto_admin_properties %} {% if auto_admin_properties.account %} <h2>{% trans "First time login" %}</h2> <p>{% trans 'You have just finished installing <strong>Mayan EDMS</strong>,congratulations!' %}</p> <p>{% trans 'Login using the following credentials:' %}</p> <p>{% blocktrans with auto_admin_properties.account as account %}Username: <strong>{{ account }}</strong>{% endblocktrans %}</p> <p>{% blocktrans with auto_admin_properties.account.email as email %}Email: <strong>{{ email }}</strong>{% endblocktrans %}</p> <p>{% blocktrans with auto_admin_properties.password as password %}Password: <strong>{{ password }}</strong>{% endblocktrans %}</p> <p>{% trans 'Be sure to change the password to increase security and to disable this message.' %}</p> {% endif %}
101. @receiver(post_save, dispatch_uid='auto_admin_account_passwd_change', sender=User) def auto_admin_account_passwd_change(sender, instance, **kwargs): auto_admin_properties = AutoAdminSingleton.objects.get() if instance == auto_admin_properties.account and instance.password != auto_admin_properties.password_hash: # Only delete the auto admin properties when the password has been changed auto_admin_properties.account = None auto_admin_properties.password = None auto_admin_properties.password_hash = None auto_admin_properties.save()
102. https://pypi.python.org/pypi/django-autoadmin
103. Media Files
104. Not all deployment environment support the same Persistent File System setup ie: Docker, Stackato
105. register_settings( namespace='documents',module='documents.settings',settings=[ { 'name': 'STORAGE_BACKEND', 'global_name': 'DOCUMENTS_STORAGE_BACKEND', 'default': 'storage.backends.filebasedstorage.FileBasedStorage' }] ) https://github.com/mayan-edms/mayan-edms/blob/master/mayan/apps/documents/setting s.py
106. mysite/ manage.py project/ __init__.py settings/ __init__.py base.py urls.py wsgi.py apps/ __init__.py
107. mysite/ manage.py project/ __init__.py settings/ __init__.py base.py media/ db.sqlite document_storage/ gpg_home/ image_cache/ static/ urls.py wsgi.py apps/ __init__.py
108. _file_path = os.path.abspath(os.path.dirname(__file__)).split('/') BASE_DIR = '/'.join(_file_path[0:-2]) MEDIA_ROOT = os.path.join(BASE_DIR, 'mayan', 'media') DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': os.path.join(MEDIA_ROOT, 'db.sqlite3'), } } STATIC_ROOT = os.path.join(BASE_DIR, 'mayan', 'media', 'static') https://github.com/mayan-edms/mayan-edms/blob/master/mayan/settings/base.py
110. # .gitignore ... *.sqlite3 gpg_home/ /mayan/media/static/ image_cache/ document_storage/
111. # MANIFEST.in ... global-exclude mayan/settings/local.py db.sqlite* mayan/media gpg_home document_storage image_cache
112. Post Install Management
113. $ /usr/share/mayan-edms/lib/python2.7/site-packages/mayan/manage.py collectstatic
115. mysite/ manage.py project/ __init__.py settings/ __init__.py base.py media/ urls.py wsgi.py apps/ __init__.py
116. mysite/ manage.py project/ bin/ mayan-edms.py # Copy of manage.py __init__.py settings/ __init__.py base.py media/ urls.py wsgi.py apps/ __init__.py
117. # setup.py scripts=['mayan/bin/mayan-edms.py'],
118. $ virtualenv venv $ source venv/bin/activate $ pip install mayan-edms $ mayan-edms.py initialsetup $ mayan-edms.py runserver $ mayan-edms.py collectstatic --noinput
119. server { location /static/ { root /usr/share/mayan-edms/lib/python2.7/site- packages/mayan/media/static; } }
120. Guided Setup
122. class MissingItem(object): _registry = [] @classmethod def get_all(cls): return cls._registry def __init__(self, label, condition, description, view): self.label = label self.condition = condition self.description = description self.view = view self.__class__._registry.append(self)
123. # /mayan/apps/documents/apps.py MissingItem( label=_('Create a document type'), description=_('Every uploaded document must be assigned a document type, it is the basic way Mayan EDMS categorizes documents.'), condition=lambda: not DocumentType.objects.exists(), view='documents:document_type_list' )
125. http://buzzerg.com
126. Disadvantages • Long paths • Media file hidden depth inside virtualenv’s directories Solution Instead of ./manage.py initialsetup, a startsite command, homologous to Django’s startproject command https://github.com/mayan-edms/mayan-edms/issues/171
127. Demo
128. @siloraptor https://github.com/rosarior http://robertorosario.com “What is your question?”
129. Thank you! @siloraptor https://github.com/rosarior http://robertorosario.com Hint: Capital of Assyria: Assyria (Assur)