Packaging Django Projects for PyPI

Slide 1

Slide 1 text

Packaging Django Projects for PyPI @siloraptor http://robertorosario.com https://github.com/rosarior

Slide 2

Slide 2 text

Who am I?

Slide 3

Slide 3 text

http://rosarior.github.io/awesome-django/

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

http://pythonlatino.com

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

$ source venv/bin/activate $ pip install python-package

Slide 14

Slide 14 text

virtualenv venv source venv/bin/activate git clone pip install -r requirements.txt vi /settings.py # Setup database # Setup media files # Set secret key # Customize other settings $ ./manage.py syncdb You have installed Django's auth system, and don't have any superusers defined. Would you like to create one now? (yes/no): yes Username (leave blank to use 'rosarior'): admin Email address: [email protected] Password: Password (again): Superuser created successfully. $ ./manage.py migrate $ ./manage.py runserver

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

One truth...

Slide 17

Slide 17 text

Using Django projects still require Django proficiency.

Slide 18

Slide 18 text

https://mayan.readthedocs.org/en/latest/topics/installation.html

Slide 19

Slide 19 text

https://mayan.readthedocs.org/en/latest/topics/installation.html

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

● Our users are rarely Django/Python programmers ● Hard to install ● Hard to override settings ● One settings.py file per instance ● Manual SECRET_KEY creation ● Flat structure: project, media files and apps at the same level ● Media files? Where? ● Initial setup is not configurable ● Insecure initial admin user creation ● Hard to manage after install ● Least amount of installation steps

Slide 22

Slide 22 text

Python package $ source venv/bin/activate $ pip install python-package Mayan EDMS $ virtualenv venv $ source venv/bin/activate $ pip install mayan-edms $ mayan-edms.py initialsetup

Slide 23

Slide 23 text

Project structure

Slide 24

Slide 24 text

mysite/ manage.py project/ __init__.py settings.py urls.py wsgi.py polls/ __init__.py admin.py migrations/ __init__.py models.py tests.py views.py

Slide 25

Slide 25 text

Namespaces are one honking great idea -- let's do more of those! https://www.python.org/dev/peps/pep-0020/

Slide 26

Slide 26 text

mysite/ manage.py project/ __init__.py settings.py urls.py wsgi.py polls/ __init__.py admin.py migrations/ __init__.py models.py tests.py views.py

Slide 27

Slide 27 text

mysite/ manage.py project/ __init__.py settings.py urls.py wsgi.py apps/ __init__.py polls/ __init__.py admin.py migrations/ __init__.py models.py tests.py views.py

Slide 28

Slide 28 text

# setup.py #!/usr/bin/env python import os import sys try: from setuptools import setup except ImportError: from distutils.core import setup Import mayan PACKAGE_NAME = 'mayan-edms' PACKAGE_DIR = 'mayan'

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

Settings

Slide 31

Slide 31 text

mysite/ project/ __init__.py settings.py

Slide 32

Slide 32 text

settings.py development.py production.py

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Solution?

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

try: from settings_local import * except ImportError: Pass if DEVELOPMENT: INTERNAL_IPS = ('127.0.0.1',) TEMPLATE_LOADERS = ( 'django.template.loaders.filesystem.Loader', 'django.template.loaders.app_directories.Loader', ) try: import rosetta INSTALLED_APPS += ('rosetta',) except ImportError: pass try: import django_extensions INSTALLED_APPS += ('django_extensions',) except ImportError: pass

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

How to fix settings spaghetti?

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

mysite/ project/ __init__.py settings.py

Slide 42

Slide 42 text

mysite/ project/ __init__.py settings/ __init__.py base.py

Slide 43

Slide 43 text

settings/base.py settings/development.py settings/production.py from .base import * INSTALLED_APPS += ( 'rosetta', 'django_extensions', 'debug_toolbar' ) from .base import * DEBUG = False

Slide 44

Slide 44 text

mysite/ project/ __init__.py settings/ __init__.py base.py

Slide 45

Slide 45 text

mysite/ project/ __init__.py settings/ __init__.py base.py development.py production.py

Slide 46

Slide 46 text

# wsgi.py import os os.environ.setdefault( "DJANGO_SETTINGS_MODULE", "mayan.settings.production" ) from django.core.wsgi import get_wsgi_application application = get_wsgi_application()

Slide 47

Slide 47 text

Explicit is better than implicit. Readability counts. https://www.python.org/dev/peps/pep-0020/

Slide 48

Slide 48 text

Instead of: ./manage.py runserver –settings = mayan.robert We get intuitive: ./manage.py runserver –settings = mayan.settings.development ./manage.py runserver –settings = mayan.settings.testing DJANGO_SETTINGS_MODULE = 'mayan.settings.celery_redis' ./manage.py runserver

Slide 49

Slide 49 text

SECRET_KEY

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

SECRET_KEY = “-- secret key --”

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

$ vi settings.py SECRET_KEY = ”qwertyuias m,1389vbqaf-0/qax/xqweqi3”

Slide 56

Slide 56 text

SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 92-93: truncated \xXX escape SyntaxError: EOL while scanning string literal

Slide 57

Slide 57 text

“I don’t know what the SECRET_KEY is used for. I fixed it by commenting the line and restarting nginx.”

Slide 58

Slide 58 text

“I don’t know what the SECRET_KEY is used for. I fixed it by commenting the line and restarting nginx.”

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

# /django/core/management/commands/startproject.py # Create a random SECRET_KEY to put it in the main settings. chars = 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)' options['secret_key'] = get_random_string(50, chars)

Slide 61

Slide 61 text

Single Step Setup

Slide 62

Slide 62 text

def _generate_secret_key(self): chars = 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)' return get_random_string(50, chars) path = os.path.join( settings.BASE_DIR, 'mayan', 'settings', 'local.py' ) with open(path, 'w+') as file_object: file_object.write('\n'.join([ 'from __future__ import absolute_import', '', 'from .base import *', '', "SECRET_KEY='{0}'".format(self._generate_secret_key()), '', ]))

Slide 63

Slide 63 text

path = os.path.join( settings.BASE_DIR, 'mayan', 'settings', 'local.py' ) ... if os.path.exists(path): print 'Existing file at: {0}. Backup, remove this file and try again.'.format(path) exit(1)

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

Keeping SECRET_KEY secret

Slide 66

Slide 66 text

No content

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

git commit a

Slide 69

Slide 69 text

git commit amend

Slide 70

Slide 70 text

git commit a

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

# .gitignore mayan/settings/local.py

Slide 73

Slide 73 text

# MANIFEST.in ... global-exclude mayan/settings/local.py

Slide 74

Slide 74 text

#!/usr/bin/env python import os import sys if __name__ == "__main__": os.environ.setdefault( "DJANGO_SETTINGS_MODULE", "mayan.settings.local" ) from django.core.management import execute_from_command_line execute_from_command_line(sys.argv)

Slide 75

Slide 75 text

$ ./manage.py initialsetup Won’t execute. Why?

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

# mayan/settings/__init__.py try: from .local import * # NOQA except ImportError: from .base import * # NOQA from mayan import settings # always works now

Slide 78

Slide 78 text

./manage.py initialsetup # tries to imports from local.py and fallsback to base.py ./manage.py runserver # imports from local.py which imports from base.py wsgi.py # imports from production.py which imports from local.py which import from base.py

Slide 79

Slide 79 text

settings/base.py settings/development.py settings/production.py From .base import * SECRET_KEY = “qwerty12345” #CELERY_ALWAYS_EAGER = False BROKER_URL = 'redis: //127.0.0.1:6379/0' CELERY_RESULT_BACKEND = 'redis://127.0.0.1: 6379/0' from . import * INSTALLED_APPS += ( 'rosetta', 'django_extensions', 'debug_toolbar' ) from . import * DEBUG = False settings/local.py

Slide 80

Slide 80 text

Single Step Setup

Slide 81

Slide 81 text

$ ./manage.py syncdb # Don’t create admin user $ ./manage.py migrate

Slide 82

Slide 82 text

No content

Slide 83

Slide 83 text

./manage.py syncdb --migrate --noinput

Slide 84

Slide 84 text

from django.core import management management.call_command('syncdb', migrate=True, interactive=False) # Don’t ask to create admin user

Slide 85

Slide 85 text

class Command(management.BaseCommand): help = 'Gets Mayan EDMS ready to be used (initializes database,creates a secret key,etc).' def _generate_secret_key(self): chars = 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)' return get_random_string(50, chars) def handle(self, *args, **options): path = os.path.join(settings.BASE_DIR, 'mayan', 'settings','local.py') if os.path.exists(path): print 'Existing file at: {0}. Backup, remove this file and try again.'.format(path) Exit(1) with open(path, 'w+') as file_object: file_object.write('\n'.join([ 'from __future__ import absolute_import', '', 'from .base import *', '', "SECRET_KEY = '{0}'".format(self._generate_secret_key()),'', ])) management.call_command('syncdb', migrate=True, interactive=False)

Slide 86

Slide 86 text

$ ./manage.py initialsetup ● Secret key ● Create settings/local.py file ● Generate database w/out creating admin ● Don’t overwrite existing settings file https://github.com/mayan-edms/mayan-edms/blob/master/mayan/apps/main/management/c ommands/initialsetup.py

Slide 87

Slide 87 text

Initial Admin Creation

Slide 88

Slide 88 text

Most common admin passwords?

Slide 89

Slide 89 text

123456 admin password

Slide 90

Slide 90 text

Worst than common admin passwords?

Slide 91

Slide 91 text

Default admin passwords!

Slide 92

Slide 92 text

No content

Slide 93

Slide 93 text

http://www.defaultpassword.com/

Slide 94

Slide 94 text

# models.py from solo.models import SingletonModel class AutoAdminSingleton(SingletonModel): account = models.ForeignKey( User, null=True, blank=True, related_name='auto_admin_account' ) password = models.CharField( null=True, blank=True, max_length=128 ) password_hash = models.CharField( null=True, blank=True, max_length=128 ) class Meta: verbose_name = verbose_name_plural = _('Auto admin properties')

Slide 95

Slide 95 text

@receiver(post_migrate, dispatch_uid='create_superuser_and_anonymous_user') def create_superuser_and_anonymous_user(sender, **kwargs): ... assert auth_models.User.objects.create_superuser( AUTO_ADMIN_USERNAME,'[email protected]', AUTO_ADMIN_PASSWORD ) admin = auth_models.User.objects.get(username=AUTO_ADMIN_USERNAME) auto_admin_properties, created = AutoAdminSingleton.objects. get_or_create() auto_admin_properties.account = admin auto_admin_properties.password = AUTO_ADMIN_PASSWORD auto_admin_properties.password_hash = admin.password auto_admin_properties.save()

Slide 96

Slide 96 text

# settings.py register_setting( namespace='common', module='common.settings', name='AUTO_ADMIN_PASSWORD', global_name='COMMON_AUTO_ADMIN_PASSWORD', default=User.objects.make_random_password(), )

Slide 97

Slide 97 text

No content

Slide 98

Slide 98 text

# templatetags/autoadmin_tags.py @register.simple_tag(takes_context=True) def auto_admin_properties(context): context['auto_admin_properties'] = AutoAdminSingleton.objects.get() return ''

Slide 99

Slide 99 text

# login.html {% auto_admin_properties %} {% if auto_admin_properties.account %}

{% trans "First time login" %}

{% trans 'You have just finished installing Mayan EDMS,congratulations!' %}

{% trans 'Login using the following credentials:' %}

{% blocktrans with auto_admin_properties.account as account %}Username: {{ account }}{% endblocktrans %}

{% blocktrans with auto_admin_properties.account.email as email %}Email: {{ email }}{% endblocktrans %}

{% blocktrans with auto_admin_properties.password as password %}Password: {{ password }}{% endblocktrans %}

{% trans 'Be sure to change the password to increase security and to disable this message.' %}

{% endif %}

Slide 100

Slide 100 text

No content

Slide 101

Slide 101 text

@receiver(post_save, dispatch_uid='auto_admin_account_passwd_change', sender=User) def auto_admin_account_passwd_change(sender, instance, **kwargs): auto_admin_properties = AutoAdminSingleton.objects.get() if instance == auto_admin_properties.account and instance.password != auto_admin_properties.password_hash: # Only delete the auto admin properties when the password has been changed auto_admin_properties.account = None auto_admin_properties.password = None auto_admin_properties.password_hash = None auto_admin_properties.save()

Slide 102

Slide 102 text

https://pypi.python.org/pypi/django-autoadmin

Slide 103

Slide 103 text

Media Files

Slide 104

Slide 104 text

Not all deployment environment support the same Persistent File System setup ie: Docker, Stackato

Slide 105

Slide 105 text

register_settings( namespace='documents',module='documents.settings',settings=[ { 'name': 'STORAGE_BACKEND', 'global_name': 'DOCUMENTS_STORAGE_BACKEND', 'default': 'storage.backends.filebasedstorage.FileBasedStorage' }] ) https://github.com/mayan-edms/mayan-edms/blob/master/mayan/apps/documents/setting s.py

Slide 106

Slide 106 text

mysite/ manage.py project/ __init__.py settings/ __init__.py base.py urls.py wsgi.py apps/ __init__.py

Slide 107

Slide 107 text

mysite/ manage.py project/ __init__.py settings/ __init__.py base.py media/ db.sqlite document_storage/ gpg_home/ image_cache/ static/ urls.py wsgi.py apps/ __init__.py

Slide 108

Slide 108 text

_file_path = os.path.abspath(os.path.dirname(__file__)).split('/') BASE_DIR = '/'.join(_file_path[0:-2]) MEDIA_ROOT = os.path.join(BASE_DIR, 'mayan', 'media') DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': os.path.join(MEDIA_ROOT, 'db.sqlite3'), } } STATIC_ROOT = os.path.join(BASE_DIR, 'mayan', 'media', 'static') https://github.com/mayan-edms/mayan-edms/blob/master/mayan/settings/base.py

Slide 109

Slide 109 text

No content

Slide 110

Slide 110 text

# .gitignore ... *.sqlite3 gpg_home/ /mayan/media/static/ image_cache/ document_storage/

Slide 111

Slide 111 text

# MANIFEST.in ... global-exclude mayan/settings/local.py db.sqlite* mayan/media gpg_home document_storage image_cache

Slide 112

Slide 112 text

Post Install Management

Slide 113

Slide 113 text

$ /usr/share/mayan-edms/lib/python2.7/site-packages/mayan/manage.py collectstatic

Slide 114

Slide 114 text

No content

Slide 115

Slide 115 text

mysite/ manage.py project/ __init__.py settings/ __init__.py base.py media/ urls.py wsgi.py apps/ __init__.py

Slide 116

Slide 116 text

mysite/ manage.py project/ bin/ mayan-edms.py # Copy of manage.py __init__.py settings/ __init__.py base.py media/ urls.py wsgi.py apps/ __init__.py

Slide 117

Slide 117 text

# setup.py scripts=['mayan/bin/mayan-edms.py'],

Slide 118

Slide 118 text

$ virtualenv venv $ source venv/bin/activate $ pip install mayan-edms $ mayan-edms.py initialsetup $ mayan-edms.py runserver $ mayan-edms.py collectstatic --noinput

Slide 119

Slide 119 text

server { location /static/ { root /usr/share/mayan-edms/lib/python2.7/site- packages/mayan/media/static; } }

Slide 120

Slide 120 text

Guided Setup

Slide 121

Slide 121 text

No content

Slide 122

Slide 122 text

class MissingItem(object): _registry = [] @classmethod def get_all(cls): return cls._registry def __init__(self, label, condition, description, view): self.label = label self.condition = condition self.description = description self.view = view self.__class__._registry.append(self)

Slide 123

Slide 123 text

# /mayan/apps/documents/apps.py MissingItem( label=_('Create a document type'), description=_('Every uploaded document must be assigned a document type, it is the basic way Mayan EDMS categorizes documents.'), condition=lambda: not DocumentType.objects.exists(), view='documents:document_type_list' )

Slide 124

Slide 124 text

No content

Slide 125

Slide 125 text

http://buzzerg.com

Slide 126

Slide 126 text

Disadvantages ● Long paths ● Media file hidden depth inside virtualenv’s directories Solution Instead of ./manage.py initialsetup, a startsite command, homologous to Django’s startproject command https://github.com/mayan-edms/mayan-edms/issues/171

Slide 127

Slide 127 text

Demo

Slide 128

Slide 128 text

@siloraptor https://github.com/rosarior http://robertorosario.com “What is your question?”

Slide 129

Slide 129 text

Thank you! @siloraptor https://github.com/rosarior http://robertorosario.com Hint: Capital of Assyria: Assyria (Assur)