django-storages NYC Django Meetup

A tour of the Django files abstraction combined with the war stories of me obtaining the django-storages packages and the trials and tribulations of a maintainer of a decently popular open source package.

Transcript

1. django-storages An Open Source Perspective
2. Me • Poker Player • Visual Revenue / Outbrain • Started programming in college • Hacker School
3. Files in Django • Static vs User Uploaded • Settings galore STATIC_ROOT STATICFILE_DIRS STATIC_URL STATICFILES_STORAGE STATICFILES_FINDERS Static Settings File Upload/Media Settings DEFAULT_FILE_STORAGE FILE_CHARSET FILE_UPLOAD_HANDLERS MEDIA_ROOT FILE_UPLOAD_PERMISSIONS FILE_UPLOAD_TEMP_DIR MEDIA_URL FILE_UPLOAD_MAX_MEMORY_SIZE
4. More Files in Django • django.core.files.File • Extremely minor abstraction • Usually used via FileField & ImageField
5. Custom Storage Backends django.core.files.storage.Storage Common Interface _open _save delete exists listdir size url get_valid_name get_available_name path accessed_time created_time modified_time
6. django-storages: History / Overview • Started by David Larlet (bitbucket.org/david) • First commit June 12, 2008 • First storage S3, followed quickly by ImageStorage, Overwrite, and Mogile
7. Abandoned • No commit since March 2014 • No release to PyPI since March 2013 • 88 Issues & 28 Pull Requests • Wealth of bugfixes in master • No Python3 Support
8. My Involvement Global Citizen
9. My Involvement • Look for alternatives - mostly lacking • No backport of fixes from main repo • Don’t test • No CI • Don’t support older Django versions Conclusion -> Fork!!
10. Fork • Adding Py3K support was fast • How did I end up with the official successor? • Mostly marketing • + avoiding the other mistakes I mentioned - a nice README and CHANGELOG never hurt either
11. Official Successor py3readiness.org
12. Personal History in Open Source • First contribution to Mozilla’s mozillians • Number of contributions to urllib3 • Various small fixes to any number of projects - Django, wagtail, Twisted, requests, pip, django-countries, fastlane etc etc • Always happy to give back
13. Maintaining something people actually use • Open source is hard • Volunteering free time, hard to find sometimes • Totally understand people getting frustrated • Constantly feel ashamed • Half the time I just delete emails about new issues and pull requests • Extremely concious of not breaking backwards compat - people are running businesses on top of this software
14. Maintaining Sanity • Basically duplicated urllib3’s contribution and guidance • Having an AUTHORS file • Requiring tests for new features • Friendly language for bothering the maintainer, everything is my fault :p
15. Tech Debt in django-storages
16. Settings • Library is essentially the intersection of 3 things: Lots of settings, the python driver for the backend and the Storage interface • Django provides robust tools for testing different settings - namely override_settings from django.test import TestCase, override_settings @override_settings(LOGIN_URL='/other/login/') class LoginTestCase(TestCase): def test_login(self): response = self.client.get('/sekrit/') self.assertRedirects(response, '/other/login/?next=/sekrit/')
17. Settings (continued) • So what’s the problem? @deconstructible class S3BotoStorage(Storage): # ...snip default_acl = setting('AWS_DEFAULT_ACL', 'public-read') bucket_acl = setting('AWS_BUCKET_ACL', default_acl) from django.conf import settings def setting(name, default=None): return getattr(settings, name, default) See the problem?
18. Settings (continued) • Makes testing annoying / a bit of a mess def test_new_file_modified_time(self): self.storage.preload_metadata = True name = 'test_storage_save.txt' content = ContentFile('new content') utcnow = datetime.datetime.utcnow() with mock.patch('storages.backends.s3boto.datetime') as mock_datetime: mock_datetime.utcnow.return_value = utcnow self.storage.save(name, content) self.assertEqual(self.storage.modified_time(name), parse_ts(utcnow.strftime(ISO8601))) • Why not just switch? • Have to wait for a 2.0 (i.e. soon)
19. The UNIX Way • Do one thing and do it well • Why did this library “win”, for some definition of win? • Why is this library used/the most popular recommendation?
20. The Way Forward • Boto3 - Long requested - PR Open • Documentation - absolutely critical but also critically lacking at the moment • New storages are constantly added • Just this morning an issue was opened asking about desire for Ceph, Swift and Riak backends
21. The Way Forward (continued) • Encourage designated sub-maintainers? • Difficult to care about modules that I don’t use • Mostly maintenance mode at the moment Interested in helping? Talk to me after or join me on GitHub!
22. Questions?