Scaling Django Codebases

Tips, tricks and lessons learned from working on a Django site with over 300 apps.

Transcript

1. Scaling Django Codebases Dan Palmer @danpalmer Thread https://www.thread.com/
2. About this talk Beginners • Learn how to structure a large Django site. Intermediate • Learn some ideas you can apply to your projects. Experts • Hear pros/cons of another way.
4. Thread Online personal-styling service A completely curated store Our Django site: • Ecommerce store with personalisation. • Payment and Delivery integrations. • CRM–like system for stylists. • Warehouse, order and supplier management. • Reporting. • Lots of web scraping and data cleaning.
5. Our codebase ‣ 1 Django site ‣ 364 first-party Django apps ‣ 368 first-party models ‣ 147 management commands ‣ 1058 URLconfs ‣ 65 LOC per file on average ‣ ~135k lines of Python, ~60k lines of Templates
6. Apps “An app is a Web application that does something” “Advanced tutorial: How to write reusable apps” “Reusability matters”
7. Apps ‣ Great for separation of concerns ‣ Great for rapid development ‣ Best part of Django
8. Apps What about circular imports?! • Not much of a problem in practice • Try to only import up the module tree What about the global namespace?
9. ‣ styleme ‣ orders ‣ account ‣ cart ‣ inventory ‣ stock_levels ‣ inventory_review ‣ inventory_review_category_review ‣ inventory_review_colour_mapping Example
10. Rule of thumb Divide now. It’s easier to merge than to separate.
11. Files in Apps ‣models.py ‣views.py ‣urls.py ‣forms.py ‣admin.py ‣tests.py ‣tasks.py ‣managers.py ‣fields.py ‣middleware.py ‣decorators.py ‣exceptions.py ‣utils.py ‣api.py ‣factories.py ‣enums.py ‣cron.py ‣prometheus.py ‣reports.py ‣sitemaps.py
12. Naming ‣ Python and Django style guides favour consistency ‣ Naming consistency through URL names, views, template hierarchy, app and module names ‣ Named URLs everywhere ‣ Model, relation, and field naming
14. Models ‣ Relational databases are great ‣ Django migrations are great ‣ Don’t import things into them ‣ Avoid Signals and overriding save ‣ Small models are great (see DjangoAutoOneToOne)
16. Just Python “A core lesson here is that settings files are just Python code.” “This works because there’s nothing special about generic views—they’re just Python functions.” “There’s nothing special about URLconfs—like anything else in Django, they’re just Python code.”
17. Just Python ‣ Extract non-Django parts into new modules ‣ Easier to test, encourages good APIs ‣ Example: Tip selection for users ‣ Django “backends” pattern. ‣ Example: Tasks, SMS, Payments, Recommendations ‣ Subsystems in management commands ‣ Example: Scraping
18. Testing ‣ Tests that write themselves ‣ Querying on import, template syntax ‣ Testing each registration funnel ‣ Custom assertions ‣ Sent email, sent SMS, enqueued task ‣ Assert barcode scanned successfully/with error
19. More Apps Be Consistent Follow Django Patterns Just Python
20. Thank you
21. Join us
22. Questions
23. Links ‣ Scaling Django to 8 billion page views:
 https://blog.disqus.com/scaling-django-to-8-billion-page-views ‣ Django documentation for apps:
 https://docs.djangoproject.com/en/1.11/ref/applications/ ‣ The Definitive Guide to Django (out of date):
 https://www.amazon.co.uk/Definitive-Guide-Django-Development- Right/dp/1590597257 ‣ Django Auto One-to-One:
 https://github.com/lamby/django-auto-one-to-one ‣ Thread’s Django Style Guide
 https://dev.thread.com/thread-style