Maintaning a Django project after 10k commits

From Stéphane "Twidi" Angel and Joachim Jablon

Let's dive into the challenges that arise when a Django project starts to become huge.

Django is extremely effective for creating a website quickly with top notch features out of the box. But in some codebases, after a while, new developments can become harder and harder.

In this talk, we'll examine some design decisions that have, and haven't, scaled successfully, in the hope that the next time you start your big scale Django project, you won't end up cursing your past self after three years.

Among the subjects we'll tackle:

- Third parties, the problems they bring and a few solutions;
- Where to put your business logic to keep your sanity;
- How to archiecture your code for fun and profit;
- Testing strategies for large projects

Video available at :
https://youtu.be/oAV73PRRWNY?t=10577

Transcript

1. MAINTAINING A DJANGO PROJECT AFTER 10.000 COMMITS Joachim Jablon - PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen
2. WHO ARE WE? 2 Joachim Jablon / ewjoachim @ewjoachim / ewjoach.im joachim@jablon.fr
3. WHO ARE WE? 3 Stéphane "Twidi" Angel @twidi / isshub.io
4. MAINTAINING A DJANGO PROJECT AFTER 10.000 COMMITS Joachim Jablon - PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen
5. MAINTAINING A DJANGO PROJECT AFTER BEFORE 10.000 COMMITS Joachim Jablon - PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen
6. MONOLI.TH 6 Let's disrupt the stone business with < https://fr.m.wikipedia.org/wiki/Fichier:Kerloas_menhir.JPG
7. Our flagship project > 1h test, > 100 dependencies > 100 models, > 1000 fields > 100M rows in some tables 7 Maintaining a Django project after 10.000 commits This is purely fictional, any resemblance to actual project living or dead is purely coincidental
8. The project zoo Unit & Integration tests Health checks & monitoring Async & cron tasks APIs i18n & l10n Logging Error reporting Packaging Deployment code Dev environment Management commands Linting Custom tooling Changelog system Partial documentation Deprecated APIs Complex
 server-side UI Half-baked initiatives
 ("ongoing efforts") Dead code Memory leaks "Don't touch that code!" Spaghetti code Help Send help please 8 Maintaining a Django project after 10.000 commits
9. Time flies Team turnover Company pivoting Scale and split Complexity grows Abandoned third parties 9 Maintaining a Django project after 10.000 commits
10. THE OF OUR CODE THE ❤ OF OUR CODE THE OF OUR CODE THE ⚔ OF OUR CODE 10
11. THIRD PARTY DEPENDENCIES 11 Part 1 - https://commons.wikimedia.org/wiki/File:Coastal-rocks.jpg >
12. What is a third party dependency? 12 Maintaining a Django project after 10.000 commits Some external code used by your project
13. Reasons to use 3rd party dependencies 13 Maintaining a Django project after 10.000 commits Lower the effort / time / cost Lower the risk
14. Reasons NOT to use 3rd party dependencies 14 Maintaining a Django project after 10.000 commits Increase the effort / time / cost Increase the risk
15. The problems 15 Maintaining a Django project after 10.000 commits
16. The problems 16 Maintaining a Django project after 10.000 commits ▸ Security updates
17. The problems 17 Maintaining a Django project after 10.000 commits Security updates ▸ Project direction
18. The problems 18 Maintaining a Django project after 10.000 commits Security updates Project direction ▸ Project lagging behind
19. The problems 19 Maintaining a Django project after 10.000 commits Security updates Project direction Project lagging behind ▸ Contributions
20. The problems 20 Maintaining a Django project after 10.000 commits Security updates Project direction Project lagging behind Contributions ▸ Project becoming inactive
21. The problems 21 Maintaining a Django project after 10.000 commits Security updates Project direction Project lagging behind Contributions Project becoming inactive ▸ Partial need
22. NEVER FORGET THE HIDDEN COSTS OF ADDING 3RD PARTIES Advice #1 Maintaining a Django project after 10.000 commits 22
23. All is not lost 23 Maintaining a Django project after 10.000 commits
24. Sometimes, it's just ok You have ownership Complex scoped problem Official packages Django backend implementation Well known apps 24 Maintaining a Django project after 10.000 commits
25. Testing Test the next versions It's ok it it fails At least, you know. 25 Maintaining a Django project after 10.000 commits
26. Participate Issues / PR Be nice 26 Maintaining a Django project after 10.000 commits
27. Taking ownership 27 Maintaining a Django project after 10.000 commits
28. Taking ownership ▸ Fork 28 Maintaining a Django project after 10.000 commits
29. Taking ownership Fork ▸ Vendor 29 Maintaining a Django project after 10.000 commits
30. Taking ownership Fork Vendor ▸ Implement 30 Maintaining a Django project after 10.000 commits
31. Adapter Build an abstraction layer Only one module can import them
 (you may enforce this with bellybutton) No *args, **kwargs 31 Maintaining a Django project after 10.000 commits YOUR CODE MY_GITHUB.PY GITHUB CLIENT
32. ISOLATE YOUR
 THIRD PARTIES Advice #2 Maintaining a Django project after 10.000 commits 32
33. Special aside for django models You need to have the entire ownership on your database External app has migrations: ❌ External app has abstract models defining fields: ⚠ 33 Maintaining a Django project after 10.000 commits
34. DON'T LET ANYONE DEFINE YOUR MODELS Advice #3 Maintaining a Django project after 10.000 commits 34
35. DON'T FORGET TO CONTRIBUTE BACK TO OPEN SOURCE Advice #4 Maintaining a Django project after 10.000 commits 35
36. Could we imagine the same approach with Django itself? 36 Maintaining a Django project after 10.000 commits
37. BUSINESS LOGIC 37 Part 2 - ❤ < https://pxhere.com/en/photo/597168
38. What is business logic? 38 Maintaining a Django project after 10.000 commits DJANGO BUSINESS LOGIC HTTP DATABASE
39. What is business logic? 39 Maintaining a Django project after 10.000 commits DJANGO FLASK BUSINESS LOGIC HTTP DATABASE
40. What is business logic? "When a user is created, they get a welcome email." "Only premium users can access this page." "On a course, start date must be before end date." 40 Maintaining a Django project after 10.000 commits
41. The problem 41 Maintaining a Django project after 10.000 commits BUSINESS LOGIC DJANGO HTTP DATABASE
42. The cause There's business logic in my templates ... And in my forms & serializers ... And in my views ... And in my models & managers And all these modules already have another purpose 42 Maintaining a Django project after 10.000 commits
43. Service layer: 2 talks to watch Hanna Kollo
 Avoiding Monoliths http://bit.ly/dce15-monolith 43 Maintaining a Django project after 10.000 commits Radoslav Georgiev
 Django structure for scale and longevity http://bit.ly/ep18-django-scale
44. A DJANGO APP Service layers: key takeaways Every non-trivial operation
 should be a service / selector: services.py for create / update selectors.py for get / list no form.save(), serializer.save(), CreateView etc 44 Maintaining a Django project after 10.000 commits From Radoslav Georgiev
 Django structure for scale and longevity SERVICES.PY SELECTORS.PY MODELS.PY FORMS.PY
45. Service layer 45 Maintaining a Django project after 10.000 commits HTTP DATABASE BUSINESS LOGIC DJANGO VIEWS, FORMS, ETC ORM
46. The sandwich architecture 46 Maintaining a Django project after 10.000 commits HTTP DATABASE PURE PYTHON BUSINESS LOGIC DJANGO VIEWS, FORMS, ETC ORM
47. And beyond? 47 Maintaining a Django project after 10.000 commits HTTP DATABASE DJANGO VIEWS, FORMS, ETC ORM PURE PYTHON BUSINESS LOGIC
48. And beyond? 48 Maintaining a Django project after 10.000 commits PURE PYTHON
 BUSINESS LOGIC HTTP, DATABASE DJANGO VIEWS, FORMS
49. SPLIT YOUR BUSINESS LOGIC, DJANGO VIEWS AND THE ORM Advice #5 Maintaining a Django project after 10.000 commits 49 Did I hear anyone say "MVC"?
50. Is this Java? ... But then so what? 50 Maintaining a Django project after 10.000 commits
51. BOXES 51 Part 3 - https://www.flickr.com/photos/ikewinski/8013010600 >
52. Software architecture Put things in the right boxes Name your boxes Don't mix boxes Define links between boxes 52 Maintaining a Django project after 10.000 commits < https://www.pexels.com/photo/gem-stones-souvenir-952681/
53. Boxes Functional core / Imperative shell + Hoist your I/Os The Hexagonal Architecture Domain Driven Design 53 Maintaining a Django project after 10.000 commits
54. Functional core / Imperative shell Gary Bernhardt Separating the logic from the glue 54 Maintaining a Django project after 10.000 commits
55. Functional core / Imperative shell Decision making lives in
 functions without side effects 55 Maintaining a Django project after 10.000 commits FUNCTIONAL CORE FUNCTIONAL CORE FUNCTIONAL CORE
56. Functional core / Imperative shell The outer shell has all
 the side effects but no logic 56 Maintaining a Django project after 10.000 commits IMPERATIVE SHELL
57. Functional core / Imperative shell 57 Maintaining a Django project after 10.000 commits FUNCTIONAL CORE FUNCTIONAL CORE FUNCTIONAL CORE IMPERATIVE SHELL IMPERATIVE SHELL IMPERATIVE SHELL From Gary Bernhardt
 Boundaries
58. Quiz time! def main(): with open("/etc/hosts") as file: for line in parse_hosts(file): print(line) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line 58 Maintaining a Django project after 10.000 commits def main(): parse_hosts() def parse_hosts(): with open("/etc/hosts") as lines: for line in lines: if line.startswith("#"): continue print(line) From Brandon Rhodes
 Hoisting your I/Os
59. Requirements have changed def main(): with open("/etc/hosts") as file: for line in parse_hosts(file): print(line) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line 59 Maintaining a Django project after 10.000 commits def main(): lines = requests.get("hosts.com").json(): with open("local_hosts", "w") as file: file.writelines(parse_hosts(lines)) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line From Brandon Rhodes
 Hoisting your I/Os
60. Hoist your I/Os Brandon Rhodes You're not reducing complexity
 by burying I/O down the stack Code manipulates
 plain data structures,
 I/O in the top level glue 60 Maintaining a Django project after 10.000 commits http://bit.ly/hoist-your-ios
61. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 61 Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN
62. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 62 Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN
63. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 63 Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN
64. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 64 Maintaining a Django project after 10.000 commits HTTP, DATABASE DJANGO VIEWS, FORMS PURE PYTHON
 BUSINESS LOGIC
65. The Clean Architecture in Python The Clean Architecture in Python
 Brandon Rhodes, PyOhio 2014 http://bit.ly/clean-architecture-python 65 Maintaining a Django project after 10.000 commits
66. Domain Driven Design A concept by Eric Evans A LOT of things, at every level 66 Maintaining a Django project after 10.000 commits
67. Domain Driven Design Domain Curiosity Bounded Contexts Ubiquitous Language 67 Maintaining a Django project after 10.000 commits
68. Being driven by your trade, not by technology The tale of the customer address 68 Maintaining a Django project after 10.000 commits USERNAME NAME ADDRESS ZIP_CODE CUSTOMER Example from Cyrille Martraire - DDD, en vrai pour le développeur
69. Being driven by your trade, not by technology The tale of the customer address 69 Maintaining a Django project after 10.000 commits USERNAME NAME ADDRESS ZIP_CODE ADDRESS_BILLING ZIP_CODE_BILLING CUSTOMER From Cyrille Martraire - DDD, en vrai pour le développeur
70. Being driven by your trade, not by technology The tale of the customer address 70 Maintaining a Django project after 10.000 commits USERNAME >SHIPPING ADDRESS >BILLING ADDRESS CUSTOMER NAME ADDRESS ZIP_CODE ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur
71. Being driven by your trade, not by technology The tale of the customer address 71 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE SHIPPING ADDRESS NAME ADDRESS ZIP_CODE BILLING ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur
72. Being driven by your trade, not by technology The tale of the customer address 72 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE RECIPIENT NAME ADDRESS ZIP_CODE BILLING ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur
73. Being driven by your trade, not by technology The tale of the customer address 73 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE RECIPIENT NAME ADDRESS ZIP_CODE ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur
74. Being driven by your trade, not by technology The tale of the customer address 74 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE BUILDING _CODE DELIVERY_HOURS RECIPIENT NAME ADDRESS ZIP_CODE ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur
75. Being driven by your trade, not by technology The tale of the customer address 75 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE BUILDING _CODE DELIVERY_HOURS RECIPIENT NAME ADDRESS ZIP_CODE TAX_CODE CURRENCY ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur
76. YOU'RE NOT "JUST" BUILDING A WEB APP Advice #6 Maintaining a Django project after 10.000 commits 76
77. The shoulders of giants 77 Maintaining a Django project after 10.000 commits THE CLEAN ARCHITECTURE BRANDON RHODES 2014 BOUNDARIES GARY BERNHARDT 2012 FAST TEST, SLOW TEST GARY BERNHARDT 2012 HOISTING YOUR I/O BRANDON RHODES 2015 RADOSLAV GEORGIEV DJANGO STRUCTURE... 2018 THE CLEAN ARCHITECTURE ROBERT MARTIN 2012 HEXAGONAL ARCHITECTURE ALISTAIR COCKBURN 2005
78. STAND ON THE SHOULDERS OF GIANTS (OR TALLER FOLKS) Advice #7 Maintaining a Django project after 10.000 commits 78
79. TESTS 79 Part 4 - ⚔ < Eduardo Otubo [CC BY 2.0]
80. The test pyramid 80 Maintaining a Django project after 10.000 commits UNIT INTEGRATION FUNCTIONAL X10? X30? 30 SEC 1 SEC 10 MS
81. What to test where for a Django view? 81 Maintaining a Django project after 10.000 commits UNIT BUSINESS LOGIC VIEW PERSISTENCE INTEGRATION FUNCTIONAL TEMPLATE NAME, URL PAGE CONTENT FLOW CONTEXT
82. DON'T JUST WRITE TESTS,
 WRITE THE RIGHT TESTS. Advice #8 Maintaining a Django project after 10.000 commits 82
83. Code quality criteria in tests Is a criteria: Dead easy to read,
 from start to tear down One test tests one thing Have a long
 and descriptive name 83 Maintaining a Django project after 10.000 commits Is not a criteria: Smart adaptable code DRY - Don't repeat yourself Layering and
 hidden complexity
84. IN YOUR TESTS, AIM FOR THE SIMPLEST
 TO READ AND WRITE Advice #9 Maintaining a Django project after 10.000 commits 84
85. Things that get complicated in the long run Test mixins self.client.get()/post() Fixture files 85 Maintaining a Django project after 10.000 commits
86. The right tools for the job Use pytest and pytest style function tests Use pytest fixtures (avoid auto-use) And use Factory Boy 86 Maintaining a Django project after 10.000 commits
87. Tests: going further 87 Maintaining a Django project after 10.000 commits Behavior Driven Development Snapshot testing
88. Behavior Driven Development Feature: Blog A site where you can publish your articles. Scenario: Publishing the article Given I'm an author user And I have an article When I go to the article page And I press the publish button Then the article should be published 88 Maintaining a Django project after 10.000 commits
89. 89 Maintaining a Django project after 10.000 commits @scenario('publish_article.feature', 'Publishing the article') def test_publish(): ... @given("I'm an author user") def author_user(auth, author): ... @given('I have an article') def article(author): ... @when('I go to the article page') def go_to_article(article, browser): ... @when('I press the publish button') def publish_article(browser): ... @then('the article should be published') def article_is_published(article): ...
90. Behavior Driven Development Document your specifications by design Functional tests from day 1 90 Maintaining a Django project after 10.000 commits
91. Snapshot testing is useful too Screenshot testing SQL testing (django-perf-rec) HTML testing? API output testing? 91 Maintaining a Django project after 10.000 commits
92. MAKE YOUR TESTS A PRIME PART OF YOUR PROCESS Advice #10 Maintaining a Django project after 10.000 commits 92
93. STAY CURIOUS, TRY THINGS, THINK BEFORE YOU CODE Final advice Maintaining a Django project after 10.000 commits 93
94. 94 Maintaining a Django project after 10.000 commits STAY CURIOUS, TRY THINGS, THINK BEFORE YOU CODE Final advice Final-Final advice DON'T OVER-ENGINEER IT!
95. THANK YOU! QUESTIONS? 95 Joachim Jablon - @ewjoachim Stéphane "Twidi" Angel - @Twidi We're hiring! Try it out!