Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Maintaning a Django project after 10k commits

Maintaning a Django project after 10k commits

From Stéphane "Twidi" Angel and Joachim Jablon

Let's dive into the challenges that arise when a Django project starts to become huge.

Django is extremely effective for creating a website quickly with top notch features out of the box. But in some codebases, after a while, new developments can become harder and harder.

In this talk, we'll examine some design decisions that have, and haven't, scaled successfully, in the hope that the next time you start your big scale Django project, you won't end up cursing your past self after three years.

Among the subjects we'll tackle:

- Third parties, the problems they bring and a few solutions;
- Where to put your business logic to keep your sanity;
- How to archiecture your code for fun and profit;
- Testing strategies for large projects

Video available at :
https://youtu.be/oAV73PRRWNY?t=10577

Joachim Jablon

April 12, 2019
Tweet

More Decks by Joachim Jablon

Other Decks in Technology

Transcript

  1. MAINTAINING A DJANGO PROJECT AFTER 10.000 COMMITS Joachim Jablon -

    PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen
  2. MAINTAINING A DJANGO PROJECT AFTER 10.000 COMMITS Joachim Jablon -

    PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen
  3. MAINTAINING A DJANGO PROJECT AFTER BEFORE 10.000 COMMITS Joachim Jablon

    - PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen
  4. Our flagship project > 1h test, > 100 dependencies >

    100 models, > 1000 fields > 100M rows in some tables 7 Maintaining a Django project after 10.000 commits This is purely fictional, any resemblance to actual project living or dead is purely coincidental
  5. The project zoo Unit & Integration tests Health checks &

    monitoring Async & cron tasks APIs i18n & l10n Logging Error reporting Packaging Deployment code Dev environment Management commands Linting Custom tooling Changelog system Partial documentation Deprecated APIs Complex
 server-side UI Half-baked initiatives
 ("ongoing efforts") Dead code Memory leaks "Don't touch that code!" Spaghetti code Help Send help please 8 Maintaining a Django project after 10.000 commits
  6. Time flies Team turnover Company pivoting Scale and split Complexity

    grows Abandoned third parties 9 Maintaining a Django project after 10.000 commits
  7. THE OF OUR CODE THE ❤ OF OUR CODE THE

    OF OUR CODE THE ⚔ OF OUR CODE 10
  8. What is a third party dependency? 12 Maintaining a Django

    project after 10.000 commits Some external code used by your project
  9. Reasons to use 3rd party dependencies 13 Maintaining a Django

    project after 10.000 commits Lower the effort / time / cost Lower the risk
  10. Reasons NOT to use 3rd party dependencies 14 Maintaining a

    Django project after 10.000 commits Increase the effort / time / cost Increase the risk
  11. The problems 18 Maintaining a Django project after 10.000 commits

    Security updates Project direction ▸ Project lagging behind
  12. The problems 19 Maintaining a Django project after 10.000 commits

    Security updates Project direction Project lagging behind ▸ Contributions
  13. The problems 20 Maintaining a Django project after 10.000 commits

    Security updates Project direction Project lagging behind Contributions ▸ Project becoming inactive
  14. The problems 21 Maintaining a Django project after 10.000 commits

    Security updates Project direction Project lagging behind Contributions Project becoming inactive ▸ Partial need
  15. NEVER FORGET THE HIDDEN COSTS OF ADDING 3RD PARTIES Advice

    #1 Maintaining a Django project after 10.000 commits 22
  16. Sometimes, it's just ok You have ownership Complex scoped problem

    Official packages Django backend implementation Well known apps 24 Maintaining a Django project after 10.000 commits
  17. Testing Test the next versions It's ok it it fails

    At least, you know. 25 Maintaining a Django project after 10.000 commits
  18. Adapter Build an abstraction layer Only one module can import

    them
 (you may enforce this with bellybutton) No *args, **kwargs 31 Maintaining a Django project after 10.000 commits YOUR CODE MY_GITHUB.PY GITHUB CLIENT
  19. Special aside for django models You need to have the

    entire ownership on your database External app has migrations: ❌ External app has abstract models defining fields: ⚠ 33 Maintaining a Django project after 10.000 commits
  20. DON'T LET ANYONE DEFINE YOUR MODELS Advice #3 Maintaining a

    Django project after 10.000 commits 34
  21. DON'T FORGET TO CONTRIBUTE BACK TO OPEN SOURCE Advice #4

    Maintaining a Django project after 10.000 commits 35
  22. Could we imagine the same approach with Django itself? 36

    Maintaining a Django project after 10.000 commits
  23. What is business logic? 38 Maintaining a Django project after

    10.000 commits DJANGO BUSINESS LOGIC HTTP DATABASE
  24. What is business logic? 39 Maintaining a Django project after

    10.000 commits DJANGO FLASK BUSINESS LOGIC HTTP DATABASE
  25. What is business logic? "When a user is created, they

    get a welcome email." "Only premium users can access this page." "On a course, start date must be before end date." 40 Maintaining a Django project after 10.000 commits
  26. The cause There's business logic in my templates ... And

    in my forms & serializers ... And in my views ... And in my models & managers And all these modules already have another purpose 42 Maintaining a Django project after 10.000 commits
  27. Service layer: 2 talks to watch Hanna Kollo
 Avoiding Monoliths

    http://bit.ly/dce15-monolith 43 Maintaining a Django project after 10.000 commits Radoslav Georgiev
 Django structure for scale and longevity http://bit.ly/ep18-django-scale
  28. A DJANGO APP Service layers: key takeaways Every non-trivial operation


    should be a service / selector: services.py for create / update selectors.py for get / list no form.save(), serializer.save(), CreateView etc 44 Maintaining a Django project after 10.000 commits From Radoslav Georgiev
 Django structure for scale and longevity SERVICES.PY SELECTORS.PY MODELS.PY FORMS.PY
  29. Service layer 45 Maintaining a Django project after 10.000 commits

    HTTP DATABASE BUSINESS LOGIC DJANGO VIEWS, FORMS, ETC ORM
  30. The sandwich architecture 46 Maintaining a Django project after 10.000

    commits HTTP DATABASE PURE PYTHON BUSINESS LOGIC DJANGO VIEWS, FORMS, ETC ORM
  31. And beyond? 47 Maintaining a Django project after 10.000 commits

    HTTP DATABASE DJANGO VIEWS, FORMS, ETC ORM PURE PYTHON BUSINESS LOGIC
  32. And beyond? 48 Maintaining a Django project after 10.000 commits

    PURE PYTHON
 BUSINESS LOGIC HTTP, DATABASE DJANGO VIEWS, FORMS
  33. SPLIT YOUR BUSINESS LOGIC, DJANGO VIEWS AND THE ORM Advice

    #5 Maintaining a Django project after 10.000 commits 49 Did I hear anyone say "MVC"?
  34. Is this Java? ... But then so what? 50 Maintaining

    a Django project after 10.000 commits
  35. Software architecture Put things in the right boxes Name your

    boxes Don't mix boxes Define links between boxes 52 Maintaining a Django project after 10.000 commits < https://www.pexels.com/photo/gem-stones-souvenir-952681/
  36. Boxes Functional core / Imperative shell + Hoist your I/Os

    The Hexagonal Architecture Domain Driven Design 53 Maintaining a Django project after 10.000 commits
  37. Functional core / Imperative shell Gary Bernhardt Separating the logic

    from the glue 54 Maintaining a Django project after 10.000 commits
  38. Functional core / Imperative shell Decision making lives in
 functions

    without side effects 55 Maintaining a Django project after 10.000 commits FUNCTIONAL CORE FUNCTIONAL CORE FUNCTIONAL CORE
  39. Functional core / Imperative shell The outer shell has all


    the side effects but no logic 56 Maintaining a Django project after 10.000 commits IMPERATIVE SHELL
  40. Functional core / Imperative shell 57 Maintaining a Django project

    after 10.000 commits FUNCTIONAL CORE FUNCTIONAL CORE FUNCTIONAL CORE IMPERATIVE SHELL IMPERATIVE SHELL IMPERATIVE SHELL From Gary Bernhardt
 Boundaries
  41. Quiz time! def main(): with open("/etc/hosts") as file: for line

    in parse_hosts(file): print(line) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line 58 Maintaining a Django project after 10.000 commits def main(): parse_hosts() def parse_hosts(): with open("/etc/hosts") as lines: for line in lines: if line.startswith("#"): continue print(line) From Brandon Rhodes
 Hoisting your I/Os
  42. Requirements have changed def main(): with open("/etc/hosts") as file: for

    line in parse_hosts(file): print(line) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line 59 Maintaining a Django project after 10.000 commits def main(): lines = requests.get("hosts.com").json(): with open("local_hosts", "w") as file: file.writelines(parse_hosts(lines)) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line From Brandon Rhodes
 Hoisting your I/Os
  43. Hoist your I/Os Brandon Rhodes You're not reducing complexity
 by

    burying I/O down the stack Code manipulates
 plain data structures,
 I/O in the top level glue 60 Maintaining a Django project after 10.000 commits http://bit.ly/hoist-your-ios
  44. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 61

    Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN
  45. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 62

    Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN
  46. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 63

    Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN
  47. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 64

    Maintaining a Django project after 10.000 commits HTTP, DATABASE DJANGO VIEWS, FORMS PURE PYTHON
 BUSINESS LOGIC
  48. The Clean Architecture in Python The Clean Architecture in Python


    Brandon Rhodes, PyOhio 2014 http://bit.ly/clean-architecture-python 65 Maintaining a Django project after 10.000 commits
  49. Domain Driven Design A concept by Eric Evans A LOT

    of things, at every level 66 Maintaining a Django project after 10.000 commits
  50. Domain Driven Design Domain Curiosity Bounded Contexts Ubiquitous Language 67

    Maintaining a Django project after 10.000 commits
  51. Being driven by your trade, not by technology The tale

    of the customer address 68 Maintaining a Django project after 10.000 commits USERNAME NAME ADDRESS ZIP_CODE CUSTOMER Example from Cyrille Martraire - DDD, en vrai pour le développeur
  52. Being driven by your trade, not by technology The tale

    of the customer address 69 Maintaining a Django project after 10.000 commits USERNAME NAME ADDRESS ZIP_CODE ADDRESS_BILLING ZIP_CODE_BILLING CUSTOMER From Cyrille Martraire - DDD, en vrai pour le développeur
  53. Being driven by your trade, not by technology The tale

    of the customer address 70 Maintaining a Django project after 10.000 commits USERNAME >SHIPPING ADDRESS >BILLING ADDRESS CUSTOMER NAME ADDRESS ZIP_CODE ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur
  54. Being driven by your trade, not by technology The tale

    of the customer address 71 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE SHIPPING ADDRESS NAME ADDRESS ZIP_CODE BILLING ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur
  55. Being driven by your trade, not by technology The tale

    of the customer address 72 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE RECIPIENT NAME ADDRESS ZIP_CODE BILLING ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur
  56. Being driven by your trade, not by technology The tale

    of the customer address 73 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE RECIPIENT NAME ADDRESS ZIP_CODE ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur
  57. Being driven by your trade, not by technology The tale

    of the customer address 74 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE BUILDING _CODE DELIVERY_HOURS RECIPIENT NAME ADDRESS ZIP_CODE ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur
  58. Being driven by your trade, not by technology The tale

    of the customer address 75 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE BUILDING _CODE DELIVERY_HOURS RECIPIENT NAME ADDRESS ZIP_CODE TAX_CODE CURRENCY ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur
  59. YOU'RE NOT "JUST" BUILDING A WEB APP Advice #6 Maintaining

    a Django project after 10.000 commits 76
  60. The shoulders of giants 77 Maintaining a Django project after

    10.000 commits THE CLEAN ARCHITECTURE BRANDON RHODES 2014 BOUNDARIES GARY BERNHARDT 2012 FAST TEST, SLOW TEST GARY BERNHARDT 2012 HOISTING YOUR I/O BRANDON RHODES 2015 RADOSLAV GEORGIEV DJANGO STRUCTURE... 2018 THE CLEAN ARCHITECTURE ROBERT MARTIN 2012 HEXAGONAL ARCHITECTURE ALISTAIR COCKBURN 2005
  61. STAND ON THE SHOULDERS OF GIANTS (OR TALLER FOLKS) Advice

    #7 Maintaining a Django project after 10.000 commits 78
  62. The test pyramid 80 Maintaining a Django project after 10.000

    commits UNIT INTEGRATION FUNCTIONAL X10? X30? 30 SEC 1 SEC 10 MS
  63. What to test where for a Django view? 81 Maintaining

    a Django project after 10.000 commits UNIT BUSINESS LOGIC VIEW PERSISTENCE INTEGRATION FUNCTIONAL TEMPLATE NAME, URL PAGE CONTENT FLOW CONTEXT
  64. DON'T JUST WRITE TESTS,
 WRITE THE RIGHT TESTS. Advice #8

    Maintaining a Django project after 10.000 commits 82
  65. Code quality criteria in tests Is a criteria: Dead easy

    to read,
 from start to tear down One test tests one thing Have a long
 and descriptive name 83 Maintaining a Django project after 10.000 commits Is not a criteria: Smart adaptable code DRY - Don't repeat yourself Layering and
 hidden complexity
  66. IN YOUR TESTS, AIM FOR THE SIMPLEST
 TO READ AND

    WRITE Advice #9 Maintaining a Django project after 10.000 commits 84
  67. Things that get complicated in the long run Test mixins

    self.client.get()/post() Fixture files 85 Maintaining a Django project after 10.000 commits
  68. The right tools for the job Use pytest and pytest

    style function tests Use pytest fixtures (avoid auto-use) And use Factory Boy 86 Maintaining a Django project after 10.000 commits
  69. Tests: going further 87 Maintaining a Django project after 10.000

    commits Behavior Driven Development Snapshot testing
  70. Behavior Driven Development Feature: Blog A site where you can

    publish your articles. Scenario: Publishing the article Given I'm an author user And I have an article When I go to the article page And I press the publish button Then the article should be published 88 Maintaining a Django project after 10.000 commits
  71. 89 Maintaining a Django project after 10.000 commits @scenario('publish_article.feature', 'Publishing

    the article') def test_publish(): ... @given("I'm an author user") def author_user(auth, author): ... @given('I have an article') def article(author): ... @when('I go to the article page') def go_to_article(article, browser): ... @when('I press the publish button') def publish_article(browser): ... @then('the article should be published') def article_is_published(article): ...
  72. Behavior Driven Development Document your specifications by design Functional tests

    from day 1 90 Maintaining a Django project after 10.000 commits
  73. Snapshot testing is useful too Screenshot testing SQL testing (django-perf-rec)

    HTML testing? API output testing? 91 Maintaining a Django project after 10.000 commits
  74. MAKE YOUR TESTS A PRIME PART OF YOUR PROCESS Advice

    #10 Maintaining a Django project after 10.000 commits 92
  75. STAY CURIOUS, TRY THINGS, THINK BEFORE YOU CODE Final advice

    Maintaining a Django project after 10.000 commits 93
  76. 94 Maintaining a Django project after 10.000 commits STAY CURIOUS,

    TRY THINGS, THINK BEFORE YOU CODE Final advice Final-Final advice DON'T OVER-ENGINEER IT!