Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Maintaning a Django project after 10k commits

Maintaning a Django project after 10k commits

From Stéphane "Twidi" Angel and Joachim Jablon

Let's dive into the challenges that arise when a Django project starts to become huge.

Django is extremely effective for creating a website quickly with top notch features out of the box. But in some codebases, after a while, new developments can become harder and harder.

In this talk, we'll examine some design decisions that have, and haven't, scaled successfully, in the hope that the next time you start your big scale Django project, you won't end up cursing your past self after three years.

Among the subjects we'll tackle:

- Third parties, the problems they bring and a few solutions;
- Where to put your business logic to keep your sanity;
- How to archiecture your code for fun and profit;
- Testing strategies for large projects

Video available at :
https://youtu.be/oAV73PRRWNY?t=10577

E122b75d171a1029ba808f7e26af1820?s=128

Joachim Jablon

April 12, 2019
Tweet

More Decks by Joachim Jablon

Other Decks in Technology

Transcript

  1. MAINTAINING A DJANGO PROJECT AFTER 10.000 COMMITS Joachim Jablon -

    PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen
  2. WHO ARE WE? 2 Joachim Jablon / ewjoachim @ewjoachim /

    ewjoach.im joachim@jablon.fr
  3. WHO ARE WE? 3 Stéphane "Twidi" Angel @twidi / isshub.io

  4. MAINTAINING A DJANGO PROJECT AFTER 10.000 COMMITS Joachim Jablon -

    PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen
  5. MAINTAINING A DJANGO PROJECT AFTER BEFORE 10.000 COMMITS Joachim Jablon

    - PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen
  6. MONOLI.TH 6 Let's disrupt the stone business with < https://fr.m.wikipedia.org/wiki/Fichier:Kerloas_menhir.JPG

  7. Our flagship project > 1h test, > 100 dependencies >

    100 models, > 1000 fields > 100M rows in some tables 7 Maintaining a Django project after 10.000 commits This is purely fictional, any resemblance to actual project living or dead is purely coincidental
  8. The project zoo Unit & Integration tests Health checks &

    monitoring Async & cron tasks APIs i18n & l10n Logging Error reporting Packaging Deployment code Dev environment Management commands Linting Custom tooling Changelog system Partial documentation Deprecated APIs Complex
 server-side UI Half-baked initiatives
 ("ongoing efforts") Dead code Memory leaks "Don't touch that code!" Spaghetti code Help Send help please 8 Maintaining a Django project after 10.000 commits
  9. Time flies Team turnover Company pivoting Scale and split Complexity

    grows Abandoned third parties 9 Maintaining a Django project after 10.000 commits
  10. THE OF OUR CODE THE ❤ OF OUR CODE THE

    OF OUR CODE THE ⚔ OF OUR CODE 10
  11. THIRD PARTY DEPENDENCIES 11 Part 1 - https://commons.wikimedia.org/wiki/File:Coastal-rocks.jpg >

  12. What is a third party dependency? 12 Maintaining a Django

    project after 10.000 commits Some external code used by your project
  13. Reasons to use 3rd party dependencies 13 Maintaining a Django

    project after 10.000 commits Lower the effort / time / cost Lower the risk
  14. Reasons NOT to use 3rd party dependencies 14 Maintaining a

    Django project after 10.000 commits Increase the effort / time / cost Increase the risk
  15. The problems 15 Maintaining a Django project after 10.000 commits

  16. The problems 16 Maintaining a Django project after 10.000 commits

    ▸ Security updates
  17. The problems 17 Maintaining a Django project after 10.000 commits

    Security updates ▸ Project direction
  18. The problems 18 Maintaining a Django project after 10.000 commits

    Security updates Project direction ▸ Project lagging behind
  19. The problems 19 Maintaining a Django project after 10.000 commits

    Security updates Project direction Project lagging behind ▸ Contributions
  20. The problems 20 Maintaining a Django project after 10.000 commits

    Security updates Project direction Project lagging behind Contributions ▸ Project becoming inactive
  21. The problems 21 Maintaining a Django project after 10.000 commits

    Security updates Project direction Project lagging behind Contributions Project becoming inactive ▸ Partial need
  22. NEVER FORGET THE HIDDEN COSTS OF ADDING 3RD PARTIES Advice

    #1 Maintaining a Django project after 10.000 commits 22
  23. All is not lost 23 Maintaining a Django project after

    10.000 commits
  24. Sometimes, it's just ok You have ownership Complex scoped problem

    Official packages Django backend implementation Well known apps 24 Maintaining a Django project after 10.000 commits
  25. Testing Test the next versions It's ok it it fails

    At least, you know. 25 Maintaining a Django project after 10.000 commits
  26. Participate Issues / PR Be nice 26 Maintaining a Django

    project after 10.000 commits
  27. Taking ownership 27 Maintaining a Django project after 10.000 commits

  28. Taking ownership ▸ Fork 28 Maintaining a Django project after

    10.000 commits
  29. Taking ownership Fork ▸ Vendor 29 Maintaining a Django project

    after 10.000 commits
  30. Taking ownership Fork Vendor ▸ Implement 30 Maintaining a Django

    project after 10.000 commits
  31. Adapter Build an abstraction layer Only one module can import

    them
 (you may enforce this with bellybutton) No *args, **kwargs 31 Maintaining a Django project after 10.000 commits YOUR CODE MY_GITHUB.PY GITHUB CLIENT
  32. ISOLATE YOUR
 THIRD PARTIES Advice #2 Maintaining a Django project

    after 10.000 commits 32
  33. Special aside for django models You need to have the

    entire ownership on your database External app has migrations: ❌ External app has abstract models defining fields: ⚠ 33 Maintaining a Django project after 10.000 commits
  34. DON'T LET ANYONE DEFINE YOUR MODELS Advice #3 Maintaining a

    Django project after 10.000 commits 34
  35. DON'T FORGET TO CONTRIBUTE BACK TO OPEN SOURCE Advice #4

    Maintaining a Django project after 10.000 commits 35
  36. Could we imagine the same approach with Django itself? 36

    Maintaining a Django project after 10.000 commits
  37. BUSINESS LOGIC 37 Part 2 - ❤ < https://pxhere.com/en/photo/597168

  38. What is business logic? 38 Maintaining a Django project after

    10.000 commits DJANGO BUSINESS LOGIC HTTP DATABASE
  39. What is business logic? 39 Maintaining a Django project after

    10.000 commits DJANGO FLASK BUSINESS LOGIC HTTP DATABASE
  40. What is business logic? "When a user is created, they

    get a welcome email." "Only premium users can access this page." "On a course, start date must be before end date." 40 Maintaining a Django project after 10.000 commits
  41. The problem 41 Maintaining a Django project after 10.000 commits

    BUSINESS LOGIC DJANGO HTTP DATABASE
  42. The cause There's business logic in my templates ... And

    in my forms & serializers ... And in my views ... And in my models & managers And all these modules already have another purpose 42 Maintaining a Django project after 10.000 commits
  43. Service layer: 2 talks to watch Hanna Kollo
 Avoiding Monoliths

    http://bit.ly/dce15-monolith 43 Maintaining a Django project after 10.000 commits Radoslav Georgiev
 Django structure for scale and longevity http://bit.ly/ep18-django-scale
  44. A DJANGO APP Service layers: key takeaways Every non-trivial operation


    should be a service / selector: services.py for create / update selectors.py for get / list no form.save(), serializer.save(), CreateView etc 44 Maintaining a Django project after 10.000 commits From Radoslav Georgiev
 Django structure for scale and longevity SERVICES.PY SELECTORS.PY MODELS.PY FORMS.PY
  45. Service layer 45 Maintaining a Django project after 10.000 commits

    HTTP DATABASE BUSINESS LOGIC DJANGO VIEWS, FORMS, ETC ORM
  46. The sandwich architecture 46 Maintaining a Django project after 10.000

    commits HTTP DATABASE PURE PYTHON BUSINESS LOGIC DJANGO VIEWS, FORMS, ETC ORM
  47. And beyond? 47 Maintaining a Django project after 10.000 commits

    HTTP DATABASE DJANGO VIEWS, FORMS, ETC ORM PURE PYTHON BUSINESS LOGIC
  48. And beyond? 48 Maintaining a Django project after 10.000 commits

    PURE PYTHON
 BUSINESS LOGIC HTTP, DATABASE DJANGO VIEWS, FORMS
  49. SPLIT YOUR BUSINESS LOGIC, DJANGO VIEWS AND THE ORM Advice

    #5 Maintaining a Django project after 10.000 commits 49 Did I hear anyone say "MVC"?
  50. Is this Java? ... But then so what? 50 Maintaining

    a Django project after 10.000 commits
  51. BOXES 51 Part 3 - https://www.flickr.com/photos/ikewinski/8013010600 >

  52. Software architecture Put things in the right boxes Name your

    boxes Don't mix boxes Define links between boxes 52 Maintaining a Django project after 10.000 commits < https://www.pexels.com/photo/gem-stones-souvenir-952681/
  53. Boxes Functional core / Imperative shell + Hoist your I/Os

    The Hexagonal Architecture Domain Driven Design 53 Maintaining a Django project after 10.000 commits
  54. Functional core / Imperative shell Gary Bernhardt Separating the logic

    from the glue 54 Maintaining a Django project after 10.000 commits
  55. Functional core / Imperative shell Decision making lives in
 functions

    without side effects 55 Maintaining a Django project after 10.000 commits FUNCTIONAL CORE FUNCTIONAL CORE FUNCTIONAL CORE
  56. Functional core / Imperative shell The outer shell has all


    the side effects but no logic 56 Maintaining a Django project after 10.000 commits IMPERATIVE SHELL
  57. Functional core / Imperative shell 57 Maintaining a Django project

    after 10.000 commits FUNCTIONAL CORE FUNCTIONAL CORE FUNCTIONAL CORE IMPERATIVE SHELL IMPERATIVE SHELL IMPERATIVE SHELL From Gary Bernhardt
 Boundaries
  58. Quiz time! def main(): with open("/etc/hosts") as file: for line

    in parse_hosts(file): print(line) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line 58 Maintaining a Django project after 10.000 commits def main(): parse_hosts() def parse_hosts(): with open("/etc/hosts") as lines: for line in lines: if line.startswith("#"): continue print(line) From Brandon Rhodes
 Hoisting your I/Os
  59. Requirements have changed def main(): with open("/etc/hosts") as file: for

    line in parse_hosts(file): print(line) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line 59 Maintaining a Django project after 10.000 commits def main(): lines = requests.get("hosts.com").json(): with open("local_hosts", "w") as file: file.writelines(parse_hosts(lines)) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line From Brandon Rhodes
 Hoisting your I/Os
  60. Hoist your I/Os Brandon Rhodes You're not reducing complexity
 by

    burying I/O down the stack Code manipulates
 plain data structures,
 I/O in the top level glue 60 Maintaining a Django project after 10.000 commits http://bit.ly/hoist-your-ios
  61. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 61

    Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN
  62. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 62

    Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN
  63. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 63

    Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN
  64. The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 64

    Maintaining a Django project after 10.000 commits HTTP, DATABASE DJANGO VIEWS, FORMS PURE PYTHON
 BUSINESS LOGIC
  65. The Clean Architecture in Python The Clean Architecture in Python


    Brandon Rhodes, PyOhio 2014 http://bit.ly/clean-architecture-python 65 Maintaining a Django project after 10.000 commits
  66. Domain Driven Design A concept by Eric Evans A LOT

    of things, at every level 66 Maintaining a Django project after 10.000 commits
  67. Domain Driven Design Domain Curiosity Bounded Contexts Ubiquitous Language 67

    Maintaining a Django project after 10.000 commits
  68. Being driven by your trade, not by technology The tale

    of the customer address 68 Maintaining a Django project after 10.000 commits USERNAME NAME ADDRESS ZIP_CODE CUSTOMER Example from Cyrille Martraire - DDD, en vrai pour le développeur
  69. Being driven by your trade, not by technology The tale

    of the customer address 69 Maintaining a Django project after 10.000 commits USERNAME NAME ADDRESS ZIP_CODE ADDRESS_BILLING ZIP_CODE_BILLING CUSTOMER From Cyrille Martraire - DDD, en vrai pour le développeur
  70. Being driven by your trade, not by technology The tale

    of the customer address 70 Maintaining a Django project after 10.000 commits USERNAME >SHIPPING ADDRESS >BILLING ADDRESS CUSTOMER NAME ADDRESS ZIP_CODE ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur
  71. Being driven by your trade, not by technology The tale

    of the customer address 71 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE SHIPPING ADDRESS NAME ADDRESS ZIP_CODE BILLING ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur
  72. Being driven by your trade, not by technology The tale

    of the customer address 72 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE RECIPIENT NAME ADDRESS ZIP_CODE BILLING ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur
  73. Being driven by your trade, not by technology The tale

    of the customer address 73 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE RECIPIENT NAME ADDRESS ZIP_CODE ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur
  74. Being driven by your trade, not by technology The tale

    of the customer address 74 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE BUILDING _CODE DELIVERY_HOURS RECIPIENT NAME ADDRESS ZIP_CODE ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur
  75. Being driven by your trade, not by technology The tale

    of the customer address 75 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE BUILDING _CODE DELIVERY_HOURS RECIPIENT NAME ADDRESS ZIP_CODE TAX_CODE CURRENCY ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur
  76. YOU'RE NOT "JUST" BUILDING A WEB APP Advice #6 Maintaining

    a Django project after 10.000 commits 76
  77. The shoulders of giants 77 Maintaining a Django project after

    10.000 commits THE CLEAN ARCHITECTURE BRANDON RHODES 2014 BOUNDARIES GARY BERNHARDT 2012 FAST TEST, SLOW TEST GARY BERNHARDT 2012 HOISTING YOUR I/O BRANDON RHODES 2015 RADOSLAV GEORGIEV DJANGO STRUCTURE... 2018 THE CLEAN ARCHITECTURE ROBERT MARTIN 2012 HEXAGONAL ARCHITECTURE ALISTAIR COCKBURN 2005
  78. STAND ON THE SHOULDERS OF GIANTS (OR TALLER FOLKS) Advice

    #7 Maintaining a Django project after 10.000 commits 78
  79. TESTS 79 Part 4 - ⚔ < Eduardo Otubo [CC

    BY 2.0]
  80. The test pyramid 80 Maintaining a Django project after 10.000

    commits UNIT INTEGRATION FUNCTIONAL X10? X30? 30 SEC 1 SEC 10 MS
  81. What to test where for a Django view? 81 Maintaining

    a Django project after 10.000 commits UNIT BUSINESS LOGIC VIEW PERSISTENCE INTEGRATION FUNCTIONAL TEMPLATE NAME, URL PAGE CONTENT FLOW CONTEXT
  82. DON'T JUST WRITE TESTS,
 WRITE THE RIGHT TESTS. Advice #8

    Maintaining a Django project after 10.000 commits 82
  83. Code quality criteria in tests Is a criteria: Dead easy

    to read,
 from start to tear down One test tests one thing Have a long
 and descriptive name 83 Maintaining a Django project after 10.000 commits Is not a criteria: Smart adaptable code DRY - Don't repeat yourself Layering and
 hidden complexity
  84. IN YOUR TESTS, AIM FOR THE SIMPLEST
 TO READ AND

    WRITE Advice #9 Maintaining a Django project after 10.000 commits 84
  85. Things that get complicated in the long run Test mixins

    self.client.get()/post() Fixture files 85 Maintaining a Django project after 10.000 commits
  86. The right tools for the job Use pytest and pytest

    style function tests Use pytest fixtures (avoid auto-use) And use Factory Boy 86 Maintaining a Django project after 10.000 commits
  87. Tests: going further 87 Maintaining a Django project after 10.000

    commits Behavior Driven Development Snapshot testing
  88. Behavior Driven Development Feature: Blog A site where you can

    publish your articles. Scenario: Publishing the article Given I'm an author user And I have an article When I go to the article page And I press the publish button Then the article should be published 88 Maintaining a Django project after 10.000 commits
  89. 89 Maintaining a Django project after 10.000 commits @scenario('publish_article.feature', 'Publishing

    the article') def test_publish(): ... @given("I'm an author user") def author_user(auth, author): ... @given('I have an article') def article(author): ... @when('I go to the article page') def go_to_article(article, browser): ... @when('I press the publish button') def publish_article(browser): ... @then('the article should be published') def article_is_published(article): ...
  90. Behavior Driven Development Document your specifications by design Functional tests

    from day 1 90 Maintaining a Django project after 10.000 commits
  91. Snapshot testing is useful too Screenshot testing SQL testing (django-perf-rec)

    HTML testing? API output testing? 91 Maintaining a Django project after 10.000 commits
  92. MAKE YOUR TESTS A PRIME PART OF YOUR PROCESS Advice

    #10 Maintaining a Django project after 10.000 commits 92
  93. STAY CURIOUS, TRY THINGS, THINK BEFORE YOU CODE Final advice

    Maintaining a Django project after 10.000 commits 93
  94. 94 Maintaining a Django project after 10.000 commits STAY CURIOUS,

    TRY THINGS, THINK BEFORE YOU CODE Final advice Final-Final advice DON'T OVER-ENGINEER IT!
  95. THANK YOU! QUESTIONS? 95 Joachim Jablon - @ewjoachim Stéphane "Twidi"

    Angel - @Twidi We're hiring! Try it out!