Maintaning a Django project after 10k commits

From Stéphane "Twidi" Angel and Joachim Jablon

Let's dive into the challenges that arise when a Django project starts to become huge.

Django is extremely effective for creating a website quickly with top notch features out of the box. But in some codebases, after a while, new developments can become harder and harder.

In this talk, we'll examine some design decisions that have, and haven't, scaled successfully, in the hope that the next time you start your big scale Django project, you won't end up cursing your past self after three years.

Among the subjects we'll tackle:

- Third parties, the problems they bring and a few solutions;
- Where to put your business logic to keep your sanity;
- How to archiecture your code for fun and profit;
- Testing strategies for large projects

Video available at :
https://youtu.be/oAV73PRRWNY?t=10577

Transcript

1. MAINTAINING A DJANGO PROJECT
   AFTER 10.000 COMMITS
   Joachim Jablon - PeopleDoc
   Stéphane "Twidi" Angel - isshub.io
   DjangoCon Europe 2019 - Copenhagen
   

   View Slide

2. WHO ARE WE?
   2
   Joachim Jablon / ewjoachim
   @ewjoachim / ewjoach.im
   joachim@jablon.fr
   

   View Slide

3. WHO ARE WE?
   3
   Stéphane "Twidi" Angel
   @twidi / isshub.io
   

   View Slide

4. MAINTAINING A DJANGO PROJECT
   AFTER 10.000 COMMITS
   Joachim Jablon - PeopleDoc
   Stéphane "Twidi" Angel - isshub.io
   DjangoCon Europe 2019 - Copenhagen
   

   View Slide

5. MAINTAINING A DJANGO PROJECT
   AFTER BEFORE 10.000 COMMITS
   Joachim Jablon - PeopleDoc
   Stéphane "Twidi" Angel - isshub.io
   DjangoCon Europe 2019 - Copenhagen
   

   View Slide

6. MONOLI.TH
   6
   Let's disrupt the stone business with
   < https://fr.m.wikipedia.org/wiki/Fichier:Kerloas_menhir.JPG
   

   View Slide

7. Our flagship project
   > 1h test, > 100 dependencies
   > 100 models, > 1000 fields
   > 100M rows in some tables
   7
   Maintaining a Django project after 10.000 commits
   This is purely fictional, any resemblance to actual project living or dead is purely coincidental
   

   View Slide

8. The project zoo
   Unit & Integration tests
   Health checks & monitoring
   Async & cron tasks
   APIs
   i18n & l10n
   Logging
   Error reporting
   Packaging
   Deployment code
   Dev environment
   Management commands
   Linting
   Custom tooling
   Changelog system
   Partial documentation
   Deprecated APIs
   Complex

   server-side UI
   Half-baked initiatives

   ("ongoing efforts")
   Dead code
   Memory leaks
   "Don't touch that code!"
   Spaghetti code
   Help
   Send help please
   8
   Maintaining a Django project after 10.000 commits
   

   View Slide

9. Time flies
   Team turnover
   Company pivoting
   Scale and split
   Complexity grows
   Abandoned third parties
   9
   Maintaining a Django project after 10.000 commits
   

   View Slide

10. THE OF OUR CODE
    THE ❤ OF OUR CODE
    THE OF OUR CODE
    THE ⚔ OF OUR CODE
    10
    

    View Slide

11. THIRD PARTY DEPENDENCIES
    11
    Part 1 -
    https://commons.wikimedia.org/wiki/File:Coastal-rocks.jpg >
    

    View Slide

12. What is a third party dependency?
    12
    Maintaining a Django project after 10.000 commits
    Some external code used by your project
    

    View Slide

13. Reasons to use 3rd party dependencies
    13
    Maintaining a Django project after 10.000 commits
    Lower the effort / time / cost
    Lower the risk
    

    View Slide

14. Reasons NOT to use 3rd party dependencies
    14
    Maintaining a Django project after 10.000 commits
    Increase the effort / time / cost
    Increase the risk
    

    View Slide

15. The problems
    15
    Maintaining a Django project after 10.000 commits
    

    View Slide

16. The problems
    16
    Maintaining a Django project after 10.000 commits
    ▸ Security updates
    

    View Slide

17. The problems
    17
    Maintaining a Django project after 10.000 commits
    Security updates
    ▸ Project direction
    

    View Slide

18. The problems
    18
    Maintaining a Django project after 10.000 commits
    Security updates
    Project direction
    ▸ Project lagging behind
    

    View Slide

19. The problems
    19
    Maintaining a Django project after 10.000 commits
    Security updates
    Project direction
    Project lagging behind
    ▸ Contributions
    

    View Slide

20. The problems
    20
    Maintaining a Django project after 10.000 commits
    Security updates
    Project direction
    Project lagging behind
    Contributions
    ▸ Project becoming inactive
    

    View Slide

21. The problems
    21
    Maintaining a Django project after 10.000 commits
    Security updates
    Project direction
    Project lagging behind
    Contributions
    Project becoming inactive
    ▸ Partial need
    

    View Slide

22. NEVER FORGET THE
    HIDDEN COSTS OF
    ADDING 3RD PARTIES
    Advice #1
    Maintaining a Django project after 10.000 commits 22
    

    View Slide

23. All is not lost
    23
    Maintaining a Django project after 10.000 commits
    

    View Slide

24. Sometimes, it's just ok
    You have ownership
    Complex scoped problem
    Official packages
    Django backend implementation
    Well known apps
    24
    Maintaining a Django project after 10.000 commits
    

    View Slide

25. Testing
    Test the next versions
    It's ok it it fails
    At least, you know.
    25
    Maintaining a Django project after 10.000 commits
    

    View Slide

26. Participate
    Issues / PR
    Be nice
    26
    Maintaining a Django project after 10.000 commits
    

    View Slide

27. Taking ownership
    27
    Maintaining a Django project after 10.000 commits
    

    View Slide

28. Taking ownership
    ▸ Fork
    28
    Maintaining a Django project after 10.000 commits
    

    View Slide

29. Taking ownership
    Fork
    ▸ Vendor
    29
    Maintaining a Django project after 10.000 commits
    

    View Slide

30. Taking ownership
    Fork
    Vendor
    ▸ Implement
    30
    Maintaining a Django project after 10.000 commits
    

    View Slide

31. Adapter
    Build an abstraction layer
    Only one module can import them

    (you may enforce this with bellybutton)
    No *args, **kwargs
    31
    Maintaining a Django project after 10.000 commits
    YOUR CODE MY_GITHUB.PY GITHUB CLIENT
    

    View Slide

32. ISOLATE YOUR

    THIRD PARTIES
    Advice #2
    Maintaining a Django project after 10.000 commits 32
    

    View Slide

33. Special aside for django models
    You need to have the entire ownership on your database
    External app has migrations: ❌
    External app has abstract models defining fields: ⚠
    33
    Maintaining a Django project after 10.000 commits
    

    View Slide

34. DON'T LET ANYONE
    DEFINE YOUR MODELS
    Advice #3
    Maintaining a Django project after 10.000 commits 34
    

    View Slide

35. DON'T FORGET TO
    CONTRIBUTE BACK TO
    OPEN SOURCE
    Advice #4
    Maintaining a Django project after 10.000 commits 35
    

    View Slide

36. Could we imagine the same approach with Django itself?
    36
    Maintaining a Django project after 10.000 commits
    

    View Slide

37. BUSINESS LOGIC
    37
    Part 2 - ❤
    < https://pxhere.com/en/photo/597168
    

    View Slide

38. What is business logic?
    38
    Maintaining a Django project after 10.000 commits
    DJANGO
    BUSINESS
    LOGIC
    HTTP DATABASE
    

    View Slide

39. What is business logic?
    39
    Maintaining a Django project after 10.000 commits
    DJANGO FLASK
    BUSINESS
    LOGIC
    HTTP DATABASE
    

    View Slide

40. What is business logic?
    "When a user is created, they get a welcome email."
    "Only premium users can access this page."
    "On a course, start date must be before end date."
    40
    Maintaining a Django project after 10.000 commits
    

    View Slide

41. The problem
    41
    Maintaining a Django project after 10.000 commits
    BUSINESS LOGIC
    DJANGO
    HTTP DATABASE
    

    View Slide

42. The cause
    There's business logic in my templates
    ... And in my forms & serializers
    ... And in my views
    ... And in my models & managers
    And all these modules already have another purpose
    42
    Maintaining a Django project after 10.000 commits
    

    View Slide

43. Service layer: 2 talks to watch
    Hanna Kollo

    Avoiding Monoliths
    http://bit.ly/dce15-monolith
    43
    Maintaining a Django project after 10.000 commits
    Radoslav Georgiev

    Django structure for scale and longevity
    http://bit.ly/ep18-django-scale
    

    View Slide

44. A DJANGO APP
    Service layers: key takeaways
    Every non-trivial operation

    should be a service / selector:
    services.py for create / update
    selectors.py for get / list
    no form.save(), serializer.save(),
    CreateView etc
    44
    Maintaining a Django project after 10.000 commits
    From Radoslav Georgiev

    Django structure for scale and longevity
    SERVICES.PY
    SELECTORS.PY
    MODELS.PY
    FORMS.PY
    

    View Slide

45. Service layer
    45
    Maintaining a Django project after 10.000 commits
    HTTP DATABASE
    BUSINESS
    LOGIC
    DJANGO
    VIEWS,
    FORMS,
    ETC
    ORM
    

    View Slide

46. The sandwich architecture
    46
    Maintaining a Django project after 10.000 commits
    HTTP DATABASE
    PURE
    PYTHON
    BUSINESS
    LOGIC
    DJANGO
    VIEWS,
    FORMS,
    ETC
    ORM
    

    View Slide

47. And beyond?
    47
    Maintaining a Django project after 10.000 commits
    HTTP DATABASE
    DJANGO
    VIEWS,
    FORMS,
    ETC
    ORM
    PURE
    PYTHON
    BUSINESS
    LOGIC
    

    View Slide

48. And beyond?
    48
    Maintaining a Django project after 10.000 commits
    PURE PYTHON

    BUSINESS LOGIC
    HTTP, DATABASE
    DJANGO VIEWS, FORMS
    

    View Slide

49. SPLIT YOUR BUSINESS
    LOGIC, DJANGO
    VIEWS AND THE ORM
    Advice #5
    Maintaining a Django project after 10.000 commits 49
    Did I hear anyone say "MVC"?
    

    View Slide

50. Is this Java?
    ... But then so what?
    50
    Maintaining a Django project after 10.000 commits
    

    View Slide

51. BOXES
    51
    Part 3 -
    https://www.flickr.com/photos/ikewinski/8013010600 >
    

    View Slide

52. Software architecture
    Put things in the right boxes
    Name your boxes
    Don't mix boxes
    Define links between boxes
    52
    Maintaining a Django project after 10.000 commits
    < https://www.pexels.com/photo/gem-stones-souvenir-952681/
    

    View Slide

53. Boxes
    Functional core / Imperative shell + Hoist your I/Os
    The Hexagonal Architecture
    Domain Driven Design
    53
    Maintaining a Django project after 10.000 commits
    

    View Slide

54. Functional core / Imperative shell
    Gary Bernhardt
    Separating the logic from the glue
    54
    Maintaining a Django project after 10.000 commits
    

    View Slide

55. Functional core / Imperative shell
    Decision making lives in

    functions without side effects
    55
    Maintaining a Django project after 10.000 commits
    FUNCTIONAL
    CORE
    FUNCTIONAL
    CORE
    FUNCTIONAL
    CORE
    

    View Slide

56. Functional core / Imperative shell
    The outer shell has all

    the side effects but no logic
    56
    Maintaining a Django project after 10.000 commits
    IMPERATIVE SHELL
    

    View Slide

57. Functional core / Imperative shell
    57
    Maintaining a Django project after 10.000 commits
    FUNCTIONAL
    CORE
    FUNCTIONAL
    CORE
    FUNCTIONAL
    CORE
    IMPERATIVE SHELL
    IMPERATIVE SHELL
    IMPERATIVE SHELL
    From Gary Bernhardt

    Boundaries
    

    View Slide

58. Quiz time!
    def main():
    with open("/etc/hosts") as file:
    for line in parse_hosts(file):
    print(line)
    def parse_hosts(lines):
    for line in lines:
    if line.startswith("#"):
    continue
    yield line
    58
    Maintaining a Django project after 10.000 commits
    def main():
    parse_hosts()
    def parse_hosts():
    with open("/etc/hosts") as lines:
    for line in lines:
    if line.startswith("#"):
    continue
    print(line)
    From Brandon Rhodes

    Hoisting your I/Os
    

    View Slide

59. Requirements have changed
    def main():
    with open("/etc/hosts") as file:
    for line in parse_hosts(file):
    print(line)
    def parse_hosts(lines):
    for line in lines:
    if line.startswith("#"):
    continue
    yield line
    59
    Maintaining a Django project after 10.000 commits
    def main():
    lines = requests.get("hosts.com").json():
    with open("local_hosts", "w") as file:
    file.writelines(parse_hosts(lines))
    def parse_hosts(lines):
    for line in lines:
    if line.startswith("#"):
    continue
    yield line
    From Brandon Rhodes

    Hoisting your I/Os
    

    View Slide

60. Hoist your I/Os
    Brandon Rhodes
    You're not reducing complexity

    by burying I/O down the stack
    Code manipulates

    plain data structures,

    I/O in the top level glue
    60
    Maintaining a Django project after 10.000 commits
    http://bit.ly/hoist-your-ios
    

    View Slide

61. The Hexagonal Architecture
    By Alistair Cockburn
    Layers
    Dependency rules
    61
    Maintaining a Django project after 10.000 commits
    INSFRASTRUCTURE
    APPLICATION
    DOMAIN
    

    View Slide

62. The Hexagonal Architecture
    By Alistair Cockburn
    Layers
    Dependency rules
    62
    Maintaining a Django project after 10.000 commits
    INSFRASTRUCTURE
    APPLICATION
    DOMAIN
    

    View Slide

63. The Hexagonal Architecture
    By Alistair Cockburn
    Layers
    Dependency rules
    63
    Maintaining a Django project after 10.000 commits
    INSFRASTRUCTURE
    APPLICATION
    DOMAIN
    

    View Slide

64. The Hexagonal Architecture
    By Alistair Cockburn
    Layers
    Dependency rules
    64
    Maintaining a Django project after 10.000 commits
    HTTP, DATABASE
    DJANGO VIEWS, FORMS
    PURE PYTHON

    BUSINESS LOGIC
    

    View Slide

65. The Clean Architecture in Python
    The Clean Architecture in Python

    Brandon Rhodes, PyOhio 2014
    http://bit.ly/clean-architecture-python
    65
    Maintaining a Django project after 10.000 commits
    

    View Slide

66. Domain Driven Design
    A concept by Eric Evans
    A LOT of things, at every level
    66
    Maintaining a Django project after 10.000 commits
    

    View Slide

67. Domain Driven Design
    Domain Curiosity
    Bounded Contexts
    Ubiquitous Language
    67
    Maintaining a Django project after 10.000 commits
    

    View Slide

68. Being driven by your trade, not by technology
    The tale of the customer address
    68
    Maintaining a Django project after 10.000 commits
    USERNAME
    NAME
    ADDRESS
    ZIP_CODE
    CUSTOMER
    Example from Cyrille Martraire - DDD, en vrai pour le développeur
    

    View Slide

69. Being driven by your trade, not by technology
    The tale of the customer address
    69
    Maintaining a Django project after 10.000 commits
    USERNAME
    NAME
    ADDRESS
    ZIP_CODE
    ADDRESS_BILLING
    ZIP_CODE_BILLING
    CUSTOMER
    From Cyrille Martraire - DDD, en vrai pour le développeur
    

    View Slide

70. Being driven by your trade, not by technology
    The tale of the customer address
    70
    Maintaining a Django project after 10.000 commits
    USERNAME
    >SHIPPING ADDRESS
    >BILLING ADDRESS
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    ADDRESS
    From Cyrille Martraire - DDD, en vrai pour le développeur
    

    View Slide

71. Being driven by your trade, not by technology
    The tale of the customer address
    71
    Maintaining a Django project after 10.000 commits
    USERNAME
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    SHIPPING ADDRESS
    NAME
    ADDRESS
    ZIP_CODE
    BILLING ADDRESS
    From Cyrille Martraire - DDD, en vrai pour le développeur
    

    View Slide

72. Being driven by your trade, not by technology
    The tale of the customer address
    72
    Maintaining a Django project after 10.000 commits
    USERNAME
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    RECIPIENT
    NAME
    ADDRESS
    ZIP_CODE
    BILLING ADDRESS
    From Cyrille Martraire - DDD, en vrai pour le développeur
    

    View Slide

73. Being driven by your trade, not by technology
    The tale of the customer address
    73
    Maintaining a Django project after 10.000 commits
    USERNAME
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    RECIPIENT
    NAME
    ADDRESS
    ZIP_CODE
    ACCOUNT
    From Cyrille Martraire - DDD, en vrai pour le développeur
    

    View Slide

74. Being driven by your trade, not by technology
    The tale of the customer address
    74
    Maintaining a Django project after 10.000 commits
    USERNAME
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    BUILDING _CODE
    DELIVERY_HOURS
    RECIPIENT
    NAME
    ADDRESS
    ZIP_CODE
    ACCOUNT
    From Cyrille Martraire - DDD, en vrai pour le développeur
    

    View Slide

75. Being driven by your trade, not by technology
    The tale of the customer address
    75
    Maintaining a Django project after 10.000 commits
    USERNAME
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    BUILDING _CODE
    DELIVERY_HOURS
    RECIPIENT
    NAME
    ADDRESS
    ZIP_CODE
    TAX_CODE
    CURRENCY
    ACCOUNT
    From Cyrille Martraire - DDD, en vrai pour le développeur
    

    View Slide

76. YOU'RE NOT "JUST"
    BUILDING A WEB APP
    Advice #6
    Maintaining a Django project after 10.000 commits 76
    

    View Slide

77. The shoulders of giants
    77
    Maintaining a Django project after 10.000 commits
    THE CLEAN ARCHITECTURE
    BRANDON RHODES
    2014
    BOUNDARIES
    GARY BERNHARDT
    2012
    FAST TEST, SLOW TEST
    GARY BERNHARDT
    2012
    HOISTING YOUR I/O
    BRANDON RHODES
    2015
    RADOSLAV GEORGIEV
    DJANGO STRUCTURE...
    2018
    THE CLEAN ARCHITECTURE
    ROBERT MARTIN
    2012
    HEXAGONAL ARCHITECTURE
    ALISTAIR COCKBURN
    2005
    

    View Slide

78. STAND ON THE
    SHOULDERS OF GIANTS
    (OR TALLER FOLKS)
    Advice #7
    Maintaining a Django project after 10.000 commits 78
    

    View Slide

79. TESTS
    79
    Part 4 - ⚔
    < Eduardo Otubo [CC BY 2.0]
    

    View Slide

80. The test pyramid
    80
    Maintaining a Django project after 10.000 commits
    UNIT
    INTEGRATION
    FUNCTIONAL
    X10?
    X30?
    30 SEC
    1 SEC
    10 MS
    

    View Slide

81. What to test where for a Django view?
    81
    Maintaining a Django project after 10.000 commits
    UNIT
    BUSINESS LOGIC
    VIEW
    PERSISTENCE
    INTEGRATION
    FUNCTIONAL
    TEMPLATE NAME, URL
    PAGE CONTENT
    FLOW CONTEXT
    

    View Slide

82. DON'T JUST WRITE TESTS,

    WRITE THE RIGHT TESTS.
    Advice #8
    Maintaining a Django project after 10.000 commits 82
    

    View Slide

83. Code quality criteria in tests
    Is a criteria:
    Dead easy to read,

    from start to tear down
    One test tests one thing
    Have a long

    and descriptive name
    83
    Maintaining a Django project after 10.000 commits
    Is not a criteria:
    Smart adaptable code
    DRY - Don't repeat yourself
    Layering and

    hidden complexity
    

    View Slide

84. IN YOUR TESTS,
    AIM FOR THE SIMPLEST

    TO READ AND WRITE
    Advice #9
    Maintaining a Django project after 10.000 commits 84
    

    View Slide

85. Things that get complicated in the long run
    Test mixins
    self.client.get()/post()
    Fixture files
    85
    Maintaining a Django project after 10.000 commits
    

    View Slide

86. The right tools for the job
    Use pytest and pytest style function tests
    Use pytest fixtures (avoid auto-use)
    And use Factory Boy
    86
    Maintaining a Django project after 10.000 commits
    

    View Slide

87. Tests: going further
    87
    Maintaining a Django project after 10.000 commits
    Behavior Driven Development
    Snapshot testing
    

    View Slide

88. Behavior Driven Development
    Feature: Blog
    A site where you can publish
    your articles.
    Scenario: Publishing the article
    Given I'm an author user
    And I have an article
    When I go to the article page
    And I press the publish button
    Then the article should be published
    88
    Maintaining a Django project after 10.000 commits
    

    View Slide

89. 89
    Maintaining a Django project after 10.000 commits
    @scenario('publish_article.feature',
    'Publishing the article')
    def test_publish(): ...
    @given("I'm an author user")
    def author_user(auth, author): ...
    @given('I have an article')
    def article(author): ...
    @when('I go to the article page')
    def go_to_article(article, browser): ...
    @when('I press the publish button')
    def publish_article(browser): ...
    @then('the article should be published')
    def article_is_published(article): ...
    

    View Slide

90. Behavior Driven Development
    Document your specifications by design
    Functional tests from day 1
    90
    Maintaining a Django project after 10.000 commits
    

    View Slide

91. Snapshot testing is useful too
    Screenshot testing
    SQL testing (django-perf-rec)
    HTML testing? API output testing?
    91
    Maintaining a Django project after 10.000 commits
    

    View Slide

92. MAKE YOUR TESTS
    A PRIME PART
    OF YOUR PROCESS
    Advice #10
    Maintaining a Django project after 10.000 commits 92
    

    View Slide

93. STAY CURIOUS,
    TRY THINGS,
    THINK BEFORE YOU CODE
    Final advice
    Maintaining a Django project after 10.000 commits 93
    

    View Slide

94. 94
    Maintaining a Django project after 10.000 commits
    STAY CURIOUS, TRY THINGS,
    THINK BEFORE YOU CODE
    Final advice
    Final-Final advice
    DON'T OVER-ENGINEER IT!
    

    View Slide

95. THANK YOU!
    QUESTIONS?
    95
    Joachim Jablon - @ewjoachim
    Stéphane "Twidi" Angel - @Twidi
    We're hiring!
    Try it out!
    

    View Slide