Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Maintaning a Django project after 10k commits

Maintaning a Django project after 10k commits

From Stéphane "Twidi" Angel and Joachim Jablon

Let's dive into the challenges that arise when a Django project starts to become huge.

Django is extremely effective for creating a website quickly with top notch features out of the box. But in some codebases, after a while, new developments can become harder and harder.

In this talk, we'll examine some design decisions that have, and haven't, scaled successfully, in the hope that the next time you start your big scale Django project, you won't end up cursing your past self after three years.

Among the subjects we'll tackle:

- Third parties, the problems they bring and a few solutions;
- Where to put your business logic to keep your sanity;
- How to archiecture your code for fun and profit;
- Testing strategies for large projects

Video available at :
https://youtu.be/oAV73PRRWNY?t=10577

Joachim Jablon

April 12, 2019
Tweet

More Decks by Joachim Jablon

Other Decks in Technology

Transcript

  1. MAINTAINING A DJANGO PROJECT
    AFTER 10.000 COMMITS
    Joachim Jablon - PeopleDoc
    Stéphane "Twidi" Angel - isshub.io
    DjangoCon Europe 2019 - Copenhagen

    View full-size slide

  2. WHO ARE WE?
    2
    Joachim Jablon / ewjoachim
    @ewjoachim / ewjoach.im
    [email protected]

    View full-size slide

  3. WHO ARE WE?
    3
    Stéphane "Twidi" Angel
    @twidi / isshub.io

    View full-size slide

  4. MAINTAINING A DJANGO PROJECT
    AFTER 10.000 COMMITS
    Joachim Jablon - PeopleDoc
    Stéphane "Twidi" Angel - isshub.io
    DjangoCon Europe 2019 - Copenhagen

    View full-size slide

  5. MAINTAINING A DJANGO PROJECT
    AFTER BEFORE 10.000 COMMITS
    Joachim Jablon - PeopleDoc
    Stéphane "Twidi" Angel - isshub.io
    DjangoCon Europe 2019 - Copenhagen

    View full-size slide

  6. MONOLI.TH
    6
    Let's disrupt the stone business with
    < https://fr.m.wikipedia.org/wiki/Fichier:Kerloas_menhir.JPG

    View full-size slide

  7. Our flagship project
    > 1h test, > 100 dependencies
    > 100 models, > 1000 fields
    > 100M rows in some tables
    7
    Maintaining a Django project after 10.000 commits
    This is purely fictional, any resemblance to actual project living or dead is purely coincidental

    View full-size slide

  8. The project zoo
    Unit & Integration tests
    Health checks & monitoring
    Async & cron tasks
    APIs
    i18n & l10n
    Logging
    Error reporting
    Packaging
    Deployment code
    Dev environment
    Management commands
    Linting
    Custom tooling
    Changelog system
    Partial documentation
    Deprecated APIs
    Complex

    server-side UI
    Half-baked initiatives

    ("ongoing efforts")
    Dead code
    Memory leaks
    "Don't touch that code!"
    Spaghetti code
    Help
    Send help please
    8
    Maintaining a Django project after 10.000 commits

    View full-size slide

  9. Time flies
    Team turnover
    Company pivoting
    Scale and split
    Complexity grows
    Abandoned third parties
    9
    Maintaining a Django project after 10.000 commits

    View full-size slide

  10. THE OF OUR CODE
    THE ❤ OF OUR CODE
    THE OF OUR CODE
    THE ⚔ OF OUR CODE
    10

    View full-size slide

  11. THIRD PARTY DEPENDENCIES
    11
    Part 1 -
    https://commons.wikimedia.org/wiki/File:Coastal-rocks.jpg >

    View full-size slide

  12. What is a third party dependency?
    12
    Maintaining a Django project after 10.000 commits
    Some external code used by your project

    View full-size slide

  13. Reasons to use 3rd party dependencies
    13
    Maintaining a Django project after 10.000 commits
    Lower the effort / time / cost
    Lower the risk

    View full-size slide

  14. Reasons NOT to use 3rd party dependencies
    14
    Maintaining a Django project after 10.000 commits
    Increase the effort / time / cost
    Increase the risk

    View full-size slide

  15. The problems
    15
    Maintaining a Django project after 10.000 commits

    View full-size slide

  16. The problems
    16
    Maintaining a Django project after 10.000 commits
    ▸ Security updates

    View full-size slide

  17. The problems
    17
    Maintaining a Django project after 10.000 commits
    Security updates
    ▸ Project direction

    View full-size slide

  18. The problems
    18
    Maintaining a Django project after 10.000 commits
    Security updates
    Project direction
    ▸ Project lagging behind

    View full-size slide

  19. The problems
    19
    Maintaining a Django project after 10.000 commits
    Security updates
    Project direction
    Project lagging behind
    ▸ Contributions

    View full-size slide

  20. The problems
    20
    Maintaining a Django project after 10.000 commits
    Security updates
    Project direction
    Project lagging behind
    Contributions
    ▸ Project becoming inactive

    View full-size slide

  21. The problems
    21
    Maintaining a Django project after 10.000 commits
    Security updates
    Project direction
    Project lagging behind
    Contributions
    Project becoming inactive
    ▸ Partial need

    View full-size slide

  22. NEVER FORGET THE
    HIDDEN COSTS OF
    ADDING 3RD PARTIES
    Advice #1
    Maintaining a Django project after 10.000 commits 22

    View full-size slide

  23. All is not lost
    23
    Maintaining a Django project after 10.000 commits

    View full-size slide

  24. Sometimes, it's just ok
    You have ownership
    Complex scoped problem
    Official packages
    Django backend implementation
    Well known apps
    24
    Maintaining a Django project after 10.000 commits

    View full-size slide

  25. Testing
    Test the next versions
    It's ok it it fails
    At least, you know.
    25
    Maintaining a Django project after 10.000 commits

    View full-size slide

  26. Participate
    Issues / PR
    Be nice
    26
    Maintaining a Django project after 10.000 commits

    View full-size slide

  27. Taking ownership
    27
    Maintaining a Django project after 10.000 commits

    View full-size slide

  28. Taking ownership
    ▸ Fork
    28
    Maintaining a Django project after 10.000 commits

    View full-size slide

  29. Taking ownership
    Fork
    ▸ Vendor
    29
    Maintaining a Django project after 10.000 commits

    View full-size slide

  30. Taking ownership
    Fork
    Vendor
    ▸ Implement
    30
    Maintaining a Django project after 10.000 commits

    View full-size slide

  31. Adapter
    Build an abstraction layer
    Only one module can import them

    (you may enforce this with bellybutton)
    No *args, **kwargs
    31
    Maintaining a Django project after 10.000 commits
    YOUR CODE MY_GITHUB.PY GITHUB CLIENT

    View full-size slide

  32. ISOLATE YOUR

    THIRD PARTIES
    Advice #2
    Maintaining a Django project after 10.000 commits 32

    View full-size slide

  33. Special aside for django models
    You need to have the entire ownership on your database
    External app has migrations: ❌
    External app has abstract models defining fields: ⚠
    33
    Maintaining a Django project after 10.000 commits

    View full-size slide

  34. DON'T LET ANYONE
    DEFINE YOUR MODELS
    Advice #3
    Maintaining a Django project after 10.000 commits 34

    View full-size slide

  35. DON'T FORGET TO
    CONTRIBUTE BACK TO
    OPEN SOURCE
    Advice #4
    Maintaining a Django project after 10.000 commits 35

    View full-size slide

  36. Could we imagine the same approach with Django itself?
    36
    Maintaining a Django project after 10.000 commits

    View full-size slide

  37. BUSINESS LOGIC
    37
    Part 2 - ❤
    < https://pxhere.com/en/photo/597168

    View full-size slide

  38. What is business logic?
    38
    Maintaining a Django project after 10.000 commits
    DJANGO
    BUSINESS
    LOGIC
    HTTP DATABASE

    View full-size slide

  39. What is business logic?
    39
    Maintaining a Django project after 10.000 commits
    DJANGO FLASK
    BUSINESS
    LOGIC
    HTTP DATABASE

    View full-size slide

  40. What is business logic?
    "When a user is created, they get a welcome email."
    "Only premium users can access this page."
    "On a course, start date must be before end date."
    40
    Maintaining a Django project after 10.000 commits

    View full-size slide

  41. The problem
    41
    Maintaining a Django project after 10.000 commits
    BUSINESS LOGIC
    DJANGO
    HTTP DATABASE

    View full-size slide

  42. The cause
    There's business logic in my templates
    ... And in my forms & serializers
    ... And in my views
    ... And in my models & managers
    And all these modules already have another purpose
    42
    Maintaining a Django project after 10.000 commits

    View full-size slide

  43. Service layer: 2 talks to watch
    Hanna Kollo

    Avoiding Monoliths
    http://bit.ly/dce15-monolith
    43
    Maintaining a Django project after 10.000 commits
    Radoslav Georgiev

    Django structure for scale and longevity
    http://bit.ly/ep18-django-scale

    View full-size slide

  44. A DJANGO APP
    Service layers: key takeaways
    Every non-trivial operation

    should be a service / selector:
    services.py for create / update
    selectors.py for get / list
    no form.save(), serializer.save(),
    CreateView etc
    44
    Maintaining a Django project after 10.000 commits
    From Radoslav Georgiev

    Django structure for scale and longevity
    SERVICES.PY
    SELECTORS.PY
    MODELS.PY
    FORMS.PY

    View full-size slide

  45. Service layer
    45
    Maintaining a Django project after 10.000 commits
    HTTP DATABASE
    BUSINESS
    LOGIC
    DJANGO
    VIEWS,
    FORMS,
    ETC
    ORM

    View full-size slide

  46. The sandwich architecture
    46
    Maintaining a Django project after 10.000 commits
    HTTP DATABASE
    PURE
    PYTHON
    BUSINESS
    LOGIC
    DJANGO
    VIEWS,
    FORMS,
    ETC
    ORM

    View full-size slide

  47. And beyond?
    47
    Maintaining a Django project after 10.000 commits
    HTTP DATABASE
    DJANGO
    VIEWS,
    FORMS,
    ETC
    ORM
    PURE
    PYTHON
    BUSINESS
    LOGIC

    View full-size slide

  48. And beyond?
    48
    Maintaining a Django project after 10.000 commits
    PURE PYTHON

    BUSINESS LOGIC
    HTTP, DATABASE
    DJANGO VIEWS, FORMS

    View full-size slide

  49. SPLIT YOUR BUSINESS
    LOGIC, DJANGO
    VIEWS AND THE ORM
    Advice #5
    Maintaining a Django project after 10.000 commits 49
    Did I hear anyone say "MVC"?

    View full-size slide

  50. Is this Java?
    ... But then so what?
    50
    Maintaining a Django project after 10.000 commits

    View full-size slide

  51. BOXES
    51
    Part 3 -
    https://www.flickr.com/photos/ikewinski/8013010600 >

    View full-size slide

  52. Software architecture
    Put things in the right boxes
    Name your boxes
    Don't mix boxes
    Define links between boxes
    52
    Maintaining a Django project after 10.000 commits
    < https://www.pexels.com/photo/gem-stones-souvenir-952681/

    View full-size slide

  53. Boxes
    Functional core / Imperative shell + Hoist your I/Os
    The Hexagonal Architecture
    Domain Driven Design
    53
    Maintaining a Django project after 10.000 commits

    View full-size slide

  54. Functional core / Imperative shell
    Gary Bernhardt
    Separating the logic from the glue
    54
    Maintaining a Django project after 10.000 commits

    View full-size slide

  55. Functional core / Imperative shell
    Decision making lives in

    functions without side effects
    55
    Maintaining a Django project after 10.000 commits
    FUNCTIONAL
    CORE
    FUNCTIONAL
    CORE
    FUNCTIONAL
    CORE

    View full-size slide

  56. Functional core / Imperative shell
    The outer shell has all

    the side effects but no logic
    56
    Maintaining a Django project after 10.000 commits
    IMPERATIVE SHELL

    View full-size slide

  57. Functional core / Imperative shell
    57
    Maintaining a Django project after 10.000 commits
    FUNCTIONAL
    CORE
    FUNCTIONAL
    CORE
    FUNCTIONAL
    CORE
    IMPERATIVE SHELL
    IMPERATIVE SHELL
    IMPERATIVE SHELL
    From Gary Bernhardt

    Boundaries

    View full-size slide

  58. Quiz time!
    def main():
    with open("/etc/hosts") as file:
    for line in parse_hosts(file):
    print(line)
    def parse_hosts(lines):
    for line in lines:
    if line.startswith("#"):
    continue
    yield line
    58
    Maintaining a Django project after 10.000 commits
    def main():
    parse_hosts()
    def parse_hosts():
    with open("/etc/hosts") as lines:
    for line in lines:
    if line.startswith("#"):
    continue
    print(line)
    From Brandon Rhodes

    Hoisting your I/Os

    View full-size slide

  59. Requirements have changed
    def main():
    with open("/etc/hosts") as file:
    for line in parse_hosts(file):
    print(line)
    def parse_hosts(lines):
    for line in lines:
    if line.startswith("#"):
    continue
    yield line
    59
    Maintaining a Django project after 10.000 commits
    def main():
    lines = requests.get("hosts.com").json():
    with open("local_hosts", "w") as file:
    file.writelines(parse_hosts(lines))
    def parse_hosts(lines):
    for line in lines:
    if line.startswith("#"):
    continue
    yield line
    From Brandon Rhodes

    Hoisting your I/Os

    View full-size slide

  60. Hoist your I/Os
    Brandon Rhodes
    You're not reducing complexity

    by burying I/O down the stack
    Code manipulates

    plain data structures,

    I/O in the top level glue
    60
    Maintaining a Django project after 10.000 commits
    http://bit.ly/hoist-your-ios

    View full-size slide

  61. The Hexagonal Architecture
    By Alistair Cockburn
    Layers
    Dependency rules
    61
    Maintaining a Django project after 10.000 commits
    INSFRASTRUCTURE
    APPLICATION
    DOMAIN

    View full-size slide

  62. The Hexagonal Architecture
    By Alistair Cockburn
    Layers
    Dependency rules
    62
    Maintaining a Django project after 10.000 commits
    INSFRASTRUCTURE
    APPLICATION
    DOMAIN

    View full-size slide

  63. The Hexagonal Architecture
    By Alistair Cockburn
    Layers
    Dependency rules
    63
    Maintaining a Django project after 10.000 commits
    INSFRASTRUCTURE
    APPLICATION
    DOMAIN

    View full-size slide

  64. The Hexagonal Architecture
    By Alistair Cockburn
    Layers
    Dependency rules
    64
    Maintaining a Django project after 10.000 commits
    HTTP, DATABASE
    DJANGO VIEWS, FORMS
    PURE PYTHON

    BUSINESS LOGIC

    View full-size slide

  65. The Clean Architecture in Python
    The Clean Architecture in Python

    Brandon Rhodes, PyOhio 2014
    http://bit.ly/clean-architecture-python
    65
    Maintaining a Django project after 10.000 commits

    View full-size slide

  66. Domain Driven Design
    A concept by Eric Evans
    A LOT of things, at every level
    66
    Maintaining a Django project after 10.000 commits

    View full-size slide

  67. Domain Driven Design
    Domain Curiosity
    Bounded Contexts
    Ubiquitous Language
    67
    Maintaining a Django project after 10.000 commits

    View full-size slide

  68. Being driven by your trade, not by technology
    The tale of the customer address
    68
    Maintaining a Django project after 10.000 commits
    USERNAME
    NAME
    ADDRESS
    ZIP_CODE
    CUSTOMER
    Example from Cyrille Martraire - DDD, en vrai pour le développeur

    View full-size slide

  69. Being driven by your trade, not by technology
    The tale of the customer address
    69
    Maintaining a Django project after 10.000 commits
    USERNAME
    NAME
    ADDRESS
    ZIP_CODE
    ADDRESS_BILLING
    ZIP_CODE_BILLING
    CUSTOMER
    From Cyrille Martraire - DDD, en vrai pour le développeur

    View full-size slide

  70. Being driven by your trade, not by technology
    The tale of the customer address
    70
    Maintaining a Django project after 10.000 commits
    USERNAME
    >SHIPPING ADDRESS
    >BILLING ADDRESS
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    ADDRESS
    From Cyrille Martraire - DDD, en vrai pour le développeur

    View full-size slide

  71. Being driven by your trade, not by technology
    The tale of the customer address
    71
    Maintaining a Django project after 10.000 commits
    USERNAME
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    SHIPPING ADDRESS
    NAME
    ADDRESS
    ZIP_CODE
    BILLING ADDRESS
    From Cyrille Martraire - DDD, en vrai pour le développeur

    View full-size slide

  72. Being driven by your trade, not by technology
    The tale of the customer address
    72
    Maintaining a Django project after 10.000 commits
    USERNAME
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    RECIPIENT
    NAME
    ADDRESS
    ZIP_CODE
    BILLING ADDRESS
    From Cyrille Martraire - DDD, en vrai pour le développeur

    View full-size slide

  73. Being driven by your trade, not by technology
    The tale of the customer address
    73
    Maintaining a Django project after 10.000 commits
    USERNAME
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    RECIPIENT
    NAME
    ADDRESS
    ZIP_CODE
    ACCOUNT
    From Cyrille Martraire - DDD, en vrai pour le développeur

    View full-size slide

  74. Being driven by your trade, not by technology
    The tale of the customer address
    74
    Maintaining a Django project after 10.000 commits
    USERNAME
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    BUILDING _CODE
    DELIVERY_HOURS
    RECIPIENT
    NAME
    ADDRESS
    ZIP_CODE
    ACCOUNT
    From Cyrille Martraire - DDD, en vrai pour le développeur

    View full-size slide

  75. Being driven by your trade, not by technology
    The tale of the customer address
    75
    Maintaining a Django project after 10.000 commits
    USERNAME
    CUSTOMER
    NAME
    ADDRESS
    ZIP_CODE
    BUILDING _CODE
    DELIVERY_HOURS
    RECIPIENT
    NAME
    ADDRESS
    ZIP_CODE
    TAX_CODE
    CURRENCY
    ACCOUNT
    From Cyrille Martraire - DDD, en vrai pour le développeur

    View full-size slide

  76. YOU'RE NOT "JUST"
    BUILDING A WEB APP
    Advice #6
    Maintaining a Django project after 10.000 commits 76

    View full-size slide

  77. The shoulders of giants
    77
    Maintaining a Django project after 10.000 commits
    THE CLEAN ARCHITECTURE
    BRANDON RHODES
    2014
    BOUNDARIES
    GARY BERNHARDT
    2012
    FAST TEST, SLOW TEST
    GARY BERNHARDT
    2012
    HOISTING YOUR I/O
    BRANDON RHODES
    2015
    RADOSLAV GEORGIEV
    DJANGO STRUCTURE...
    2018
    THE CLEAN ARCHITECTURE
    ROBERT MARTIN
    2012
    HEXAGONAL ARCHITECTURE
    ALISTAIR COCKBURN
    2005

    View full-size slide

  78. STAND ON THE
    SHOULDERS OF GIANTS
    (OR TALLER FOLKS)
    Advice #7
    Maintaining a Django project after 10.000 commits 78

    View full-size slide

  79. TESTS
    79
    Part 4 - ⚔
    < Eduardo Otubo [CC BY 2.0]

    View full-size slide

  80. The test pyramid
    80
    Maintaining a Django project after 10.000 commits
    UNIT
    INTEGRATION
    FUNCTIONAL
    X10?
    X30?
    30 SEC
    1 SEC
    10 MS

    View full-size slide

  81. What to test where for a Django view?
    81
    Maintaining a Django project after 10.000 commits
    UNIT
    BUSINESS LOGIC
    VIEW
    PERSISTENCE
    INTEGRATION
    FUNCTIONAL
    TEMPLATE NAME, URL
    PAGE CONTENT
    FLOW CONTEXT

    View full-size slide

  82. DON'T JUST WRITE TESTS,

    WRITE THE RIGHT TESTS.
    Advice #8
    Maintaining a Django project after 10.000 commits 82

    View full-size slide

  83. Code quality criteria in tests
    Is a criteria:
    Dead easy to read,

    from start to tear down
    One test tests one thing
    Have a long

    and descriptive name
    83
    Maintaining a Django project after 10.000 commits
    Is not a criteria:
    Smart adaptable code
    DRY - Don't repeat yourself
    Layering and

    hidden complexity

    View full-size slide

  84. IN YOUR TESTS,
    AIM FOR THE SIMPLEST

    TO READ AND WRITE
    Advice #9
    Maintaining a Django project after 10.000 commits 84

    View full-size slide

  85. Things that get complicated in the long run
    Test mixins
    self.client.get()/post()
    Fixture files
    85
    Maintaining a Django project after 10.000 commits

    View full-size slide

  86. The right tools for the job
    Use pytest and pytest style function tests
    Use pytest fixtures (avoid auto-use)
    And use Factory Boy
    86
    Maintaining a Django project after 10.000 commits

    View full-size slide

  87. Tests: going further
    87
    Maintaining a Django project after 10.000 commits
    Behavior Driven Development
    Snapshot testing

    View full-size slide

  88. Behavior Driven Development
    Feature: Blog
    A site where you can publish
    your articles.
    Scenario: Publishing the article
    Given I'm an author user
    And I have an article
    When I go to the article page
    And I press the publish button
    Then the article should be published
    88
    Maintaining a Django project after 10.000 commits

    View full-size slide

  89. 89
    Maintaining a Django project after 10.000 commits
    @scenario('publish_article.feature',
    'Publishing the article')
    def test_publish(): ...
    @given("I'm an author user")
    def author_user(auth, author): ...
    @given('I have an article')
    def article(author): ...
    @when('I go to the article page')
    def go_to_article(article, browser): ...
    @when('I press the publish button')
    def publish_article(browser): ...
    @then('the article should be published')
    def article_is_published(article): ...

    View full-size slide

  90. Behavior Driven Development
    Document your specifications by design
    Functional tests from day 1
    90
    Maintaining a Django project after 10.000 commits

    View full-size slide

  91. Snapshot testing is useful too
    Screenshot testing
    SQL testing (django-perf-rec)
    HTML testing? API output testing?
    91
    Maintaining a Django project after 10.000 commits

    View full-size slide

  92. MAKE YOUR TESTS
    A PRIME PART
    OF YOUR PROCESS
    Advice #10
    Maintaining a Django project after 10.000 commits 92

    View full-size slide

  93. STAY CURIOUS,
    TRY THINGS,
    THINK BEFORE YOU CODE
    Final advice
    Maintaining a Django project after 10.000 commits 93

    View full-size slide

  94. 94
    Maintaining a Django project after 10.000 commits
    STAY CURIOUS, TRY THINGS,
    THINK BEFORE YOU CODE
    Final advice
    Final-Final advice
    DON'T OVER-ENGINEER IT!

    View full-size slide

  95. THANK YOU!
    QUESTIONS?
    95
    Joachim Jablon - @ewjoachim
    Stéphane "Twidi" Angel - @Twidi
    We're hiring!
    Try it out!

    View full-size slide