Tweet
Tweet

Slide 1

Slide 1 text

MAINTAINING A DJANGO PROJECT AFTER 10.000 COMMITS Joachim Jablon - PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen

Slide 2

Slide 2 text

WHO ARE WE? 2 Joachim Jablon / ewjoachim @ewjoachim / ewjoach.im [email protected]

Slide 3

Slide 3 text

WHO ARE WE? 3 Stéphane "Twidi" Angel @twidi / isshub.io

Slide 4

Slide 4 text

MAINTAINING A DJANGO PROJECT AFTER 10.000 COMMITS Joachim Jablon - PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen

Slide 5

Slide 5 text

MAINTAINING A DJANGO PROJECT AFTER BEFORE 10.000 COMMITS Joachim Jablon - PeopleDoc Stéphane "Twidi" Angel - isshub.io DjangoCon Europe 2019 - Copenhagen

Slide 6

Slide 6 text

MONOLI.TH 6 Let's disrupt the stone business with < https://fr.m.wikipedia.org/wiki/Fichier:Kerloas_menhir.JPG

Slide 7

Slide 7 text

Our flagship project > 1h test, > 100 dependencies > 100 models, > 1000 fields > 100M rows in some tables 7 Maintaining a Django project after 10.000 commits This is purely fictional, any resemblance to actual project living or dead is purely coincidental

Slide 8

Slide 8 text

The project zoo Unit & Integration tests Health checks & monitoring Async & cron tasks APIs i18n & l10n Logging Error reporting Packaging Deployment code Dev environment Management commands Linting Custom tooling Changelog system Partial documentation Deprecated APIs Complex
 server-side UI Half-baked initiatives
 ("ongoing efforts") Dead code Memory leaks "Don't touch that code!" Spaghetti code Help Send help please 8 Maintaining a Django project after 10.000 commits

Slide 9

Slide 9 text

Time flies Team turnover Company pivoting Scale and split Complexity grows Abandoned third parties 9 Maintaining a Django project after 10.000 commits

Slide 10

Slide 10 text

THE OF OUR CODE THE ❤ OF OUR CODE THE OF OUR CODE THE ⚔ OF OUR CODE 10

Slide 11

Slide 11 text

THIRD PARTY DEPENDENCIES 11 Part 1 - https://commons.wikimedia.org/wiki/File:Coastal-rocks.jpg >

Slide 12

Slide 12 text

What is a third party dependency? 12 Maintaining a Django project after 10.000 commits Some external code used by your project

Slide 13

Slide 13 text

Reasons to use 3rd party dependencies 13 Maintaining a Django project after 10.000 commits Lower the effort / time / cost Lower the risk

Slide 14

Slide 14 text

Reasons NOT to use 3rd party dependencies 14 Maintaining a Django project after 10.000 commits Increase the effort / time / cost Increase the risk

Slide 15

Slide 15 text

The problems 15 Maintaining a Django project after 10.000 commits

Slide 16

Slide 16 text

The problems 16 Maintaining a Django project after 10.000 commits ▸ Security updates

Slide 17

Slide 17 text

The problems 17 Maintaining a Django project after 10.000 commits Security updates ▸ Project direction

Slide 18

Slide 18 text

The problems 18 Maintaining a Django project after 10.000 commits Security updates Project direction ▸ Project lagging behind

Slide 19

Slide 19 text

The problems 19 Maintaining a Django project after 10.000 commits Security updates Project direction Project lagging behind ▸ Contributions

Slide 20

Slide 20 text

The problems 20 Maintaining a Django project after 10.000 commits Security updates Project direction Project lagging behind Contributions ▸ Project becoming inactive

Slide 21

Slide 21 text

The problems 21 Maintaining a Django project after 10.000 commits Security updates Project direction Project lagging behind Contributions Project becoming inactive ▸ Partial need

Slide 22

Slide 22 text

NEVER FORGET THE HIDDEN COSTS OF ADDING 3RD PARTIES Advice #1 Maintaining a Django project after 10.000 commits 22

Slide 23

Slide 23 text

All is not lost 23 Maintaining a Django project after 10.000 commits

Slide 24

Slide 24 text

Sometimes, it's just ok You have ownership Complex scoped problem Official packages Django backend implementation Well known apps 24 Maintaining a Django project after 10.000 commits

Slide 25

Slide 25 text

Testing Test the next versions It's ok it it fails At least, you know. 25 Maintaining a Django project after 10.000 commits

Slide 26

Slide 26 text

Participate Issues / PR Be nice 26 Maintaining a Django project after 10.000 commits

Slide 27

Slide 27 text

Taking ownership 27 Maintaining a Django project after 10.000 commits

Slide 28

Slide 28 text

Taking ownership ▸ Fork 28 Maintaining a Django project after 10.000 commits

Slide 29

Slide 29 text

Taking ownership Fork ▸ Vendor 29 Maintaining a Django project after 10.000 commits

Slide 30

Slide 30 text

Taking ownership Fork Vendor ▸ Implement 30 Maintaining a Django project after 10.000 commits

Slide 31

Slide 31 text

Adapter Build an abstraction layer Only one module can import them
 (you may enforce this with bellybutton) No *args, **kwargs 31 Maintaining a Django project after 10.000 commits YOUR CODE MY_GITHUB.PY GITHUB CLIENT

Slide 32

Slide 32 text

ISOLATE YOUR
 THIRD PARTIES Advice #2 Maintaining a Django project after 10.000 commits 32

Slide 33

Slide 33 text

Special aside for django models You need to have the entire ownership on your database External app has migrations: ❌ External app has abstract models defining fields: ⚠ 33 Maintaining a Django project after 10.000 commits

Slide 34

Slide 34 text

DON'T LET ANYONE DEFINE YOUR MODELS Advice #3 Maintaining a Django project after 10.000 commits 34

Slide 35

Slide 35 text

DON'T FORGET TO CONTRIBUTE BACK TO OPEN SOURCE Advice #4 Maintaining a Django project after 10.000 commits 35

Slide 36

Slide 36 text

Could we imagine the same approach with Django itself? 36 Maintaining a Django project after 10.000 commits

Slide 37

Slide 37 text

BUSINESS LOGIC 37 Part 2 - ❤ < https://pxhere.com/en/photo/597168

Slide 38

Slide 38 text

What is business logic? 38 Maintaining a Django project after 10.000 commits DJANGO BUSINESS LOGIC HTTP DATABASE

Slide 39

Slide 39 text

What is business logic? 39 Maintaining a Django project after 10.000 commits DJANGO FLASK BUSINESS LOGIC HTTP DATABASE

Slide 40

Slide 40 text

What is business logic? "When a user is created, they get a welcome email." "Only premium users can access this page." "On a course, start date must be before end date." 40 Maintaining a Django project after 10.000 commits

Slide 41

Slide 41 text

The problem 41 Maintaining a Django project after 10.000 commits BUSINESS LOGIC DJANGO HTTP DATABASE

Slide 42

Slide 42 text

The cause There's business logic in my templates ... And in my forms & serializers ... And in my views ... And in my models & managers And all these modules already have another purpose 42 Maintaining a Django project after 10.000 commits

Slide 43

Slide 43 text

Service layer: 2 talks to watch Hanna Kollo
 Avoiding Monoliths http://bit.ly/dce15-monolith 43 Maintaining a Django project after 10.000 commits Radoslav Georgiev
 Django structure for scale and longevity http://bit.ly/ep18-django-scale

Slide 44

Slide 44 text

A DJANGO APP Service layers: key takeaways Every non-trivial operation
 should be a service / selector: services.py for create / update selectors.py for get / list no form.save(), serializer.save(), CreateView etc 44 Maintaining a Django project after 10.000 commits From Radoslav Georgiev
 Django structure for scale and longevity SERVICES.PY SELECTORS.PY MODELS.PY FORMS.PY

Slide 45

Slide 45 text

Service layer 45 Maintaining a Django project after 10.000 commits HTTP DATABASE BUSINESS LOGIC DJANGO VIEWS, FORMS, ETC ORM

Slide 46

Slide 46 text

The sandwich architecture 46 Maintaining a Django project after 10.000 commits HTTP DATABASE PURE PYTHON BUSINESS LOGIC DJANGO VIEWS, FORMS, ETC ORM

Slide 47

Slide 47 text

And beyond? 47 Maintaining a Django project after 10.000 commits HTTP DATABASE DJANGO VIEWS, FORMS, ETC ORM PURE PYTHON BUSINESS LOGIC

Slide 48

Slide 48 text

And beyond? 48 Maintaining a Django project after 10.000 commits PURE PYTHON
 BUSINESS LOGIC HTTP, DATABASE DJANGO VIEWS, FORMS

Slide 49

Slide 49 text

SPLIT YOUR BUSINESS LOGIC, DJANGO VIEWS AND THE ORM Advice #5 Maintaining a Django project after 10.000 commits 49 Did I hear anyone say "MVC"?

Slide 50

Slide 50 text

Is this Java? ... But then so what? 50 Maintaining a Django project after 10.000 commits

Slide 51

Slide 51 text

BOXES 51 Part 3 - https://www.flickr.com/photos/ikewinski/8013010600 >

Slide 52

Slide 52 text

Software architecture Put things in the right boxes Name your boxes Don't mix boxes Define links between boxes 52 Maintaining a Django project after 10.000 commits < https://www.pexels.com/photo/gem-stones-souvenir-952681/

Slide 53

Slide 53 text

Boxes Functional core / Imperative shell + Hoist your I/Os The Hexagonal Architecture Domain Driven Design 53 Maintaining a Django project after 10.000 commits

Slide 54

Slide 54 text

Functional core / Imperative shell Gary Bernhardt Separating the logic from the glue 54 Maintaining a Django project after 10.000 commits

Slide 55

Slide 55 text

Functional core / Imperative shell Decision making lives in
 functions without side effects 55 Maintaining a Django project after 10.000 commits FUNCTIONAL CORE FUNCTIONAL CORE FUNCTIONAL CORE

Slide 56

Slide 56 text

Functional core / Imperative shell The outer shell has all
 the side effects but no logic 56 Maintaining a Django project after 10.000 commits IMPERATIVE SHELL

Slide 57

Slide 57 text

Functional core / Imperative shell 57 Maintaining a Django project after 10.000 commits FUNCTIONAL CORE FUNCTIONAL CORE FUNCTIONAL CORE IMPERATIVE SHELL IMPERATIVE SHELL IMPERATIVE SHELL From Gary Bernhardt
 Boundaries

Slide 58

Slide 58 text

Quiz time! def main(): with open("/etc/hosts") as file: for line in parse_hosts(file): print(line) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line 58 Maintaining a Django project after 10.000 commits def main(): parse_hosts() def parse_hosts(): with open("/etc/hosts") as lines: for line in lines: if line.startswith("#"): continue print(line) From Brandon Rhodes
 Hoisting your I/Os

Slide 59

Slide 59 text

Requirements have changed def main(): with open("/etc/hosts") as file: for line in parse_hosts(file): print(line) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line 59 Maintaining a Django project after 10.000 commits def main(): lines = requests.get("hosts.com").json(): with open("local_hosts", "w") as file: file.writelines(parse_hosts(lines)) def parse_hosts(lines): for line in lines: if line.startswith("#"): continue yield line From Brandon Rhodes
 Hoisting your I/Os

Slide 60

Slide 60 text

Hoist your I/Os Brandon Rhodes You're not reducing complexity
 by burying I/O down the stack Code manipulates
 plain data structures,
 I/O in the top level glue 60 Maintaining a Django project after 10.000 commits http://bit.ly/hoist-your-ios

Slide 61

Slide 61 text

The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 61 Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN

Slide 62

Slide 62 text

The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 62 Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN

Slide 63

Slide 63 text

The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 63 Maintaining a Django project after 10.000 commits INSFRASTRUCTURE APPLICATION DOMAIN

Slide 64

Slide 64 text

The Hexagonal Architecture By Alistair Cockburn Layers Dependency rules 64 Maintaining a Django project after 10.000 commits HTTP, DATABASE DJANGO VIEWS, FORMS PURE PYTHON
 BUSINESS LOGIC

Slide 65

Slide 65 text

The Clean Architecture in Python The Clean Architecture in Python
 Brandon Rhodes, PyOhio 2014 http://bit.ly/clean-architecture-python 65 Maintaining a Django project after 10.000 commits

Slide 66

Slide 66 text

Domain Driven Design A concept by Eric Evans A LOT of things, at every level 66 Maintaining a Django project after 10.000 commits

Slide 67

Slide 67 text

Domain Driven Design Domain Curiosity Bounded Contexts Ubiquitous Language 67 Maintaining a Django project after 10.000 commits

Slide 68

Slide 68 text

Being driven by your trade, not by technology The tale of the customer address 68 Maintaining a Django project after 10.000 commits USERNAME NAME ADDRESS ZIP_CODE CUSTOMER Example from Cyrille Martraire - DDD, en vrai pour le développeur

Slide 69

Slide 69 text

Being driven by your trade, not by technology The tale of the customer address 69 Maintaining a Django project after 10.000 commits USERNAME NAME ADDRESS ZIP_CODE ADDRESS_BILLING ZIP_CODE_BILLING CUSTOMER From Cyrille Martraire - DDD, en vrai pour le développeur

Slide 70

Slide 70 text

Being driven by your trade, not by technology The tale of the customer address 70 Maintaining a Django project after 10.000 commits USERNAME >SHIPPING ADDRESS >BILLING ADDRESS CUSTOMER NAME ADDRESS ZIP_CODE ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur

Slide 71

Slide 71 text

Being driven by your trade, not by technology The tale of the customer address 71 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE SHIPPING ADDRESS NAME ADDRESS ZIP_CODE BILLING ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur

Slide 72

Slide 72 text

Being driven by your trade, not by technology The tale of the customer address 72 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE RECIPIENT NAME ADDRESS ZIP_CODE BILLING ADDRESS From Cyrille Martraire - DDD, en vrai pour le développeur

Slide 73

Slide 73 text

Being driven by your trade, not by technology The tale of the customer address 73 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE RECIPIENT NAME ADDRESS ZIP_CODE ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur

Slide 74

Slide 74 text

Being driven by your trade, not by technology The tale of the customer address 74 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE BUILDING _CODE DELIVERY_HOURS RECIPIENT NAME ADDRESS ZIP_CODE ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur

Slide 75

Slide 75 text

Being driven by your trade, not by technology The tale of the customer address 75 Maintaining a Django project after 10.000 commits USERNAME CUSTOMER NAME ADDRESS ZIP_CODE BUILDING _CODE DELIVERY_HOURS RECIPIENT NAME ADDRESS ZIP_CODE TAX_CODE CURRENCY ACCOUNT From Cyrille Martraire - DDD, en vrai pour le développeur

Slide 76

Slide 76 text

YOU'RE NOT "JUST" BUILDING A WEB APP Advice #6 Maintaining a Django project after 10.000 commits 76

Slide 77

Slide 77 text

The shoulders of giants 77 Maintaining a Django project after 10.000 commits THE CLEAN ARCHITECTURE BRANDON RHODES 2014 BOUNDARIES GARY BERNHARDT 2012 FAST TEST, SLOW TEST GARY BERNHARDT 2012 HOISTING YOUR I/O BRANDON RHODES 2015 RADOSLAV GEORGIEV DJANGO STRUCTURE... 2018 THE CLEAN ARCHITECTURE ROBERT MARTIN 2012 HEXAGONAL ARCHITECTURE ALISTAIR COCKBURN 2005

Slide 78

Slide 78 text

STAND ON THE SHOULDERS OF GIANTS (OR TALLER FOLKS) Advice #7 Maintaining a Django project after 10.000 commits 78

Slide 79

Slide 79 text

TESTS 79 Part 4 - ⚔ < Eduardo Otubo [CC BY 2.0]

Slide 80

Slide 80 text

The test pyramid 80 Maintaining a Django project after 10.000 commits UNIT INTEGRATION FUNCTIONAL X10? X30? 30 SEC 1 SEC 10 MS

Slide 81

Slide 81 text

What to test where for a Django view? 81 Maintaining a Django project after 10.000 commits UNIT BUSINESS LOGIC VIEW PERSISTENCE INTEGRATION FUNCTIONAL TEMPLATE NAME, URL PAGE CONTENT FLOW CONTEXT

Slide 82

Slide 82 text

DON'T JUST WRITE TESTS,
 WRITE THE RIGHT TESTS. Advice #8 Maintaining a Django project after 10.000 commits 82

Slide 83

Slide 83 text

Code quality criteria in tests Is a criteria: Dead easy to read,
 from start to tear down One test tests one thing Have a long
 and descriptive name 83 Maintaining a Django project after 10.000 commits Is not a criteria: Smart adaptable code DRY - Don't repeat yourself Layering and
 hidden complexity

Slide 84

Slide 84 text

IN YOUR TESTS, AIM FOR THE SIMPLEST
 TO READ AND WRITE Advice #9 Maintaining a Django project after 10.000 commits 84

Slide 85

Slide 85 text

Things that get complicated in the long run Test mixins self.client.get()/post() Fixture files 85 Maintaining a Django project after 10.000 commits

Slide 86

Slide 86 text

The right tools for the job Use pytest and pytest style function tests Use pytest fixtures (avoid auto-use) And use Factory Boy 86 Maintaining a Django project after 10.000 commits

Slide 87

Slide 87 text

Tests: going further 87 Maintaining a Django project after 10.000 commits Behavior Driven Development Snapshot testing

Slide 88

Slide 88 text

Behavior Driven Development Feature: Blog A site where you can publish your articles. Scenario: Publishing the article Given I'm an author user And I have an article When I go to the article page And I press the publish button Then the article should be published 88 Maintaining a Django project after 10.000 commits

Slide 89

Slide 89 text

89 Maintaining a Django project after 10.000 commits @scenario('publish_article.feature', 'Publishing the article') def test_publish(): ... @given("I'm an author user") def author_user(auth, author): ... @given('I have an article') def article(author): ... @when('I go to the article page') def go_to_article(article, browser): ... @when('I press the publish button') def publish_article(browser): ... @then('the article should be published') def article_is_published(article): ...

Slide 90

Slide 90 text

Behavior Driven Development Document your specifications by design Functional tests from day 1 90 Maintaining a Django project after 10.000 commits

Slide 91

Slide 91 text

Snapshot testing is useful too Screenshot testing SQL testing (django-perf-rec) HTML testing? API output testing? 91 Maintaining a Django project after 10.000 commits

Slide 92

Slide 92 text

MAKE YOUR TESTS A PRIME PART OF YOUR PROCESS Advice #10 Maintaining a Django project after 10.000 commits 92

Slide 93

Slide 93 text

STAY CURIOUS, TRY THINGS, THINK BEFORE YOU CODE Final advice Maintaining a Django project after 10.000 commits 93

Slide 94

Slide 94 text

94 Maintaining a Django project after 10.000 commits STAY CURIOUS, TRY THINGS, THINK BEFORE YOU CODE Final advice Final-Final advice DON'T OVER-ENGINEER IT!

Slide 95

Slide 95 text

THANK YOU! QUESTIONS? 95 Joachim Jablon - @ewjoachim Stéphane "Twidi" Angel - @Twidi We're hiring! Try it out!