Dungeon Master's guide to Django's ORM

Transcript

1. Dungeon Master's guide to Django's ORM Malcolm Tredinnick

2. Me • Python user since 1997 • Database dabbler since

   the 1980's • Django user since September 2005 • Django committer since May 2006 • Fingers in most parts of the code base

3. Some History

4. 12 July 2005, 8:25 pm CDT • “Imported Django from

   private SVN repository” • django/core/meta.py was ORM code • 2,142 lines

5. 30 January, 2006 • “Added 'When will you release Django

   1.0?' to FAQ” • “[...]This should happen in a couple of months or so, although it's entirely possible that it could happen earlier. That translates into February or March 2006.”

6. 1 May, 2006 • “MERGED MAGIC-REMOVAL BRANCH TO TRUNK. This

   change is highly backwards- incompatible.” • django/db/models/query.py appears. • parse_lookup() and lookup_inner().

7. Late July 2006 • I start to look at restructuring

   of SQL generation. • Three fairly different approaches attempted – first one worked, but was probably too complex. – second one sucked. I don't like to talk about it. – third one is what we have now.

8. 4 July 2007 • “Merged Unicode branch into trunk (r4952:5608).

   This should be fully backwards compatible for all practical purposes.” • Made database encoding much easier. • Bought some goodwill.

9. Sunday, 27 April 2008 Merged the queryset-refactor branch into trunk.

   This is a big internal change, but mostly backwards compatible with existing code. Also adds a couple of new features. Fixed #245, #1050, #1656, #1801, #2076, #2091, #2150, #2253, #2306, #2400, #2430, #2482, #2496, #2676, #2737, #2874, #2902, #2939, #3037, #3141, #3288, #3440, #3592, #3739, #4088, #4260, #4289, #4306, #4358, #4464, #4510, #4858, #5012, #5020, #5261, #5295, #5321, #5324, #5325, #5555, #5707, #5796, #5817, #5987, #6018, #6074, #6088, #6154, #6177, #6180, #6203, #6658

10. 3 September 2008, 7:38pm EDT • Django 1.0 • First

    DjangoCon on weekend of 6 – 7 September.

11. 15 January, 2009 • “Fixed #3566 -- Added support for

    aggregation to the ORM.” • Mostly a Google Summer of Code project by Nicolas Lara. • Supervision, carrot and stick provided by Russell Keith-Magee.

12. Was it worth it?

13. Was it worth it? • Code structure has remained fairly

    stable. • The layers of abstraction feel roughly correct. • Minor code duplication in recent times. – Hmm... code sprints later this week • Mostly logical code flow. (Stop laughing!)

14. How does it work?

15. Interlude: a useful rule The developers who were here before

    you were probably not insane.

16. Code layers • django/db/models/ – managers.py – query.py (~1800 lines)

    • django/db/models/sql/ – query.py (~2000 lines) – compiler.py (~1100 lines) • django/db/backends/*/ – base.py – operations.py

17. A trip down the rabbit hole Article.objects.filter(date__lte=today, author__byline=”anonymous”)

18. The classes • django.db.models.query.QuerySet • django.db.models.sql.query.Query • django.db.models.sql.where.WhereNode • django.db.models.sql.compiler.SQLCompiler

19. filter(date__lte=today, author__byline=”anonymous”) QuerySet._filter_or_exclude(negate=False, date__lte=today, author__byline=”anonymous”)

20. QuerySet._filter_or_exclude(negate=False, date__lte=today, author__byline=”anonymous”) Query.add_q( Q(date__lte=today, author__byline=”anonymous”) )

21. Query.add_q( Q(date__lte=today, author__byline=”anonymous”) ) (“date__lte”, today) (“author__byline”, “anonymous”) Query.add_filter( ...

    )

22. Query.add_filter((“author__byline”, “anonymous”), ... ) Query.setup_joins((“author”, “byline”), ...) fields = (“author”,

    “byline”) lookup_type = “exact” value = “anonymous”

23. Query.setup_joins((“author”, “byline”), ...) Query.alias_map = { “T1”: (“app_article”, “T1”, None,

    None, None, None, False), “T2”: (“app_author”, “T2”, INNER, “T1”, “author”, “id”, False), ... } nullable join?

24. Query.add_filter((“author__byline”, “anonymous”), ... ) WhereNode.add( ..., “exact”, “anonymous”) fields =

    (“author”, “byline”) lookup_type = “exact” value = “anonymous”

25. In the background... • Aliases are reference counted – sometimes

    joins can be trimmed away (for speed) • order() can cause new joins to be added • So can model inheritance • SQL “HAVING” is like “WHERE” – also uses WhereNode

26. In the background... • All aliases in a QuerySet can

    be changed at once – T1, T2, T3, .. --> U1, U2, U3, ... – for nested queries • QuerySets can be merged • Same table can appear with different aliases

27. In the background... • select_related(), defer(), only() affect columns to

    be selected

28. Cloning QuerySets • Happens a lot! – qs . filter(...)

    . filter( ...) <-- 3 clones • Effectively as fast as possible; still slow. • We need to add a “no_clone” experts-only feature.

29. Executing a query QuerySet.__iter__() QuerySet.iterator() Query.get_compiler( ... ) SQLCompiler.results_iter() SQLCompiler.as_sql()

    Create SQL Return results (rows)

30. Food for thought • Look at Query.add_q() – can handle

    custom Q-like objects – E.g. create some to utilise advanced DB features • Look at expression objects – django.db.models.expressions.ExpressionNode – create custom F-like objects

31. Anybody still awake? Question time http://speakerdeck.com/u/malcolmt/