Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Digging Into Django's Migrations

Andrew Godwin
September 03, 2014

Digging Into Django's Migrations

My keynote from DjangoCon US 2014

Andrew Godwin

September 03, 2014
Tweet

More Decks by Andrew Godwin

Other Decks in Programming

Transcript

  1. 's migrations
    digging into

    View Slide

  2. Andrew Godwin
    Author of South migrations library
    Hi, I'm
    Author of 1.7 Django migrations
    Senior Software Engineer at

    View Slide

  3. south migrations for django
    south.aeracode.org
    DjangoCon 2008
    https://speakerdeck.com/andrewgodwin/south-migrations-for-django

    View Slide

  4. Migrations:
    They're pretty good.

    View Slide

  5. South
    0.1
    0.2
    0.3
    0.4
    0.5
    0.6
    0.7
    0.8
    1.0
    Aug 2008
    Aug 2008
    Sep 2008
    Jan 2009
    Apr 2009
    Jun 2009
    Mar 2010
    May 2013
    Jul 2014

    View Slide

  6. The Early Years
    0.1
    0.2
    0.3
    0.4
    0.5
    0.6
    0.7
    0.8
    1.0
    Aug 2008
    Aug 2008
    Sep 2008
    Jan 2009
    Apr 2009
    Jun 2009
    Mar 2010
    May 2013
    Jul 2014

    View Slide

  7. Approaching Stability
    0.1
    0.2
    0.3
    0.4
    0.5
    0.6
    0.7
    0.8
    1.0
    Aug 2008
    Aug 2008
    Sep 2008
    Jan 2009
    Apr 2009
    Jun 2009
    Mar 2010
    May 2013
    Jul 2014
    Well, in the short term,
    South 1.0 will be released.


    June 2010

    View Slide

  8. The Long Gap & Kickstarter
    0.1
    0.2
    0.3
    0.4
    0.5
    0.6
    0.7
    0.8
    1.0
    Aug 2008
    Aug 2008
    Sep 2008
    Jan 2009
    Apr 2009
    Jun 2009
    Mar 2010
    May 2013
    Jul 2014

    View Slide

  9. Today
    0.1
    0.2
    0.3
    0.4
    0.5
    0.6
    0.7
    0.8
    1.0
    Aug 2008
    Aug 2008
    Sep 2008
    Jan 2009
    Apr 2009
    Jun 2009
    Mar 2010
    May 2013
    Jul 2014

    View Slide

  10. For at least a year now, people have
    been suggesting to me that South
    should be in Django core.


    June 2010

    View Slide

  11. The Initial Plan
    Django
    Schema backend
    ORM Hooks
    South 2
    Migration handling
    User interface

    View Slide

  12. The Revised Plan
    Django
    Schema backend
    ORM Hooks
    South 2
    Migration handling
    User interface
    Backport for 1.4 - 1.6

    View Slide

  13. The Revised Revised Plan
    Django
    Schema backend
    ORM Hooks
    Migration handling
    User interface

    View Slide

  14. Moving South into Django
    instead, "Adding migrations to Django"

    View Slide

  15. Logically Separate
    SchemaEditor
    Schema Migrations
    field.deconstruct()
    ModelOptions.apps
    Operations
    Loader / Graph
    Executor
    Autodetector
    Optimiser
    State

    View Slide

  16. Operations & State

    View Slide

  17. Operations are a declarative
    representation of model changes
    State is an in-memory representation
    of entire project state

    View Slide

  18. State State
    Operation
    AddModel
    No Models 1 Model

    View Slide

  19. State State
    Operation
    State
    Operation
    State
    Operation
    State
    Operation
    Migration 1 Migration 2

    View Slide

  20. AddModel DeleteModel RenameModel
    AddField DeleteField AlterModelOptions
    RenameField AlterModelTable AlterField
    AlterUniqueTogether AlterIndexTogether
    RunSQL RunPython

    View Slide

  21. SchemaEditor

    View Slide

  22. connection.schema_editor()
    Abstraction over database DDL
    Takes Django models and fields

    View Slide

  23. with connection.schema_editor() as ed:
    ed.create_model(Author)
    ed.add_field(
    Book,
    "author",
    models.ForeignKey(Author),
    )

    View Slide

  24. Field Deconstruction

    View Slide

  25. Every field has deconstruct()
    Returns arguments passed to
    __init__ to duplicate itself

    View Slide

  26. >>> field = CharField(max_length=255)
    >>> field.deconstruct()
    (
    None,
    'django.db.models.CharField',
    [],
    {'max_length': 255}
    )

    View Slide

  27. The Graph

    View Slide

  28. Represents migrations and
    dependencies as directed graph
    Loop detection, leaf/root selection

    View Slide

  29. myapp/0001
    myapp/0002 otherapp/0001
    contenttypes/0001
    appthree/0001
    myapp/0003 otherapp/0002

    View Slide

  30. The Autodetector

    View Slide

  31. Compares two States and outputs
    a set of operations to perform
    Operations have own dependencies
    and resolver

    View Slide

  32. Model A Model A w/FK, Model B
    diff
    AddModel(B)
    AddField(A, "b", FK(B))
    dependency sort
    AddModel(B) AddField(A, "b", FK(B))
    BEFORE AFTER

    View Slide

  33. The Optimiser

    View Slide

  34. Takes one set of Operations and
    outputs another with the same effect
    Steps over pairs and checks if they
    combine and what they pass over

    View Slide

  35. can't
    reduce
    reduce
    possible
    check
    conflicts
    reduced,
    reset

    View Slide

  36. A New Format
    More concise
    Declarative
    Introspectable

    View Slide

  37. Migration actions
    Frozen ORM

    View Slide

  38. View Slide

  39. makemigrations
    field.deconstruct()
    Loader / Graph
    Autodetector Optimiser
    State
    Writer
    1
    2
    State
    3
    5
    4

    View Slide

  40. migrate
    SchemaEditor
    ModelOptions.apps
    Operations
    Loader / Graph Executor
    State
    1 2
    Recorder
    3

    View Slide

  41. In-memory running
    Creates models from migration sets
    Autodetector diffs created from on-disk
    Used to feed SchemaEditor / ORM

    View Slide

  42. State State
    Operation
    State
    Operation
    State
    Operation
    State
    Operation
    Migration 1 Migration 2
    Models
    Models
    Render Render

    View Slide

  43. But what went wrong?

    View Slide

  44. Swappable Models

    View Slide

  45. Your migration dependencies
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001

    View Slide

  46. myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    Your migration dependencies
    on swappable models

    View Slide

  47. what?
    Your migration dependencies
    on swappable models
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    ???

    View Slide

  48. what?
    Your migration dependencies
    on swappable models
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    ???

    View Slide

  49. Unmigrated Apps

    View Slide

  50. Test persistence

    View Slide

  51. Test persistence
    on MySQL

    View Slide

  52. Random Meta options
    order_with_respect_to? Really?

    View Slide

  53. Proxy Models

    View Slide

  54. But we're there!
    Django 1.7, out now!

    View Slide

  55. Thanks!
    Andrew Godwin
    [email protected]

    View Slide