Digging Into Django's Migrations

My keynote from DjangoCon US 2014

Transcript

1. 's migrations
   digging into
   

   View Slide

2. Andrew Godwin
   Author of South migrations library
   Hi, I'm
   Author of 1.7 Django migrations
   Senior Software Engineer at
   

   View Slide

3. south migrations for django
   south.aeracode.org
   DjangoCon 2008
   https://speakerdeck.com/andrewgodwin/south-migrations-for-django
   

   View Slide

4. Migrations:
   They're pretty good.
   

   View Slide

5. South
   0.1
   0.2
   0.3
   0.4
   0.5
   0.6
   0.7
   0.8
   1.0
   Aug 2008
   Aug 2008
   Sep 2008
   Jan 2009
   Apr 2009
   Jun 2009
   Mar 2010
   May 2013
   Jul 2014
   

   View Slide

6. The Early Years
   0.1
   0.2
   0.3
   0.4
   0.5
   0.6
   0.7
   0.8
   1.0
   Aug 2008
   Aug 2008
   Sep 2008
   Jan 2009
   Apr 2009
   Jun 2009
   Mar 2010
   May 2013
   Jul 2014
   

   View Slide

7. Approaching Stability
   0.1
   0.2
   0.3
   0.4
   0.5
   0.6
   0.7
   0.8
   1.0
   Aug 2008
   Aug 2008
   Sep 2008
   Jan 2009
   Apr 2009
   Jun 2009
   Mar 2010
   May 2013
   Jul 2014
   Well, in the short term,
   South 1.0 will be released.
   “
   “
   June 2010
   

   View Slide

8. The Long Gap & Kickstarter
   0.1
   0.2
   0.3
   0.4
   0.5
   0.6
   0.7
   0.8
   1.0
   Aug 2008
   Aug 2008
   Sep 2008
   Jan 2009
   Apr 2009
   Jun 2009
   Mar 2010
   May 2013
   Jul 2014
   

   View Slide

9. Today
   0.1
   0.2
   0.3
   0.4
   0.5
   0.6
   0.7
   0.8
   1.0
   Aug 2008
   Aug 2008
   Sep 2008
   Jan 2009
   Apr 2009
   Jun 2009
   Mar 2010
   May 2013
   Jul 2014
   

   View Slide

10. For at least a year now, people have
    been suggesting to me that South
    should be in Django core.
    “
    “
    June 2010
    

    View Slide

11. The Initial Plan
    Django
    Schema backend
    ORM Hooks
    South 2
    Migration handling
    User interface
    

    View Slide

12. The Revised Plan
    Django
    Schema backend
    ORM Hooks
    South 2
    Migration handling
    User interface
    Backport for 1.4 - 1.6
    

    View Slide

13. The Revised Revised Plan
    Django
    Schema backend
    ORM Hooks
    Migration handling
    User interface
    

    View Slide

14. Moving South into Django
    instead, "Adding migrations to Django"
    

    View Slide

15. Logically Separate
    SchemaEditor
    Schema Migrations
    field.deconstruct()
    ModelOptions.apps
    Operations
    Loader / Graph
    Executor
    Autodetector
    Optimiser
    State
    

    View Slide

16. Operations & State
    

    View Slide

17. Operations are a declarative
    representation of model changes
    State is an in-memory representation
    of entire project state
    

    View Slide

18. State State
    Operation
    AddModel
    No Models 1 Model
    

    View Slide

19. State State
    Operation
    State
    Operation
    State
    Operation
    State
    Operation
    Migration 1 Migration 2
    

    View Slide

20. AddModel DeleteModel RenameModel
    AddField DeleteField AlterModelOptions
    RenameField AlterModelTable AlterField
    AlterUniqueTogether AlterIndexTogether
    RunSQL RunPython
    

    View Slide

21. SchemaEditor
    

    View Slide

22. connection.schema_editor()
    Abstraction over database DDL
    Takes Django models and fields
    

    View Slide

23. with connection.schema_editor() as ed:
    ed.create_model(Author)
    ed.add_field(
    Book,
    "author",
    models.ForeignKey(Author),
    )
    

    View Slide

24. Field Deconstruction
    

    View Slide

25. Every field has deconstruct()
    Returns arguments passed to
    __init__ to duplicate itself
    

    View Slide

26. >>> field = CharField(max_length=255)
    >>> field.deconstruct()
    (
    None,
    'django.db.models.CharField',
    [],
    {'max_length': 255}
    )
    

    View Slide

27. The Graph
    

    View Slide

28. Represents migrations and
    dependencies as directed graph
    Loop detection, leaf/root selection
    

    View Slide

29. myapp/0001
    myapp/0002 otherapp/0001
    contenttypes/0001
    appthree/0001
    myapp/0003 otherapp/0002
    

    View Slide

30. The Autodetector
    

    View Slide

31. Compares two States and outputs
    a set of operations to perform
    Operations have own dependencies
    and resolver
    

    View Slide

32. Model A Model A w/FK, Model B
    diff
    AddModel(B)
    AddField(A, "b", FK(B))
    dependency sort
    AddModel(B) AddField(A, "b", FK(B))
    BEFORE AFTER
    

    View Slide

33. The Optimiser
    

    View Slide

34. Takes one set of Operations and
    outputs another with the same effect
    Steps over pairs and checks if they
    combine and what they pass over
    

    View Slide

35. can't
    reduce
    reduce
    possible
    check
    conflicts
    reduced,
    reset
    

    View Slide

36. A New Format
    More concise
    Declarative
    Introspectable
    

    View Slide

37. Migration actions
    Frozen ORM
    

    View Slide

38. View Slide

39. makemigrations
    field.deconstruct()
    Loader / Graph
    Autodetector Optimiser
    State
    Writer
    1
    2
    State
    3
    5
    4
    

    View Slide

40. migrate
    SchemaEditor
    ModelOptions.apps
    Operations
    Loader / Graph Executor
    State
    1 2
    Recorder
    3
    

    View Slide

41. In-memory running
    Creates models from migration sets
    Autodetector diffs created from on-disk
    Used to feed SchemaEditor / ORM
    

    View Slide

42. State State
    Operation
    State
    Operation
    State
    Operation
    State
    Operation
    Migration 1 Migration 2
    Models
    Models
    Render Render
    

    View Slide

43. But what went wrong?
    

    View Slide

44. Swappable Models
    

    View Slide

45. Your migration dependencies
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    

    View Slide

46. myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    Your migration dependencies
    on swappable models
    

    View Slide

47. what?
    Your migration dependencies
    on swappable models
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    ???
    

    View Slide

48. what?
    Your migration dependencies
    on swappable models
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    ???
    

    View Slide

49. Unmigrated Apps
    

    View Slide

50. Test persistence
    

    View Slide

51. Test persistence
    on MySQL
    

    View Slide

52. Random Meta options
    order_with_respect_to? Really?
    

    View Slide

53. Proxy Models
    

    View Slide

54. But we're there!
    Django 1.7, out now!
    

    View Slide

55. Thanks!
    Andrew Godwin
    andrewgodwin@eventbrite.com
    

    View Slide