Migrations Under The Hood

A talk I gave at Django Under The Hood 2014.

Transcript

1. Migrations
   UNDER THE HOOD
   Andrew Godwin
   Migrations
   

   View Slide

2. Andrew Godwin
   Author of South migrations library
   Hi, I'm
   Author of 1.7 Django migrations
   Senior Software Engineer at Eventbrite
   Apparently at every Django conference
   

   View Slide

3. Migrations
   

   View Slide

4. Migrations
   

   View Slide

5. Other talks are available.
   This is a technical dive!
   

   View Slide

6. The Initial Plan
   Django
   Schema backend
   ORM Hooks
   South 2
   Migration handling
   User interface
   

   View Slide

7. TheBRevisedBPlan
   Django
   SchemaBbackend
   ORMBHooks
   SouthB2
   MigrationBhandling
   UserBinterface
   BackportBforB1.4B-B1.6
   

   View Slide

8. The Revised Revised Plan
   Django
   Schema backend
   ORM Hooks
   Migration handling
   User interface
   

   View Slide

9. Logically Separate
   SchemaEditor
   Schema Migrations
   field.deconstruct()
   ModelOptions.apps
   Operations
   Loader / Graph
   Executor
   Autodetector
   Optimiser
   State
   

   View Slide

10. Logically Separate
    SchemaEditor
    Schema Migrations
    field.deconstruct()
    ModelOptions.apps
    Operations
    Loader / Graph
    Executor
    Autodetector
    Optimiser
    State
    

    View Slide

11. myapp/0001
    myapp/0002 otherapp/0001
    contenttypes/0001
    appthree/0001
    myapp/0003 otherapp/0002
    

    View Slide

12. (insert wavy memory effect!)
    Why do we need dependencies?
    

    View Slide

13. Makes all tables
    syncdb
    Sets all constraints
    Adds all ForeignKeys
    

    View Slide

14. Makes appA tables
    migrations
    Makes appB tables
    Adds FKs from appB to appA
    Adds FKs from appA to appB
    

    View Slide

15. Makes appA tables
    migrations
    Makes appB tables
    Adds FKs from appB to appA
    Adds FKs from appA to appB
    

    View Slide

16. The obvious stuff.
    Basic Dependencies
    

    View Slide

17. Or your database gets really mopey.
    FK target add before FK added
    

    View Slide

18. In the general case; some DBs less fussy
    FK target delete after FK deleted
    

    View Slide

19. Good luck doing it otherwise.
    Model create before field create
    

    View Slide

20. Also after field create
    Model delete after field alter
    

    View Slide

21. Stuff you occasionally forget Django has.
    Trickier Dependencies
    

    View Slide

22. And, of course, the reverse for deletion.
    M2M through model before M2M
    

    View Slide

23. I'll let you figure out deletion reversals from now on.
    MTI parent before MTI child
    

    View Slide

24. Index together, too
    Unique together after fields
    

    View Slide

25. Well done if you've actually used this feature.
    Order with respect to after target
    

    View Slide

26. This happens when you turn proxies into real models.
    Same delete before same create
    

    View Slide

27. Impossible dependencies
    

    View Slide

28. OH PLEASE NO TAKE IT AWAY
    Swappable models
    

    View Slide

29. Your migration dependencies
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    

    View Slide

30. myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    Your migration dependencies
    on swappable models
    

    View Slide

31. what?
    Your migration dependencies
    on swappable models
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    ???
    

    View Slide

32. what?
    Your migration dependencies
    on swappable models
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    ???
    argh
    

    View Slide

33. End of presentation. Click to exit.
    

    View Slide

34. They can't change at runtime, for swappable
    models or anything else.
    Dependencies are static
    

    View Slide

35. (for a while, the answer was "careful function ordering")
    How is it implemented?
    

    View Slide

36. no
    issues
    depend
    match
    shift
    operations
    restart
    

    View Slide

37. Using dependencies to define boundaries
    Chop operations into migrations
    

    View Slide

38. Optimisation
    

    View Slide

39. There's really quite a lot.
    Even more rules
    

    View Slide

40. create model + delete model = nothing
    create model + create field = create model
    create field + alter field = create field
    alter field + delete field = delete field
    alter field + alter field = alter field
    

    View Slide

41. This time, with collapses rather than swaps
    Even more shuffling
    

    View Slide

42. can't
    reduce
    reduce
    possible
    check
    conflicts
    reduced,
    reset
    

    View Slide

43. Lets us be more verbose and clear with operations
    Applied after autodetection
    

    View Slide

44. Makes your history a bit shorter
    Applied after squash
    

    View Slide

45. Loading & Graph
    

    View Slide

46. Builds graph of all basic migrations in memory
    Loads from disk first
    

    View Slide

47. Marks applied state on each node
    Loads from db second
    

    View Slide

48. Only if all replaced nodes have same applied state
    Applies squashes third
    

    View Slide

49. In-order list of all a node's ancestors in the tree
    Graph returns "plans"
    

    View Slide

50. • Autodetector is slow
    • Optimizer is not great
    • Graph state building is inefficient
    Room for improvement!
    

    View Slide

51. django/db/migrations/autodetector.py
    Autodetector
    django/db/migrations/optimizer.py
    Optimiser
    reduce()GfunctionGisGentryGforGlogic
    StartGatG_detect_changes()GandGfollowGcalls
    django/db/migrations/graph.py
    Graph
    django/db/migrations/loader.py
    

    View Slide

52. Thank you.
    Andrew Godwin
    @andrewgodwin
    

    View Slide