Migrations Under The Hood

Transcript

1. 1.

   Migrations UNDER THE HOOD Andrew Godwin Migrations

2. 2.

   Andrew Godwin Author of South migrations library Hi, I'm Author

   of 1.7 Django migrations Senior Software Engineer at Eventbrite Apparently at every Django conference
3. 3.

   Migrations

4. 4.

   Migrations

5. 5.

   Other talks are available. This is a technical dive!

6. 6.

   The Initial Plan Django Schema backend ORM Hooks South 2

   Migration handling User interface
7. 7.

   TheBRevisedBPlan Django SchemaBbackend ORMBHooks SouthB2 MigrationBhandling UserBinterface BackportBforB1.4B-B1.6

8. 8.

   The Revised Revised Plan Django Schema backend ORM Hooks Migration

   handling User interface
9. 9.

   Logically Separate SchemaEditor Schema Migrations field.deconstruct() ModelOptions.apps Operations Loader /

   Graph Executor Autodetector Optimiser State
10. 10.

    Logically Separate SchemaEditor Schema Migrations field.deconstruct() ModelOptions.apps Operations Loader /

    Graph Executor Autodetector Optimiser State
11. 11.

    myapp/0001 myapp/0002 otherapp/0001 contenttypes/0001 appthree/0001 myapp/0003 otherapp/0002

12. 12.

    (insert wavy memory effect!) Why do we need dependencies?

13. 13.

    Makes all tables syncdb Sets all constraints Adds all ForeignKeys

14. 14.

    Makes appA tables migrations Makes appB tables Adds FKs from

    appB to appA Adds FKs from appA to appB
15. 15.

    Makes appA tables migrations Makes appB tables Adds FKs from

    appB to appA Adds FKs from appA to appB
16. 16.

    The obvious stuff. Basic Dependencies

17. 17.

    Or your database gets really mopey. FK target add before

    FK added
18. 18.

    In the general case; some DBs less fussy FK target

    delete after FK deleted
19. 19.

    Good luck doing it otherwise. Model create before field create

20. 20.

    Also after field create Model delete after field alter

21. 21.

    Stuff you occasionally forget Django has. Trickier Dependencies

22. 22.

    And, of course, the reverse for deletion. M2M through model

    before M2M
23. 23.

    I'll let you figure out deletion reversals from now on.

    MTI parent before MTI child
24. 24.

    Index together, too Unique together after fields

25. 25.

    Well done if you've actually used this feature. Order with

    respect to after target
26. 26.

    This happens when you turn proxies into real models. Same

    delete before same create
27. 27.

    Impossible dependencies

28. 28.

    OH PLEASE NO TAKE IT AWAY Swappable models

29. 29.

    Your migration dependencies myapp/0001 myapp/0002 otherapp/0001 auth/0001 contenttypes/0001

30. 30.

    myapp/0001 myapp/0002 otherapp/0001 auth/0001 contenttypes/0001 thirdapp/0001 Your migration dependencies on

    swappable models
31. 31.

    what? Your migration dependencies on swappable models myapp/0001 myapp/0002 otherapp/0001

    auth/0001 contenttypes/0001 thirdapp/0001 ???
32. 32.

    what? Your migration dependencies on swappable models myapp/0001 myapp/0002 otherapp/0001

    auth/0001 contenttypes/0001 thirdapp/0001 ??? argh
33. 33.

    End of presentation. Click to exit.

34. 34.

    They can't change at runtime, for swappable models or anything

    else. Dependencies are static
35. 35.

    (for a while, the answer was "careful function ordering") How

    is it implemented?
36. 36.

    no issues depend match shift operations restart

37. 37.

    Using dependencies to define boundaries Chop operations into migrations

38. 38.

    Optimisation

39. 39.

    There's really quite a lot. Even more rules

40. 40.

    create model + delete model = nothing create model +

    create field = create model create field + alter field = create field alter field + delete field = delete field alter field + alter field = alter field
41. 41.

    This time, with collapses rather than swaps Even more shuffling

42. 42.

    can't reduce reduce possible check conflicts reduced, reset

43. 43.

    Lets us be more verbose and clear with operations Applied

    after autodetection
44. 44.

    Makes your history a bit shorter Applied after squash

45. 45.

    Loading & Graph

46. 46.

    Builds graph of all basic migrations in memory Loads from

    disk first
47. 47.

    Marks applied state on each node Loads from db second

48. 48.

    Only if all replaced nodes have same applied state Applies

    squashes third
49. 49.

    In-order list of all a node's ancestors in the tree

    Graph returns "plans"
50. 50.

    • Autodetector is slow • Optimizer is not great •

    Graph state building is inefficient Room for improvement!
51. 51.

    django/db/migrations/autodetector.py Autodetector django/db/migrations/optimizer.py Optimiser reduce()GfunctionGisGentryGforGlogic StartGatG_detect_changes()GandGfollowGcalls django/db/migrations/graph.py Graph django/db/migrations/loader.py

52. 52.

    Thank you. Andrew Godwin @andrewgodwin