Migrations Under The Hood

A talk I gave at Django Under The Hood 2014.

Transcript

1. Migrations UNDER THE HOOD Andrew Godwin Migrations
2. Andrew Godwin Author of South migrations library Hi, I'm Author of 1.7 Django migrations Senior Software Engineer at Eventbrite Apparently at every Django conference
3. Migrations
4. Migrations
5. Other talks are available. This is a technical dive!
6. The Initial Plan Django Schema backend ORM Hooks South 2 Migration handling User interface
7. TheBRevisedBPlan Django SchemaBbackend ORMBHooks SouthB2 MigrationBhandling UserBinterface BackportBforB1.4B-B1.6
8. The Revised Revised Plan Django Schema backend ORM Hooks Migration handling User interface
9. Logically Separate SchemaEditor Schema Migrations field.deconstruct() ModelOptions.apps Operations Loader / Graph Executor Autodetector Optimiser State
10. Logically Separate SchemaEditor Schema Migrations field.deconstruct() ModelOptions.apps Operations Loader / Graph Executor Autodetector Optimiser State
11. myapp/0001 myapp/0002 otherapp/0001 contenttypes/0001 appthree/0001 myapp/0003 otherapp/0002
12. (insert wavy memory effect!) Why do we need dependencies?
13. Makes all tables syncdb Sets all constraints Adds all ForeignKeys
14. Makes appA tables migrations Makes appB tables Adds FKs from appB to appA Adds FKs from appA to appB
15. Makes appA tables migrations Makes appB tables Adds FKs from appB to appA Adds FKs from appA to appB
16. The obvious stuff. Basic Dependencies
17. Or your database gets really mopey. FK target add before FK added
18. In the general case; some DBs less fussy FK target delete after FK deleted
19. Good luck doing it otherwise. Model create before field create
20. Also after field create Model delete after field alter
21. Stuff you occasionally forget Django has. Trickier Dependencies
22. And, of course, the reverse for deletion. M2M through model before M2M
23. I'll let you figure out deletion reversals from now on. MTI parent before MTI child
24. Index together, too Unique together after fields
25. Well done if you've actually used this feature. Order with respect to after target
26. This happens when you turn proxies into real models. Same delete before same create
27. Impossible dependencies
28. OH PLEASE NO TAKE IT AWAY Swappable models
29. Your migration dependencies myapp/0001 myapp/0002 otherapp/0001 auth/0001 contenttypes/0001
30. myapp/0001 myapp/0002 otherapp/0001 auth/0001 contenttypes/0001 thirdapp/0001 Your migration dependencies on swappable models
31. what? Your migration dependencies on swappable models myapp/0001 myapp/0002 otherapp/0001 auth/0001 contenttypes/0001 thirdapp/0001 ???
32. what? Your migration dependencies on swappable models myapp/0001 myapp/0002 otherapp/0001 auth/0001 contenttypes/0001 thirdapp/0001 ??? argh
33. End of presentation. Click to exit.
34. They can't change at runtime, for swappable models or anything else. Dependencies are static
35. (for a while, the answer was "careful function ordering") How is it implemented?
36. no issues depend match shift operations restart
37. Using dependencies to define boundaries Chop operations into migrations
38. Optimisation
39. There's really quite a lot. Even more rules
40. create model + delete model = nothing create model + create field = create model create field + alter field = create field alter field + delete field = delete field alter field + alter field = alter field
41. This time, with collapses rather than swaps Even more shuffling
42. can't reduce reduce possible check conflicts reduced, reset
43. Lets us be more verbose and clear with operations Applied after autodetection
44. Makes your history a bit shorter Applied after squash
45. Loading & Graph
46. Builds graph of all basic migrations in memory Loads from disk first
47. Marks applied state on each node Loads from db second
48. Only if all replaced nodes have same applied state Applies squashes third
49. In-order list of all a node's ancestors in the tree Graph returns "plans"
50. • Autodetector is slow • Optimizer is not great • Graph state building is inefficient Room for improvement!
51. django/db/migrations/autodetector.py Autodetector django/db/migrations/optimizer.py Optimiser reduce()GfunctionGisGentryGforGlogic StartGatG_detect_changes()GandGfollowGcalls django/db/migrations/graph.py Graph django/db/migrations/loader.py
52. Thank you. Andrew Godwin @andrewgodwin