Pro Yearly is on sale from $80 to $50! »

Migrations Under The Hood

077e9a0cb34fa3eba2699240c9509717?s=47 Andrew Godwin
November 14, 2014

Migrations Under The Hood

A talk I gave at Django Under The Hood 2014.

077e9a0cb34fa3eba2699240c9509717?s=128

Andrew Godwin

November 14, 2014
Tweet

Transcript

  1. Migrations UNDER THE HOOD Andrew Godwin Migrations

  2. Andrew Godwin Author of South migrations library Hi, I'm Author

    of 1.7 Django migrations Senior Software Engineer at Eventbrite Apparently at every Django conference
  3. Migrations

  4. Migrations

  5. Other talks are available. This is a technical dive!

  6. The Initial Plan Django Schema backend ORM Hooks South 2

    Migration handling User interface
  7. TheBRevisedBPlan Django SchemaBbackend ORMBHooks SouthB2 MigrationBhandling UserBinterface BackportBforB1.4B-B1.6

  8. The Revised Revised Plan Django Schema backend ORM Hooks Migration

    handling User interface
  9. Logically Separate SchemaEditor Schema Migrations field.deconstruct() ModelOptions.apps Operations Loader /

    Graph Executor Autodetector Optimiser State
  10. Logically Separate SchemaEditor Schema Migrations field.deconstruct() ModelOptions.apps Operations Loader /

    Graph Executor Autodetector Optimiser State
  11. myapp/0001 myapp/0002 otherapp/0001 contenttypes/0001 appthree/0001 myapp/0003 otherapp/0002

  12. (insert wavy memory effect!) Why do we need dependencies?

  13. Makes all tables syncdb Sets all constraints Adds all ForeignKeys

  14. Makes appA tables migrations Makes appB tables Adds FKs from

    appB to appA Adds FKs from appA to appB
  15. Makes appA tables migrations Makes appB tables Adds FKs from

    appB to appA Adds FKs from appA to appB
  16. The obvious stuff. Basic Dependencies

  17. Or your database gets really mopey. FK target add before

    FK added
  18. In the general case; some DBs less fussy FK target

    delete after FK deleted
  19. Good luck doing it otherwise. Model create before field create

  20. Also after field create Model delete after field alter

  21. Stuff you occasionally forget Django has. Trickier Dependencies

  22. And, of course, the reverse for deletion. M2M through model

    before M2M
  23. I'll let you figure out deletion reversals from now on.

    MTI parent before MTI child
  24. Index together, too Unique together after fields

  25. Well done if you've actually used this feature. Order with

    respect to after target
  26. This happens when you turn proxies into real models. Same

    delete before same create
  27. Impossible dependencies

  28. OH PLEASE NO TAKE IT AWAY Swappable models

  29. Your migration dependencies myapp/0001 myapp/0002 otherapp/0001 auth/0001 contenttypes/0001

  30. myapp/0001 myapp/0002 otherapp/0001 auth/0001 contenttypes/0001 thirdapp/0001 Your migration dependencies on

    swappable models
  31. what? Your migration dependencies on swappable models myapp/0001 myapp/0002 otherapp/0001

    auth/0001 contenttypes/0001 thirdapp/0001 ???
  32. what? Your migration dependencies on swappable models myapp/0001 myapp/0002 otherapp/0001

    auth/0001 contenttypes/0001 thirdapp/0001 ??? argh
  33. End of presentation. Click to exit.

  34. They can't change at runtime, for swappable models or anything

    else. Dependencies are static
  35. (for a while, the answer was "careful function ordering") How

    is it implemented?
  36. no issues depend match shift operations restart

  37. Using dependencies to define boundaries Chop operations into migrations

  38. Optimisation

  39. There's really quite a lot. Even more rules

  40. create model + delete model = nothing create model +

    create field = create model create field + alter field = create field alter field + delete field = delete field alter field + alter field = alter field
  41. This time, with collapses rather than swaps Even more shuffling

  42. can't reduce reduce possible check conflicts reduced, reset

  43. Lets us be more verbose and clear with operations Applied

    after autodetection
  44. Makes your history a bit shorter Applied after squash

  45. Loading & Graph

  46. Builds graph of all basic migrations in memory Loads from

    disk first
  47. Marks applied state on each node Loads from db second

  48. Only if all replaced nodes have same applied state Applies

    squashes third
  49. In-order list of all a node's ancestors in the tree

    Graph returns "plans"
  50. • Autodetector is slow • Optimizer is not great •

    Graph state building is inefficient Room for improvement!
  51. django/db/migrations/autodetector.py Autodetector django/db/migrations/optimizer.py Optimiser reduce()GfunctionGisGentryGforGlogic StartGatG_detect_changes()GandGfollowGcalls django/db/migrations/graph.py Graph django/db/migrations/loader.py

  52. Thank you. Andrew Godwin @andrewgodwin