Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Migrations Under The Hood

Andrew Godwin
November 14, 2014

Migrations Under The Hood

A talk I gave at Django Under The Hood 2014.

Andrew Godwin

November 14, 2014
Tweet

More Decks by Andrew Godwin

Other Decks in Programming

Transcript

  1. Migrations
    UNDER THE HOOD
    Andrew Godwin
    Migrations

    View Slide

  2. Andrew Godwin
    Author of South migrations library
    Hi, I'm
    Author of 1.7 Django migrations
    Senior Software Engineer at Eventbrite
    Apparently at every Django conference

    View Slide

  3. Migrations

    View Slide

  4. Migrations

    View Slide

  5. Other talks are available.
    This is a technical dive!

    View Slide

  6. The Initial Plan
    Django
    Schema backend
    ORM Hooks
    South 2
    Migration handling
    User interface

    View Slide

  7. TheBRevisedBPlan
    Django
    SchemaBbackend
    ORMBHooks
    SouthB2
    MigrationBhandling
    UserBinterface
    BackportBforB1.4B-B1.6

    View Slide

  8. The Revised Revised Plan
    Django
    Schema backend
    ORM Hooks
    Migration handling
    User interface

    View Slide

  9. Logically Separate
    SchemaEditor
    Schema Migrations
    field.deconstruct()
    ModelOptions.apps
    Operations
    Loader / Graph
    Executor
    Autodetector
    Optimiser
    State

    View Slide

  10. Logically Separate
    SchemaEditor
    Schema Migrations
    field.deconstruct()
    ModelOptions.apps
    Operations
    Loader / Graph
    Executor
    Autodetector
    Optimiser
    State

    View Slide

  11. myapp/0001
    myapp/0002 otherapp/0001
    contenttypes/0001
    appthree/0001
    myapp/0003 otherapp/0002

    View Slide

  12. (insert wavy memory effect!)
    Why do we need dependencies?

    View Slide

  13. Makes all tables
    syncdb
    Sets all constraints
    Adds all ForeignKeys

    View Slide

  14. Makes appA tables
    migrations
    Makes appB tables
    Adds FKs from appB to appA
    Adds FKs from appA to appB

    View Slide

  15. Makes appA tables
    migrations
    Makes appB tables
    Adds FKs from appB to appA
    Adds FKs from appA to appB

    View Slide

  16. The obvious stuff.
    Basic Dependencies

    View Slide

  17. Or your database gets really mopey.
    FK target add before FK added

    View Slide

  18. In the general case; some DBs less fussy
    FK target delete after FK deleted

    View Slide

  19. Good luck doing it otherwise.
    Model create before field create

    View Slide

  20. Also after field create
    Model delete after field alter

    View Slide

  21. Stuff you occasionally forget Django has.
    Trickier Dependencies

    View Slide

  22. And, of course, the reverse for deletion.
    M2M through model before M2M

    View Slide

  23. I'll let you figure out deletion reversals from now on.
    MTI parent before MTI child

    View Slide

  24. Index together, too
    Unique together after fields

    View Slide

  25. Well done if you've actually used this feature.
    Order with respect to after target

    View Slide

  26. This happens when you turn proxies into real models.
    Same delete before same create

    View Slide

  27. Impossible dependencies

    View Slide

  28. OH PLEASE NO TAKE IT AWAY
    Swappable models

    View Slide

  29. Your migration dependencies
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001

    View Slide

  30. myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    Your migration dependencies
    on swappable models

    View Slide

  31. what?
    Your migration dependencies
    on swappable models
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    ???

    View Slide

  32. what?
    Your migration dependencies
    on swappable models
    myapp/0001
    myapp/0002
    otherapp/0001
    auth/0001
    contenttypes/0001
    thirdapp/0001
    ???
    argh

    View Slide

  33. End of presentation. Click to exit.

    View Slide

  34. They can't change at runtime, for swappable
    models or anything else.
    Dependencies are static

    View Slide

  35. (for a while, the answer was "careful function ordering")
    How is it implemented?

    View Slide

  36. no
    issues
    depend
    match
    shift
    operations
    restart

    View Slide

  37. Using dependencies to define boundaries
    Chop operations into migrations

    View Slide

  38. Optimisation

    View Slide

  39. There's really quite a lot.
    Even more rules

    View Slide

  40. create model + delete model = nothing
    create model + create field = create model
    create field + alter field = create field
    alter field + delete field = delete field
    alter field + alter field = alter field

    View Slide

  41. This time, with collapses rather than swaps
    Even more shuffling

    View Slide

  42. can't
    reduce
    reduce
    possible
    check
    conflicts
    reduced,
    reset

    View Slide

  43. Lets us be more verbose and clear with operations
    Applied after autodetection

    View Slide

  44. Makes your history a bit shorter
    Applied after squash

    View Slide

  45. Loading & Graph

    View Slide

  46. Builds graph of all basic migrations in memory
    Loads from disk first

    View Slide

  47. Marks applied state on each node
    Loads from db second

    View Slide

  48. Only if all replaced nodes have same applied state
    Applies squashes third

    View Slide

  49. In-order list of all a node's ancestors in the tree
    Graph returns "plans"

    View Slide

  50. • Autodetector is slow
    • Optimizer is not great
    • Graph state building is inefficient
    Room for improvement!

    View Slide

  51. django/db/migrations/autodetector.py
    Autodetector
    django/db/migrations/optimizer.py
    Optimiser
    reduce()GfunctionGisGentryGforGlogic
    StartGatG_detect_changes()GandGfollowGcalls
    django/db/migrations/graph.py
    Graph
    django/db/migrations/loader.py

    View Slide

  52. Thank you.
    Andrew Godwin
    @andrewgodwin

    View Slide