$30 off During Our Annual Pro Sale. View Details »

Migrating The Future

Migrating The Future

A talk I gave at DjangoCon EU 2013 about South, Django, and the future of migrations in Django core.

Andrew Godwin

May 15, 2013
Tweet

More Decks by Andrew Godwin

Other Decks in Programming

Transcript

  1. Andrew Godwin
    @andrewgodwin
    Migrating
    THE
    Future

    View Slide

  2. Andrew Godwin
    Django core developer
    South author
    Database Wrangler

    View Slide

  3. South

    View Slide

  4. South
    django.db.migrations

    View Slide

  5. South
    django.db.migrations
    Kickstarter

    View Slide

  6. £2,500

    View Slide

  7. £2,500
    Raised!

    View Slide

  8. £2,500
    Raised!
    65 minutes in.

    View Slide

  9. £17,952
    From 502 backers

    View Slide

  10. View Slide

  11. Tom Christie
    Alex Gaynor
    Pixtape.com
    Ulrich Petri (@ulope)
    Sam Wilson
    Dan Fairs
    Nyaruka Ltd
    Jorge Bastida
    Aaron Bassett
    Stefan Foulis
    Aymeric Augustin
    Rich Leland
    Ray Schamp
    CookDrop Inc.
    fitclipboard.com
    Type/Code
    Bruno Renié
    Bryan Veloso
    Smashing Ideas
    John P. Neumann
    Filipe Regadas
    Simon Law ([email protected])
    Ryan Kaskel
    CYeong
    Baptiste Mispelon
    Eric Florenzano
    Albert O'Connor, Web Developer
    Jeremy Dunck
    Daniel Lindsley
    Dan Cundiff
    Automatique
    another one GmbH
    Vincent Driessen (@nvie)
    Daniel Greenfeld
    Joey Wilhelm
    Marek Stępniowski
    Sil Westerveld (SURFsara)
    http://Scal.io
    RMSA - the makers of Fresco
    Nitty Gritty Software, LLC
    Greg Aker
    Christopher Dickes
    SJM Steffann
    Messiah Studio (New Zealand)
    Dirty Monkey
    Łukasz Langa (ambv)
    Dougal Matthews
    Joel Crocker

    View Slide

  12. Aaron Bassett Aaron Ho Aaron Larner Aaron Malone Adam Connor Adam Fast Adam Foster Adam Gross Adam Joseph Cook Adam Rich Adrian Espinosa Adrian
    Rego Adrian Smith Agostino Albert O'Connor Alec Perkins Ales Alexander Rudakov Alexandre Lessard Alexey Sveshnikov Alex Gaynor Alex Hung Alex Louden Alex
    Tziallas Alex Yuzhakov Alfredo Aguirre Ames Cornish Anatoly Popov Andreas Louca Andreas Madsack Andreas Rieger André Simões Andrew Austin Andrew Dunn
    Andrew Gorcester Andrew Jesaitis Andrzej Voss Andy Baker Andy Dirnberger Angel Freire Anselm Lingnau Arild Martin Møller Haugstad Arnaud Art van
    Scheppingen Athanassios Diacakis August Bigelow Aymeric Augustin Backer Name Backer Name Backer Name Backer Name Backer Name Backer Name Backer
    Name Backer Name Baldur Þór Emilsson Baptiste Millou Baptiste Mispelon Barbara Shaurette Bayo opadeyi Ben Ben Firshman Ben Graham Benjamin Wohlwend
    Ben Lopatin Ben Spaulding BeProud Bernhard Janetzki bitprophet Bob Barcklay Bob Haugen Bob Lozano Bolster Boris Chervenkov Bouke Haarsma Bozidar Benko
    Brad Montgomery Brad Wright Brandon Bennett Brandon Konkle Brett Epps Brian O'Dell Brian Reigle Brian Rosner Brian Wisti Bruno Renié Bruno Tikami Bryan
    Koroleski Bryan Veloso Carl Meyer Carlos Ribeiro Carny Cheng Casper S. Jensen Chad Catlett Chris Adams Chris Beaven Chris Drackett Chris Johnston Chris
    Kleinknecht Chris Streeter Christian Bertschy Christian Lange Christian Toivola Christian Treutler Christophe Pettus Christopher Dickes Christopher Grebs
    Christopher J. Hasenpflug Christopher Reeves Christoph Heer Chris Vigelius "Chris ""Wraithan"" McDonald" Cícero Raupp Rolim Claes Ström Colin Woods
    Cuauhtemoc Ramirez Damon Ciarelli Dan Cundiff DanEE Dan Fairs Daniel Gagnon Daniel Greenfeld Daniel Lindsley Daniel Perry Daniel Watkins Danilo Dan Koch
    Dan Langer Danny Roa Dan Tyan Darian Moody Dave Arter Dave Masse David David Cramer David Feinzeig David Grant David Jeffrey David Novák David Sinsky
    Denis Darii Dennis Seymour Derek Leverenz Deric Crago Devin Hunt Dj Gilcrease Dmitriy Narkevich Dong Hee Kim Dougal Matthews Doug Tabacco Dustin Davis
    Edgars Jēkabsons Elliot Murphy Enrico Berti Ensorceled Eric Eric Florenzano Ernesto Rico-Schmidt Esa Evan Culver Ezequiel Pochiero Federico Giordano Felipe
    Coury Felipe Kellermann Felix Yuan Fernando Gutierrez Fernando Rocha Filipe Regadas Filipp Lepalaan Florian Apolloner fm 10.4 Frederik De Bleser Fusionbox
    Gabriel Gamil Gabriel Grant g chow George Hickman Gert Van Gool Giles Thomas Giovanni Totaro Graham Ullrich Greg Aker Guilherme Gondim Håkon Erichsen
    Hans Lawrenz harijay Herbert Poul Horst Gutmann Hrishikesh Ballal Hugo Rodger-Brown Hugo Wang Ian Ildar Samit Ivan Habunek Iván Raskovsky Ivan Yurchenko
    Ivo Teel Jacek Jacob Briggs Jacob Kaplan-Moss Jaime Irurzun Jakub Wiśniowski James Cleveland James Johnson James Robert James Tiplady James Walker
    Jamie Wilson Jannis Leidel Jason Jason Cartwright Jason Mayfield Jason Simeone Jasper den Hertog Javed Khan Jeff jens quade Jérémie Blaser Jeremy Cantrell
    Jeremy Dunck Jeremy Keeshin Jesse Michelsen Jesus Noland Jim Ray Jirka Vejrazka JMatt Peterson João Oliveira Joel Crocker Joel Shapiro Joel Squire Joey
    Castillo Joey Leingang Joey Wilhelm John Paulett John P. Neumann John Sykora John Wegis Jonas Nockert Jonathan Biddle Jonathan Morgan Jon Danao Jon
    Fedyk Joonathan Mägi Jorge Bastida Jorge Camargo Jörg Kreß Jose Padilla Joseph Wenninger Josh Ourisman Josh Steiner Joshua Blount Joshua Johanan Josh
    Wright Julen Ruiz Aizpuru Julian Wachholz Julio Oña Julius Welby Jure Ham Kalle Karjalainen Katie Cunningham Katsumi Honda Keith Gray Kenneth Love Kenneth
    Reitz Kirill Sibirev Kiwi Koh Chuan Yeong kortina Kura Kuratko Lachlan Musicman Larry Hamel Larry Osborn Lars Wikman Laust M. Ladefoged Leo Leo Hourvitz
    Leonidas Tsementzis Leon Matthews Leon Smith Le Thanh Duong Logit internet services Ltd. Loic Bistuer Lucas Eisenzimmer Łukasz Langa Luke Hatcher Maciej
    Konieczny Maciek Gryka Marc Egli Marcelo Alves Marcel Rudolph Marco Bonetti Marc Tamlyn Marek Olszewski Marek Stępniowski Margus Holland Marianne
    Marielle Volz Mark Hepburn Mark Hurley Mark Longair Mark Lyon Mark Ng Mark Norman Francis Marko Samastur Markus Zapke-Gründemann marteinn Martin
    Brochhaus Mathieu Fenniak Mathieu Meylan Mathieu Richardoz Matt Austin Matthew Tretter Mattias Linnap Matt Lenc Matt McGlincy Matt Stevens Matt Woodward
    Matus Moravcik Mauro Doglio Max Arnold Meir Kriheli Michael Blatherwick Michael Fogel Michael Gisi Michael Sandford Michael Twomey Miguel Pilar Mihael
    Konjević Mike Gallagher MIKe O'Toole Mirus Research Mitch Berkson Mladen Kovacevic moberley Nate Graves Nate Meyvis Nate Thelen Nephila Nick Albright Nick
    Siemsen Nicolas Perriault Nicolas Pottier Nicole Foster niels Niels Sandholt Busch Nikolai Prokoschenko Noah Kantrowitz Noemi notanumber Numan Sachwani
    numbsafari Nurzhan Bakibayev Ola Sitarska Oliver Beattie Oliver George Oliver Steele Omen Ondrej Slintak ORDOQUY Xavier Ori Oskar Jakieła Ousmane Wilane
    Parham Pascal Hartig Pathwright Press Patrick Arminio Patrick Craston Patrick Taylor Paul Bersch Paul Ebreo Paul Mattal Paul Medlock-Walton Peppe Bergqvist
    Peter Baumgartner + Lincoln Loop Peter Evans Peter Farrell Peter Hyben Peter Inglesby Peter Kropf Peter Sanchez Petr Marhoun Petr Timofeev Philip Kimmey
    Philipp Bosch Philipp Kruse Philipp Läubli Philipp Weidenhiller Philip Roche Pierre-Julien Grizel pmdhazy Preston Holmes Promeritum quinox Raj Jalan Ralph
    Ramji Srinivasan Ray Remco Wendt Renato Garcia Pedigoni Renato Oliveira riccardo magliocchetti Richard Beales Rich Leland Rinat Akhtamov Rishabh Manocha
    Rivo Laks RJ Walsh Rob Berwick Robert Medeiros Roberto Rosario Rod Afshar Rodrigo Chacon Rogelio Rolo Mawlabaux Ross Crawford-d'Heureuse Ross Duggan
    Russell Keith-Magee Ryan Kaskel Ryan McDevitt Ryan van Roode Sævar Öfjörð Magnússon Sahas Katta Sam Bull Sam Hartsfield Sam Johnson Sam Kitonyi Samuel
    Clay Samuel Linde Sander Steffann Sandor Weisz Sau Ramirez Scott Henley Scott Nixon Sean O'Connor Sean Powell Sebastian Brandt Sebastian Desimone
    Sebastian vetter Sebastian Wain seibert Senko Rašić Sergey Leschina Sergey Silaev Sergey Vilgelm Seth Burke Shawn Milochik Silvan Spross Sil Westerveld
    Simeon Franklin Simon Bernier simon hayward Simon Hewitt Simon Law Simon Pantzare Simon Willison Skirmantas Jurgaitis Solomon Matthews Sören Schlömer
    Stefan Foulis Stefano Delli Ponti Stein Magnus Jodal Stephan Huiser Stephen Steve Steve Holden Steve Jalim Sumit Chachraa Sungjoo Ha Sven Grunewaldt Taavi
    Taijala Takuya Tozawa Taymon Thijs van Dien Thomas Gruner Tijs Teulings Tim Carter Timothy Zwolak TinFoil Tom Christie Tom Dyson Tomek Paczkowski Tommy
    Gudmundsen Tom Usher Tony Hauber Tony Meyer Trevor Riles TrialReach Tuk Bredsdorff Tunde Ashafa Twidi Type/Code Ulrich Petri Valentin V. Bartenev Victor Allen
    Vidir Gudmundsson ViKing Ville Alatalo Vincent Driessen Visa Kopu Vitaliy Fuks Vladimir Volodymyr Hotsyk Wade Austin Walker Hamilton Warren Packard Wiley
    Kestner William Arnold William Hogan Williams Mendez Wil Linssen Yarden Michael Shem-Tov Yury Yurevich zaebee Павел Тявин

    View Slide

  13. Why replace South?

    View Slide

  14. 5 years of learning
    Poor VCS branching
    Huge migration file size
    Migration sets get too large

    View Slide

  15. The Future
    New! Exciting!
    95% less migration noise!

    View Slide

  16. django.db.migrations
    django.db.backends.schema

    View Slide

  17. django.db.migrations
    Migration commands
    Autodetection
    Dependencies

    View Slide

  18. django.db.backends.schema
    SQL generation
    Database abstraction

    View Slide

  19. ed = connection.schema_editor()
    ed.start()
    ed.create_model(MyModel)
    ed.add_field(OtherModel, "foo", BooleanField())
    ed.commit()

    View Slide

  20. PostgreSQL
    MySQL
    SQLite
    Oracle
    MSSQL
    Yep!
    Hopefully!
    DB2
    MongoDB Wait, what?

    View Slide

  21. New migration format
    Finally! Please stop complaining now.

    View Slide

  22. Old Format

    View Slide

  23. Old Format
    Migration actions
    Frozen ORM

    View Slide

  24. New Format

    View Slide

  25. New Format

    View Slide

  26. How?
    Create Model Page with fields content, date
    Create Model Author with fields name, age
    Add title to Page
    Remove age from Author
    Migration 1
    Migration 2
    Migration 3
    Model Page with fields title, content, date
    Model Author with field name

    View Slide

  27. Dependencies
    Better. Faster. Stronger. Not harder.

    View Slide

  28. 0001_initial
    Before
    0002_ponies
    0003_foo 0003_bar

    View Slide

  29. 0001_initial
    Now
    0002_ponies
    0003_foo 0003_bar

    View Slide

  30. 0001_initial
    Now
    0002_ponies
    0003_foo 0003_bar
    0004_merge

    View Slide

  31. Squashing
    Saves precious, precious bytes.

    View Slide

  32. 0001
    0002
    0003
    0004
    0005
    0006

    View Slide

  33. 0001
    0002
    0003
    0004
    0005
    0006
    0005_alt

    View Slide

  34. 0006
    0005_alt

    View Slide

  35. 0006
    0001
    0002
    0003
    0004
    0005
    0006
    0001
    0002
    0003
    0004
    0005
    APP A APP B

    View Slide

  36. 0006
    0001
    0002
    0003
    0004
    0005
    0006
    0001
    0002
    0003
    0004
    0005
    APP A APP B

    View Slide

  37. How's it going?
    Kickstarter is funding so many holidays work days

    View Slide

  38. Schema Backends
    Mostly done, ready for merge

    View Slide

  39. Migration code
    Still going, most complex part

    View Slide

  40. Field API changes
    Incoming. Watch django-developers if interested.

    View Slide

  41. The Code
    https://github.com/andrewgodwin/
    django/tree/schema-alteration/

    View Slide

  42. Blog Posts
    aeracode.org/category/django-diaries/

    View Slide

  43. Thoughts, feedback and fully-
    complete migration frameworks to:
    [email protected]

    View Slide

  44. Fin.
    Andrew Godwin
    Thank you.
    @andrewgodwin
    [email protected]
    http://www.aeracode.org

    View Slide