Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MongoDB Schema Evolution and Data Migration

MongoDB Schema Evolution and Data Migration

MongoDB has arrived in the industry and runs for years in production systems now. But schema evolution and schema maintenance is neccessary. In his talk, Timmo will present solutions for (Java) Developers to get theire data into MongoDB and make data migration explicit, straight-forward and testable. You will learn: - how to map your Java domain model to a schema - how you can support model and schema evolution - that TDD is very easy with MongoDB - why MongoDB supports continuous deployment

Timmo Freudl-Gierke

February 26, 2013
Tweet

More Decks by Timmo Freudl-Gierke

Other Decks in Technology

Transcript

  1. Schema / JSON Structure { name : "Timmo Freudl-Gierke", company

    : "Hypoport AG" street : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } Application Domain Model Speaker name company street zipCode city
  2. Speaker name company street zipCode city Speaker firstName lastName Company

    name street zipCode city 0,1 Speaker firstName lastName company street zipCode city Version 1 Version 2 Version 3
  3. Speaker name company street zipCode city Speaker firstName lastName company

    street zipCode city Version 1 Version 2 db.speakers.find({}).forEach( function(speaker) { var original = speaker.name var first = original.match(/[A-Za-z]+/)[0] var last = original.match(/[A-Za-z-]+$/)[0] speaker.firstName = first speaker.lastName = last db. speakers.save(speaker) })
  4. Speaker firstName lastName Company name street zipCode city 0,1 Speaker

    firstName lastName company street zipCode city Version 2 Version 3 db.speakers.update({},{$rename:{"company":"company.name"}}) db.speakers.update({},{$rename:{"street":"company.street"}}) db.speakers.update({},{$rename:{"zipCode":"company.zipCode"}}) db.speakers.update({},{$rename:{"city":"company.city"}}) http://docs.mongodb.org/manual/reference/operator/rename/
  5. Speaker firstName lastName Company name street zipCode city 0,1 Speaker

    firstName lastName company street zipCode city Version 2 Version 3 DBObject speaker = ... DBObject company = new BasicDBObject(); company.put("name", speaker.get("name")); company.put("street", speaker.get("street")); company.put("zipCode", speaker.get("zipCode")); company.put("city", speaker.get("city")); speaker.put("company", company); speaker.removeField("street"); speaker.removeField("zipCode"); speaker.removeField("city"); collection.save(speaker);
  6. @Test public void companySeperation() { // given String json =

    getFileAsJson("speaker_v2.json") DBObject speaker = (DBObject) JSON.parse(json); // when converter.seperateCompany(speaker); // then assertEquals(speaker.get("company.name"), "Hypoport AG"); assertFalse(speaker.containsField("street"); . . . } { firstName : "Timmo", lastName : "Freudl-Gierke" company : "Hypoport AG" street : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } speaker_v2.json Unit Testing
  7. MongoDB speakers {name: "Timmo Freudl-Gierke ", modelVersion:1, ...} {name: "Joe

    Morrissey", modelVersion:1, ...} {name: "Stefan Rudnitzki ", modelVersion:2, ...} {name: "Christian Kvalheim ", modelVersion:1, ...} {name: "Ross Lawley ", modelVersion:2, ...} {name: "Thomas Rueckstiess ", modelVersion:3, ...} {name: "Johannes Hoppe ", modelVersion:3, ...} . . .
  8. { name : "Timmo Freudl-Gierke", company : "Hypoport AG" street

    : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } { name : "Timmo Freudl-Gierke", company : "Hypoport AG" street : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } { name : "Timmo Freudl-Gierke", company : "Hypoport AG" street : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } Multiple Version Converter Integration Test speaker_v1.json speaker_v2.json speaker_v3.json src/test/resources/
  9. Multiple Version Converter Integration Test src/test/java/ @DataProvider public Object[][] allVersions()

    { return new Object[][] { {"speaker_v1.json"}, {"speaker_v2.json"}, {"speaker_v3.json"}, }} @Test public void deserializationOfAllVersion(String filename) { String json = getFileAsJson(filename) DBObject speaker = (DBObject) JSON.parse(json); speakerConverter.convert(speaker); Speaker s = deserializeSpeaker(speaker); // assert structure and relevant attributes . . . }
  10. SpringData-Integration @Component public class SpeakerMongoEventListener extends AbstractMongoEventListener<Speaker> { @Inject SpeakerVersionConverter

    versionConverter; @Override public void onBeforeSave(Speaker source, DBObject dbo) { setModelVersion(dbo); } @Override public void onAfterLoad(DBObject dbo) { versionConverter.convert(dbo); } }
  11. Testing speakers {name: "Timmo Freudl-Gierke ", modelVersion: {name: "Joe Morrissey",

    modelVersion:1, ...} {name: "Stefan Rudnitzki ", modelVersion:2, ...} {name: "Christian Kvalheim ", modelVersion:1, ...} {name: "Ross Lawley ", modelVersion:2, ...} {name: "Thomas Rueckstiess ", modelVersion:3, ...} {name: "Johannes Hoppe ", modelVersion:3, ...} . . . $rename { name : "Timmo Freudl-Gierke", company : "Hypoport AG" street : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } Speaker name company street zipCode city Text