MongoEngine: Lessons learnt building an ODM

My PyCon Ireland 2012 talk about lessons learnt maintaining MongoEngine and working with the community.

1. Lessons learnt building @RossC0 http://github.com/rozza
2. WHAT IS MONGODB? A document database Highly scalable Developer friendly http://mongodb.org In BSON { _id : ObjectId("..."), author : "Ross", date : ISODate("2012-07-05..."), text : "About MongoDB...", tags : [ "tech", "databases" ], comments : [{ author : "Tim", date : ISODate("2012-07-05..."), text : "Best Post Ever!" }], comment_count : 1 }
3. WHAT IS MONGODB? In Python In BSON { "_id" : ObjectId("..."), "author" : "Ross", "date" : datetime(2012,7,5,10,0), "text" : "About MongoDB...", "tags" : ["tech", "databases"], "comments" : [{ "author" : "Tim", "date" : datetime(2012,7,5,11,35), "text" : "Best Post Ever!" }], "comment_count" : 1 } { _id : ObjectId("..."), author : "Ross", date : ISODate("2012-07-05..."), text : "About MongoDB...", tags : [ "tech", "databases" ], comments : [{ author : "Tim", date : ISODate("2012-07-05..."), text : "Best Post Ever!" }], comment_count : 1 }
4. http://education.10gen.com Want to know more?
5. http://www.flickr.com/photos/51838104@N02/5841690990 WH Y DO YO U EVE N NE ED AN ODM?
6. MongoDB a good fit Documents schema in code Enforces schema Data validation Speeds up development Build tooling off it Can DRY up code... SCHEMA LESS != CHAOS
7. Inspired by Django's ORM Supports Python 2.5 - Python 3.3 Originally authored by Harry Marr 2010 I took over development in May 2011 Current release 0.7.5 http://github.com/MongoEngine/mongoengine
8. INTRODUCING MONGOENGINE class Post(Document): title = StringField(max_length=120, required=True) author = ReferenceField('User') tags = ListField(StringField(max_length=30)) comments = ListField(EmbeddedDocumentField('Comment')) class Comment(EmbeddedDocument): content = StringField() name = StringField(max_length=120) class User(Document): email = StringField(required=True) first_name = StringField(max_length=50) last_name = StringField(max_length=50)
9. CREATING A MODEL class Post(Document): title = StringField(max_length=120, required=True) author = ReferenceField('User') tags = ListField(StringField(max_length=30)) comments = ListField(EmbeddedDocumentField('Comment')) Define a class inheriting from Document Map a field to a defined data type strings, ints, binary, files, lists etc.. By default all declared fields aren't required Pass keyword arguments to apply constraints eg set if unique, max_length, default values.
10. INSERTING DATA # Pass data into the constructor user = User(email="ross@10gen.com", name="Ross").save() # Create instance and edit / update in place post = Post() post.title = "mongoengine" post.author = user post.tags = ['odm', 'mongodb', 'python'] post.save() Create instance of the object Update its attributes Call save, insert, update to persist the data
11. QUERYING DATA # An `objects` manager is added to every `Document` class users = User.objects(email='ross@10gen.com') # Pass kwargs to commands are lazy and be extended as needed users.filter(auth=True) # Iterating evaluates the queryset print [u for u in users] Documents have a queryset manager (objects) for querying You can continually extend it Queryset evaluated on iteration
12. 6 LESSONS LEARNT
13. LE S SO N 1: DI VE IN ! http://www.flickr.com/photos/jackace/565857899/
14. In May 2011 >200 forks >100 issues ~50 pull requests I needed it Volunteered to help Started reviewing issues Supported Harry and community PROJECT STALLED
15. LE S SO N 2 : METACL ASSES http://www.flickr.com/photos/ubique/135848053
16. WHATS NEEDED TO MAKE AN ORM? Instance methods validation data manipulate data convert data to and from mongodb Queryset methods Finding data Bulk changes
17. METACLASSES class Document(object): __metaclass__ = TopLevelDocumentMetaclass ... class EmbeddedDocument(object): __metaclass__ = DocumentMetaclass ... Needed for: 1. inspect the object inheritance 2. inject functionality to the class Its surprisingly simple - all we need is: __new__
18. METACLASSES 101 TopLevelDocument Document python's type cls, name, bases, attrs IN new class Out
19. METACLASSES TopLevelDocument Document python's type Creates default meta data inheritance rules, id_field, index information, default ordering. Merges in parents meta Validation abstract flag on an inherited class collection set on a subclass Manipulates the attrs going in. IN
20. METACLASSES TopLevelDocument Document python's type Merges all fields from parents Adds in own field definitions Creates lookups _db_field_map _reverse_db_field_map Determine superclasses (for model inheritance) IN
21. METACLASSES TopLevelDocument Document python's type Adds in handling for delete rules So we can handle deleted References Adds class to the registry So we can load the data into the correct class OUT
22. METACLASSES TopLevelDocument Document python's type Builds index specifications Injects queryset manager Sets primary key (if needed) OUT
23. LESSONS LEARNT Spend time learning what is being done and why Don't continually patch: Rewrote the metaclasses in 0.7
24. LE S SO N 3: S TR AYI NG F ROM THE PATH http://www.flickr.com/photos/51838104@N02/5841690990
25. REWRITING THE QUERY LANGUAGE # In pymongo you pass dictionaries to query uk_pages = db.page.find({"published": True}) # In mongoengine uk_pages = Page.objects(published=True) # pymongo dot syntax to query sub documents uk_pages = db.page.find({"author.country": "uk"}) # In mongoengine we use dunder instead uk_pages = Page.objects(author__country='uk')
26. REWRITING THE QUERY LANGUAGE #Somethings are nicer - regular expresion search db.post.find({'title': re.compile('MongoDB', re.IGNORECASE)}) Post.objects(title__icontains='MongoDB') # In mongoengine # Chaining is nicer db.post.update({"published": False}, {"$set": {"published": True}}, multi=True) Post.objects(published=False).update(set__published=True)
27. LE S SO N 4 : NOT ALL IDEAS ARE GOOD http://www.flickr.com/photos/abiding_silence/6951229015
28. CHANGING SAVE # In pymongo save replaces the whole document db.post.save({'_id': 'my_id', 'title': 'MongoDB', 'published': True}) # In mongoengine we track changes post = Post.objects(_id='my_id').first() post.published = True post.save() # Results in: db.post.update({'_id': 'my_id'}, {'$set': {'published': True}})
29. CHANGING SAVE Any field changes are noted How to monitor lists and dicts? Custom List and Dict classes Observes changes and marks as dirty
30. HOW IT WORKS class Post(Document): title = StringField(max_length=120, required=True) author = ReferenceField('User') tags = ListField(StringField(max_length=30)) comments = ListField(EmbeddedDocumentField('Comment')) class User(Document): email = StringField(required=True) first_name = StringField(max_length=50) last_name = StringField(max_length=50) class Comment(EmbeddedDocument): content = StringField() name = StringField(max_length=120)
31. Post HOW IT WORKS - comments comment comment comment post = Post.objects.first() post.comments[1].name = 'Fred' post.save()
32. Post HOW IT WORKS - comments comment 1.Convert the comments data to a BaseList BaseList Stores the instance and name / location comment comment post.comments[1].name = 'Fred'
33. Post HOW IT WORKS - comments comment 2.Convert the comment data to BaseDict sets name as: 'comments.1' comment comment post.comments[1].name = 'Fred'
34. Post HOW IT WORKS - comments comment 3.Change name to "Fred" 4. Tell Post 'comments.1.name' has changed comment comment post.comments[1].name = 'Fred'
35. Post HOW IT WORKS - comments comment 5.On save() Iterate all the changes on post and run $set / $unset queries comment comment post.save() db.post.update( {'_id': 'my_id'}, {'$set': { 'comments.1.name': 'Fred'}} )
36. A GOOD IDEA? + Makes it easier to use + save acts how people think it should - Its expensive - Doesn't help people understand MongoDB
37. LE S SO N 5: M ANAGI NG A COMMUNIT Y http://kingscross.the-hub.net/
38. Github effect >10 django mongoengine projects None active on pypi Little cross project communication CODERS JUST WANT TO CODE * * Side effect of being stalled?
39. Flask-mongoengine on pypi There were 2 different projects Now has extra maintainers from the flask-mongorest Django-mongoengine* Spoke to authors of 7 projects and merged their work together to form a single library * Coming soon! REACH OUT
40. THE COMMUNITY Not all ideas are good! Vocal people don't always have great ideas Travis is great* * but you still have to read the pull request Communities have to be managed I've only just started to learn how to herding cats
41. LE S SO N 6 : DON' T BE AF R A ID TO ASK http://www.flickr.com/photos/kandyjaxx/2012468692
42. Website Documentation Tutorials Framework support Core mongoengine http://twitter.com/RossC0 I NEED HELP ;) http://github.com/MongoEngine
43. Q UE S T I ONS ? http://www.flickr.com/photos/9550033@N04/5020799468