Round Pegs and Square Holes

The talk that Audrey Roy and I gave at DjangoCon Europe on June 4, 2012.

Transcript

1. Round Pegs and Square Holes Django and MongoDB by Daniel Greenfeld and Audrey Roy Monday, June 4, 12
2. @pydanny / @audreyr cartwheelweb.com Audrey / Danny • Principals at Cartwheel Web • Co-founders of consumer.io • Met at PyCon 2010 • Now affianced Photo credit: Christopher Neugebauer Monday, June 4, 12
3. @pydanny / @audreyr cartwheelweb.com What is MongoDB? • NoSQL • Fast, Scalable, Indexable, Shardable • Schema-less Monday, June 4, 12
4. @pydanny / @audreyr cartwheelweb.com What is MongoDB? • Written in C++ • Stores data in BSON (kinda like JSON) • Uses JavaScript internally for scripting • Has Python, Ruby, PHP, C, other drivers • Huge community Monday, June 4, 12
5. @pydanny / @audreyr cartwheelweb.com MongoDB: SQL Analogy • Collections are like tables • Documents are like records (rows) Monday, June 4, 12
6. @pydanny / @audreyr cartwheelweb.com What is a MongoDB queryset? Minimalist view As served by pymongo Monday, June 4, 12
7. @pydanny / @audreyr cartwheelweb.com What is a MongoDB queryset? A list of dictionaries Minimalist view As served by pymongo Monday, June 4, 12
8. @pydanny / @audreyr cartwheelweb.com What it looks like Minimalist view collection = [] document = { '_objectId': ObjectId('4f844e916c97c1000c000003'), 'username': 'pydanny', 'fiancee': { 'username': 'audreyr', 'location': 'Zurich' } } collection = [document, ] list Monday, June 4, 12
9. @pydanny / @audreyr cartwheelweb.com What it looks like Minimalist view collection = [] document = { '_objectId': ObjectId('4f844e916c97c1000c000003'), 'username': 'pydanny', 'fiancee': { 'username': 'audreyr', 'location': 'Zurich' } } collection = [document, ] list dictionary Monday, June 4, 12
10. @pydanny / @audreyr cartwheelweb.com What it looks like Minimalist view collection = [] document = { '_objectId': ObjectId('4f844e916c97c1000c000003'), 'username': 'pydanny', 'fiancee': { 'username': 'audreyr', 'location': 'Zurich' } } collection = [document, ] list dictionary list of dictionaries! Monday, June 4, 12
11. Tools for using MongoDB with Django (actually, most of these are not Django-dependent) Monday, June 4, 12
12. @pydanny / @audreyr cartwheelweb.com “MongoDB connectors” • pymongo • mongoengine • mongokit • Django-nonrel • the only one here requiring Django Monday, June 4, 12
13. Option 1 pymongo Monday, June 4, 12
14. @pydanny / @audreyr cartwheelweb.com pymongo http://api.mongodb.org/python/current/ Official Python binding for MongoDB >>> from pymongo import Connection >>> connection = Connection() >>> my_data = {'rating': 3, 'title': 'I like ice cream'} >>> connection.reviews.insert(my_data) >>> your_data = {'rating': 3, 'subject': 'You like ice cream'} >>> connection.reviews.insert(your_data) Monday, June 4, 12
15. @pydanny / @audreyr cartwheelweb.com pymongo http://api.mongodb.org/python/current/ Official Python binding for MongoDB >>> from pymongo import Connection >>> connection = Connection() >>> my_data = {'rating': 3, 'title': 'I like ice cream'} >>> connection.reviews.insert(my_data) >>> your_data = {'rating': 3, 'subject': 'You like ice cream'} >>> connection.reviews.insert(your_data) my_data with title Monday, June 4, 12
16. @pydanny / @audreyr cartwheelweb.com pymongo http://api.mongodb.org/python/current/ Official Python binding for MongoDB >>> from pymongo import Connection >>> connection = Connection() >>> my_data = {'rating': 3, 'title': 'I like ice cream'} >>> connection.reviews.insert(my_data) >>> your_data = {'rating': 3, 'subject': 'You like ice cream'} >>> connection.reviews.insert(your_data) my_data with title your_data with subject Monday, June 4, 12
17. @pydanny / @audreyr cartwheelweb.com pymongo >>> connection = pymongo.Connection() >>> db = connection.db >>> for review in db.reviews.find({'rating': 3}): ... review['title'] >>> for review in db.reviews.find( ... {"title": {"$regex": "ice cream"} } ... ): ... review['title'] [ {'rating': 3,'title': 'I like ice cream'}, {'rating': 3, 'subject': 'You like ice cream'} ] Monday, June 4, 12
18. @pydanny / @audreyr cartwheelweb.com pymongo >>> connection = pymongo.Connection() >>> db = connection.db >>> for review in db.reviews.find({'rating': 3}): ... review['title'] >>> for review in db.reviews.find( ... {"title": {"$regex": "ice cream"} } ... ): ... review['title'] [ {'rating': 3,'title': 'I like ice cream'}, {'rating': 3, 'subject': 'You like ice cream'} ] Finds all reviews with a rating of 3 Monday, June 4, 12
19. @pydanny / @audreyr cartwheelweb.com pymongo >>> connection = pymongo.Connection() >>> db = connection.db >>> for review in db.reviews.find({'rating': 3}): ... review['title'] >>> for review in db.reviews.find( ... {"title": {"$regex": "ice cream"} } ... ): ... review['title'] [ {'rating': 3,'title': 'I like ice cream'}, {'rating': 3, 'subject': 'You like ice cream'} ] Monday, June 4, 12
20. @pydanny / @audreyr cartwheelweb.com pymongo >>> connection = pymongo.Connection() >>> db = connection.db >>> for review in db.reviews.find({'rating': 3}): ... review['title'] >>> for review in db.reviews.find( ... {"title": {"$regex": "ice cream"} } ... ): ... review['title'] [ {'rating': 3,'title': 'I like ice cream'}, {'rating': 3, 'subject': 'You like ice cream'} ] Only finds the document with ‘title’ in it. Monday, June 4, 12
21. @pydanny / @audreyr cartwheelweb.com pymongo • Really fast, bare metal • Lets you go schema-crazy • Supported directly by 10gen • They say “PyMongo is the recommended way to work with MongoDB from Python.” Pros Monday, June 4, 12
22. @pydanny / @audreyr cartwheelweb.com pymongo • “PyMongo introspection” is an oxymoron • Very low-level • Lose out on ModelForms, Auth, Admin, etc • Syntax not as clean as with an object mapper Cons Monday, June 4, 12
23. Option 2 MongoEngine Monday, June 4, 12
24. @pydanny / @audreyr cartwheelweb.com MongoEngine import mongoengine as me class Review(me.Document): title = me.StringField() body = me.StringField() author = me.StringField() created = me.DateTimeField(default=datetime.utcnow) rating = me.IntField() http://mongoengine.org/ Doesn’t this look like the Django ORM? Monday, June 4, 12
25. @pydanny / @audreyr cartwheelweb.com MongoEngine Doesn’t this look like a Django query? >>> from reviews.models import Review >>> for review in Review.objects.all(): ... review.title Monday, June 4, 12
26. @pydanny / @audreyr cartwheelweb.com • Looks similar to Django ORM code • You can develop SUPER-QUICKLY • Can use with django-mongonaut for introspection • Light schema, unenforced by the db • Supports some inter-document connections MongoEngine Pros Monday, June 4, 12
27. @pydanny / @audreyr cartwheelweb.com • Some feel there’s too much structure • Validation messages sometimes unclear • Using it with Django, you lose out on the Django Admin’s introspection* and ModelForms MongoEngine Cons * django-mongonaut addresses this. Monday, June 4, 12
28. @pydanny / @audreyr cartwheelweb.com mongoengine and django-mongonaut http://bit.ly/django-mongonaut Monday, June 4, 12
29. Option 3 MongoKit Monday, June 4, 12
30. @pydanny / @audreyr cartwheelweb.com MongoKit http://namlook.github.com/mongokit/ from mongokit import Document, Connection connection = Connection() @connection.register class Review(Document): structure = { 'title':unicode, 'body':unicode, 'author':unicode, 'created':datetime.datetime, 'rating':int } required_fields = ['title', 'author', 'created'] default_values = {'rating': 0, 'created': datetime.utcnow} Monday, June 4, 12
31. @pydanny / @audreyr cartwheelweb.com MongoKit http://namlook.github.com/mongokit/ from mongokit import Document, Connection connection = Connection() @connection.register class Review(Document): structure = { 'title':unicode, 'body':unicode, 'author':unicode, 'created':datetime.datetime, 'rating':int } required_fields = ['title', 'author', 'created'] default_values = {'rating': 0, 'created': datetime.utcnow} Review model representing MongoDB collection Monday, June 4, 12
32. @pydanny / @audreyr cartwheelweb.com MongoKit http://namlook.github.com/mongokit/ from mongokit import Document, Connection connection = Connection() @connection.register class Review(Document): structure = { 'title':unicode, 'body':unicode, 'author':unicode, 'created':datetime.datetime, 'rating':int } required_fields = ['title', 'author', 'created'] default_values = {'rating': 0, 'created': datetime.utcnow} Review model representing MongoDB collection Expected fields Monday, June 4, 12
33. @pydanny / @audreyr cartwheelweb.com MongoKit http://namlook.github.com/mongokit/ from mongokit import Document, Connection connection = Connection() @connection.register class Review(Document): structure = { 'title':unicode, 'body':unicode, 'author':unicode, 'created':datetime.datetime, 'rating':int } required_fields = ['title', 'author', 'created'] default_values = {'rating': 0, 'created': datetime.utcnow} Review model representing MongoDB collection Expected fields Connect to reviews collection Monday, June 4, 12
34. @pydanny / @audreyr cartwheelweb.com MongoKit http://namlook.github.com/mongokit/ >>> from mongokit import Connection >>> connection = Connection() >>> for review in connection.Review.find({'rating': 3}): ... review['title'] Identical to pymongo queries Monday, June 4, 12
35. @pydanny / @audreyr cartwheelweb.com • Light schema, unenforced by the db • Or can go all out schemaless! • Speed • Types are a mix of Python & MongoDB • Uses pymongo-style queries MongoKit Pros Monday, June 4, 12
36. @pydanny / @audreyr cartwheelweb.com • Using it with Django, you lose out on the Django Admin’s introspection, ModelForms, auth, etc • Introspection is hard MongoKit Cons Monday, June 4, 12
37. Option 4 Django-nonrel + mongodb-engine Monday, June 4, 12
38. @pydanny / @audreyr cartwheelweb.com • A patch to Django that adds NoSQL support to the Django ORM • Works with GAE, MongoDB, even SQL DBs http://docs.django-nonrel.org Django-nonrel + mongodb-engine Monday, June 4, 12
39. @pydanny / @audreyr cartwheelweb.com Django-nonrel + mongodb-engine • Can use Django as you normally would • Mirrors the ORM functionality • Introspection via djangotoolbox Pros Monday, June 4, 12
40. @pydanny / @audreyr cartwheelweb.com • Fork of the whole Django project. • Dependent on others to maintain parity with Django core (still on Django 1.3). • Multi-db usage is confusing Cons Django-nonrel + mongodb-engine Monday, June 4, 12
41. @pydanny / @audreyr cartwheelweb.com Summary • pymongo is low-level and well-supported by 10gen. • MongoEngine is like schemaless Django models. • MongoKit is like pymongo with extra structure. • Django-nonrel is a fork of Django 1.3. Monday, June 4, 12
42. The Take Away? Monday, June 4, 12
43. @pydanny / @audreyr cartwheelweb.com What about a single third-party app that lets you combine critical Django apps and MongoDB? Danny’s Thoughts Can we build a “simple” bridge? • django.contrib.auth • django.forms • django-social-auth / registration • others... Monday, June 4, 12
44. @pydanny / @audreyr cartwheelweb.com Danny’s Thoughts I wonder, why add schemas to schema-less when: * cache-machine or johnny-cache Relational Databases South High level Caching tools* allow you to develop fast moving datastores with transactions and built-in Django support? Monday, June 4, 12
45. @pydanny / @audreyr cartwheelweb.com Danny’s Thoughts Introspection Tool Idea Treat introspection like MongoDB Queries Immediate introspection tool (no ‘title’ then don’t show title) Monday, June 4, 12
46. @pydanny / @audreyr cartwheelweb.com Audrey’s Thoughts • Schemaless dbs promise performance advantages • Especially for distributed systems • Tradeoff: ACID compliance http://stackoverflow.com/questions/3856222/whats-the-attraction-of-schemaless-database-systems Monday, June 4, 12
47. @pydanny / @audreyr cartwheelweb.com Audrey’s Thoughts “Schemaless database” == ACID-noncompliant database OK when performance is more important than being consistent 100% of the time. Monday, June 4, 12
48. @pydanny / @audreyr cartwheelweb.com Audrey’s Thoughts Schemaless Python models != Schemaless MongoDB collections I think MongoEngine is best unless your use case calls for schema anarchy. Monday, June 4, 12
49. Going Forward Monday, June 4, 12
50. @pydanny / @audreyr cartwheelweb.com • Big hurdles: ORM, Admin, ModelForms, Auth • Were built for relational data • But the situation is improving rapidly Using Django With MongoDB Monday, June 4, 12
51. @pydanny / @audreyr cartwheelweb.com • New introspection tools (working on it) • django.forms bridge. • django.contrib.admin bridge. • Drop-in replacement for Django Auth (Django 1.5?) • Creation of best practices document for use of MongoDB with Django. What needs to be done Monday, June 4, 12
52. @pydanny / @audreyr cartwheelweb.com Django Mongonaut • Introspection tool for MongoEngine • Works well so far • Integrate graphing tools • Make independent from mongoengine • Contributors wanted: • https://github.com/pydanny/django-mongonaut Monday, June 4, 12
53. @pydanny / @audreyr cartwheelweb.com Django Mongonaut Mockup Based off of immediate introspection - not definition Monday, June 4, 12
54. @pydanny / @audreyr cartwheelweb.com Django Mongonaut Mockup Based off of immediate introspection - not definition SVG w/links for examining nested structures Monday, June 4, 12
55. @pydanny / @audreyr cartwheelweb.com Final Summary pymongo mongoengine mongokit django-nonrel django-mongodb django-mongonaut Python/MongoDB tools to consider { Monday, June 4, 12