Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Round Pegs and Square Holes

Round Pegs and Square Holes

The talk that Audrey Roy and I gave at DjangoCon Europe on June 4, 2012.

67e05420d4dd3492097aeb77f44f7867?s=128

Daniel Greenfeld

June 04, 2012
Tweet

More Decks by Daniel Greenfeld

Other Decks in Programming

Transcript

  1. Round Pegs and Square Holes Django and MongoDB by Daniel

    Greenfeld and Audrey Roy Monday, June 4, 12
  2. @pydanny / @audreyr cartwheelweb.com Audrey / Danny • Principals at

    Cartwheel Web • Co-founders of consumer.io • Met at PyCon 2010 • Now affianced Photo credit: Christopher Neugebauer Monday, June 4, 12
  3. @pydanny / @audreyr cartwheelweb.com What is MongoDB? • NoSQL •

    Fast, Scalable, Indexable, Shardable • Schema-less Monday, June 4, 12
  4. @pydanny / @audreyr cartwheelweb.com What is MongoDB? • Written in

    C++ • Stores data in BSON (kinda like JSON) • Uses JavaScript internally for scripting • Has Python, Ruby, PHP, C, other drivers • Huge community Monday, June 4, 12
  5. @pydanny / @audreyr cartwheelweb.com MongoDB: SQL Analogy • Collections are

    like tables • Documents are like records (rows) Monday, June 4, 12
  6. @pydanny / @audreyr cartwheelweb.com What is a MongoDB queryset? Minimalist

    view As served by pymongo Monday, June 4, 12
  7. @pydanny / @audreyr cartwheelweb.com What is a MongoDB queryset? A

    list of dictionaries Minimalist view As served by pymongo Monday, June 4, 12
  8. @pydanny / @audreyr cartwheelweb.com What it looks like Minimalist view

    collection = [] document = { '_objectId': ObjectId('4f844e916c97c1000c000003'), 'username': 'pydanny', 'fiancee': { 'username': 'audreyr', 'location': 'Zurich' } } collection = [document, ] list Monday, June 4, 12
  9. @pydanny / @audreyr cartwheelweb.com What it looks like Minimalist view

    collection = [] document = { '_objectId': ObjectId('4f844e916c97c1000c000003'), 'username': 'pydanny', 'fiancee': { 'username': 'audreyr', 'location': 'Zurich' } } collection = [document, ] list dictionary Monday, June 4, 12
  10. @pydanny / @audreyr cartwheelweb.com What it looks like Minimalist view

    collection = [] document = { '_objectId': ObjectId('4f844e916c97c1000c000003'), 'username': 'pydanny', 'fiancee': { 'username': 'audreyr', 'location': 'Zurich' } } collection = [document, ] list dictionary list of dictionaries! Monday, June 4, 12
  11. Tools for using MongoDB with Django (actually, most of these

    are not Django-dependent) Monday, June 4, 12
  12. @pydanny / @audreyr cartwheelweb.com “MongoDB connectors” • pymongo • mongoengine

    • mongokit • Django-nonrel • the only one here requiring Django Monday, June 4, 12
  13. Option 1 pymongo Monday, June 4, 12

  14. @pydanny / @audreyr cartwheelweb.com pymongo http://api.mongodb.org/python/current/ Official Python binding for

    MongoDB >>> from pymongo import Connection >>> connection = Connection() >>> my_data = {'rating': 3, 'title': 'I like ice cream'} >>> connection.reviews.insert(my_data) >>> your_data = {'rating': 3, 'subject': 'You like ice cream'} >>> connection.reviews.insert(your_data) Monday, June 4, 12
  15. @pydanny / @audreyr cartwheelweb.com pymongo http://api.mongodb.org/python/current/ Official Python binding for

    MongoDB >>> from pymongo import Connection >>> connection = Connection() >>> my_data = {'rating': 3, 'title': 'I like ice cream'} >>> connection.reviews.insert(my_data) >>> your_data = {'rating': 3, 'subject': 'You like ice cream'} >>> connection.reviews.insert(your_data) my_data with title Monday, June 4, 12
  16. @pydanny / @audreyr cartwheelweb.com pymongo http://api.mongodb.org/python/current/ Official Python binding for

    MongoDB >>> from pymongo import Connection >>> connection = Connection() >>> my_data = {'rating': 3, 'title': 'I like ice cream'} >>> connection.reviews.insert(my_data) >>> your_data = {'rating': 3, 'subject': 'You like ice cream'} >>> connection.reviews.insert(your_data) my_data with title your_data with subject Monday, June 4, 12
  17. @pydanny / @audreyr cartwheelweb.com pymongo >>> connection = pymongo.Connection() >>>

    db = connection.db >>> for review in db.reviews.find({'rating': 3}): ... review['title'] >>> for review in db.reviews.find( ... {"title": {"$regex": "ice cream"} } ... ): ... review['title'] [ {'rating': 3,'title': 'I like ice cream'}, {'rating': 3, 'subject': 'You like ice cream'} ] Monday, June 4, 12
  18. @pydanny / @audreyr cartwheelweb.com pymongo >>> connection = pymongo.Connection() >>>

    db = connection.db >>> for review in db.reviews.find({'rating': 3}): ... review['title'] >>> for review in db.reviews.find( ... {"title": {"$regex": "ice cream"} } ... ): ... review['title'] [ {'rating': 3,'title': 'I like ice cream'}, {'rating': 3, 'subject': 'You like ice cream'} ] Finds all reviews with a rating of 3 Monday, June 4, 12
  19. @pydanny / @audreyr cartwheelweb.com pymongo >>> connection = pymongo.Connection() >>>

    db = connection.db >>> for review in db.reviews.find({'rating': 3}): ... review['title'] >>> for review in db.reviews.find( ... {"title": {"$regex": "ice cream"} } ... ): ... review['title'] [ {'rating': 3,'title': 'I like ice cream'}, {'rating': 3, 'subject': 'You like ice cream'} ] Monday, June 4, 12
  20. @pydanny / @audreyr cartwheelweb.com pymongo >>> connection = pymongo.Connection() >>>

    db = connection.db >>> for review in db.reviews.find({'rating': 3}): ... review['title'] >>> for review in db.reviews.find( ... {"title": {"$regex": "ice cream"} } ... ): ... review['title'] [ {'rating': 3,'title': 'I like ice cream'}, {'rating': 3, 'subject': 'You like ice cream'} ] Only finds the document with ‘title’ in it. Monday, June 4, 12
  21. @pydanny / @audreyr cartwheelweb.com pymongo • Really fast, bare metal

    • Lets you go schema-crazy • Supported directly by 10gen • They say “PyMongo is the recommended way to work with MongoDB from Python.” Pros Monday, June 4, 12
  22. @pydanny / @audreyr cartwheelweb.com pymongo • “PyMongo introspection” is an

    oxymoron • Very low-level • Lose out on ModelForms, Auth, Admin, etc • Syntax not as clean as with an object mapper Cons Monday, June 4, 12
  23. Option 2 MongoEngine Monday, June 4, 12

  24. @pydanny / @audreyr cartwheelweb.com MongoEngine import mongoengine as me class

    Review(me.Document): title = me.StringField() body = me.StringField() author = me.StringField() created = me.DateTimeField(default=datetime.utcnow) rating = me.IntField() http://mongoengine.org/ Doesn’t this look like the Django ORM? Monday, June 4, 12
  25. @pydanny / @audreyr cartwheelweb.com MongoEngine Doesn’t this look like a

    Django query? >>> from reviews.models import Review >>> for review in Review.objects.all(): ... review.title Monday, June 4, 12
  26. @pydanny / @audreyr cartwheelweb.com • Looks similar to Django ORM

    code • You can develop SUPER-QUICKLY • Can use with django-mongonaut for introspection • Light schema, unenforced by the db • Supports some inter-document connections MongoEngine Pros Monday, June 4, 12
  27. @pydanny / @audreyr cartwheelweb.com • Some feel there’s too much

    structure • Validation messages sometimes unclear • Using it with Django, you lose out on the Django Admin’s introspection* and ModelForms MongoEngine Cons * django-mongonaut addresses this. Monday, June 4, 12
  28. @pydanny / @audreyr cartwheelweb.com mongoengine and django-mongonaut http://bit.ly/django-mongonaut Monday, June

    4, 12
  29. Option 3 MongoKit Monday, June 4, 12

  30. @pydanny / @audreyr cartwheelweb.com MongoKit http://namlook.github.com/mongokit/ from mongokit import Document,

    Connection connection = Connection() @connection.register class Review(Document): structure = { 'title':unicode, 'body':unicode, 'author':unicode, 'created':datetime.datetime, 'rating':int } required_fields = ['title', 'author', 'created'] default_values = {'rating': 0, 'created': datetime.utcnow} Monday, June 4, 12
  31. @pydanny / @audreyr cartwheelweb.com MongoKit http://namlook.github.com/mongokit/ from mongokit import Document,

    Connection connection = Connection() @connection.register class Review(Document): structure = { 'title':unicode, 'body':unicode, 'author':unicode, 'created':datetime.datetime, 'rating':int } required_fields = ['title', 'author', 'created'] default_values = {'rating': 0, 'created': datetime.utcnow} Review model representing MongoDB collection Monday, June 4, 12
  32. @pydanny / @audreyr cartwheelweb.com MongoKit http://namlook.github.com/mongokit/ from mongokit import Document,

    Connection connection = Connection() @connection.register class Review(Document): structure = { 'title':unicode, 'body':unicode, 'author':unicode, 'created':datetime.datetime, 'rating':int } required_fields = ['title', 'author', 'created'] default_values = {'rating': 0, 'created': datetime.utcnow} Review model representing MongoDB collection Expected fields Monday, June 4, 12
  33. @pydanny / @audreyr cartwheelweb.com MongoKit http://namlook.github.com/mongokit/ from mongokit import Document,

    Connection connection = Connection() @connection.register class Review(Document): structure = { 'title':unicode, 'body':unicode, 'author':unicode, 'created':datetime.datetime, 'rating':int } required_fields = ['title', 'author', 'created'] default_values = {'rating': 0, 'created': datetime.utcnow} Review model representing MongoDB collection Expected fields Connect to reviews collection Monday, June 4, 12
  34. @pydanny / @audreyr cartwheelweb.com MongoKit http://namlook.github.com/mongokit/ >>> from mongokit import

    Connection >>> connection = Connection() >>> for review in connection.Review.find({'rating': 3}): ... review['title'] Identical to pymongo queries Monday, June 4, 12
  35. @pydanny / @audreyr cartwheelweb.com • Light schema, unenforced by the

    db • Or can go all out schemaless! • Speed • Types are a mix of Python & MongoDB • Uses pymongo-style queries MongoKit Pros Monday, June 4, 12
  36. @pydanny / @audreyr cartwheelweb.com • Using it with Django, you

    lose out on the Django Admin’s introspection, ModelForms, auth, etc • Introspection is hard MongoKit Cons Monday, June 4, 12
  37. Option 4 Django-nonrel + mongodb-engine Monday, June 4, 12

  38. @pydanny / @audreyr cartwheelweb.com • A patch to Django that

    adds NoSQL support to the Django ORM • Works with GAE, MongoDB, even SQL DBs http://docs.django-nonrel.org Django-nonrel + mongodb-engine Monday, June 4, 12
  39. @pydanny / @audreyr cartwheelweb.com Django-nonrel + mongodb-engine • Can use

    Django as you normally would • Mirrors the ORM functionality • Introspection via djangotoolbox Pros Monday, June 4, 12
  40. @pydanny / @audreyr cartwheelweb.com • Fork of the whole Django

    project. • Dependent on others to maintain parity with Django core (still on Django 1.3). • Multi-db usage is confusing Cons Django-nonrel + mongodb-engine Monday, June 4, 12
  41. @pydanny / @audreyr cartwheelweb.com Summary • pymongo is low-level and

    well-supported by 10gen. • MongoEngine is like schemaless Django models. • MongoKit is like pymongo with extra structure. • Django-nonrel is a fork of Django 1.3. Monday, June 4, 12
  42. The Take Away? Monday, June 4, 12

  43. @pydanny / @audreyr cartwheelweb.com What about a single third-party app

    that lets you combine critical Django apps and MongoDB? Danny’s Thoughts Can we build a “simple” bridge? • django.contrib.auth • django.forms • django-social-auth / registration • others... Monday, June 4, 12
  44. @pydanny / @audreyr cartwheelweb.com Danny’s Thoughts I wonder, why add

    schemas to schema-less when: * cache-machine or johnny-cache Relational Databases South High level Caching tools* allow you to develop fast moving datastores with transactions and built-in Django support? Monday, June 4, 12
  45. @pydanny / @audreyr cartwheelweb.com Danny’s Thoughts Introspection Tool Idea Treat

    introspection like MongoDB Queries Immediate introspection tool (no ‘title’ then don’t show title) Monday, June 4, 12
  46. @pydanny / @audreyr cartwheelweb.com Audrey’s Thoughts • Schemaless dbs promise

    performance advantages • Especially for distributed systems • Tradeoff: ACID compliance http://stackoverflow.com/questions/3856222/whats-the-attraction-of-schemaless-database-systems Monday, June 4, 12
  47. @pydanny / @audreyr cartwheelweb.com Audrey’s Thoughts “Schemaless database” == ACID-noncompliant

    database OK when performance is more important than being consistent 100% of the time. Monday, June 4, 12
  48. @pydanny / @audreyr cartwheelweb.com Audrey’s Thoughts Schemaless Python models !=

    Schemaless MongoDB collections I think MongoEngine is best unless your use case calls for schema anarchy. Monday, June 4, 12
  49. Going Forward Monday, June 4, 12

  50. @pydanny / @audreyr cartwheelweb.com • Big hurdles: ORM, Admin, ModelForms,

    Auth • Were built for relational data • But the situation is improving rapidly Using Django With MongoDB Monday, June 4, 12
  51. @pydanny / @audreyr cartwheelweb.com • New introspection tools (working on

    it) • django.forms bridge. • django.contrib.admin bridge. • Drop-in replacement for Django Auth (Django 1.5?) • Creation of best practices document for use of MongoDB with Django. What needs to be done Monday, June 4, 12
  52. @pydanny / @audreyr cartwheelweb.com Django Mongonaut • Introspection tool for

    MongoEngine • Works well so far • Integrate graphing tools • Make independent from mongoengine • Contributors wanted: • https://github.com/pydanny/django-mongonaut Monday, June 4, 12
  53. @pydanny / @audreyr cartwheelweb.com Django Mongonaut Mockup Based off of

    immediate introspection - not definition Monday, June 4, 12
  54. @pydanny / @audreyr cartwheelweb.com Django Mongonaut Mockup Based off of

    immediate introspection - not definition SVG w/links for examining nested structures Monday, June 4, 12
  55. @pydanny / @audreyr cartwheelweb.com Final Summary pymongo mongoengine mongokit django-nonrel

    django-mongodb django-mongonaut Python/MongoDB tools to consider { Monday, June 4, 12