Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Round Pegs and Square Holes

Round Pegs and Square Holes

The talk that Audrey Roy and I gave at DjangoCon Europe on June 4, 2012.

Daniel Greenfeld

June 04, 2012
Tweet

More Decks by Daniel Greenfeld

Other Decks in Programming

Transcript

  1. Round Pegs
    and
    Square Holes
    Django and MongoDB
    by Daniel Greenfeld and Audrey Roy
    Monday, June 4, 12

    View full-size slide

  2. @pydanny / @audreyr cartwheelweb.com
    Audrey / Danny
    • Principals at Cartwheel Web
    • Co-founders of consumer.io
    • Met at PyCon 2010
    • Now affianced
    Photo credit: Christopher Neugebauer
    Monday, June 4, 12

    View full-size slide

  3. @pydanny / @audreyr cartwheelweb.com
    What is MongoDB?
    • NoSQL
    • Fast, Scalable, Indexable, Shardable
    • Schema-less
    Monday, June 4, 12

    View full-size slide

  4. @pydanny / @audreyr cartwheelweb.com
    What is MongoDB?
    • Written in C++
    • Stores data in BSON (kinda like JSON)
    • Uses JavaScript internally for scripting
    • Has Python, Ruby, PHP, C, other drivers
    • Huge community
    Monday, June 4, 12

    View full-size slide

  5. @pydanny / @audreyr cartwheelweb.com
    MongoDB: SQL Analogy
    • Collections are like tables
    • Documents are like records (rows)
    Monday, June 4, 12

    View full-size slide

  6. @pydanny / @audreyr cartwheelweb.com
    What is a MongoDB queryset?
    Minimalist view
    As served by pymongo
    Monday, June 4, 12

    View full-size slide

  7. @pydanny / @audreyr cartwheelweb.com
    What is a MongoDB queryset?
    A list
    of dictionaries
    Minimalist view
    As served by pymongo
    Monday, June 4, 12

    View full-size slide

  8. @pydanny / @audreyr cartwheelweb.com
    What it looks like
    Minimalist view
    collection = []
    document = {
    '_objectId': ObjectId('4f844e916c97c1000c000003'),
    'username': 'pydanny',
    'fiancee': {
    'username': 'audreyr',
    'location': 'Zurich'
    }
    }
    collection = [document, ]
    list
    Monday, June 4, 12

    View full-size slide

  9. @pydanny / @audreyr cartwheelweb.com
    What it looks like
    Minimalist view
    collection = []
    document = {
    '_objectId': ObjectId('4f844e916c97c1000c000003'),
    'username': 'pydanny',
    'fiancee': {
    'username': 'audreyr',
    'location': 'Zurich'
    }
    }
    collection = [document, ]
    list
    dictionary
    Monday, June 4, 12

    View full-size slide

  10. @pydanny / @audreyr cartwheelweb.com
    What it looks like
    Minimalist view
    collection = []
    document = {
    '_objectId': ObjectId('4f844e916c97c1000c000003'),
    'username': 'pydanny',
    'fiancee': {
    'username': 'audreyr',
    'location': 'Zurich'
    }
    }
    collection = [document, ]
    list
    dictionary
    list of dictionaries!
    Monday, June 4, 12

    View full-size slide

  11. Tools for using
    MongoDB with Django
    (actually, most of these are not Django-dependent)
    Monday, June 4, 12

    View full-size slide

  12. @pydanny / @audreyr cartwheelweb.com
    “MongoDB connectors”
    • pymongo
    • mongoengine
    • mongokit
    • Django-nonrel
    • the only one here requiring Django
    Monday, June 4, 12

    View full-size slide

  13. Option 1
    pymongo
    Monday, June 4, 12

    View full-size slide

  14. @pydanny / @audreyr cartwheelweb.com
    pymongo
    http://api.mongodb.org/python/current/
    Official Python binding for MongoDB
    >>> from pymongo import Connection
    >>> connection = Connection()
    >>> my_data = {'rating': 3, 'title': 'I like ice cream'}
    >>> connection.reviews.insert(my_data)
    >>> your_data = {'rating': 3, 'subject': 'You like ice cream'}
    >>> connection.reviews.insert(your_data)
    Monday, June 4, 12

    View full-size slide

  15. @pydanny / @audreyr cartwheelweb.com
    pymongo
    http://api.mongodb.org/python/current/
    Official Python binding for MongoDB
    >>> from pymongo import Connection
    >>> connection = Connection()
    >>> my_data = {'rating': 3, 'title': 'I like ice cream'}
    >>> connection.reviews.insert(my_data)
    >>> your_data = {'rating': 3, 'subject': 'You like ice cream'}
    >>> connection.reviews.insert(your_data)
    my_data with title
    Monday, June 4, 12

    View full-size slide

  16. @pydanny / @audreyr cartwheelweb.com
    pymongo
    http://api.mongodb.org/python/current/
    Official Python binding for MongoDB
    >>> from pymongo import Connection
    >>> connection = Connection()
    >>> my_data = {'rating': 3, 'title': 'I like ice cream'}
    >>> connection.reviews.insert(my_data)
    >>> your_data = {'rating': 3, 'subject': 'You like ice cream'}
    >>> connection.reviews.insert(your_data)
    my_data with title
    your_data with subject
    Monday, June 4, 12

    View full-size slide

  17. @pydanny / @audreyr cartwheelweb.com
    pymongo
    >>> connection = pymongo.Connection()
    >>> db = connection.db
    >>> for review in db.reviews.find({'rating': 3}):
    ... review['title']
    >>> for review in db.reviews.find(
    ... {"title": {"$regex": "ice cream"} }
    ... ):
    ... review['title']
    [
    {'rating': 3,'title': 'I like ice cream'},
    {'rating': 3, 'subject': 'You like ice cream'}
    ]
    Monday, June 4, 12

    View full-size slide

  18. @pydanny / @audreyr cartwheelweb.com
    pymongo
    >>> connection = pymongo.Connection()
    >>> db = connection.db
    >>> for review in db.reviews.find({'rating': 3}):
    ... review['title']
    >>> for review in db.reviews.find(
    ... {"title": {"$regex": "ice cream"} }
    ... ):
    ... review['title']
    [
    {'rating': 3,'title': 'I like ice cream'},
    {'rating': 3, 'subject': 'You like ice cream'}
    ]
    Finds all reviews
    with a rating of 3
    Monday, June 4, 12

    View full-size slide

  19. @pydanny / @audreyr cartwheelweb.com
    pymongo
    >>> connection = pymongo.Connection()
    >>> db = connection.db
    >>> for review in db.reviews.find({'rating': 3}):
    ... review['title']
    >>> for review in db.reviews.find(
    ... {"title": {"$regex": "ice cream"} }
    ... ):
    ... review['title']
    [
    {'rating': 3,'title': 'I like ice cream'},
    {'rating': 3, 'subject': 'You like ice cream'}
    ]
    Monday, June 4, 12

    View full-size slide

  20. @pydanny / @audreyr cartwheelweb.com
    pymongo
    >>> connection = pymongo.Connection()
    >>> db = connection.db
    >>> for review in db.reviews.find({'rating': 3}):
    ... review['title']
    >>> for review in db.reviews.find(
    ... {"title": {"$regex": "ice cream"} }
    ... ):
    ... review['title']
    [
    {'rating': 3,'title': 'I like ice cream'},
    {'rating': 3, 'subject': 'You like ice cream'}
    ]
    Only finds the document
    with ‘title’ in it.
    Monday, June 4, 12

    View full-size slide

  21. @pydanny / @audreyr cartwheelweb.com
    pymongo
    • Really fast, bare metal
    • Lets you go schema-crazy
    • Supported directly by 10gen
    • They say “PyMongo is the recommended
    way to work with MongoDB from
    Python.”
    Pros
    Monday, June 4, 12

    View full-size slide

  22. @pydanny / @audreyr cartwheelweb.com
    pymongo
    • “PyMongo introspection” is an oxymoron
    • Very low-level
    • Lose out on ModelForms, Auth, Admin, etc
    • Syntax not as clean as with an object mapper
    Cons
    Monday, June 4, 12

    View full-size slide

  23. Option 2
    MongoEngine
    Monday, June 4, 12

    View full-size slide

  24. @pydanny / @audreyr cartwheelweb.com
    MongoEngine
    import mongoengine as me
    class Review(me.Document):
    title = me.StringField()
    body = me.StringField()
    author = me.StringField()
    created = me.DateTimeField(default=datetime.utcnow)
    rating = me.IntField()
    http://mongoengine.org/
    Doesn’t this look
    like the Django ORM?
    Monday, June 4, 12

    View full-size slide

  25. @pydanny / @audreyr cartwheelweb.com
    MongoEngine
    Doesn’t this look
    like a Django query?
    >>> from reviews.models import Review
    >>> for review in Review.objects.all():
    ... review.title
    Monday, June 4, 12

    View full-size slide

  26. @pydanny / @audreyr cartwheelweb.com
    • Looks similar to Django ORM code
    • You can develop SUPER-QUICKLY
    • Can use with django-mongonaut for introspection
    • Light schema, unenforced by the db
    • Supports some inter-document connections
    MongoEngine
    Pros
    Monday, June 4, 12

    View full-size slide

  27. @pydanny / @audreyr cartwheelweb.com
    • Some feel there’s too much structure
    • Validation messages sometimes unclear
    • Using it with Django, you lose out on the
    Django Admin’s introspection* and
    ModelForms
    MongoEngine
    Cons
    * django-mongonaut addresses this.
    Monday, June 4, 12

    View full-size slide

  28. @pydanny / @audreyr cartwheelweb.com
    mongoengine and
    django-mongonaut
    http://bit.ly/django-mongonaut
    Monday, June 4, 12

    View full-size slide

  29. Option 3
    MongoKit
    Monday, June 4, 12

    View full-size slide

  30. @pydanny / @audreyr cartwheelweb.com
    MongoKit
    http://namlook.github.com/mongokit/
    from mongokit import Document, Connection
    connection = Connection()
    @connection.register
    class Review(Document):
    structure = {
    'title':unicode,
    'body':unicode,
    'author':unicode,
    'created':datetime.datetime,
    'rating':int
    }
    required_fields = ['title', 'author', 'created']
    default_values = {'rating': 0,
    'created': datetime.utcnow}
    Monday, June 4, 12

    View full-size slide

  31. @pydanny / @audreyr cartwheelweb.com
    MongoKit
    http://namlook.github.com/mongokit/
    from mongokit import Document, Connection
    connection = Connection()
    @connection.register
    class Review(Document):
    structure = {
    'title':unicode,
    'body':unicode,
    'author':unicode,
    'created':datetime.datetime,
    'rating':int
    }
    required_fields = ['title', 'author', 'created']
    default_values = {'rating': 0,
    'created': datetime.utcnow}
    Review model
    representing
    MongoDB collection
    Monday, June 4, 12

    View full-size slide

  32. @pydanny / @audreyr cartwheelweb.com
    MongoKit
    http://namlook.github.com/mongokit/
    from mongokit import Document, Connection
    connection = Connection()
    @connection.register
    class Review(Document):
    structure = {
    'title':unicode,
    'body':unicode,
    'author':unicode,
    'created':datetime.datetime,
    'rating':int
    }
    required_fields = ['title', 'author', 'created']
    default_values = {'rating': 0,
    'created': datetime.utcnow}
    Review model
    representing
    MongoDB collection
    Expected
    fields
    Monday, June 4, 12

    View full-size slide

  33. @pydanny / @audreyr cartwheelweb.com
    MongoKit
    http://namlook.github.com/mongokit/
    from mongokit import Document, Connection
    connection = Connection()
    @connection.register
    class Review(Document):
    structure = {
    'title':unicode,
    'body':unicode,
    'author':unicode,
    'created':datetime.datetime,
    'rating':int
    }
    required_fields = ['title', 'author', 'created']
    default_values = {'rating': 0,
    'created': datetime.utcnow}
    Review model
    representing
    MongoDB collection
    Expected
    fields
    Connect to
    reviews
    collection
    Monday, June 4, 12

    View full-size slide

  34. @pydanny / @audreyr cartwheelweb.com
    MongoKit
    http://namlook.github.com/mongokit/
    >>> from mongokit import Connection
    >>> connection = Connection()
    >>> for review in connection.Review.find({'rating': 3}):
    ... review['title']
    Identical to pymongo queries
    Monday, June 4, 12

    View full-size slide

  35. @pydanny / @audreyr cartwheelweb.com
    • Light schema, unenforced by the db
    • Or can go all out schemaless!
    • Speed
    • Types are a mix of Python & MongoDB
    • Uses pymongo-style queries
    MongoKit
    Pros
    Monday, June 4, 12

    View full-size slide

  36. @pydanny / @audreyr cartwheelweb.com
    • Using it with Django, you lose out on the
    Django Admin’s introspection, ModelForms,
    auth, etc
    • Introspection is hard
    MongoKit
    Cons
    Monday, June 4, 12

    View full-size slide

  37. Option 4
    Django-nonrel
    +
    mongodb-engine
    Monday, June 4, 12

    View full-size slide

  38. @pydanny / @audreyr cartwheelweb.com
    • A patch to Django that adds NoSQL support
    to the Django ORM
    • Works with GAE, MongoDB, even SQL DBs
    http://docs.django-nonrel.org
    Django-nonrel
    + mongodb-engine
    Monday, June 4, 12

    View full-size slide

  39. @pydanny / @audreyr cartwheelweb.com
    Django-nonrel
    + mongodb-engine
    • Can use Django as you normally would
    • Mirrors the ORM functionality
    • Introspection via djangotoolbox
    Pros
    Monday, June 4, 12

    View full-size slide

  40. @pydanny / @audreyr cartwheelweb.com
    • Fork of the whole Django project.
    • Dependent on others to maintain parity with
    Django core (still on Django 1.3).
    • Multi-db usage is confusing
    Cons
    Django-nonrel
    + mongodb-engine
    Monday, June 4, 12

    View full-size slide

  41. @pydanny / @audreyr cartwheelweb.com
    Summary
    • pymongo is low-level and well-supported by 10gen.
    • MongoEngine is like schemaless Django models.
    • MongoKit is like pymongo with extra structure.
    • Django-nonrel is a fork of Django 1.3.
    Monday, June 4, 12

    View full-size slide

  42. The Take Away?
    Monday, June 4, 12

    View full-size slide

  43. @pydanny / @audreyr cartwheelweb.com
    What about a single third-party app that lets you
    combine critical Django apps and MongoDB?
    Danny’s Thoughts
    Can we build a “simple” bridge?
    • django.contrib.auth
    • django.forms
    • django-social-auth / registration
    • others...
    Monday, June 4, 12

    View full-size slide

  44. @pydanny / @audreyr cartwheelweb.com
    Danny’s Thoughts
    I wonder, why add schemas to schema-less when:
    * cache-machine or johnny-cache
    Relational Databases
    South
    High level Caching tools*
    allow you to develop fast moving datastores
    with transactions and built-in Django support?
    Monday, June 4, 12

    View full-size slide

  45. @pydanny / @audreyr cartwheelweb.com
    Danny’s Thoughts
    Introspection Tool Idea
    Treat introspection like MongoDB Queries
    Immediate introspection tool
    (no ‘title’ then don’t show title)
    Monday, June 4, 12

    View full-size slide

  46. @pydanny / @audreyr cartwheelweb.com
    Audrey’s Thoughts
    • Schemaless dbs promise performance
    advantages
    • Especially for distributed systems
    • Tradeoff: ACID compliance
    http://stackoverflow.com/questions/3856222/whats-the-attraction-of-schemaless-database-systems
    Monday, June 4, 12

    View full-size slide

  47. @pydanny / @audreyr cartwheelweb.com
    Audrey’s Thoughts
    “Schemaless database”
    ==
    ACID-noncompliant database
    OK when performance is more important than being
    consistent 100% of the time.
    Monday, June 4, 12

    View full-size slide

  48. @pydanny / @audreyr cartwheelweb.com
    Audrey’s Thoughts
    Schemaless Python models
    !=
    Schemaless MongoDB collections
    I think MongoEngine is best unless your use case calls for
    schema anarchy.
    Monday, June 4, 12

    View full-size slide

  49. Going Forward
    Monday, June 4, 12

    View full-size slide

  50. @pydanny / @audreyr cartwheelweb.com
    • Big hurdles: ORM, Admin, ModelForms, Auth
    • Were built for relational data
    • But the situation is improving rapidly
    Using Django With
    MongoDB
    Monday, June 4, 12

    View full-size slide

  51. @pydanny / @audreyr cartwheelweb.com
    • New introspection tools (working on it)
    • django.forms bridge.
    • django.contrib.admin bridge.
    • Drop-in replacement for Django Auth (Django 1.5?)
    • Creation of best practices document for use of
    MongoDB with Django.
    What needs to be
    done
    Monday, June 4, 12

    View full-size slide

  52. @pydanny / @audreyr cartwheelweb.com
    Django Mongonaut
    • Introspection tool for MongoEngine
    • Works well so far
    • Integrate graphing tools
    • Make independent from mongoengine
    • Contributors wanted:
    • https://github.com/pydanny/django-mongonaut
    Monday, June 4, 12

    View full-size slide

  53. @pydanny / @audreyr cartwheelweb.com
    Django Mongonaut
    Mockup
    Based off of immediate
    introspection - not definition
    Monday, June 4, 12

    View full-size slide

  54. @pydanny / @audreyr cartwheelweb.com
    Django Mongonaut
    Mockup
    Based off of immediate
    introspection - not definition
    SVG w/links
    for examining
    nested structures
    Monday, June 4, 12

    View full-size slide

  55. @pydanny / @audreyr cartwheelweb.com
    Final Summary
    pymongo
    mongoengine
    mongokit
    django-nonrel
    django-mongodb
    django-mongonaut
    Python/MongoDB
    tools to consider
    {
    Monday, June 4, 12

    View full-size slide