Python-CouchDB Training at PyCon PL 2012

Transcript

1. Using CouchDB with Python Stefan Kögl @skoegl
2. What we will cover • What is CouchDB? – Access from Python though couchdbkit – Key-value Store Functionality – MapReduce Queries – HTTP API • When is CouchDB useful and when not? – Multi-Master Replication – Scaling up and down • Pointers to other resources, CouchDB ecosystem
3. What we won't cover • CouchApps – browser-based apps that are served by CouchDB • Detailled Security, Scaling and other operative issues • Other functionality that didn't fit
4. Training Modes • Code-Along – Follow Examples, write your own code – Small Scripts or REPL • Learning-by-Watching – Example Application at https://github.com/stefankoegl/python-couchdb-examples – Slides at https://slideshare.net/skoegl/couch-db-pythonpyconpl2012 – Use example scripts and see what happens – Submit Pull-Requests!
5. Contents • Intro – Contents – CouchDB – Example Application • DB Initialization • Key-Value Store • Simple MapReduce Queries • The _changes Feed • Complex MapReduce Queries • Replication • Additional Features and the Couch Ecosystem
6. CouchDB • Apache Project • https://couchdb.apache.org/ • Current Version: 1.2 • Apache CouchDB™ is a database that uses JSON for documents, JavaScript for MapReduce queries, and regular HTTP for an API
7. Example Application • Lending Database – Stores Items that you might want to lend – Stores when you have lent what to whom • Stand-alone or distributed • Small Scripts that do one task each • Look at HTTP traffic
8. Contents • Intro • DB Initialization – Setting Up CouchDB – Installing couchdbkit – Creating a Database • Key-Value Store • Simple MapReduce Queries • The _changes Feed • Complex MapReduce Queries • Replication • Additional Features and the Couch Ecosystem
9. Getting Set Up: CouchDB • Provided by me (not valid anymore after the training) • http://couch.skoegl.net:5984/<yourname> • Authentication: username training, password training • Setup your DB_URL in settings.py • If you want to install your own – Tutorials: https://wiki.apache.org/couchdb/Installation – Ubuntu: https://launchpad.net/~longsleep/+archive/couchdb – Mac, Windows: https://couchdb.apache.org/#download
10. Getting Set Up: couchdbkit • http://couchdbkit.org/ • Python client library # install with pip pip install couchdbkit # or from source git clone git://github.com/benoitc/couchdbkit.git cd couchdbkit sudo python setup.py install # and then you should be able to import import couchdbkit
11. Contents • Intro • DB Initialization – Setting Up CouchDB – Installing couchdbkit – Creating a Database • Key-Value Store • Simple MapReduce Queries • Complex MapReduce Queries • The _changes Feed • Replication • Additional Features and the Couch Ecosystem
12. Creating a Database • What we have: a CouchDB server and its URL eg http://127.0.0.1:5984 • What we want: a database there eg http://127.0.0.1:5984/myname • http://wiki.apache.org/couchdb/HTTP_database_API
13. A note on Debugging • Apache-style log files • Locally – $ tail ­f /var/log/couchdb/couch.log • HTTP – http://127.0.0.1:5984/_log?bytes=5000 – http://wiki.apache.org/couchdb/HttpGetLog
14. Creating a Database # ldb-init.py from restkit import BasicAuth from couchdbkit import Database from couchdbkit.exceptions import ResourceNotFound auth_filter = BasicAuth('username', 'pwd') db = Database(dburl, filters=[auth_filter]) server = db.server try: server.delete_db(db.dbname) except ResourceNotFound: pass db = server.get_or_create_db(db.dbname)
15. Creating a Database [Thu, 06 Sep 2012 16:44:30 GMT] [info] [<0.1435.0>] 127.0.0.1 - - DELETE /myname/ 200 [Thu, 06 Sep 2012 16:44:30 GMT] [info] [<0.1435.0>] 127.0.0.1 - - HEAD /myname/ 404 [Thu, 06 Sep 2012 16:44:30 GMT] [info] [<0.1440.0>] 127.0.0.1 - - PUT /myname/ 201
16. Contents • Intro • DB Initialization • Key-Value Store – Modelling Documents – Storing and Retrieving Documents – Updating Documents • Simple MapReduce Queries • Complex MapReduce Queries • The _changes Feed • Replication • Additional Features and the Couch Ecosystem
17. Key-Value Store • Core of CouchDB • Keys (_id): any valid JSON string • Values (documents): any valid JSON objects • Stored in B+-Trees • http://guide.couchdb.org/draft/btree.html
18. Modelling a Thing • A thing that we want to lend – Name – Owner – Dynamic properties like • Description • Movie rating • etc
19. Modelling a Thing • In CouchDB documents are JSON objects • You can store any dict – Wrapped in couchdbkit's Document classes for convenience • Documents can be serialized to JSON … mydict = mydoc.to_json() • … and deserialized from JSON mydoc = DocClass.wrap(mydict)
20. Modelling a Thing # models.py from couchdbkit import Database, Document, StringProperty class Thing(Document): owner = StringProperty(required=True) name = StringProperty(required=True) db = Database(DB_URL) Thing.set_db(db)
21. Storing a Document • Document identified by _id – Auto-assigned by Database (bad) – Provided when storing the database (good) – Think about lost responses – couchdbkit does that for us • couchdbkit adds property doc_type with value „Thing“
22. Internal Storage • Database File /var/lib/couchdb/dbname.couch • B+-Tree of _id • Access: O(log n) • Append-only storage • Accessible in historic order (we'll come to that later)
23. Storing a Document # ldb-new-thing.py couchguide = Thing(owner='stefan', name='CouchDB The Definitive Guide') couchguide.publisher = "O'Reilly“ couchguide.to_json() # {'owner': u'stefan', 'doc_type': 'Thing', # 'name': u'CouchDB The Definitive Guide', # 'publisher': u"O'Reilly"} couchguide.save() print couchguide._id # 448aaecfe9bc1cde5d6564a4c93f79c2
24. Storing a Document [Thu, 06 Sep 2012 19:40:26 GMT] [info] [<0.962.0>] 127.0.0.1 - - GET /_uuids?count=1000 200 [Thu, 06 Sep 2012 19:40:26 GMT] [info] [<0.962.0>] 127.0.0.1 - - PUT /lendb/8f14ef7617b8492fdbd800f1101ebb35 201
25. Retrieving a Document • Retrieve Documents by its _id – Limited use – Does not allow queries by other properties # ldb­get­thing.py thing = Thing.get(thing_id)
26. Retrieving a Document [Thu, 06 Sep 2012 19:45:30 GMT] [info] [<0.962.0>] 127.0.0.1 - - GET /lendb/8f14ef7617b8492fdbd800f1101ebb35 200
27. Updating a Document • Optimistic Concurrency Control • Each Document has a revision • Each Operation includes revision • Operation fails if revision doesn't match
28. Updating a Document >>> thing1 = Thing.get(some_id) >>> thing1._rev '1­110e1e46bcde6ed3c2d9b1073f0b26' >>> thing1.something = True >>> thing1.save() >>> thing1._rev '2­3f800dffa62f4414b2f8c84f7cb1a1' Success >>> thing2 = Thing.get(some_id) >>> thing2._rev '1­110e1e46bcde6ed3c2d9b1073f0b26' >>> thing2._rev '1­110e1e46bcde6ed3c2d9b1073f0b26' >>> thing2.conflicting = 'test' >>> thing2.save() couchdbkit.exceptions.ResourceConfl ict: Document update conflict. Failed
29. Updating a Document [Thu, 13 Sep 2012 06:16:52 GMT] [info] [<0.7977.0>] 127.0.0.1 - - GET /lendb/d46d311d9a0f64b1f7322d20721f9f1d 200 [Thu, 13 Sep 2012 06:16:55 GMT] [info] [<0.7977.0>] 127.0.0.1 - - GET /lendb/d46d311d9a0f64b1f7322d20721f9f1d 200 [Thu, 13 Sep 2012 06:17:34 GMT] [info] [<0.7977.0>] 127.0.0.1 - - PUT /lendb/d46d311d9a0f64b1f7322d20721f9f1d 201 [Thu, 13 Sep 2012 06:17:48 GMT] [info] [<0.7977.0>] 127.0.0.1 - - PUT /lendb/d46d311d9a0f64b1f7322d20721f9f1d 409
30. Contents • Intro • DB Initialization • Key-Value Store • Simple MapReduce Queries – Create a View – Query the View • Complex MapReduce Queries • The _changes Feed • Replication • Additional Features and the Couch Ecosystem
31. Views • A specific „view“ on (parts of) the data in a database • Indexed incrementally • Query is just reading a range of a view sequentially • Generated using MapReduce
32. MapReduce Views • Map Function – Called for each document – Has to be side-effect free – Emits zero or more intermediate key-value pairs • Reduce Function (optional) – Aggregates intermediate pairs • View Results stored in B+-Tree – Incrementally pre-computed at query-time – Queries are just a O(log n)
33. List all Things • Implemented as MapReduce View • Contained in a Design Document – Create – Store – Query
34. Create a Design Document • Regular document, interpreted by the database • Views Mapped to Filesystem by directory structure _design/<ddoc name>/views/<view name>/{map,reduce}.js • Written in JavaScript or Erlang • Pluggable View Servers – http://wiki.apache.org/couchdb/View_server – http://packages.python.org/CouchDB/views.html – Lisp, PHP, Ruby, Python, Clojure, Perl, etc
35. Design Document # _design/things/views/by_owner_name/map.js function(doc) { if(doc.doc_type == “Thing“) { emit([doc.owner, doc.name], null); } }
36. Intermediate Results Key Value [„stefan“, „couchguide“] null [„stefan“, „Polish Dictionary“] null [„marek“, „robot“] null
37. Design Document # _design/things/views/by_owner_name/reduce.js _count
38. Reduced Results • Result depends on group level Key Value [„stefan“, „couchguide“] 1 [„stefan“, „Polish Dictionary“] 1 [„marek“, „robot“] 1 Key Value [„stefan“] 2 [„marek“] 1 Key Value null 3
39. Synchronize Design Docs • Upload the design document • _id: _design/<ddoc name> • couchdbkit syncs ddocs from filesystem • We'll need this a few more times – Put the following in its own script – or run $ ./ldb­sync­ddocs.py
40. Synchronize Design Docs # ldb­sync­ddocs.py from couchdbkit.loaders import FileSystemDocsLoader auth_filter = BasicAuth('username', 'pwd') db = Database(dburl, filters=[auth_filter]) loader = FileSystemDocsLoader('_design') loader.sync(db, verbose=True)
41. View things/by_name • Emitted key-value pairs • Sorted by key http://wiki.apache.org/couchdb/View_collation • Keys can be complex (lists, dicts) • Query http://127.0.0.1:5984/myname/_design/things/_view/by_name?reduce=false Key Value _id (implicit) Document (implicit) [“stefan“, “couchguide“] null { … } [“stefan“, “Polish Dictionary“] null { … }
42. Query a View # ldb­list­things.py things = Thing.view('things/by_owner_name', include_docs=True, reduce=False) for thing in things: print thing._id, thing.name, thing.owner
43. Query a View – Reduced # ldb­overview.py owners = Thing.view('things/by_owner_name', group_level=1) for owner_status in owners: owner = owner_status['key'][0] count = owner_status['value'] print owner, count
44. Break
45. From the Break • Filtering by Price – startkey = 5 – endkey = 10 • Structure: ddoc name / view name – Logical Grouping – Performance
46. Contents • Intro • DB Initialization • Key-Value Store • Simple MapReduce Queries • The _changes Feed – Accessing the _changes Feed – Lending Objects • Advanced MapReduce Queries • Replication • Additional Features and the Couch Ecosystem
47. Changes Sequence • With every document update, a change is recorded • local history, ordered by _seq value • Only the latest _seq is kept
48. Changes Feed • List of all documents, in the order they were last modified • Possibility to – React on changes – Process all documents without skipping any – Continue at some point with since parameter • CouchDB as a distributed, persistent MQ • http://guide.couchdb.org/draft/notifications.html • http://wiki.apache.org/couchdb/HTTP_database_API#Changes
49. Changes Feed # ldb­changes­log.py def callback(line): seq = line['seq'] doc = line['doc'] # get obj according to doc['doc_type'] print seq, obj consumer = Consumer(db) consumer.wait(callback, since=since, include_docs=True)
50. „Lending“ Objects • Thing that is lent • Who lent it (ie who is the owner of the thing) • To whom it is lent • When it was lent • When it was returned
51. Modelling a „Lend“ Object # models.py class Lending(Document): thing = StringProperty(required=True) owner = StringProperty(required=True) to_user = StringProperty(required=True) lent = DateTimeProperty(default=datetime.now) returned = DateTimeProperty() Lending.set_db(db)
52. Lending a Thing # ldb­lend­thing.py lending = Lending(thing=thing_id, owner=username, to_user=to_user) lending.save()
53. Returning a Thing # ldb­return­thing.py lending = Lending.get(lend_id) lending.returned = datetime.now() lending.save()
54. Contents • Intro • DB Initialization • Key-Value Store • Simple MapReduce Queries • The _changes Feed • Advanced MapReduce Queries – Imitating Joins with „Mixed“ Views • Replication • Additional Features and the Couch Ecosystem
55. Current Thing Status • View to get the current status of a thing • No Joins • We emit with keys, that group together
56. Complex View # _design/things/_view/history/map.js function(doc) { if(doc.doc_type == "Thing") { emit([doc.owner, doc._id, 1], doc.name); } if(doc.doc_type == "Lending") { if(doc.lent && !doc.returned) { emit([doc.owner, doc.thing, 2], doc.to_user); } } }
57. Intermediate View Results Key Value [„stefan“, 12345, 1] „couchguide“ [„stefan“, 12345, 2] [„someone“, „2012-09-12“] [„marek“, 34544, 1] „robot“
58. Reduce Intermediate Results # _design/things/_view/status/reduce.js /* use with group_level = 2 */ function(keys, values) { /* there is at least one „Lending“ row */ if(keys.length > 1) { return "lent"; } else { return "available"; } }
59. • Don't forget to synchronize your design docs! • Group Level: 2 • Reduce Function receives rows with same grouped value Intermediate – not reduced reduced Reduce Intermediate Results Key Value [„owner“, 12345] „lent“ [„owner“, 34544] „available“ Key Value [„stefan“, 12345, 1] „couchguide“ [„stefan“, 12345, 2] [„someone“, „2012-09-12“] [„marek“, 34544, 1] „robot“
60. Get Status # ldb­status.py things = Thing.view('things/status', group_level = 2) for result in things: owner = result['key'][0] thing_id = result['key'][1] status = result['value']) Print owner, thing_id, status
61. Contents • Intro • DB Initialization • Key-Value Store • Simple MapReduce Queries • The _changes Feed • Advanced MapReduce Queries • Replication – Setting up filters – Find Friends and Replicate from them – Eventual Consistency and Conflicts • Additional Features and the Couch Ecosystem
62. Replication • Replicate Things and their status from friends • Don't replicate things from friends of friends – we don't want to borrow anything from them
63. Replication • Pull replication – Pull documents from our friends, and store them locally • There's also Push replication, but we won't use it • Goes through the source's _changes feed • Compares with local documents, updates or creates conflicts
64. Set up a Filter • A Filter is a JavaScript function that takes – a document – a request object • and returns – true, if the document passes the filter – false otherwise • A filter is evaluated at the source
65. Replication Filter # _design/things/filters/from_friend.js /* doc is the document, req is the request that uses the filter */ function(doc, req) { /* Allow only if entry is owned by the friend */ return (doc.owner == req.query.friend); }
66. Replication • Sync design docs to your own database! • Find friends to borrow from – Post your nickname and Database URL to http://piratepad.net/pycouchpl – Pick at least two friends
67. Replication • _replicator database • Objects describe Replication tasks – Source – Target – Continuous – Filter – etc • http://wiki.apache.org/couchdb/Replication
68. Replication # ldb­replicate­friend.py auth_filter = BasicAuth(username, password) db = Database(db_url, filters=[auth_filter]) replicator_db = db.server['_replicator'] replication_doc = { "source": friend_db_url, "target": db_url, "continuous": True, "filter": "things/from_friend", "query_params": { "friend": friend_name } } replicator_db[username+“­“+friend_name]=replication_doc
69. Replication • Documents should be propagated into own database • Views should contain both own and friends' things
70. Dealing with Conflicts • Conflicts introduces by – Replication – „forcing“ a document update • _rev calculated based on – Previous _rev – document content • Conflict when two documents have – The same _id – Distinct _rev
71. Dealing with Conflicts • Select a Winner • Database can't do this for you • Automatic strategy selects a (temporary) winner – Deterministic: always the same winner on each node – leaves them in conflict state • View that contains all conflicts • Resolve conflict programmatically • http://guide.couchdb.org/draft/conflicts.html • http://wiki.apache.org/couchdb/Replication_and_conflicts
72. Contents • Intro • DB Initialization • Key-Value Store • Simple MapReduce Queries • The _changes Feed • Advanced MapReduce Queries • Replication • Additional Features and the Couch Ecosystem – Scaling and related Projects – Fulltext Search – Further Reading
73. Scaling Up / Out • BigCouch – Cluster of CouchDB nodes that appears as a single server – http://bigcouch.cloudant.com/ – will be merged into CouchDB soon • refuge – Fully decentralized data platform based on CouchDB – Includes fork of GeoCouch for spatial indexing – http://refuge.io/
74. Scaling Down • CouchDB-compatible Databases on a smaller scale • PouchDB – JavaScript library http://pouchdb.com/ • TouchDB • IOS: https://github.com/couchbaselabs/TouchDB-iOS • Android: https://github.com/couchbaselabs/TouchDB-Android
75. Fulltext and Relational Search • http://wiki.apache.org/couchdb/Full_text_search • CouchDB Lucene – http://www.slideshare.net/martin.rehfeld/couchdblucene – https://github.com/rnewson/couchdb-lucene • Elastic Search – http://www.elasticsearch.org/
76. Operations Considerations • Append Only Storage • Your backup tools: cp, rsync • Regular Compaction needed
77. Further Features • Update Handlers: JavaScript code that carries out update in the database server • External Processes: use CouchDB as a proxy to other processes (eg search engines) • Attachments: attach binary files to documents • Update Validation: JavaScript code to validate doc updates • CouchApps: Web-Apps served directly by CouchDB • Bulk APIs: Several Updates in one Request • List and Show Functions: Transforming responses before serving them
78. Summing Up • Apache CouchDB™ is a database that uses JSON for documents, JavaScript for MapReduce queries, and regular HTTP for an API • couchdbkit is a a Python library providing access to Apache CouchDB
79. Thanks! Time for Questions and Discussion Downloads https://slideshare.net/skoegl/couch-db-pythonpyconpl2012 https://github.com/stefankoegl/python-couchdb-examples Stefan Kögl stefan@skoegl.net @skoegl