Embracing Constraints with CouchDB (IPC2010 2010-10-13)

Presentation given at International PHP Conference 2010 in Mainz, Germany.

Transcript

1. EMBRACING CONSTRAINTS WITH COUCHDB
2. David Zülke
3. David Zuelke
4. http://en.wikipedia.org/wiki/File:München_Panorama.JPG
5. Founder
7. Lead Developer
9. @dzuelke
10. A DISCLAIMER FIRST Before You All Figure This Out Yourselves...
11. 
13. NEIN NEIN NEIN NEIN DAS IST BETRUG
14. This talk is not really about embracing constraints
15. I’ll tell you what it’s really about when we’re finished
16. I’ll also apologize to you for lying at that point, and not now
17. (it’s always easier to apologize than to ask for permission)
18. COUCHDB IN THREE SLIDES Full Of DIS IS SRS BSNS Bullet Points
19. COUCHDB STORES DOCUMENTS • CouchDB stores documents with arbitrary keys and values • Each document is identified by an ID and has a revision • Documents can have file attachments • Stored as JSON, so it’s easy to interface with
20. COUCHDB SPEAKS HTTP • CouchDB uses HTTP to communicate with clients & servers • That means scalability • That means a lot of kick ass stuff totally for free • Caching • Load Balancing • Content Negotiation
21. COUCHDB USES MVCC • Multiversion Concurrency Control • When updating, you must supply a revision number • Your change will be rejected if the revision is not the latest • All writes are serialized • No need for locks, but puts some responsibility on developers
22. SOME DETAILS An In-Depth Look At What Makes CouchDB Different
23. CAP consistency availability partition tolerance X Do you know the theorem?
24. “So, CouchDB does not have consistency of CAP?”
25. “Booh, that means my data will be inconsistent. Fail!”
26. psssshhh
27. YOUR MOM IS INCONSISTENT
28. CouchDB is eventually consistent
29. When replicating, conflicting revisions will be marked as such
30. These conflicts can then be resolved (users, daemons,...)
31. and everything will be fine \o/
32. which brings us to...
33. REPLICATION • You can do Master-Master replication • Conflicts are detected and marked automatically • Conflicts are supposed to be resolved by applications • Or by users, who usually know best what to do!
34. CouchDB is Ground Computing
35. Imagine a world where every computer runs CouchDB
36. Ubuntu One already does, to sync bookmarks etc!
37. MAP/REDUCE
38. BASIC PRINCIPLE: MAPPER • The Mapper reads records and emits <key, value> pairs • Example: Apache access.log • Each line is a record • Extract client IP address and number of bytes transferred • Emit IP address as key, number of bytes as value • For hourly rotating logs, the job can be split across 24 nodes* * In pratice, it’s a lot smarter than that
39. BASIC PRINCIPLE: REDUCER • A Reducer is given a key and all values for this specific key • Even if there are many Mappers on many computers; the results are aggregated before they are handed to Reducers • Example: Apache access.log • The Reducer is called once for each client IP (that’s our key), with a list of values (transferred bytes) • We simply sum up the bytes to get the total traffic per IP!
40. EXAMPLE OF MAPPED INPUT IP Bytes 212.122.174.13 18271 212.122.174.13 191726 212.122.174.13 198 74.119.8.111 91272 74.119.8.111 8371 212.122.174.13 43
41. REDUCER WILL RECEIVE THIS IP Bytes 212.122.174.13 18271 212.122.174.13 191726 212.122.174.13 198 212.122.174.13 43 74.119.8.111 91272 74.119.8.111 8371
42. AFTER REDUCTION IP Bytes 212.122.174.13 210238 74.119.8.111 99643
43. COUCHDB INCREMENTAL MAPREDUCE
44. THE KEY DIFFERENCE • Maps and Reduces are incremental: • If one document changes, only that one document needs: • mapping • reduction • Then a few new reduce runs are performed to compute the final result
45. MAPPER: DOCS BY TAGS function(doc)  {    if(doc.type  ==  'product')  {        (doc.tags  ||  []).forEach(function(tag)  {            emit(tag,  doc);        });    } }
46. MAPREDUCE: COUNT TAGS function(doc)  {    if(doc.type  ==  'product')  {        (doc.tags  ||  []).forEach(function(tag)  {            emit(tag,  1);        });    } } function(key,  values)  {    return  sum(values); }
47. BUT WAIT! There are no tables :(
48. so... how do you join data from related documents?
49. JOIN CATEGORIES & ITEMS function(doc)  {    if(doc.type  ==  'product')  {        emit([doc._id,  0],  doc);        emit([doc._id,  1],  {  _id:  doc.category_id  });    } } ["123",  0]            {_id:  "123",  _rev:  "5-­‐a72",  type:  "product",  "name":  "Laser  Beam"} ["123",  1]            {_id:  "est",  _rev:  "2-­‐9af",  type:  "category",  "name":  "Evil  Stuff"} ["817",  0]            {_id:  "817",  _rev:  "2-­‐aa8",  type:  "product",  "name":  "Rocketship"} ["817",  1]            {_id:  "cst",  _rev:  "3-­‐d8a",  type:  "category",  "name":  "Cool  Stuff"} ["441",  0]            {_id:  "441",  _rev:  "19-­‐fdf",  type:  "product",  "name":  "Sharks"} ["441",  1]            {_id:  "est",  _rev:  "2-­‐9af",  type:  "category",  "name":  "Evil  Stuff"}
50. JOIN PRODUCTS & ITEMS function(doc)  {    if(doc.type  ==  'category')  {        emit([doc._id,  0],  doc);    }  elseif(doc.type  ==  'product')  {        emit([doc.category_id,  doc._id],  doc);    } } ["est",  0]            {_id:  "est",  _rev:  "2-­‐9af",  type:  "category",  "name":  "Evil  Stuff"} ["est",  "123"]    {_id:  "123",  _rev:  "5-­‐a72",  type:  "product",  "name":  "Laser  Beam"} ["est",  "441"]    {_id:  "441",  _rev:  "19-­‐fdf",  type:  "product",  "name":  "Sharks"} ["cst",  0]            {_id:  "cst",  _rev:  "3-­‐d8a",  type:  "category",  "name":  "Cool  Stuff"} ["cst",  "817"]    {_id:  "817",  _rev:  "2-­‐aa8",  type:  "product",  "name":  "Rocketship"}
51. BUT... BUT... WAIT! How do you validate document structures if it’s all schema-less?
52. VALIDATE DOCUMENTS function  (newDoc,  savedDoc,  userCtx)  {    if(savedDoc  &&  savedDoc.created_at  !=  newDoc.created_at)  {        throw({forbidden:  'created_at  is  immutable'});    }    if(doc.type  ==  'product')  {        if(!doc.price)  {            throw({forbidden:  'product  must  have  a  price'});        }    } }
53. VALIDATE DOCUMENTS function  (newDoc,  savedDoc,  userCtx)  {    function  require(beTrue,  message)  {        if(!beTrue)  throw({forbidden:  message});    }    require(savedDoc  &&  savedDoc.created_at  !=  newDoc.created_at,        'created_at  is  immutable'    );    if(doc.type  ==  'product')  {        require(!doc.price,            'product  must  have  a  price'        );    } }
54. LUCENE INTEGRATION Full Control Over What Is Indexed, And How
55. COUCHAPP Python Tool For Development And Deployment
56. DEMO TIME Let’s Relax On The Couch
57. !e End
58. FURTHER READING • http://books.couchdb.org/ • http://couchdb.apache.org/ • http://github.com/couchapp/couchapp • http://github.com/rnewson/couchdb-lucene/ • http://janl.github.com/couchdbx/ • http://j.mp/oqbQs (E4X in CouchDB for XML parsing)
59. DID YOU SEE THE HEAD FAKE? This Talk Was Not About Embracing Constraints It Was About Embracing Awesomeness
60. Questions?
61. THANK YOU! This was http://joind.in/2188 by @dzuelke