MongoDB: In- and Outside

An intro to MongoDB with some discussion if internals. Also a bit about how we use MongoDB at "Fiesta.":https://fiesta.cc Given at DevChatt in June 2011.

Transcript

1. in-­‐  and  outside Mike Dirolf @mdirolf mike@corp.fiesta.cc
5. Horizontally Scalable Architectures
6. Horizontally Scalable Architectures no  joins no  complex  transactions +
7. New Data Models no  joins no  complex  transactions +
8. New Data Models improved  ways  to  develop  applications?
9. JSON-style Documents {“hello”:  “world”}
10. A Fiesta User {addresses:  ["mike@dirolf.com",                          "mike@corp.fiesta.cc"],  first:  "Mike",  last:  "Dirolf",  pw:  "...",  groups:  [      {address:  "mike@dirolf.com",        name:  "family",        id:  ObjectId("...")},      {address:  "mike@dirolf.com",        name:  "rovers",        id:  ObjectId("...")}],  owned:  [ObjectId("...")]}
11. Flexible “Schemas” {“author”:  “mike”,  “text”:  “...”} {“author”:  “eliot”,  “text”:  “...”,  “tags”:  [“mongodb”]}
12. To munge, or not to munge...
13. if user.get("munge", True): do_munge(...)
14. Query Language “query  by  example”  plus  $  modifiers: {first:  “Mike”,  age:  {$gte:  20,  $lt:  40}}
15. Dynamic Queries db.users.find({“groups.id”:  ObjectId(...)}) db.users.find({...}).sort({last:  -­‐1})
16. Indexing db.users.ensureIndex({“groups.id”:  1})
17. What’s Easy About MongoDB Indexing? It’s  almost  the  same  as  in  your   RDBMS
18. What’s Hard About MongoDB Indexing? It’s  almost  the  same  as  in  your   RDBMS
19. Indexes Maintain Order Index  on  {a:  1} {a:  0,  b:  9} {a:  2,  b:  0} {a:  7,  b:  1} {a:  3,  b:  2} {a:  3,  b:  5} {a:  3,  b:  7} {a:  9,  b:  1}
20. Indexes Maintain Order Index  on  {a:  1,  b:  -­‐1} {a:  0,  b:  9} {a:  2,  b:  0} {a:  7,  b:  1} {a:  3,  b:  7} {a:  3,  b:  2} {a:  3,  b:  5} {a:  9,  b:  1}
21. B-tree Structure Index  on  {a:  1} {...}  {...}  {...}  {...}  {...}  {...}  {...}  {...}  {...}  {...}  {...} [-∞, 5) [5, 10) [10, ∞) [5, 7) [7, 9) [9, 10) [10, ∞) buckets [-∞, 5) buckets
22. Query for {a: 7} {...}  {...}  {...}  {...}  {...}  {...}  {...}  {...}  {...}  {...}  {...} [-∞, 5) [5, 10) [10, ∞) [5, 7) [7, 9) [9, 10) [10, ∞) buckets [-∞, 5) buckets Index Scan
23. Picking an Index find({x:  10,  y:  “foo”})    scan    index  on  x    index  on  y remember terminate
24. Replica Sets primary secondary secondary offline secondary secondary offline secondary primary secondary secondary primary
25. Replication @ Fiesta primary passive secondary primary backups backups
26. Auto-sharding client mongos ... mongos mongod mongod mongod mongod mongod mongod ... Shards mongod mongod mongod Config Servers
27. db.test.insert({hello:  “world”})
28. _id if  not  specified  drivers  will  add  default: ObjectId("4bface1a2231316e04f3c434") timestamp machine  id process  id counter
29. BSON Encoding {_id:  ObjectId(XXXXXXXXXXXX),  hello:  “world”} \x27\x00\x00\x00\x07      _      i      d\x00      X          X      X      X      X      X      X      X      X      X      X      X\x02      h      e      l      l      o\x00\x06\x00 \x00\x00      w      o      r      l      d\x00\x00
30. Insert Message (TCP/IP) message  length request  id response  id op  code  (insert) \x68\x00\x00\x00 \xXX\xXX\xXX\xXX \x00\x00\x00\x00 \xd2\x07\x00\x00 reserved collection  name document(s) \x00\x00\x00\x00 f  o  o  .  t  e  s  t  \x00 BSON  Data
31. Data File Allocation $  ls  -­‐sk  /data/db/  16384  foo.ns  65536  foo.0 131072  foo.1  16384  bar.ns              ... }allocated  per   database { double  in  size   (up  to  2  gigs)
32. Memory Management
33. Extent Allocation foo.0 foo.1 foo.2 00000000000 00000000000 00000000000 00000000000 00000000000 00000000000 00000000000 preallocated  space 00000000000 0000 foo.$freelist foo.baz foo.bar foo.test allocated  per  namespace: ns  details  stored  in   foo.ns
34. Deleted  Record  (Size,  Offset,  Next) BSON  Data Header  (Size,  Offset,  Next,  Prev) Record Allocation Padding ... ...
35. Cursors >  var  c  =  db.test.find({x:  20}).skip(20).limit(10) >  c.next() >  c.next()      ... query first  N  results  +  cursor  id getMore  w/  cursor  id next  N  results  +  cursor  id  or  0 ...
36. Download MongoDB and  try
37. Capped Collections preallocated auto  LRI  age-­‐out no  default  _id  index always  in  insertion  order
38. Replication Oplog >  use  local switched  to  db  local >  db.oplog.$main.find() {ts:  ...,  op:  "n",  ns:  "",  o:  {}} {ts:  ...,  op:  "n",  ns:  "",  o:  {}} {ts:  ...,  op:  "i",  ns:  "foo.test",    o:  {_id:  ObjectId("..."),          x:  1,          url:  "http://dirolf.com"}} {ts:  ...,  op:  "n",  ns:  "",  o:  {}} {ts:  ...,  op:  "u",  ns:  "foo.test",  o2:  {_id:  ObjectId("...")},  o:  {$set:  {x:  2}}} >  use  foo switched  to  db  foo >  db.test.insert({x:  1,  url:   "http://dirolf.com"}); >  db.test.update({url:  "http:// dirolf.com"},  {$inc:  {x:  1}});
39. Geohashing (20,  10) (0001  0100,  0000  1010) 0000  0010  0110  0100 0000  0010  0110  0011 tricky  part  happens  at  bit-­‐flips  (127  vs  128) maps  close  coordinates  (21,  9)  to  close  hashes: