MongoDB: In- and Outside

Transcript

1. in-­‐  and  outside Mike Dirolf @mdirolf mike@corp.fiesta.cc

2. None
3. None
4. None

5. Horizontally Scalable Architectures

6. Horizontally Scalable Architectures no  joins no  complex  transactions +

7. New Data Models no  joins no  complex  transactions +

8. New Data Models improved  ways  to  develop  applications?

9. JSON-style Documents {“hello”:  “world”}

10. A Fiesta User {addresses:  ["[email protected]",          

                   "[email protected]"],  first:  "Mike",  last:  "Dirolf",  pw:  "...",  groups:  [      {address:  "[email protected]",        name:  "family",        id:  ObjectId("...")},      {address:  "[email protected]",        name:  "rovers",        id:  ObjectId("...")}],  owned:  [ObjectId("...")]}

11. Flexible “Schemas” {“author”:  “mike”,  “text”:  “...”} {“author”:  “eliot”,  “text”:  “...”,

     “tags”:  [“mongodb”]}

12. To munge, or not to munge...

13. if user.get("munge", True): do_munge(...)

14. Query Language “query  by  example”  plus  $  modifiers: {first:  “Mike”,

     age:  {$gte:  20,  $lt:  40}}

15. Dynamic Queries db.users.find({“groups.id”:  ObjectId(...)}) db.users.find({...}).sort({last:  -­‐1})

16. Indexing db.users.ensureIndex({“groups.id”:  1})

17. What’s Easy About MongoDB Indexing? It’s  almost  the  same  as

     in  your   RDBMS

18. What’s Hard About MongoDB Indexing? It’s  almost  the  same  as

     in  your   RDBMS

19. Indexes Maintain Order Index  on  {a:  1} {a:  0,  b:

     9} {a:  2,  b:  0} {a:  7,  b:  1} {a:  3,  b:  2} {a:  3,  b:  5} {a:  3,  b:  7} {a:  9,  b:  1}

20. Indexes Maintain Order Index  on  {a:  1,  b:  -­‐1} {a:

     0,  b:  9} {a:  2,  b:  0} {a:  7,  b:  1} {a:  3,  b:  7} {a:  3,  b:  2} {a:  3,  b:  5} {a:  9,  b:  1}

21. B-tree Structure Index  on  {a:  1} {...}  {...}  {...}  {...}

     {...}  {...}  {...}  {...}  {...}  {...}  {...} [-∞, 5) [5, 10) [10, ∞) [5, 7) [7, 9) [9, 10) [10, ∞) buckets [-∞, 5) buckets

22. Query for {a: 7} {...}  {...}  {...}  {...}  {...}  {...}

     {...}  {...}  {...}  {...}  {...} [-∞, 5) [5, 10) [10, ∞) [5, 7) [7, 9) [9, 10) [10, ∞) buckets [-∞, 5) buckets Index Scan

23. Picking an Index find({x:  10,  y:  “foo”})    scan  

     index  on  x    index  on  y remember terminate

24. Replica Sets primary secondary secondary offline secondary secondary offline secondary

    primary secondary secondary primary

25. Replication @ Fiesta primary passive secondary primary backups backups

26. Auto-sharding client mongos ... mongos mongod mongod mongod mongod mongod

    mongod ... Shards mongod mongod mongod Config Servers

27. db.test.insert({hello:  “world”})

28. _id if  not  specified  drivers  will  add  default: ObjectId("4bface1a2231316e04f3c434") timestamp

    machine  id process  id counter

29. BSON Encoding {_id:  ObjectId(XXXXXXXXXXXX),  hello:  “world”} \x27\x00\x00\x00\x07      _

         i      d\x00      X          X      X      X      X      X      X      X      X      X      X      X\x02      h      e      l      l      o\x00\x06\x00 \x00\x00      w      o      r      l      d\x00\x00

30. Insert Message (TCP/IP) message  length request  id response  id op

     code  (insert) \x68\x00\x00\x00 \xXX\xXX\xXX\xXX \x00\x00\x00\x00 \xd2\x07\x00\x00 reserved collection  name document(s) \x00\x00\x00\x00 f  o  o  .  t  e  s  t  \x00 BSON  Data

31. Data File Allocation $  ls  -­‐sk  /data/db/  16384  foo.ns  65536

     foo.0 131072  foo.1  16384  bar.ns              ... }allocated  per   database { double  in  size   (up  to  2  gigs)

32. Memory Management

33. Extent Allocation foo.0 foo.1 foo.2 00000000000 00000000000 00000000000 00000000000 00000000000

    00000000000 00000000000 preallocated  space 00000000000 0000 foo.$freelist foo.baz foo.bar foo.test allocated  per  namespace: ns  details  stored  in   foo.ns

34. Deleted  Record  (Size,  Offset,  Next) BSON  Data Header  (Size,  Offset,

     Next,  Prev) Record Allocation Padding ... ...

35. Cursors >  var  c  =  db.test.find({x:  20}).skip(20).limit(10) >  c.next() >

     c.next()      ... query first  N  results  +  cursor  id getMore  w/  cursor  id next  N  results  +  cursor  id  or  0 ...

36. Download MongoDB and  try

37. Capped Collections preallocated auto  LRI  age-­‐out no  default  _id  index

    always  in  insertion  order

38. Replication Oplog >  use  local switched  to  db  local >

     db.oplog.$main.find() {ts:  ...,  op:  "n",  ns:  "",  o:  {}} {ts:  ...,  op:  "n",  ns:  "",  o:  {}} {ts:  ...,  op:  "i",  ns:  "foo.test",    o:  {_id:  ObjectId("..."),          x:  1,          url:  "http://dirolf.com"}} {ts:  ...,  op:  "n",  ns:  "",  o:  {}} {ts:  ...,  op:  "u",  ns:  "foo.test",  o2:  {_id:  ObjectId("...")},  o:  {$set:  {x:  2}}} >  use  foo switched  to  db  foo >  db.test.insert({x:  1,  url:   "http://dirolf.com"}); >  db.test.update({url:  "http:// dirolf.com"},  {$inc:  {x:  1}});

39. Geohashing (20,  10) (0001  0100,  0000  1010) 0000  0010  0110

     0100 0000  0010  0110  0011 tricky  part  happens  at  bit-­‐flips  (127  vs  128) maps  close  coordinates  (21,  9)  to  close  hashes: