Upgrade to Pro — share decks privately, control downloads, hide ads and more …

High Performance Real Time Analytics - MongoSV 2012

High Performance Real Time Analytics - MongoSV 2012

Real time analytics is a compelling use case for MongoDB, but only if everything is up and running smoothly. We'll talk about how to set up and configure MongoDB to maintain high performance and redundancy. This will cover what to consider for high write throughput performance from hardware configuration through to the use of replica sets, multi-data centre deployments, monitoring and sharding to ensure your database is fast and stays online.

Presented by David Mytton at MongoSV 2012

Abc013bd0be76d8d25656c3ab7bd1fa2?s=128

Server Density

December 04, 2012
Tweet

Transcript

  1. High Performance Real Time Analytics

  2. ~3333 write ops/s 0.07 - 0.05 ms response

  3. David Mytton Woop Japan!

  4. None
  5. None
  6. MongoDB at Server Density

  7. •27 nodes MongoDB at Server Density

  8. • June 2009 - +3yrs •27 nodes MongoDB at Server

    Density
  9. •MySQL -> MongoDB •27 nodes MongoDB at Server Density •

    June 2009 - +3yrs
  10. •MySQL -> MongoDB •17TB data per month •27 nodes MongoDB

    at Server Density • June 2009 - +3yrs
  11. Why?

  12. Why? • Replication

  13. Why? • Replication • Official drivers

  14. Why? • Replication • Official drivers • Easy deployment

  15. Why? • Replication • Official drivers • Easy deployment •

    Fast out of the box1
  16. Fast out of the box? Photo: dannychoo.com

  17. ~3333 write ops/s 0.07 - 0.05 ms response

  18. Fast out of the box? Photo: dannychoo.com • Softlayer cloud

    (1 core, 8GB)
  19. Fast out of the box? Photo: dannychoo.com • Local instance

    storage • Softlayer cloud (1 core, 8GB)
  20. Fast out of the box? Photo: dannychoo.com • Local instance

    storage • Ubuntu 10.04 LTS • Softlayer cloud (1 core, 8GB)
  21. None
  22. Fast out of the box? Photo: dannychoo.com • Journaling

  23. Fast out of the box? Photo: dannychoo.com • Journaling •

    Replication
  24. Fast out of the box? Photo: dannychoo.com • Journaling •

    Write concern • Replication
  25. Picture is unrelated! Mmm, ice cream. Fast out of the

    box?
  26. Picture is unrelated! Mmm, ice cream. Fast out of the

    box? • Fast network
  27. Picture is unrelated! Mmm, ice cream. Fast out of the

    box? • Fast network • Working set in RAM
  28. mongos> db.metrics_20120508_15_1m.stats() { "sharded" : true, "flags" : 1, "ns"

    : "metrics.metrics_20120508_15_1m", "count" : 2752934, "numExtents" : 46, "size" : 746837640, "storageSize" : 823717888, "totalIndexSize" : 517581680, "indexSizes" : { "_id_" : 130358144, "a_1_i_1" : 155711920, "a_1_i_1_m_1_t_1" : 231511616 }, "avgObjSize" : 271.2878841265355, "nindexes" : 3, "nchunks" : 61,
  29. "size" : 746837640, "totalIndexSize" : 517581680 Indexes = 493MB Data

    = 712MB
  30. "size" : 746837640, "totalIndexSize" : 517581680 Indexes = 493MB Data

    = 712MB
  31. "size" : 746837640, "totalIndexSize" : 517581680 Indexes = 493MB Data

    = 712MB Total = 1205MB
  32. http://www.flickr.com/photos/comedynose/4388430444/ Where should it go? What? Should it be in

    memory? Indexes Always Data If you can
  33. How you’ll know 1) Slow queries Thu Oct 14 17:01:11

    [conn7410] update sd.apiLog query: { c: "android/setDeviceToken", a: 1466, u: "blah", ua: "Server Density Android" } 51926ms www.flickr.com/photos/tonivc/2283676770/
  34. How you’ll know 2) Timeouts cursor timed out (20000 ms)

  35. How you’ll know 3) Disk i/o spikes www.flickr.com/photos/daddo83/3406962115/

  36. Picture is unrelated! Mmm, ice cream. Fast out of the

    box? • Fast network • Working set in RAM
  37. Picture is unrelated! Mmm, ice cream. Fast out of the

    box? • Fast network • Working set in RAM • Fast disks (optional)
  38. None
  39. None
  40. Updates in place

  41. Updates in place db.my_collection.update( { _id : ... }, {

    $inc : { y : 2 } } )
  42. Updates in place > db.coll.stats() { "ns" : "...", ...,

    "paddingFactor" : 1, ..., "ok" : 1 }
  43. Picture is unrelated! Mmm, ice cream. Fast out of the

    box? • Fast network • Working set in RAM • Fast disks (optional)
  44. Picture is unrelated! Mmm, ice cream. Fast out of the

    box? • Fast network • Working set in RAM • Fast disks (optional) • Sharding (optional)
  45. Scaling writes

  46. • Global DB lock Scaling writes

  47. • Concurrency Scaling writes • Global DB lock

  48. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

  49. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

  50. http://bit.ly/mongolock

  51. • Sharding • Concurrency Scaling writes • Global DB lock

  52. Scaling writes

  53. None
  54. • Collection location Scaling writes

  55. • Collection location Scaling writes • Pre-split / moveChunk

  56. • Collection location Scaling writes • Hashing (v2.4) • Pre-split

    / moveChunk
  57. Failover •Replica sets

  58. Failover •Replica sets •Master/slave

  59. Failover •Replica sets •Min 3 nodes •Master/slave

  60. Failover •Replica sets •Min 3 nodes •Master/slave •Automatic failover

  61. rs.status() www.ex-astris-scientia.org/inconsistencies/ent_vs_tng.htm (yes it’s a replicator from Star Trek) {

    ! "_id" : 1, ! "name" : "rs3b:27018", ! "health" : 1, ! "state" : 2, ! "stateStr" : "SECONDARY", ! "uptime" : 1886098, ! "optime" : { ! ! "t" : 1291252178000, ! ! "i" : 13 ! }, ! "optimeDate" : ISODate("2010-12-02T01:09:38Z"), "lastHeartbeat" : ISODate("2010-12-02T01:09:38Z") },
  62. 1) myState rs.status() Value Meaning 0 Starting up (phase 1)

    1 Primary 2 Secondary 3 Recovering 4 Fatal error 5 Starting up (phase 2) 6 Unknown state 7 Arbiter 8 Down
  63. 2) Optime rs.status() www.flickr.com/photos/robbie73/4244846566/ "optimeDate" : ISODate("2010-12-02T01:09:38Z")

  64. 3) Heartbeat rs.status() www.flickr.com/photos/drawblindfaith/3400981091/ "lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")

  65. Scaling reads

  66. Scaling reads •Replica slaves

  67. Scaling reads •Replica slaves •Consistency

  68. •Replica slaves Scaling reads •Consistency •w flag / tags

  69. WriteConcern Changed Nov 27 2012

  70. • Safe by default WriteConcern >>> from pymongo import MongoClient

    >>> connection = MongoClient()
  71. • Safe by default WriteConcern >>> from pymongo import MongoClient

    >>> connection = MongoClient(w=int/str) Value Meaning 0 Unsafe 1 Primary 2 Primary + x1 secondary 3 Primary + x2 secondaries
  72. Tags { _id : "someSet", members : [ {_id :

    0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"})
  73. { _id : "someSet", members : [ {_id : 0,

    host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"}) (A or B) + (C or D) + E Tags
  74. { _id : "someSet", members : [ {_id : 0,

    host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "sortOfImportant"}) (A + C) or (D + E) ... Tags
  75. • J WriteConcern • Safe by default

  76. • J • fsync WriteConcern • Safe by default

  77. Bottlenecks •EC2

  78. Bottlenecks •EC2 •Local storage

  79. Bottlenecks •EC2 •EBS: RAID10 4-8 volumes •Local storage

  80. Bottlenecks •EC2 •EBS: RAID10 4-8 volumes •Local storage •i/o: rand

    but not sequential
  81. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs

  82. http://bit.ly/ ec2mongodb

  83. Bottlenecks •CPU •Index building

  84. mongostat

  85. Locks/Queues mongostat

  86. Diagnostics mongostat

  87. Current operations www.flickr.com/photos/jeffhester/2784666811/ db.currentOp(); { ! ! ! "opid" :

    "shard1:299939199", ! ! ! "active" : true, ! ! ! "lockType" : "write", ! ! ! "waitingForLock" : false, ! ! ! "secs_running" : 15419, ! ! ! "op" : "remove", ! ! ! "ns" : "sd.metrics", ! ! ! "query" : { ! ! ! ! "accId" : 1391, ! ! ! ! "tA" : { ! ! ! ! ! "$lte" : ISODate("2010-11-24T19:53:00Z") ! ! ! ! } ! ! ! }, ! ! ! "client" : "10.121.12.228:44426", ! ! ! "desc" : "conn" ! ! },
  88. Monitoring tools Run yourself Ganglia

  89. Monitoring tools Server Density

  90. Picture is unrelated! Mmm, ice cream. Fast out of the

    box? • Fast network • Working set in RAM
  91. Picture is unrelated! Mmm, ice cream. Fast out of the

    box? • Fast network • Working set in RAM • bit.ly/benchrun
  92. David Mytton david@serverdensity.com @davidmytton Woop Japan! www.serverdensity.com