High Performance Real Time Analytics - MongoSV 2012

Real time analytics is a compelling use case for MongoDB, but only if everything is up and running smoothly. We'll talk about how to set up and configure MongoDB to maintain high performance and redundancy. This will cover what to consider for high write throughput performance from hardware configuration through to the use of replica sets, multi-data centre deployments, monitoring and sharding to ensure your database is fast and stays online.

Presented by David Mytton at MongoSV 2012

Transcript

1. High Performance Real Time Analytics
2. ~3333 write ops/s 0.07 - 0.05 ms response
3. David Mytton Woop Japan!
6. MongoDB at Server Density
7. •27 nodes MongoDB at Server Density
8. • June 2009 - +3yrs •27 nodes MongoDB at Server Density
9. •MySQL -> MongoDB •27 nodes MongoDB at Server Density • June 2009 - +3yrs
10. •MySQL -> MongoDB •17TB data per month •27 nodes MongoDB at Server Density • June 2009 - +3yrs
11. Why?
12. Why? • Replication
13. Why? • Replication • Official drivers
14. Why? • Replication • Official drivers • Easy deployment
15. Why? • Replication • Official drivers • Easy deployment • Fast out of the box1
16. Fast out of the box? Photo: dannychoo.com
17. ~3333 write ops/s 0.07 - 0.05 ms response
18. Fast out of the box? Photo: dannychoo.com • Softlayer cloud (1 core, 8GB)
19. Fast out of the box? Photo: dannychoo.com • Local instance storage • Softlayer cloud (1 core, 8GB)
20. Fast out of the box? Photo: dannychoo.com • Local instance storage • Ubuntu 10.04 LTS • Softlayer cloud (1 core, 8GB)
22. Fast out of the box? Photo: dannychoo.com • Journaling
23. Fast out of the box? Photo: dannychoo.com • Journaling • Replication
24. Fast out of the box? Photo: dannychoo.com • Journaling • Write concern • Replication
25. Picture is unrelated! Mmm, ice cream. Fast out of the box?
26. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network
27. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM
28. mongos> db.metrics_20120508_15_1m.stats() { "sharded" : true, "flags" : 1, "ns" : "metrics.metrics_20120508_15_1m", "count" : 2752934, "numExtents" : 46, "size" : 746837640, "storageSize" : 823717888, "totalIndexSize" : 517581680, "indexSizes" : { "_id_" : 130358144, "a_1_i_1" : 155711920, "a_1_i_1_m_1_t_1" : 231511616 }, "avgObjSize" : 271.2878841265355, "nindexes" : 3, "nchunks" : 61,
29. "size" : 746837640, "totalIndexSize" : 517581680 Indexes = 493MB Data = 712MB
30. "size" : 746837640, "totalIndexSize" : 517581680 Indexes = 493MB Data = 712MB
31. "size" : 746837640, "totalIndexSize" : 517581680 Indexes = 493MB Data = 712MB Total = 1205MB
32. http://www.flickr.com/photos/comedynose/4388430444/ Where should it go? What? Should it be in memory? Indexes Always Data If you can
33. How you’ll know 1) Slow queries Thu Oct 14 17:01:11 [conn7410] update sd.apiLog query: { c: "android/setDeviceToken", a: 1466, u: "blah", ua: "Server Density Android" } 51926ms www.flickr.com/photos/tonivc/2283676770/
34. How you’ll know 2) Timeouts cursor timed out (20000 ms)
35. How you’ll know 3) Disk i/o spikes www.flickr.com/photos/daddo83/3406962115/
36. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM
37. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • Fast disks (optional)
40. Updates in place
41. Updates in place db.my_collection.update( { _id : ... }, { $inc : { y : 2 } } )
42. Updates in place > db.coll.stats() { "ns" : "...", ..., "paddingFactor" : 1, ..., "ok" : 1 }
43. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • Fast disks (optional)
44. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • Fast disks (optional) • Sharding (optional)
45. Scaling writes
46. • Global DB lock Scaling writes
47. • Concurrency Scaling writes • Global DB lock
48. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
49. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
50. http://bit.ly/mongolock
51. • Sharding • Concurrency Scaling writes • Global DB lock
52. Scaling writes
54. • Collection location Scaling writes
55. • Collection location Scaling writes • Pre-split / moveChunk
56. • Collection location Scaling writes • Hashing (v2.4) • Pre-split / moveChunk
57. Failover •Replica sets
58. Failover •Replica sets •Master/slave
59. Failover •Replica sets •Min 3 nodes •Master/slave
60. Failover •Replica sets •Min 3 nodes •Master/slave •Automatic failover
61. rs.status() www.ex-astris-scientia.org/inconsistencies/ent_vs_tng.htm (yes it’s a replicator from Star Trek) { ! "_id" : 1, ! "name" : "rs3b:27018", ! "health" : 1, ! "state" : 2, ! "stateStr" : "SECONDARY", ! "uptime" : 1886098, ! "optime" : { ! ! "t" : 1291252178000, ! ! "i" : 13 ! }, ! "optimeDate" : ISODate("2010-12-02T01:09:38Z"), "lastHeartbeat" : ISODate("2010-12-02T01:09:38Z") },
62. 1) myState rs.status() Value Meaning 0 Starting up (phase 1) 1 Primary 2 Secondary 3 Recovering 4 Fatal error 5 Starting up (phase 2) 6 Unknown state 7 Arbiter 8 Down
63. 2) Optime rs.status() www.flickr.com/photos/robbie73/4244846566/ "optimeDate" : ISODate("2010-12-02T01:09:38Z")
64. 3) Heartbeat rs.status() www.flickr.com/photos/drawblindfaith/3400981091/ "lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")
65. Scaling reads
66. Scaling reads •Replica slaves
67. Scaling reads •Replica slaves •Consistency
68. •Replica slaves Scaling reads •Consistency •w flag / tags
69. WriteConcern Changed Nov 27 2012
70. • Safe by default WriteConcern >>> from pymongo import MongoClient >>> connection = MongoClient()
71. • Safe by default WriteConcern >>> from pymongo import MongoClient >>> connection = MongoClient(w=int/str) Value Meaning 0 Unsafe 1 Primary 2 Primary + x1 secondary 3 Primary + x2 secondaries
72. Tags { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"})
73. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"}) (A or B) + (C or D) + E Tags
74. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "sortOfImportant"}) (A + C) or (D + E) ... Tags
75. • J WriteConcern • Safe by default
76. • J • fsync WriteConcern • Safe by default
77. Bottlenecks •EC2
78. Bottlenecks •EC2 •Local storage
79. Bottlenecks •EC2 •EBS: RAID10 4-8 volumes •Local storage
80. Bottlenecks •EC2 •EBS: RAID10 4-8 volumes •Local storage •i/o: rand but not sequential
81. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
82. http://bit.ly/ ec2mongodb
83. Bottlenecks •CPU •Index building
84. mongostat
85. Locks/Queues mongostat
86. Diagnostics mongostat
87. Current operations www.flickr.com/photos/jeffhester/2784666811/ db.currentOp(); { ! ! ! "opid" : "shard1:299939199", ! ! ! "active" : true, ! ! ! "lockType" : "write", ! ! ! "waitingForLock" : false, ! ! ! "secs_running" : 15419, ! ! ! "op" : "remove", ! ! ! "ns" : "sd.metrics", ! ! ! "query" : { ! ! ! ! "accId" : 1391, ! ! ! ! "tA" : { ! ! ! ! ! "$lte" : ISODate("2010-11-24T19:53:00Z") ! ! ! ! } ! ! ! }, ! ! ! "client" : "10.121.12.228:44426", ! ! ! "desc" : "conn" ! ! },
88. Monitoring tools Run yourself Ganglia
89. Monitoring tools Server Density
90. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM
91. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • bit.ly/benchrun
92. David Mytton david@serverdensity.com @davidmytton Woop Japan! www.serverdensity.com