MongoDB performance at scale - WebPerfDays 2012

A technical look at how we get fast performance out of the box processing over 15TB of data a month with MongoDB.

Presented at http://webperfdays.org/ 2012 by David Mytton

Transcript

1. MongoDB performance at scale
2. ~3333 write ops/s 0.07 - 0.05 ms response
3. David Mytton Woop Japan!
5. MongoDB at Server Density
6. •27 nodes MongoDB at Server Density
7. • June 2009 - +3yrs •27 nodes MongoDB at Server Density
8. •MySQL -> MongoDB •27 nodes MongoDB at Server Density • June 2009 - +3yrs
9. •MySQL -> MongoDB •15TB data per month •27 nodes MongoDB at Server Density • June 2009 - +3yrs
10. Why?
11. Why? • Replication
12. Why? • Replication • Official drivers
13. Why? • Replication • Official drivers • Easy deployment
14. Why? • Replication • Official drivers • Easy deployment • Fast out of the box
15. Fast out of the box? Photo: dannychoo.com
16. ~3333 write ops/s 0.07 - 0.05 ms response
18. Fast out of the box? Photo: dannychoo.com • Softlayer cloud (1 core, 8GB)
19. Fast out of the box? Photo: dannychoo.com • Local instance storage • Softlayer cloud (1 core, 8GB)
20. Fast out of the box? Photo: dannychoo.com • Local instance storage • Ubuntu 10.04 LTS • Softlayer cloud (1 core, 8GB)
22. Fast out of the box? Photo: dannychoo.com • Journaling
23. Fast out of the box? Photo: dannychoo.com • Journaling • Replication
24. Fast out of the box? Photo: dannychoo.com • Journaling • Unsafe writes • Replication
25. Picture is unrelated! Mmm, ice cream. Fast out of the box?
26. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network
27. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM
28. mongos> db.metrics_20120508_15_1m.stats() { "sharded" : true, "flags" : 1, "ns" : "metrics.metrics_20120508_15_1m", "count" : 2752934, "numExtents" : 46, "size" : 746837640, "storageSize" : 823717888, "totalIndexSize" : 517581680, "indexSizes" : { "_id_" : 130358144, "a_1_i_1" : 155711920, "a_1_i_1_m_1_t_1" : 231511616 }, "avgObjSize" : 271.2878841265355, "nindexes" : 3, "nchunks" : 61,
29. "size" : 746837640, "totalIndexSize" : 517581680 Indexes = 493MB Data = 712MB
30. "size" : 746837640, "totalIndexSize" : 517581680 Indexes = 493MB Data = 712MB
31. "size" : 746837640, "totalIndexSize" : 517581680 Indexes = 493MB Data = 712MB Total = 1205MB
32. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM
33. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • Fast disks (optional)
36. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • Fast disks (optional)
37. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • Fast disks (optional) • Sharding (optional)
38. Shards are the secret ingredient in the web scale sauce.
39. Scaling writes
40. • Global DB lock Scaling writes
41. • Concurrency Scaling writes • Global DB lock
42. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
43. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
44. http://bit.ly/mongolock
45. • Sharding • Concurrency Scaling writes • Global DB lock
46. Scaling writes
47. Failover •Replica sets
48. Failover •Replica sets •Master/slave
49. Failover •Replica sets •Min 3 nodes •Master/slave
50. Failover •Replica sets •Min 3 nodes •Master/slave •Automatic failover
51. Scaling reads
52. Scaling reads •Replica slaves
53. Scaling reads •Replica slaves •Consistency
54. •Replica slaves Scaling reads •Consistency •w flag / tags
55. Unsafe writes
56. Unsafe writes • Fast by default
57. Unsafe writes • Fast by default • J
58. Unsafe writes • Fast by default • J • fsync
59. Unsafe writes • Fast by default • J • w • fsync
60. Tags { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"})
61. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"}) (A or B) + (C or D) + E Tags
62. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "sortOfImportant"}) (A + C) or (D + E) ... Tags
63. Bottlenecks •EC2
64. Bottlenecks •EC2 •Local storage
65. Bottlenecks •EC2 •EBS: RAID10 4-8 volumes •Local storage
66. Bottlenecks •EC2 •EBS: RAID10 4-8 volumes •Local storage •i/o: rand but not sequential
67. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
68. http://bit.ly/ ec2mongodb
69. Bottlenecks •CPU •Index building
70. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM
71. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • bit.ly/benchrun
72. David Mytton david@serverdensity.com @davidmytton Woop Japan! www.serverdensity.com