Real time, high performance big data - Ubuntu Enterprise Summit

How Server Density built an infrastructure for handling high volume, real time data analysis, using high availability cloud and dedicated Ubuntu instances.

Presented by David Mytton at Ubuntu Enterprise Summit 2012

Transcript

1. Real time, high-performance big data
2. ~3333 write ops/s 0.07 - 0.05 ms response
3. David Mytton Woop Japan!
6. Similarities
7. Similarities • Document stores
8. Similarities • Document stores • Flexible schema
9. Similarities • Document stores • Flexible schema • Purpose
10. Similarities • Document stores • Flexible schema • NoSQL • Purpose
11. It’s a little different.
12. Differences
13. Differences • Implementation
14. Differences • Implementation • Durability
15. Differences • Implementation • Durability • Queries
16. MongoDB at Server Density
17. •27 nodes MongoDB at Server Density
18. • June 2009 - +3yrs •27 nodes MongoDB at Server Density
19. •MySQL -> MongoDB •27 nodes MongoDB at Server Density • June 2009 - +3yrs
20. •MySQL -> MongoDB •15TB data per month •27 nodes MongoDB at Server Density • June 2009 - +3yrs
21. Case Study
22. Case Study •Britain’s Got Talent •2 nodes •10k votes/s •RDS m1.large = 300/s
23. Case Study
24. Case Study •BBC •8 nodes per DC •Eventual consistency •DNS failover •8 nodes per DC
25. Case Study •BBC •Max 1k PUT/s/node •24 PUT/s •500 GET/s
26. Case Study •BBC •No direct access •Caching •k/v store
27. Why?
28. Why? • Replication
29. Why? • Replication • Official drivers
30. Why? • Replication • Official drivers • Easy deployment
31. Why? • Replication • Official drivers • Easy deployment • Fast out of the box
32. Fast out of the box? Photo: dannychoo.com
33. ~3333 write ops/s 0.07 - 0.05 ms response
34. Fast out of the box? Photo: dannychoo.com • Softlayer cloud (1 core, 8GB)
35. Fast out of the box? Photo: dannychoo.com • Local instance storage • Softlayer cloud (1 core, 8GB)
36. Fast out of the box? Photo: dannychoo.com • Local instance storage • Ubuntu 10.04 LTS • Softlayer cloud (1 core, 8GB)
37. •Dedicated servers Vertical scaling
38. •Dedicated servers Vertical scaling •Single tenant
39. •Dedicated servers Vertical scaling •Single tenant •SSDs
40. Picture is unrelated! Mmm, ice cream. Fast out of the box?
41. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network
42. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM
43. mongos> db.metrics_20120508_15_1m.stats() { "sharded" : true, "flags" : 1, "ns" : "metrics.metrics_20120508_15_1m", "count" : 2752934, "numExtents" : 46, "size" : 746837640, "storageSize" : 823717888, "totalIndexSize" : 517581680, "indexSizes" : { "_id_" : 130358144, "a_1_i_1" : 155711920, "a_1_i_1_m_1_t_1" : 231511616 }, "avgObjSize" : 271.2878841265355, "nindexes" : 3, "nchunks" : 61,
44. http://www.flickr.com/photos/comedynose/4388430444/ Where should it go? What? Should it be in memory? Indexes Always Data If you can
45. How you’ll know 1) Slow queries Thu Oct 14 17:01:11 [conn7410] update sd.apiLog query: { c: "android/setDeviceToken", a: 1466, u: "blah", ua: "Server Density Android" } 51926ms www.flickr.com/photos/tonivc/2283676770/
46. How you’ll know 2) Timeouts cursor timed out (20000 ms)
47. How you’ll know 3) Disk i/o spikes www.flickr.com/photos/daddo83/3406962115/
48. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM
49. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • Fast disks (optional)
52. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • Fast disks (optional)
53. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • Fast disks (optional) • Sharding (optional)
54. Scaling writes
55. • Global DB lock Scaling writes
56. • Concurrency Scaling writes • Global DB lock
57. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
58. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
59. http://bit.ly/mongolock
60. • Sharding • Concurrency Scaling writes • Global DB lock
61. Scaling writes
62. Failover •Replica sets
63. Failover •Replica sets •Master/slave
64. Failover •Replica sets •Min 3 nodes •Master/slave
65. Failover •Replica sets •Min 3 nodes •Master/slave •Automatic failover
66. Scaling reads
67. Scaling reads •Replica slaves
68. Scaling reads •Replica slaves •Consistency
69. •Replica slaves Scaling reads •Consistency •w flag / tags
70. Unsafe writes
71. Unsafe writes • Fast by default
72. Unsafe writes • Fast by default • J
73. Unsafe writes • Fast by default • J • fsync
74. Unsafe writes • Fast by default • J • w • fsync
75. Tags { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"})
76. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"}) (A or B) + (C or D) + E Tags
77. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "sortOfImportant"}) (A + C) or (D + E) ... Tags
78. Bottlenecks •EC2
79. Bottlenecks •EC2 •Local storage
80. Bottlenecks •EC2 •EBS: RAID10 4-8 volumes •Local storage
81. Bottlenecks •EC2 •EBS: RAID10 4-8 volumes •Local storage •i/o: rand but not sequential
82. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
83. http://bit.ly/ ec2mongodb
84. Bottlenecks •CPU •Index building
85. mongostat
86. Locks/Queues mongostat
87. Maintenance
88. Maintenance Scheduled updates
89. Maintenance Scheduled updates Critical updates
90. Maintenance Scheduled updates Critical updates Canaries
91. Maintenance Scheduled updates Critical updates Canaries Pinning
92. Monitoring tools Run yourself Ganglia
93. Monitoring tools Server Density
94. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM
95. Picture is unrelated! Mmm, ice cream. Fast out of the box? • Fast network • Working set in RAM • bit.ly/benchrun
96. David Mytton david@serverdensity.com @davidmytton Woop Japan! www.serverdensity.com