MongoDB at the Guardian

A quick overview of our experience with MongoDB at the Guardian.

Transcript

1. MongoDB at the Guardian @philwills Tuesday, 11 October 2011
2. Tuesday, 11 October 2011
3. Tuesday, 11 October 2011
4. Tuesday, 11 October 2011
5. Why use mongo? • Flexible documents fit our domain well • Query flexibility important Tuesday, 11 October 2011
6. Guardian MongoDB usage • Development for over a year • Three projects in production • Internal and cloud deployments • Most common choice for new projects • At present, used for a minority of pages Tuesday, 11 October 2011
7. Tuesday, 11 October 2011
8. • “Schema” design • Performance • Operations • Overall Impressions Tuesday, 11 October 2011
9. Schema Design • More thought required • Importance of Aggregates • Flexibility vs. Consistency • Traps Tuesday, 11 October 2011
10. More thought required • Much less prior art • Can’t just plump for 3NF Tuesday, 11 October 2011
11. Aggregates • Domain Driven Design • Atomicity of transactions • Distribution Boundaries http://www.flickr.com/photos/cdeimages/3584213567/ Tuesday, 11 October 2011
12. Flexibility vs. Consistency • Application has to match flexibility of documents • Design on the basis that flexibility has a cost • Lack of datastore enforced schema means loss of documentation Tuesday, 11 October 2011
13. Traps • So fast you don’t notice problems • DBRefs look a bit like a foreign key... • ...but just add complexity Tuesday, 11 October 2011
14. Performance Tuesday, 11 October 2011
15. Something old... • Indexes, indexes, indexes • “Select *” considered harmful • Structure of data storage is crucial Tuesday, 11 October 2011
16. ...something new • mongostat • db.currentOp() • db.system.profile.find() Tuesday, 11 October 2011
17. • It’s easy when all your data is in memory • Determining what’s really your working set not always trivial Tuesday, 11 October 2011
18. Reading from secondary • In general eventual consistency is fine • db.requestStart() • db.requestDone() Tuesday, 11 October 2011
19. Operations Tuesday, 11 October 2011
20. Replica sets work • Hardware upgrade with no downtime • Compaction with no downtime • Spite Thor! Tuesday, 11 October 2011
21. 32-bit OS = :o( • Rules out small EC2 boxes Tuesday, 11 October 2011
22. Zoom, zoom • New features coming in fast. Yay! • Lots of time can be wasted on scripts that rapidly become redundant • Roadmap not always as visible as it might be Tuesday, 11 October 2011
23. Security • Granularity of access is coarse • No connection encryption without tunnelling • Auditing could be easier Tuesday, 11 October 2011
24. One global lock • Scary... • ...but not yet seen a real problem Tuesday, 11 October 2011
25. Manual Updates • Not ideal, but always necessary • db.collection.update() is a banana skin • Representation in query results may not distinguish non-JS types Tuesday, 11 October 2011
26. • Still maturing • More thought required • Not a golden hammer Overall Impressions Tuesday, 11 October 2011
27. • Fast • Manageable • Suits our data • Still our first choice Overall Impressions Tuesday, 11 October 2011
28. Questions? Tuesday, 11 October 2011