Community Engine, our experience with MongoDB - Mathieu Kempe, Community Engine

MongoDB Sydney 2012

This talk will cover lessons learned at Community Engine regarding MongoDB, including: why we moved away from an Hybrid solution using SQL and MongoDB; an outline of the technologies and what we learned using MongoDB on Amazon Web Services; the MongoDB C# driver; MongoDB with SOLR for Full Text Search; how we do migration, deployment and more.

Transcript

1. MongoDB   at
2. About  me Lead  pla/orm  engineer   mathieu.kempe@communityengine.com @mathieukempe
3. Agenda Brain  dump  on  our  experience  with  MongoDB • Why  NoSQL • Why  we  chose  MongoDB • Moving  away  from  an  hybrid  solu@on • MongoDB  and  Amazon • SOLR  and  MongoDB • Ease  of  development • Zero  down@me  database  deployment
4. About  Community  Engine • Social  network  based  on  locality  and   small  business • Launching  in  April • Built  on: •  ASP.NET  MVC  3 • Amazon  Web  services • MongoDB • SOLR • Mahout • …
5. NoSQL? • How  to  store  the  big  amounts  of  data  required   in  social  networking  applica@ons   • Data  complexity,  NoSQL  handle  hierarchical   and  graph  data  structures  beKer • Change  management  is  always  difficult  with   RDBMS • Scaling
6. Why  we  chose  MongoDB? • Reviewed  different  products  RavenDB,  CouchDB... • Selected  MongoDB  because  we  had  the  best   experience – Very  easy  to  install  and  get  started – Great  developer  experience – Replica@on  very  easy  to  setup – Good  documenta@on – Much  of  the  convenience  of  SQL,  Dynamic  Queries,   Indexing
8. Why Hybrid? • Team  had  a  lot  of  experience  with  SQL  Server   and  En@ty  Framework • Repor@ng • Transac@on
10. No more SQL Server • Simplify  our  infrastructure • Easier  to  Backup • BeKer  performance,  not  slowed  down  by  SQL   Server,  too  many  queries  joined  in  the   applica@on • Development  speed • Lower  cost
11. Transaction? Transac@ons  we  could  go  around  that  using  the  atomic   document  updates  and  a  good  schema  design   MongoDB  supports  atomic  opera@ons  on  single   document. When  transac@ons  across  documents  are  needed Two  phase  commits
12. Hosting with Amazon Web Service • Elas@city  and  scalability • Configure  MongoDB  using  Amazon  EC2  instance   bundled  into  an  AMI. • 64  bits  EC2  instance • Raid10  +  EBS  volumes • Mul@-­‐datacenter  3-­‐node  replica  set  in  different   availability  zone • Use  secondaries  for  zero  down@me  backup • We  are  not  yet  using  sharded  replica  sets
14. What about durability? Use  journaling Use  replica  sets
15. Critical writes Verify  that  replica:on  is  working  at  write  :me mongodb://host1,host2,host3/?safe=true;w=2; • safe=true  :  Use  safemode • w=2:  wmode,  connect  to  a  replica  set  wai@ng  for   replica@on  to  succeed  on  the  majority  of  nodes
21. Why we kept SOLR? • Right  tool  for  the  right  job • Proven  technology • SOLR  best  solu@on  for  Full  Text  Indexing • Faceted  search,  Spelling  sugges@ons…   • Team  already  skilled  with  SOLR • SOLR  scales  well
22. MongoDB/SOLR How we do it
23. MongoDB/SOLR How we do it
24. MongoDB/SOLR How we do it
25. MongoDB/SOLR How we do it
26. MongoDB/SOLR How we do it
27. Ease of development
28. Hierarchical data in SQL Server
29. Single table
30. Mongo  Database  Schema Using  Type  discriminator { "_id"  :  ObjectId("4f504e7acd3e1c190ce04198"), "_t"  :  "PhotoSpark", "Photo"  :  "MyMotorcycle.png  "   "DateCreated"  :  ISODate("2012-­‐02-­‐24T09:23:12.246Z") } { "_id"  :  ObjectId("4f504e7ccd3e1c190ce04199"), "_t"  :  "PostSpark", "Body"  :  "Hello  World“   "  DateCreated"  :  ISODate("2012-­‐02-­‐28T10:44:12.858Z") }
34. Views
35. Deployment  of  database  with  zero   downtime • We  release  every  week • We  aim  at  zero  downeme • Our  domain  model  change  ofen  
36. Deployment  of  database  with  zero   downtime Make  sure  that  our  code  can  handle  both   "versions"  of  the  data  structure When  saving  we  updates  to  the  new  structure
37. Deployment  of  database  with  zero   downtime • Use  a  migra@on  script
38. Expansion  Script Deploy  new  version Compression  Script t Deployment of database with zero downtime
39. Questions?
40. Thank you! mathieu.kempe@communityengine.com @mathieukempe