mongoDB at Intercom

These slides are from a talk I gave at the first meeting of the Dublin mongoDB users group.

Transcript

1. Wednesday 23 May 12
2. Ciaran Lee @ciaran_lee CTO of @intercom Wednesday 23 May 12
3. MongoDB at Intercom Wednesday 23 May 12
4. • What we do • Why we started using mongoDB • Learnings • Problems • Hosting Overview Wednesday 23 May 12
5. CRM & Messaging Tool for Web Apps Wednesday 23 May 12
6. <script> var intercomSettings = { app_id:"tx2p130c", email:"ciaran@intercom.io" }; </script> <script src="https://api.intercom.io/api/ js/library.js"</script> Wednesday 23 May 12
7. Wednesday 23 May 12
8. Wednesday 23 May 12
9. • Rails • MySQL • MongoDB • Redis • Memcached • Solr Technology Stack Wednesday 23 May 12
10. • Open beta • Installed on thousands of apps • Tracking millions of end users Current status Wednesday 23 May 12
11. MySQL • Initially used for all data • We had a built high traffic apps with it before (@exceptional) Wednesday 23 May 12
12. Wednesday 23 May 12
13. var intercomSettings = { app_id:"tx2p130c", email:"ciaran@intercom.io", custom_data: { plan: "Small", teammates: 4, revenue: 312, } }; Wednesday 23 May 12
14. http://en.wikipedia.org/wiki/Entity-attribute-value_model Entity–attribute–value pattern Wednesday 23 May 12
15. Wednesday 23 May 12
16. class User < ActiveRecord::Base serialize :custom_data end user.custom_data = { :plan => "Small", :team_mates => 4, :revenue => 312, } Wednesday 23 May 12
17. • Reading and writing worked fine • Sorting or filtering was a disaster Results Wednesday 23 May 12
18. Why MongoDB? • Handles arbitrary document structure • Flexible querying • Can use an ActiveRecord compatible ORM (mongomapper) • Available as a heroku addon Wednesday 23 May 12
19. Users -> mongoDB Wednesday 23 May 12
20. Wednesday 23 May 12
21. The Mongo way • Different way of thinking to relational DB • Often denormalised • Desired read scenarios influence design Wednesday 23 May 12
22. Tagging Wednesday 23 May 12
23. Tagging • Relational way is users + tags + join table • Mongo way is a tag_ids array in the user record Wednesday 23 May 12
24. db.users.update( { app_id: 6, 'custom_data.apps': { $gte: 5 } }, { $addToSet: { tag_ids:1234 } } ); Wednesday 23 May 12
25. db.users.update( { app_id: 6, 'custom_data.apps': { $gte: 5 } }, { $addToSet: { tag_ids:1234 } } ); Wednesday 23 May 12
26. db.users.update( { app_id: 6, 'custom_data.apps': { $gte: 5 } }, { $addToSet: { tag_ids:1234 } } ); Wednesday 23 May 12
27. Challenges Wednesday 23 May 12
28. Activity Stream Wednesday 23 May 12
29. Wednesday 23 May 12
30. Problem #1 • Querying users has gotten slower • Users collection is 1/4 the size of activities • Activity stream growing too fast • Knocks the users collection out of RAM Wednesday 23 May 12
31. Solution • Move the activity stream over to a separate database • We can use the more powerful database for complex user queries • sorting + filtering should be quicker Wednesday 23 May 12
32. Problem #2 • Can’t index every custom data attribute • Intend to experiment with app specific user collections (+ index all custom data attrs on that app’s collection) Wednesday 23 May 12
33. Slowness • Always seems to involve disk access • Slow performance of simple queries after big addToSet operations (> 200k users) Wednesday 23 May 12
34. Hosting • MongoHQ • Replica set - heroku addon • ec2 cluster • Dedicated hardware with SSDs • Amazing service Wednesday 23 May 12
35. Thanks for listening! Wednesday 23 May 12
36. Wednesday 23 May 12