How BillGuard Does MongoDB

A description of the move BillGuard did to put its crowd sourcing brain on MongoDB given at the MongoIL meetup on March 7, 2012.

Transcript

1. Crowd Sourcing with MongoDB How BillGuard Does Mongo!
2. Us Aviv Ben-Yosef @avivby David Brailovsky @davidbrai
3. BillGuard
4. People Powered Anti Virus for Bills
5. A lot of data
6. Crowd source knowledge between users
7. How we started
8. Everything in MySQL
9. 1,000s of Transactions Per User
10. Crowd Sourcing for n00bs
12. Recalculate in memory when server boots Works as long as you have one server and little data
13. Time to upgrade Boot time became problematic as data grew Needed to sync across multiple servers
14. Y U NOSQL?!
15. Cluster data requires different sharding from user data
16. Our models are dynamic and sparse
17. Why Mongo?
18. Document Oriented = Easiest paradigm shift + Felt natural
19. Sounds cool!
20. Our CEO loooves saying “Mongo DB”
21. How we made the move
22. What we use • Java: Morphia • Python: pymongo
23. MySQL -> Mongo Foreign Key
24. We were cautious You have to gain our trust
25. Aggregated recomputable data Not mission critical
26. 2 hours later Mongo
27. 1 month later Time for the big guns We made it “Highly Available”
28. Replica Set 2 regular instances 1 arbiter
29. A breeze to configure config = { "_id" : "bg", "members" : [ { "_id" : 0, "host" : "$MONGO_SERVER_A:27017" }, { "_id" : 1, "host" : "$MONGO_SERVER_B:27017" }, { "_id" : 2, "host" : "$MONGO_ARBITER:27017", "arbiterOnly": true } ] } rs.initiate(config); Now configure a master-master in MySQL...
30. Automatic promotions upon failure = Better sleep at night!
31. More big guns • Backing up the EBSs it sits on • Mongo Monitoring System (MMS)
32. Mongo Monitoring System nohup  python  agent.py  >   /LOG_DIRECTORY/agent.log  2>&1  &
33. Why we <3 Mongo
34. No Migrations • No downtime • Never worry about backwards compatibility
35. What about transactions?
36. Upserts and atomic • No need to lock • No need to rollback • Just write atomically!
37. Atomic use case - clusters • 2 users register with transactions from same merchant • no problem!
38. >  db.merchants.update(        {name:  "yamit  2000"},        {$inc:  {users:  1}},        true) >  db.merchants.find() {  "name"  :  "yamit  2000",  "users"  :  1  } >  db.merchants.update(        {name:  "yamit  2000"},        {$inc:  {users:  1}},        true) >  db.merchants.find() {  "name"  :  "yamit  2000",  "users"  :  2  }
39. Easily Indexable class MyEntity { @Id private ObjectId id; @Indexed private String widget; // ... } class MyEntityDao { public MyEntityDao() { // ... datastore.ensureIndexes(); } }
40. Optimal for sparse structures Example: Enrichment of Merchants
41. ...and lotsa stuff • sweet queries: regex, awesome operators ($in, $inc, …) • map reduce (but no group by) • ids have timestamps and unique across servers
42. Pitfalls & Downsides
43. Write Concerns “Read what you write”
44. Data Analysts LOVE SQL
45. No good clients
46. User management is a pain
47. Questions?