MongoDB NYC 2012: 80 Million Voters and Counting

MongoDB NYC 2012: 80 Million Voters and Counting, Daniel Weitzenfeld, 5ivePoints. At 5ivePoints, we have a MongoDB database with 85+million US voters, every one of them geocoded. Our application allows supporters of political campaigns across the country to pull up nearby voters on their mobile phone to canvas. Moreover, campaigns can create custom filters using any combination of voter attributes, and we provide data-driven filters that update in near real-time based on campaign results. But with a collection of this size, each index is on the order of GBs, and un-indexed queries aren't feasible. I'll discuss how we are able to support flexible user queries - integrating both geo-spatial and other attributes - with only 4 indexes, minimizing page faults caused by pulling rarely-used indexes into memory.

Transcript

1. 130 Million Voters and Counting @
2. Daniel Weitzenfeld, Director of Analytics @DanielWDanielW @5ivepoints
3. What 5ivepoints does
4. •  On demand door-to-door canvassing (hence MongoDB) •  Voter lookup by name •  Call lists •  Social web integration
5. MongoDB stack
6. Use clever application/schema design to minimize necessary indexes.
7. The Voter-Retrieval Challenge •  Retrieve nearby voters matching a complex query. •  Query can be generated by user, or by app $$ Voter Restaurant $$
8. Constraint •  Minimize indexes •  with 130+ m records, each index is on order of 3~5 GB •  MongoDB needs indexes in memory for best results
9. Without constraint… $$ party background gender religion income issues + geospatial, name, ID numbers… districts
10. Our Two-Part Solution 1.  Client-side filtering with decision trees 2.  Store all those districts as polygon objects Result: –  Only 4 indexes on voter collection –  Indexes fit in 17 GB RAM
11. Part 1: Client side filtering with decision trees
12. What’s a decision tree? 100 95 75 65 50 25
13. Demonstration 100 95 75 65 50 25 50 65 95 100 75 75 85 1. db.V.find({g:{$near:[x,y]}}).limit(50) 2. Run voters through appropriate decision tree 3. Return only high performers, with “Query-Matching” score Regardless of tree/filter complexity: Index: geospatial 1
14. Why Decision Trees? •  Easy to implement, using recursion •  Easy fuzzy matching •  Easy handling of missing values, categorical and continuous data •  Non-linearity •  Easily stored in Mongo, and translated to/ from PHP Array •  Wide usage in machine learning means ML output can plug right in
15. Why Decision Trees? •  Easy to run an object through a tree using recursion 100 95 75 65 50 25 subtree subtree
16. Why Decision Trees? •  Easy fuzzy matching 100 95 25 $$$ $$ $$$
17. Why Decision Trees? •  Easy handling of missing values, categorical and continuous data 100 95 65 50 25 , missing >=45 <45
18. Why Decision Trees? •  Non-linearity Voter Income Score 100 0
19. Why Decision Trees? •  Easily stored in Mongo, and translated to/from PHP Array
20. Why Decision Trees? •  Wide usage in machine learning means ML output can plug right in – Single Tree – Multiple Trees: •  Random Forests •  Gradient Boosted Trees
21. Part 2: Store all those districts as polygon objects
22. Index on all districts? I’m running for Dog Catcher in ____________. What voters do you have in my district? Many levels of district: •  Town •  School District •  County •  State Legislature, lower house •  State Legislature, upper house •  Congressional
23. Or index on none? 1. db.DISTRICTS.findOne({‘name’:’Hicksville’}) Hicksville polygon from Census shapefiles 2. db.VOTERS.find({‘g’:{$within:<Hicksville polygon>}}) Index: geospatial 1
24. Bonus: Random Sampling Hicksville Bounding Box Much easier/faster/randomer than: db.VOTERS.find({‘Town’:’Hicksville’}).skip(random int)
25. Use clever application/schema design to minimize necessary indexes.