MongoDB NYC 2012: 80 Million Voters and Counting

Transcript

1. 130 Million Voters and Counting @

2. Daniel Weitzenfeld, Director of Analytics @DanielWDanielW @5ivepoints

3. What 5ivepoints does

4. •  On demand door-to-door canvassing (hence MongoDB) •  Voter lookup

   by name •  Call lists •  Social web integration

5. MongoDB stack

6. Use clever application/schema design to minimize necessary indexes.

7. The Voter-Retrieval Challenge •  Retrieve nearby voters matching a complex

   query. •  Query can be generated by user, or by app $$ Voter Restaurant $$

8. Constraint •  Minimize indexes •  with 130+ m records, each

   index is on order of 3~5 GB •  MongoDB needs indexes in memory for best results

9. Without constraint… $$ party background gender religion income issues +

   geospatial, name, ID numbers… districts

10. Our Two-Part Solution 1.  Client-side filtering with decision trees 2. 

    Store all those districts as polygon objects Result: –  Only 4 indexes on voter collection –  Indexes fit in 17 GB RAM

11. Part 1: Client side filtering with decision trees

12. What’s a decision tree? 100 95 75 65 50 25

13. Demonstration 100 95 75 65 50 25 50 65 95

    100 75 75 85 1. db.V.find({g:{$near:[x,y]}}).limit(50) 2. Run voters through appropriate decision tree 3. Return only high performers, with “Query-Matching” score Regardless of tree/filter complexity: Index: geospatial 1

14. Why Decision Trees? •  Easy to implement, using recursion • 

    Easy fuzzy matching •  Easy handling of missing values, categorical and continuous data •  Non-linearity •  Easily stored in Mongo, and translated to/ from PHP Array •  Wide usage in machine learning means ML output can plug right in

15. Why Decision Trees? •  Easy to run an object through

    a tree using recursion 100 95 75 65 50 25 subtree subtree

16. Why Decision Trees? •  Easy fuzzy matching 100 95 25

    $$$ $$ $$$

17. Why Decision Trees? •  Easy handling of missing values, categorical

    and continuous data 100 95 65 50 25 , missing >=45 <45

18. Why Decision Trees? •  Non-linearity Voter Income Score 100 0

19. Why Decision Trees? •  Easily stored in Mongo, and translated

    to/from PHP Array

20. Why Decision Trees? •  Wide usage in machine learning means

    ML output can plug right in – Single Tree – Multiple Trees: •  Random Forests •  Gradient Boosted Trees

21. Part 2: Store all those districts as polygon objects

22. Index on all districts? I’m running for Dog Catcher in

    ____________. What voters do you have in my district? Many levels of district: •  Town •  School District •  County •  State Legislature, lower house •  State Legislature, upper house •  Congressional

23. Or index on none? 1. db.DISTRICTS.findOne({‘name’:’Hicksville’}) Hicksville polygon from Census

    shapefiles 2. db.VOTERS.find({‘g’:{$within:<Hicksville polygon>}}) Index: geospatial 1

24. Bonus: Random Sampling Hicksville Bounding Box Much easier/faster/randomer than: db.VOTERS.find({‘Town’:’Hicksville’}).skip(random

    int)

25. Use clever application/schema design to minimize necessary indexes.