MongoDB at Groupon

How the Merchant Data team at Groupon uses Mongo to manage its place data. Presented by Peter Bakkum.

Transcript

1. MongoDB at Groupon Peter Bakkum @pbbakkum
2. Merchant   Data   CRM   Merchant   Pages  
4. Merchant   Data   CRM   Merchant   Pages   Self-­‐Service   Others  
5. Arnold: Declarative Crowd-Machine Data Integration Shawn Jeffery, Liwen Sun, Matt DeLand, Nick Pendar, Rick Barber, Andrew Galdi CIDR 2013 cidrdb.org/cidr2013/Papers/CIDR13_Paper22.pdf
6. Concordance { name: “Joe’s Pizza”, location: { address: “1000 Market St.”, postal_code: “94100-1001” }, source: 1 } { name: “Joes Pizza”, location: { address: “1000 Market Street”, postal_code: “94100” }, source: 2 } { name: “Joes”, location: { address: “1000 Market”, postal_code: “94100” }, source: 3 }
7. Data Systems
8. Content  Input   Data   Processing   Serving  
9. Content  Input       Data  Sets   Input   Feeds   Normaliza@on   Crowd   Sourcing   Web   Crawling  
10. Content  Input   Crawl Store Web Crawler Configuration Normalized Data Recent Feed History
11. Data   Processing   Storm  Topology   Storm
12. Serving   HTTP  Access  Layer   Varnish
13. Data Processing
14. storm topology bolts parsing normalization concordance geocoding persistence
15. place model record tree
16. places.find({ _id: “013e4e2afc26” }) placeCollection.find({ location.postcode: “94100”, location.country: “US” }) places.findAndModify( { _id: “013e4e2afc26” persisted_at: “2013-02-01T0:00:00Z” }, { place model }) Concordance Persistence
17. config cluster 4 arbiters 4 shards of 2 nodes replica set failover 64 GB dedicated hardware storm workers mongos routers
18. ID Scheme
19. UUID v1 82d991c6-b098-11e2-8fc0-c82a14fffe86 82d9996e-b098-11e2-8fc0-c82a14fffe86 82d99f04-b098-11e2-8fc0-c82a14fffe86 82d9a40e-b098-11e2-8fc0-c82a14fffe86
21. UUID vB wwwwwwww-xxxx-byyy-yyyy-zzzzzzzzzzzz w: controllable counter x: process id b: literal 'b’ y: fragment of MAC address z: milliseconds since epoch (UTC)
22. toggle c8c9cef9-7a7f-bd53-7a50-013e4e2afbde 14951cfa-7a7f-bd53-7a50-013e4e2afbde 6f5169fb-7a7f-bd53-7a50-013e4e2afbde ba2da6fc-7a7f-bd53-7a50-013e4e2afbde f5166777-7a7f-bd53-7a50-013e4e2afc26 f5166778-7a7f-bd53-7a50-013e4e2afc26 f5166779-7a7f-bd53-7a50-013e4e2afc26 f516677a-7a7f-bd53-7a50-013e4e2afc26
24. github.com/groupon/locality-uuid.java github.com/groupon/locality-uuid.rb
25. Backup and MapReduce
26. Hadoop Cluster
27. places.ns places.0 places.1 places.2 … char[128] name DiskLoc firstExtent DiskLoc lastExtent places.ns places.0 places.1 places.2 …
28. places.ns places.0 places.1 places.2 … places.0 places.1 extent   extent   extent  
29. places.ns places.0 places.1 places.2 … places.0 places.1 extent   extent   extent   MapReduce   Input  Split   MapReduce   Input  Split   MapReduce   Input  Split  
30. public void map( Text key, WritableBSONObject value, Context context) { String id = (String) value.get(“_id”); ... }
31. Mongo Cluster Hadoop Cluster MapReduce Job Backs up Mongo data to Hadoop Much faster data export Exploits our Hadoop cluster
32. Peter Bakkum @pbbakkum pbakkum@groupon.com