MongoNYC 2012: Document Schema Design and Data Consistency

MongoNYC 2012: Document Schema Design and Data Consistency, Kiril Savino, GameChanger. GameChanger is a giant data pipeline where high throughput on dynamic content is the top priority. We route live sports data in and back out to content generation systems, phones, dynamic web viewers, mobile apps, and email, peaking out at 10s of 1000s of constantly changing data streams. Our play-by-play collection will soon pass 1 billion 1k-sized, sport-variable documents- it's not something you can just query. We've had to re-learn how to lay out data in the document world: you must validate and maintain data differently, change access patterns, and cater to the strengths of the platform.

Transcript

1. Document Schema Design & Data Consistency A.K.A. awesome-scale data Kiril Savino CTO, Founder GameChanger Media
2. GameChanger
3. Abundance & Urgency • 15,000 games, or 5,000,000 events per DAY • 1,100,000 games scored to date • 320 million events and counting • Impatient consumers
6. Solving Problems • Many small objects • Rapid interleaved writing • Immediate availability • Complex representation
7. The Single Query Rule
8. The Single Document Rule
9. Multiple Access Patterns • Write patterns • Edit patterns • Read patterns • Multiple representations
11. Consistent, Eventually • Canonical location • Write contract • Completeness guarantee
12. Canonical Location
13. Write Contract
14. Completeness Rule
15. The Machine
16. Thanks! gc.io/about @kirilnyc