High Volume Event Voting and Data Collection at Viacom with MongoDB

High Volume Event Voting and Data Collection at Viacom with MongoDB (or how to support a few million kids sliming their favorite celebrities all at once)

This was presented at MongoDB World 2015.

Note: There were two videos shown at points in the deck that are not included in the slides.

Transcript

1. MongoDB @ Viacom 1   (or how to support a few million kids sliming their favorite celebrities all at once.)
2. Who Are We? Michael Makunas Director - Multiplatform Engineering @mmakunas Ramesh Nuthalapati Architect - Multiplatform Engineering @nuthalapati 2  
3. Who Are We? Viacom Media Networks Viacom creates original entertainment content for every audience, on every screen around the world. The company's brands strive for creative excellence and cultural impact across programming, motion pictures and digital experiences that live online, on mobile, and via apps. 3  
4. Today We Are Talking About... Funnel Viacom’s internal high volume polling, voting, and data collection service. 4  
5. …and How MongoDB just works. 5   •  Performance and scalability doesn’t mean complexity •  Launched in 2011 on v 2.0 •  Currently on v 2.4
6. Use Cases Award Show Voting • MTV Video Music Awards (VMAs) • European Movie Awards (EMAs) • Kids Choice Awards (KCAs) • CMT Music Awards Polling • MTV March Musical Madness • Spike Bellator On Air Match Voting Contests • MTV Ultimate Fan Experience 6  
7. What is “High Volume” to Viacom? • From big to little, continuous “horizontal” support across 100+ global sites • Voting, polling, and data collection occurs 24/7 • No “maintenance windows” • Sudden spikes to 15K+ votes per second from on- air “Throws” during prime time viewing: “Go to the site and Vote Now!” • Storage of 100s of millions of votes for analysis after events 7  
8. A Very Important Question... 8  
9. Who will survive the Human Slime Car Wash!?!?!? 9  
10. Statistics (Pushing The Limit) 2015 Kids Choice Awards • 531 Million votes over 38 days (a new record!) • 7 Million votes during the show 10  
11. Some Definitions Funnel • Our internal name for our system Site • A brand and/or site specific namespace for collecting data Tent-pole • A major programming (entertainment) event 11  
12. Some Definitions (cont.) Collection • An individual poll, voting event, sweepstakes, etc. • NOT a MongoDB Collection Entry • A single vote or poll entry Processor • Performs an action on a entry (validates it, persists it, etc.) 12  
13. History • Why did we build Funnel? • Why did we choose Mongo? 13  
14. Why Did We Build It? • Previous systems unmanageable • One system for data collection • Contests, sweepstakes, small polls • Built in-house with ATG Dynamo and MySQL • Per-site deployment model meant multiple installations per brand • Flexible schema == MySql “xml blobs” • Separate system for high volume polling • Commercial • Lack of customization • Per-brand installation 14  
15. Why Did We Choose Mongo? • Flexible schema document style database provides the quick-to-market support needed by 100+ sites, brands, and countries (All who want things “just a little bit different”) • Proven scalability • JSON in and JSON out 15  
16. Architectural Goals • High write throughput over read throughput and consistency. • Handle up to 10,000 1KB writes per second. Read throughput is secondary. • High Availability and scalability. • Replace dozens of separate installations and services. • Wide adoption across internal brands. 16  
17. Server Layout 17   Secondary   Primary   Secondary   Replica Set 1 (day-to-day use) Replica Set 2 (tent-pole events) Secondary   Primary   Secondary  
18. Data Layout livefunnelmdb02:PRIMARY>  show  dbs   Dba  XXGB   kca_mtvi_com  XXGB   kca_nick_com  XXGB   millennial_mtvi_com  XXGB   movieawards_mtv_com  XXGB   woodies_mtv_com  XXGB   vma_mtv_com  XXGB     livefunnelmdb02:PRIMARY>  show  collections   funnel.metadata   funnel.tags   site.kca-­‐vote-­‐2015.entries   site.kca-­‐vote-­‐2015.hour.stats   site.kca-­‐vote-­‐2015.tallies  
19. MMS • Monitoring with MongoDB Management Service (MMS). • Email Alerts from MMS for high volume op inserts. • MMS Visualizations help us pick problem patterns quickly. 19  
20. More on MMS 20   KCA 2015 MMS Graph
21. Funnel UI
22. Schema For a Typical Collection "_id"  :  "kca-­‐live-­‐vote-­‐2015-­‐event",    "_class"  :  "com.mtvi.funnel.domain.model.collection.Collection",    "site"  :  "kca.nick.com",    "name"  :  "kca-­‐live-­‐vote-­‐2015-­‐event",    "description"  :  "KCA  Live  Event",    "owner"  :  "nuthalar",    "start"  :  ISODate("2015-­‐03-­‐02T05:00:00Z"),    "end"  :  ISODate("2015-­‐05-­‐10T04:00:00Z"),    "createdDate"  :  ISODate("2015-­‐03-­‐03T17:34:55.113Z"),    "lastUpdateDate"  :  ISODate("2015-­‐03-­‐26T14:40:15.665Z"),    "tags"  :  ["live-­‐show"],    "processorChain"  :  [...],    “questions”  :  [...]        -­‐-­‐  Questions  has  Options  and  Validators   22  
23. Processors • An entry received by Funnel will be handled by an ordered list of processors (the ProcessorChain) • As each processor in the list is visited, the processor will decide whether to continue to the next one or to terminate further processing • Processing handled serially; no parallelization 23  
24. Processors A Processor can raise an error, and its detail message will be configurable and localizable. When an error is raised, it can signal the system to take a few different actions depending on the severity of the error condition: •  Log the error but continue to processing •  Bubble up the error to the client and cease further processing •  Ignore the error, cease further processing, and return an OK response to the client 24  
25. Processors (UI) 25  
26. Questions: Email 26  
27. Questions: Email {            "name":"email",        "displayText":"email",        "persistable":true,        "source":"ENTITY_BODY",        "validators":[            {       "enabled":true,   "failureMessage":"Please  enter  a  valid  email  address  to  proceed.",   "contentType":"EmailAddressValidator",                     "_class":"com.mtvi.funnel.domain.model.collection.questions.validators.E mailAddressValidator"              },              {                        "enabled":true,                    "failureMessage":"This  field  is  required.  Please  enter  a  response  to  proceed.",                    "contentType":"RequiredValidator",                    "_class":"com.mtvi.funnel.domain.model.collection.questions.validators.RequiredValidator"              }        ],        "seqNum":NumberLong(0),        "tallyEnabled":false,        "isMandatory":false,        "contentType":"EmailQuestion"   }   27  
28. Questions: Single Choice 28  
29. Questions: Single Choice 29   {            "multipleAnswers":false,        "options":[                  {                        "optionValue":"andrew-­‐garfield-­‐male-­‐action-­‐star",                    "displayText":"andrew-­‐garfield-­‐male-­‐action-­‐star",                    "customOption":false              },      {       "optionValue":"channing-­‐tatum-­‐male-­‐action-­‐ star",   "displayText":"channing-­‐tatum-­‐male-­‐action-­‐ star",   "customOption":false   },   {       "optionValue":"chris-­‐evans-­‐male-­‐action-­‐ star",   "displayText":"chris-­‐evans-­‐male-­‐action-­‐ star",   "customOption":false    },              {                        "optionValue":"chris-­‐pratt-­‐male-­‐action-­‐star",                    "displayText":"chris-­‐pratt-­‐male-­‐action-­‐star",                    "customOption":false              },   ],        "writeInOptionEnabled":false,        "name":"favorite-­‐male-­‐action-­‐star",        "displayText":"favorite-­‐male-­‐action-­‐star",        "persistable":true,        "source":"ENTITY_BODY",          "validators":[                  {                        "enabled":false,                    "failureMessage":"This  field  is  required.  Please  enter  a   response  to  proceed.",                    "contentType":"RequiredValidator",                     "_class":"com.mtvi.funnel.domain.model.collection.questions.validators.R equiredValidator"              },              {                        "enabled":true,                    "failureMessage":"Please  select  a  valid  choice  to  proceed.",                    "contentType":"ChoiceValidator",                     "_class":"com.mtvi.funnel.domain.model.collection.questions.validators.C hoiceValidator"              },              {                        "enabled":true,                    "failureMessage":"Please  fill  in  the  'Other'  value  to   proceed.",                    "contentType":"WriteInRequiredValidator",                     "_class":"com.mtvi.funnel.domain.model.collection.questions.validators.W riteInRequiredValidator"              }        ],        "seqNum":NumberLong(0),        "tallyEnabled":true,        "isMandatory":false,        "sm4MappingEnabled":false,        "sm4Mapping":"",        "exodusMapping":"",        "contentType":"SingleChoiceQuestion",         "_class":"com.mtvi.funnel.domain.model.collection.questions.QuestionWith Options"   }    
30. An Entry would look like… 30   {    "_id"  :  ObjectId("53c698c1ebd268ff9b76034d"),    "answers"  :  {      "question"  :  "option-­‐3",      "source"  :  "mobile",      "True-­‐Client-­‐IP"  :  "166.77.6.8"    },    "createdDate"  :  ISODate("2014-­‐07-­‐16T15:22:41.593Z")   }  
31. Entry Trend Graph 31  
32. A Tally would look like… 32   {    "_id"  :  ObjectId("53c698c51180cc552e79c12d"),    "option-­‐1"  :  1249596,    "option-­‐2"  :  341404,    "option-­‐3"  :  466646,    "questionName"  :  "question",    "timeWindowType"  :  "ALL_TIME"   }  
33. Optimizations We didn’t get it perfect the first time: • Correct Indices - Index on created Date of Entry Collection. EnsureIndex is called during collection creation and update • Batch update of vote tallies - Update the count per second instead of per Entry • Hourly Stats – Pre-calculating hourly totals saved resources during live events to get stats for past 24 hours, 7 days and 30 days 33  
34. SPAM Prevention • De-duplicator processor: De-duplicates X number of entries for a defined period (eg. 30 Entries per minute). Internal to Funnel application • VERISIGN DDoS Protection service • Akamai WAF (part of Kona Web Application Firewall) • CAPTCHA • Authentication (FLUX - Viacom Internal System) 34  
35. Before & After Previous Systems • Decentralized • Multiple Installations • Outdated App Server • Rigid Schema requiring rigid change control process 35   Funnel • Service Oriented • Performant • Flexible Schema; easily extended
36. Our Track Record • 100+ Sites • 15K Collections • 10+ Billion Votes to date 36  
37. Where else MongoDB is Used @ Viacom • Used in production since v 1.6 • Content Management System • User Activy Tracking (Social Media Platform) • Viacom Identity Platform (Auth & User Profiles) 37  
38. Want to help? • We’re hiring! • Software Architect, Big Data • http://viacomcareers.com • https://lnkd.in/e2eauDn 38  
39. Wait… 39  
40. Who survived the Human Slime Carwash!?!?!? 40  
41. Questions ? 41  
42. Thank You! 42     • Michael  Makunas  -­‐-­‐  @mmakunas   • Ramesh  NuthalapaD  -­‐-­‐  @nuthalapaD