MongoNYC 2012: How MongoDB is Helping Shape the Future of the Music Industry

MongoNYC 2012: How MongoDB is Helping Shape the Future of the Music Industry, Buck Heroux, Next Big Sound. MongoDB is helping Next Big Sound change how the music industry uses big data. By marrying social media data with traditional sales data, Next Big Sound creates a central platform for all of the music industries needs with MongoDB working hard behind the scenes. This use case analysis will hit key points such as cluster architecture, document schema, some growing pains, and how Next Big Sound uses MongoDB to store rich event data as well as index hundreds of thousands of artists by their network distribution and geolocation.

Transcript

1. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com How MongoDB is Helping Shape the Future of the Music Industry ® & Thursday, May 24, 2012
2. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com 1991 Soundscan Thursday, May 24, 2012
3. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com twitter.com/keshasuxxx last.fm/ke$ha facebook.com/kesha Thursday, May 24, 2012
4. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com 2011 Thursday, May 24, 2012
5. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Traditional Industry Metrics + Social Media Data + Relevant Event Information = Thursday, May 24, 2012
6. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Who is going to be popular next? Thursday, May 24, 2012
7. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com soundcloud.com/skrillex Thursday, May 24, 2012
8. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Thursday, May 24, 2012
9. Thursday, May 24, 2012
10. Thursday, May 24, 2012
11. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Storage Exportation Presentation Importation Processing/Analytics Data Sources Customers Behind the Scenes Operations Thursday, May 24, 2012
12. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com MongoDB @ Next Big Sound Thursday, May 24, 2012
13. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Events Artist Indexing R&D > 30 million documents Rich Query Language Flexibility Java, Python, and PHP drivers Thursday, May 24, 2012
14. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com R&D and the value of the MVP Thursday, May 24, 2012
15. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Events Thursday, May 24, 2012
16. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Thursday, May 24, 2012
17. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Thursday, May 24, 2012
18. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Thursday, May 24, 2012
19. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Thursday, May 24, 2012
20. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com db.events.findOne({artist_id:356}) (dat schema) { "artist_id" : 356 "date" : 1307054669, "display" : "Kanye West shared http://on.fb.me/h6Yydg on facebook", "event" : { "summary" : "http://www.facebook.com/WatchTheThrone", "title" : "Kanye West shared http://on.fb.me/h6Yydg on facebook" }, "source" : "facebook_links" } Thursday, May 24, 2012
21. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com db.events.stats() { "sharded" : true, "flags" : 1, "count" : 31705294(31.7M), "numExtents" : 77, "size" : 24103642176(22.44 GB), "storageSize" : 42046062448, "totalIndexSize" : 10484861520(9.76 GB) } Thursday, May 24, 2012
22. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Maintainable Horizontal Scaling Flexible Schema Fast Querying Why MongoDB? Thursday, May 24, 2012
23. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Hindsight Thursday, May 24, 2012
24. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Index Smartly watch slow queries use mms Thursday, May 24, 2012
25. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Moving was a pain fsync + data files Thursday, May 24, 2012
26. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Artist Indexing Thursday, May 24, 2012
27. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Artists Profiles Networks Metrics Timeseries 300k 1m 45 150 3 years Thursday, May 24, 2012
28. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Metric Data Metadata Thursday, May 24, 2012
29. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com key => {artist,network,metric,date}.join(“_”) { } , , , Thursday, May 24, 2012
30. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Aggregate, then Filter & Breakdown Thursday, May 24, 2012
31. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com json.dumps(kanye_west) { "name" : "Kanye West" "signed" : true, "location" : { "city" : "Bedford Park", "iso2" : "US-IL", "geo" : { "lon": -87.6780, "lat": 41.8707 } }, "data" : { "twitter" : { "fans" : { "value": 7160559, "quantile": 999 }, ...}, ...}, } Thursday, May 24, 2012
32. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Find me the top 20 unsigned bands that have at least 10,000 Facebook Likes and more Twitter Followers than 80% of all other bands, but at most more Followers than 95% and located within 10 miles of New York City Thursday, May 24, 2012
33. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com db.artists.find({ ! "signed": false, ! "data.twitter.fans.quantile": { ! ! "$gte": 800, ! ! "$lte": 950 ! }, ! "data.facebook.fans.value": { ! ! "$gt": 10000 ! }, ! "location.geo": { ! ! "$within": {"$centerSphere": [ [-73.9, 40.6], 10/3959 ]} ! } }) .sort({ ! "data.facebook.fans.value": -1 }) .limit(20) Dat Query Thursday, May 24, 2012
34. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Results [ { "name": "Kayvon Zand"}, { "name": "Hoodie Allen"}, { "name": "Katrina Bradley"}, { "name": "Mike Wax"}, { "name": "Range"}, { "name": "DJ Mia Moretti"}, { "name": "Amanda Lepore"}, { "name": "DJ Spinna"}, { "name": "Anamanaguchi"}, { "name": "Wakey!Wakey!"}, { "name": "Holy Ghost!"}, ... ] Thursday, May 24, 2012
35. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Thursday, May 24, 2012
36. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com This is like, really cool Thursday, May 24, 2012
37. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com http://server.com/database_name/collection?query Thursday, May 24, 2012
38. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com http://appserver.com/find/artists? ! data.twitter.fans.quantile=[(gte:800),(lte:950)]& ! data.facebook.fans.value=[(gt:10000)]& ! geo:location.geo=[(40.6:-73.9),10]& ! sort=[(data.twitter.fans.value:desc)]& ! fields=[nbs_id,name]& ! signed=False& ! limit=30 Asynchronous Mongo REST API { 1 count: 20, 2 results: [ 1 { 1 nbs_id: 80025, 2 name: "Kayvon Zand" 2 }, 3 { 1 nbs_id: 77313, 2 name: "Hoodie Allen" 4 }, 5 { 1 nbs_id: 269573, 2 name: "Karina Bradley" 6 }, 7 { 1 nbs_id: 56276, 2 name: "Mikey Wax" Thursday, May 24, 2012
39. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Useful? Thursday, May 24, 2012
40. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com What’s Next? Thursday, May 24, 2012
41. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Thursday, May 24, 2012
42. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Thursday, May 24, 2012
43. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Change the way the music industry uses data Thursday, May 24, 2012
44. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com ʼS Thursday, May 24, 2012
45. © 2009 - 2012 Next Big Sound, Inc. buck@nextbigsound.com Buck Heroux buck@nextbigsound.com @buckheroux nextbigsound.com/jobs Thursday, May 24, 2012