Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MongoNYC 2012: How MongoDB is Helping Shape the Future of the Music Industry

mongodb
May 29, 2012
730

MongoNYC 2012: How MongoDB is Helping Shape the Future of the Music Industry

MongoNYC 2012: How MongoDB is Helping Shape the Future of the Music Industry, Buck Heroux, Next Big Sound. MongoDB is helping Next Big Sound change how the music industry uses big data. By marrying social media data with traditional sales data, Next Big Sound creates a central platform for all of the music industries needs with MongoDB working hard behind the scenes. This use case analysis will hit key points such as cluster architecture, document schema, some growing pains, and how Next Big Sound uses MongoDB to store rich event data as well as index hundreds of thousands of artists by their network distribution and geolocation.

mongodb

May 29, 2012
Tweet

Transcript

  1. © 2009 - 2012 Next Big Sound, Inc. [email protected] How

    MongoDB is Helping Shape the Future of the Music Industry ® & Thursday, May 24, 2012
  2. © 2009 - 2012 Next Big Sound, Inc. [email protected] twitter.com/keshasuxxx

    last.fm/ke$ha facebook.com/kesha Thursday, May 24, 2012
  3. © 2009 - 2012 Next Big Sound, Inc. [email protected] Traditional

    Industry Metrics + Social Media Data + Relevant Event Information = Thursday, May 24, 2012
  4. © 2009 - 2012 Next Big Sound, Inc. [email protected] Who

    is going to be popular next? Thursday, May 24, 2012
  5. © 2009 - 2012 Next Big Sound, Inc. [email protected] Storage

    Exportation Presentation Importation Processing/Analytics Data Sources Customers Behind the Scenes Operations Thursday, May 24, 2012
  6. © 2009 - 2012 Next Big Sound, Inc. [email protected] MongoDB

    @ Next Big Sound Thursday, May 24, 2012
  7. © 2009 - 2012 Next Big Sound, Inc. [email protected] Events

    Artist Indexing R&D > 30 million documents Rich Query Language Flexibility Java, Python, and PHP drivers Thursday, May 24, 2012
  8. © 2009 - 2012 Next Big Sound, Inc. [email protected] R&D

    and the value of the MVP Thursday, May 24, 2012
  9. © 2009 - 2012 Next Big Sound, Inc. [email protected] db.events.findOne({artist_id:356})

    (dat schema) { "artist_id" : 356 "date" : 1307054669, "display" : "Kanye West shared http://on.fb.me/h6Yydg on facebook", "event" : { "summary" : "http://www.facebook.com/WatchTheThrone", "title" : "Kanye West shared http://on.fb.me/h6Yydg on facebook" }, "source" : "facebook_links" } Thursday, May 24, 2012
  10. © 2009 - 2012 Next Big Sound, Inc. [email protected] db.events.stats()

    { "sharded" : true, "flags" : 1, "count" : 31705294(31.7M), "numExtents" : 77, "size" : 24103642176(22.44 GB), "storageSize" : 42046062448, "totalIndexSize" : 10484861520(9.76 GB) } Thursday, May 24, 2012
  11. © 2009 - 2012 Next Big Sound, Inc. [email protected] Maintainable

    Horizontal Scaling Flexible Schema Fast Querying Why MongoDB? Thursday, May 24, 2012
  12. © 2009 - 2012 Next Big Sound, Inc. [email protected] Index

    Smartly watch slow queries use mms Thursday, May 24, 2012
  13. © 2009 - 2012 Next Big Sound, Inc. [email protected] Moving

    was a pain fsync + data files Thursday, May 24, 2012
  14. © 2009 - 2012 Next Big Sound, Inc. [email protected] Artists

    Profiles Networks Metrics Timeseries 300k 1m 45 150 3 years Thursday, May 24, 2012
  15. © 2009 - 2012 Next Big Sound, Inc. [email protected] key

    => {artist,network,metric,date}.join(“_”) { } , , , Thursday, May 24, 2012
  16. © 2009 - 2012 Next Big Sound, Inc. [email protected] Aggregate,

    then Filter & Breakdown Thursday, May 24, 2012
  17. © 2009 - 2012 Next Big Sound, Inc. [email protected] json.dumps(kanye_west)

    { "name" : "Kanye West" "signed" : true, "location" : { "city" : "Bedford Park", "iso2" : "US-IL", "geo" : { "lon": -87.6780, "lat": 41.8707 } }, "data" : { "twitter" : { "fans" : { "value": 7160559, "quantile": 999 }, ...}, ...}, } Thursday, May 24, 2012
  18. © 2009 - 2012 Next Big Sound, Inc. [email protected] Find

    me the top 20 unsigned bands that have at least 10,000 Facebook Likes and more Twitter Followers than 80% of all other bands, but at most more Followers than 95% and located within 10 miles of New York City Thursday, May 24, 2012
  19. © 2009 - 2012 Next Big Sound, Inc. [email protected] db.artists.find({

    ! "signed": false, ! "data.twitter.fans.quantile": { ! ! "$gte": 800, ! ! "$lte": 950 ! }, ! "data.facebook.fans.value": { ! ! "$gt": 10000 ! }, ! "location.geo": { ! ! "$within": {"$centerSphere": [ [-73.9, 40.6], 10/3959 ]} ! } }) .sort({ ! "data.facebook.fans.value": -1 }) .limit(20) Dat Query Thursday, May 24, 2012
  20. © 2009 - 2012 Next Big Sound, Inc. [email protected] Results

    [ { "name": "Kayvon Zand"}, { "name": "Hoodie Allen"}, { "name": "Katrina Bradley"}, { "name": "Mike Wax"}, { "name": "Range"}, { "name": "DJ Mia Moretti"}, { "name": "Amanda Lepore"}, { "name": "DJ Spinna"}, { "name": "Anamanaguchi"}, { "name": "Wakey!Wakey!"}, { "name": "Holy Ghost!"}, ... ] Thursday, May 24, 2012
  21. © 2009 - 2012 Next Big Sound, Inc. [email protected] This

    is like, really cool Thursday, May 24, 2012
  22. © 2009 - 2012 Next Big Sound, Inc. [email protected] http://appserver.com/find/artists?

    ! data.twitter.fans.quantile=[(gte:800),(lte:950)]& ! data.facebook.fans.value=[(gt:10000)]& ! geo:location.geo=[(40.6:-73.9),10]& ! sort=[(data.twitter.fans.value:desc)]& ! fields=[nbs_id,name]& ! signed=False& ! limit=30 Asynchronous Mongo REST API { 1 count: 20, 2 results: [ 1 { 1 nbs_id: 80025, 2 name: "Kayvon Zand" 2 }, 3 { 1 nbs_id: 77313, 2 name: "Hoodie Allen" 4 }, 5 { 1 nbs_id: 269573, 2 name: "Karina Bradley" 6 }, 7 { 1 nbs_id: 56276, 2 name: "Mikey Wax" Thursday, May 24, 2012
  23. © 2009 - 2012 Next Big Sound, Inc. [email protected] Change

    the way the music industry uses data Thursday, May 24, 2012
  24. © 2009 - 2012 Next Big Sound, Inc. [email protected] Buck

    Heroux [email protected] @buckheroux nextbigsound.com/jobs Thursday, May 24, 2012