Popular not Significant

Popular not Significant

What is the difference between being popular and significant in data analysis.

A8c1cd7556870cf906064041cc5db121?s=128

Pablo Musa

May 18, 2017
Tweet

Transcript

  1. @DataBeersMLG 18-May-2017 Pablo Musa Popular != Significant

  2. Pablo Musa • MSc. Computer Science • Backend Developer •

    Software Architect • Infra Lover • 2 years Hadoop DevOps • 3 years Elastic Enthusiast 2
  3. Motivation • Marketing, Recommendation, Analysis,... • It is all about

    understanding the "audience", the group, the niche • Sometimes understanding the group can be hard • In which city should I focus my Marketing Campaign? • A user watched "Mar Adentro" what should we recommend next? 3
  4. Popularity • Are blue cars more common in London or

    Birmingham? • We will probably have more blue cars in London. However, not because Londoners like it more than others, but because London has a huge population. • Are there more people accessing "elastic.co" in London or Birmingham? 4
  5. Popularity • A user watched "Mar Adentro" what movies should

    we recommend next? • We can look into the most common movies that users that watched "Mar Adentro" also watched and use it as recommendation. Will it be good? 5
  6. Popular • Are blue cars more common in London or

    Birmingham? • We can look into the most common movies that...? • Common: occurring, found, or done often; prevalent 6
  7. Top Popular Cities per Color 7

  8. Significant • Are blue cars more significant in London or

    Birmingham? • We can look into the most significant movies that...? • Significant: sufficiently great or important to be worthy of attention; noteworthy. 8
  9. Top Significant Cities per Color 9

  10. 10 999 users watched Recommended Watched ?

  11. 11 999 users watched 708 also watched 701 also watched

    699 also watched Recommended Watched
  12. 12 999 users watched 371 also watched 263 also watched

    354 also watched Recommended Watched
  13. 13 999 users watched 371 also watched 263 also watched

    354 also watched Recommended Watched bg:2437
 sc:7.45 bg:1311
 sc:7.04 bg:2619
 sc:6.28
  14. 14 Significant Popular

  15. What about bands? 15

  16. What about bands? 16

  17. The Coldplay Effect 17

  18. The Coldplay Effect Fixed 18

  19. Another way to understand the coldplay effect 19 q=random q=mecano

    s=10% s=all
  20. Another way to understand the coldplay effect 20 q=[mecano, amaral,

    pereza, coldplay] q=[mecano, amaral, pereza, coldplay] s=all s=500
  21. Another way to understand the coldplay effect 21 q=[mecano, amaral,

    pereza, coldplay] q=[mecano, amaral, pereza, coldplay] s=all s=500 NOISE SIGNAL
  22. References • https://www.elastic.co/guide/en/elasticsearch/reference/5.4/search- aggregations-bucket- • terms-aggregation.html • significantterms-aggregation.html • sampler-aggregation.html

    • Datasets • https://grouplens.org/datasets/movielens/20M/ • http://www.dtic.upf.edu/~ocelma/MusicRecommendationDataset/ lastfm-360K.html • http://data.dft.gov.uk/ 22
  23. Thanks!! * JONTB - 16:30 - Mollete Hall - Managing

    your Black Friday Logs Pablo Musa @pablitomusa