Popular not Significant

What is the difference between being popular and significant in data analysis.

Transcript

1. @DataBeersMLG 18-May-2017 Pablo Musa Popular != Significant
2. Pablo Musa • MSc. Computer Science • Backend Developer • Software Architect • Infra Lover • 2 years Hadoop DevOps • 3 years Elastic Enthusiast 2
3. Motivation • Marketing, Recommendation, Analysis,... • It is all about understanding the "audience", the group, the niche • Sometimes understanding the group can be hard • In which city should I focus my Marketing Campaign? • A user watched "Mar Adentro" what should we recommend next? 3
4. Popularity • Are blue cars more common in London or Birmingham? • We will probably have more blue cars in London. However, not because Londoners like it more than others, but because London has a huge population. • Are there more people accessing "elastic.co" in London or Birmingham? 4
5. Popularity • A user watched "Mar Adentro" what movies should we recommend next? • We can look into the most common movies that users that watched "Mar Adentro" also watched and use it as recommendation. Will it be good? 5
6. Popular • Are blue cars more common in London or Birmingham? • We can look into the most common movies that...? • Common: occurring, found, or done often; prevalent 6
7. Top Popular Cities per Color 7
8. Significant • Are blue cars more significant in London or Birmingham? • We can look into the most significant movies that...? • Significant: sufficiently great or important to be worthy of attention; noteworthy. 8
9. Top Significant Cities per Color 9
10. 10 999 users watched Recommended Watched ?
11. 11 999 users watched 708 also watched 701 also watched 699 also watched Recommended Watched
12. 12 999 users watched 371 also watched 263 also watched 354 also watched Recommended Watched
13. 13 999 users watched 371 also watched 263 also watched 354 also watched Recommended Watched bg:2437
 sc:7.45 bg:1311
 sc:7.04 bg:2619
 sc:6.28
14. 14 Significant Popular
15. What about bands? 15
16. What about bands? 16
17. The Coldplay Effect 17
18. The Coldplay Effect Fixed 18
19. Another way to understand the coldplay effect 19 q=random q=mecano s=10% s=all
20. Another way to understand the coldplay effect 20 q=[mecano, amaral, pereza, coldplay] q=[mecano, amaral, pereza, coldplay] s=all s=500
21. Another way to understand the coldplay effect 21 q=[mecano, amaral, pereza, coldplay] q=[mecano, amaral, pereza, coldplay] s=all s=500 NOISE SIGNAL
22. References • https://www.elastic.co/guide/en/elasticsearch/reference/5.4/search- aggregations-bucket- • terms-aggregation.html • significantterms-aggregation.html • sampler-aggregation.html • Datasets • https://grouplens.org/datasets/movielens/20M/ • http://www.dtic.upf.edu/~ocelma/MusicRecommendationDataset/ lastfm-360K.html • http://data.dft.gov.uk/ 22
23. Thanks!! * JONTB - 16:30 - Mollete Hall - Managing your Black Friday Logs Pablo Musa @pablitomusa