Context-aware Image Tweet Modelling and Recommendation

Presented at the 2016 ACM Multimedia Conference (MM '16) in Amsterdam, The Netherlands.

Transcript

1. 2016 Context-aware Image Tweet Modelling and Recommendation Tao Chen, Xiangnan He, Min-Yen Kan
2. 10/18/2016 We are visual beings！ 2
3. 10/18/2016 We love posting images! - Image tweets constitute 14% of Twitter posts and 56% of Weibo posts [Chen 2016] Image courtesy: Martin Parr/Magnum Photos 3
4. Interpreting Microblog Images •  Vital for downstream applications, such as – User interest modelling, retrieval, event detection, summarization •  Image understanding – Low-level features (e.g., SIFT) do not work well due to the semantic gap – How about visual objects? 10/18/2016 4
5. little cute child girl indoor China ends the one-child policy
6. transportation system people car asphalt road 10/18/2016 6
7. Interpreting Microblog Images •  Vital for downstream applications, e.g., – User interest modelling, retrieval, event detection, summarization •  Image understanding – Low-level features (e.g., SIFT) do not work well due to the semantic gap – How about visual objects? •  Not sufficient for microblog images 10/18/2016 7
8. Context is the key to interpret microblog images! 10/18/2016 Context-aware Image Tweet Modelling and Recommendation 8
9. •  The most obvious context is post’s text 10/18/2016 9
10. •  The most obvious context is post’s text •  We focus on conflating the variants of hashtags –  #icebucket, #ALSIceBucketChallenge 10/18/2016 10
11. •  The most obvious context is post’s text •  We focus on conflating the variants of hashtags –  #icebucket, #ALSIceBucketChallenge 10/18/2016 11
12. 1. Hashtag Enhanced Text •  The most obvious context is post’s text •  We focus on conflating the variants of hashtags –  #icebucket, #ALSIceBucketChallenge – 14.3% of image tweets have multi-words hashtags 10/18/2016 Microsoft Word Breaker API [Wang et al. NAACL’10] ice bucket ALS ice bucket challenge 12
13. 2. Text in the Image 10/18/2016 •  Apply an OCR tool (Google Tesseract) to extract text from images •  26.4% of the images have at least one recognized textual word Coming soon!!! imdb.to/IGxE9f Pretty much 13
14. 3. External URLs 10/18/2016 Coming soon!!! imdb.to/IGxE9f -  22.7% of image tweets have URLs 14
15. 3. External URLs 10/18/2016 Coming soon!!! imdb.to/IGxE9f -  22.7% of image tweets have URLs -  82.1% of external pages contain the image in the post 15
16. 4. Search Engine as a Context Miner •  Not all images in microblogs are user generated – Used in other places with a similar context 10/18/2016 Pages that contain the image Named entity Best guess Google Image Search - 76.0% of images have been indexed by Google 16
17. 4. Search Engine as a Context Miner •  Not all images in microblogs are user generated – Used in other places with a similar context 10/18/2016 Google Image Search 17
18. 4. Search Engine as a Context Miner •  Not all images in microblogs are user generated – Used in other places with a similar context 10/18/2016 Google Image Search Named entity Best guess Pages that contain the image 18
19. 4. Search Engine as a Context Miner •  Not all images in microblogs are user generated – Used in other places with a similar context 10/18/2016 Pages that contain the image Named entity Best guess Google Image Search - 76.0% of images have been indexed by Google 19
20. CITING: Context-aware Image Tweet Modelling 10/18/2016 Coming soon!!! imdb.to/IGxE9f 1. Hashtag enhanced text URLs 3. External web page Word Breaker 2. Text in Image OCR Tool Google Image Search Best guess Named entity Pages that contain the image 4. Search Result 20
21. Overlap of the three major sources 10/18/2016 14.3% of image tweets have multiple-word hashtags Text in Image External Web Pages Google Image Search 21.8% 79.3% 26.4% 21
22. 10/18/2016 Text in Image External Web Pages Google Image Search 2.8% 12.5% 22.2% 2.5% 21.8% 79.3% 26.4% 14.3% of image tweets have multiple-word hashtags Overlap of the three major sources 22
23. Hashtag > External pages > OCR text > Search results 10/18/2016 23
24. Hashtag > External pages > OCR text > Search results •  External pages contain rich and more relevant information than overlaid text •  Errors introduced by OCR tool 10/18/2016 24
25. Hashtag > External pages > OCR text > Search results •  Google Image Search can not differentiate the pure text-style and meme images well. 10/18/2016 25
26. CITING: Context-aware Image Tweet Modelling 10/18/2016 Coming soon!!! imdb.to/IGxE9f 1. Hashtag enhanced text URLs 3. External web page Word Breaker 2. Text in Image OCR Tool Google Image Search Best guess Named entity Pages that contain the image 4. Search Result Text quality: Hashtag > External pages > OCR text > Search results 26
27. Rules for Fusing Contextual Text 10/18/2016 Coming soon!!! imdb.to/IGxE9f Basic text (94.8%): Text from post + enhanced hashtags (14.3%) URLs Basic + Text from External Pages Basic + OCR Text Basic + Text from Search Result OCR Text Yes 14.4% No Yes 23.5% Search Engine No Yes 48.9% Reduce contextual text acquisition cost by 18% No Basic text Text quality: Hashtag > External pages > OCR text > Search results 27
28. Outline 1.  Introduction 2.  Motivation 5. Conclusion 10/18/2016 3.  CITING framework 4. Personalized image tweet recommendation 28 We are the first one!
29. Personalized Image Tweet Recommendation 10/18/2016 1 1 0 0 1 1 0 0 0 1 1 0 1 1 0 ? U1 U2 U3 U4 Matrix Factorization (MF) -  The state-of-the-art collaborative filtering algorithm -  Learn a vector representation for each user and Item in a latent space Will U4 retweet I4 ? User’s latent factor Item’s latent factor 29 I1 I2 I3 I4
30. Standard MF does not work for image tweets 10/18/2016 1 1 0 0 1 1 0 0 0 1 1 0 1 1 0 ? U1 U2 U3 U4 I1 I2 I3 I4 ? ? ? ? ? ? ? ? I5 I6 Cold start -  Take the features of image tweets into consideration User-item interac@on User-feature interac@on decompose 30
31. Feature-aware Matrix Factorization •  A generic model that incorporates various types of features into users’ interest modeling •  Not susceptible to cold start 10/18/2016 N types of features (e.g., CITING text, visual objects) A feature’s latent factor Item’s latent factor User’s latent factor 31
32. Model Learning •  Pair-wise Learning to Rank – Positive tweets (retweets) has a better rank than negative ones (non-retweets) – Bayesian Personalized Ranking [Rendle et al. 2009] – Minimize loss function •  Infer the parameters via stochastic gradient descent (SGD) 10/18/2016 Regularization term 32
33. Time-aware Negative Sampling •  Retweets are positive instances •  We sample negative instances based on the time of retweets 10/18/2016 33 Retweet time Non-retweet 1 Non-retweet 2 2 is more likely to be a real negative instance
34. Experimental Setting •  Keep users’ 10 most recent retweets as a test set •  Evaluation metrics – Mean Average Precision (MAP) – Average Precision at top ranks 10/18/2016 Users Retweets All Tweets Ra8ngs Training 926 174,765 1,316,645 1,592,837 Test 9,021 77,061 82,743 34
35. Method Feature P@1 P@3 P@5 MAP 1 Random 0.114 0.115 0.115 0.156 2 Length Post’s text 0.176 0.158 0.150 0.173 3 Profiling Post’s text 0.336 0.227 0.197 0.202 4 FAMF Visual Objects (VO) 0.211 0.205 0.192 0.211 5 FAMF Post’s text 0.359 0.325 0.287 0.275 6 FAMF CITING 0.419 0.355 0.319 0.298 Effectiveness of CITING Framework 10/18/2016 35
36. Method Feature P@1 P@3 P@5 MAP 1 Random 0.114 0.115 0.115 0.156 2 Length Post’s text 0.176 0.158 0.150 0.173 3 Profiling Post’s text 0.336 0.227 0.197 0.202 4 FAMF Visual Objects (VO) 0.211 0.205 0.192 0.211 5 FAMF Post’s text 0.359 0.325 0.287 0.275 6 FAMF CITING 0.419 0.355 0.319 0.298 Effectiveness of CITING Framework 10/18/2016 36
37. Method Feature P@1 P@3 P@5 MAP 1 Random 0.114 0.115 0.115 0.156 2 Length Post’s text 0.176 0.158 0.150 0.173 3 Profiling Post’s text 0.336 0.227 0.197 0.202 4 FAMF Visual Objects (VO) 0.211 0.205 0.192 0.211 5 FAMF Post’s text 0.359 0.325 0.287 0.275 6 FAMF CITING 0.419 0.355 0.319 0.298 Effectiveness of CITING Framework 10/18/2016 37 -  Visual objects are not sufficient to model semantics of microblog images
38. Method Feature P@1 P@3 P@5 MAP 1 Random 0.114 0.115 0.115 0.156 2 Length Post’s text 0.176 0.158 0.150 0.173 3 Profiling Post’s text 0.336 0.227 0.197 0.202 4 FAMF Visual Objects (VO) 0.211 0.205 0.192 0.211 5 FAMF Post’s text 0.359 0.325 0.287 0.275 6 FAMF CITING 0.419 0.355 0.319 0.298 Effectiveness of CITING Framework 10/18/2016 38
39. Method Feature P@1 P@3 P@5 MAP 1 Random 0.114** 0.115 0.115 0.156** 2 Length Post’s text 0.176** 0.158 0.150 0.173** 3 Profiling Post’s text 0.336** 0.227 0.197 0.202** 4 FAMF Visual Objects (VO) 0.211** 0.205 0.192 0.211** 5 FAMF Post’s text 0.359* 0.325 0.287 0.275** 6 FAMF CITING 0.419 0.355 0.319 0.298 Effectiveness of CITING Framework 10/18/2016 39 -  Our proposal significantly outperforms the other approaches **: p<0.01, *: p<0.05
40. Effectiveness of CITING Framework Method Feature P@1 P@3 P@5 MAP 1 Random 0.114** 0.115 0.115 0.156** 2 Length Post’s text 0.176** 0.158 0.150 0.173** 3 Profiling Post’s text 0.336** 0.227 0.197 0.202** 4 FAMF Visual Objects (VO) 0.211** 0.205 0.192 0.211** 5 FAMF Post’s text 0.359* 0.325 0.287 0.275** 6 FAMF CITING 0.419 0.355 0.319 0.298 7 FAMF Non-filtered Context 0.413 0.352 0.319 0.296 10/18/2016 **: p<0.01, *: p<0.05 40
41. Effectiveness of CITING Framework Method Feature P@1 P@3 P@5 MAP 1 Random 0.114** 0.115 0.115 0.156** 2 Length Post’s text 0.176** 0.158 0.150 0.173** 3 Profiling Post’s text 0.336** 0.227 0.197 0.202** 4 FAMF Visual Objects (VO) 0.211** 0.205 0.192 0.211** 5 FAMF Post’s text 0.359* 0.325 0.287 0.275** 6 FAMF CITING 0.419 0.355 0.319 0.298 7 FAMF Non-filtered Context 0.413 0.352 0.319 0.296 10/18/2016 **: p<0.01, *: p<0.05 -  Filtered fusion improves contextual text quality 41
42. Effectiveness of CITING Framework Method Feature P@1 P@3 P@5 MAP 1 Random 0.114** 0.115 0.115 0.156** 2 Length Post’s text 0.176** 0.158 0.150 0.173** 3 Profiling Post’s text 0.336** 0.227 0.197 0.202** 4 FAMF Visual Objects (VO) 0.211** 0.205 0.192 0.211** 5 FAMF Post’s text 0.359* 0.325 0.287 0.275** 6 FAMF CITING 0.419 0.355 0.319 0.298 7 FAMF Non-filtered Context 0.413 0.352 0.319 0.296 8 FAMF CITING+ VO 0.425 0.350 0.313 0.298 10/18/2016 **: p<0.01, *: p<0.05 42
43. Effectiveness of CITING Framework Method Feature P@1 P@3 P@5 MAP 1 Random 0.114** 0.115 0.115 0.156** 2 Length Post’s text 0.176** 0.158 0.150 0.173** 3 Profiling Post’s text 0.336** 0.227 0.197 0.202** 4 FAMF Visual Objects (VO) 0.211** 0.205 0.192 0.211** 5 FAMF Post’s text 0.359* 0.325 0.287 0.275** 6 FAMF CITING 0.419 0.355 0.319 0.298 7 FAMF Non-filtered Context 0.413 0.352 0.319 0.296 8 FAMF CITING+ VO 0.425 0.350 0.313 0.298 10/18/2016 **: p<0.01, *: p<0.05 43 -  The incorporation of visual objects does not consistently improve the recommendation performance
44. Case Study 10/18/2016 Average Precision: 0.226 (visual objects) -> 0.592 (CITING) 44
45. Conclusion •  Released code and datasets: https://github.com/kite1988/famf •  Future work –  Other contexts: geo-location, time, author –  Other fusion approaches, e.g., learn weights of each contextual source 10/18/2016 CITING framework to model image tweets -  Hashtag enhanced text -  OCR text -  External pages -  Search results Feature-aware MF to recommend image tweets -  Decompose user-item interaction to user-feature interaction -  Alleviate cold-start problem 45