10/18/2016 We love posting images! - Image tweets constitute 14% of Twitter posts and 56% of Weibo posts [Chen 2016] Image courtesy: Martin Parr/Magnum Photos 3
Interpreting Microblog Images • Vital for downstream applications, such as – User interest modelling, retrieval, event detection, summarization • Image understanding – Low-level features (e.g., SIFT) do not work well due to the semantic gap – How about visual objects? 10/18/2016 4
Interpreting Microblog Images • Vital for downstream applications, e.g., – User interest modelling, retrieval, event detection, summarization • Image understanding – Low-level features (e.g., SIFT) do not work well due to the semantic gap – How about visual objects? • Not sufficient for microblog images 10/18/2016 7
1. Hashtag Enhanced Text • The most obvious context is post’s text • We focus on conflating the variants of hashtags – #icebucket, #ALSIceBucketChallenge – 14.3% of image tweets have multi-words hashtags 10/18/2016 Microsoft Word Breaker API [Wang et al. NAACL’10] ice bucket ALS ice bucket challenge 12
2. Text in the Image 10/18/2016 • Apply an OCR tool (Google Tesseract) to extract text from images • 26.4% of the images have at least one recognized textual word Coming soon!!! imdb.to/IGxE9f Pretty much 13
4. Search Engine as a Context Miner • Not all images in microblogs are user generated – Used in other places with a similar context 10/18/2016 Pages that contain the image Named entity Best guess Google Image Search - 76.0% of images have been indexed by Google 16
4. Search Engine as a Context Miner • Not all images in microblogs are user generated – Used in other places with a similar context 10/18/2016 Google Image Search 17
4. Search Engine as a Context Miner • Not all images in microblogs are user generated – Used in other places with a similar context 10/18/2016 Google Image Search Named entity Best guess Pages that contain the image 18
4. Search Engine as a Context Miner • Not all images in microblogs are user generated – Used in other places with a similar context 10/18/2016 Pages that contain the image Named entity Best guess Google Image Search - 76.0% of images have been indexed by Google 19
CITING: Context-aware Image Tweet Modelling 10/18/2016 Coming soon!!! imdb.to/IGxE9f 1. Hashtag enhanced text URLs 3. External web page Word Breaker 2. Text in Image OCR Tool Google Image Search Best guess Named entity Pages that contain the image 4. Search Result 20
Overlap of the three major sources 10/18/2016 14.3% of image tweets have multiple-word hashtags Text in Image External Web Pages Google Image Search 21.8% 79.3% 26.4% 21
10/18/2016 Text in Image External Web Pages Google Image Search 2.8% 12.5% 22.2% 2.5% 21.8% 79.3% 26.4% 14.3% of image tweets have multiple-word hashtags Overlap of the three major sources 22
Hashtag > External pages > OCR text > Search results • External pages contain rich and more relevant information than overlaid text • Errors introduced by OCR tool 10/18/2016 24
Hashtag > External pages > OCR text > Search results • Google Image Search can not differentiate the pure text-style and meme images well. 10/18/2016 25
CITING: Context-aware Image Tweet Modelling 10/18/2016 Coming soon!!! imdb.to/IGxE9f 1. Hashtag enhanced text URLs 3. External web page Word Breaker 2. Text in Image OCR Tool Google Image Search Best guess Named entity Pages that contain the image 4. Search Result Text quality: Hashtag > External pages > OCR text > Search results 26
Rules for Fusing Contextual Text 10/18/2016 Coming soon!!! imdb.to/IGxE9f Basic text (94.8%): Text from post + enhanced hashtags (14.3%) URLs Basic + Text from External Pages Basic + OCR Text Basic + Text from Search Result OCR Text Yes 14.4% No Yes 23.5% Search Engine No Yes 48.9% Reduce contextual text acquisition cost by 18% No Basic text Text quality: Hashtag > External pages > OCR text > Search results 27
Feature-aware Matrix Factorization • A generic model that incorporates various types of features into users’ interest modeling • Not susceptible to cold start 10/18/2016 N types of features (e.g., CITING text, visual objects) A feature’s latent factor Item’s latent factor User’s latent factor 31
Model Learning • Pair-wise Learning to Rank – Positive tweets (retweets) has a better rank than negative ones (non-retweets) – Bayesian Personalized Ranking [Rendle et al. 2009] – Minimize loss function • Infer the parameters via stochastic gradient descent (SGD) 10/18/2016 Regularization term 32
Time-aware Negative Sampling • Retweets are positive instances • We sample negative instances based on the time of retweets 10/18/2016 33 Retweet time Non-retweet 1 Non-retweet 2 2 is more likely to be a real negative instance
Experimental Setting • Keep users’ 10 most recent retweets as a test set • Evaluation metrics – Mean Average Precision (MAP) – Average Precision at top ranks 10/18/2016 Users Retweets All Tweets Ra8ngs Training 926 174,765 1,316,645 1,592,837 Test 9,021 77,061 82,743 34
Conclusion • Released code and datasets: https://github.com/kite1988/famf • Future work – Other contexts: geo-location, time, author – Other fusion approaches, e.g., learn weights of each contextual source 10/18/2016 CITING framework to model image tweets - Hashtag enhanced text - OCR text - External pages - Search results Feature-aware MF to recommend image tweets - Decompose user-item interaction to user-feature interaction - Alleviate cold-start problem 45
kite1988
abemii
isayan
naoto0804
ahspragu
shuntaros
chokkan
sensetime_japan
mhangyo
xtalent
masakat0
eqpresents
thc1006
tenderlove
myddelton
orderedlist
yeseniaperezcruz
thekraken
lynnandtonic
sugarenia
thoeni
revolveconf
vanstee
speakerdeck
chrislema