THUIR Study Group 08-22

Transcript

1. Identifying Modes of User Engagement with Online News and Their Relationship to Information Gain in Text Nir Grinberg Network Science Institute, Northeastern University
2. Motivation • Understand user engagement within an article page. • Distinguish between different modes of engagement with an article (scan, skim, in-depth reading) • Understand the relationship between articles’text and the engagement of readers with it. • Existing engagement measures do not take into account the text information.(The development of ideas within the text.)
3. Related Work • The use of post-click engagement measures in systems’ design • Dwell time, scrolling, other post-click behavior. • Our: Using summary statistics. • The relationship between content properties and user engagement • Load time • Visual complexity, aesthetics of web pages, visual saliency • Article length, image and videos count. • Emotional dimensions of news articles (sentiment and polarity.) • Readability of text and general topic. • Our: The development of ideas within an article.
4. Engagement Data • Large dataset (7.7 million page views) • View ~ [User, Article, behaviors] • Articles • From 11 major news publications • Crawlled the content of articles. [text]
5. Engagement Data • Behaviors: • Dwell time • Maximal depth (pixels) • Active engagment the amount of user interactionwith a page in any form: mouse move, scroll, swipe, key press, etc. • Others: • The length of article content • The height of user’s viewport
6. Transformed Measures • Absolute measures are not universal • Different across sites • Different across devices: desktop/mobile/tablet • Inter-dependency • Dwell time, scrolling depth, active engagement are all positive correlated.
7. Transformed Measures • Relative Depth • Average Scrolling Speed • Normalized Engagement d: the maximal position reached h: the user’s viewport height l: the length of article content t: the total time on the page e: the raw measure c: a constant scaler (c=10) l/h: the number of screens it would take a visitor to read an entire article The proportion of an article that was visible on the user’s screen. How quickly the user scrolled through the visible part of the page.
8. Transformed Measures • Log-transformed. • After the relative transformations, site difference are notably less prominent. [dwell time, scrollingdepth,active engagement, relative depth,average scrollingspeed,normalizedengagement]
9. Model of Engagement • Characterizing page views • Modeled by a Probabilistic Multivariate Mixture Model • Training on a balanced dataset (sites and devices) • Remove the page views quickly leaved. (<10 sec./no inter.) V: the set of page view summaries K: the number of multivariate normals Σ" : covariance matrix (tied, diag, spherical, full) Leaning robust patterns of engagement. Model each summaryvector as the result of a probabilisticmutivariatemixturemodel.
10. Model of Engagement • Characterizing page views • Choosing the optimal K (Bayesian information criteria BIC) • Full covariance matrix outperforms all other covariance structures. (choose: full, k=5)
11. Model of Engagement 1. has the fastest speed: Scanning 3. Readlongger 4. Low reading speed: longer period of inactivity. 5. Low values across all engagement measures.
12. Model of Engagement • Characterizing engagement with news articles • Engagement vector for article as the mean assignment vector across all page view of the article. • Different levels of engagement with articles from different sites. % of page view in the site are Scan
13. Semantic Information Gain (SIG) • Inspired by the theory of Information Foraging. • Semantic information gain with each paragraph of a text. • 48-dim. Doc2Vec trained on the entire corpus. • Paragraph -> relative pixel depth in an article.
14. Semantic Information Gain • The SIG at the beginning of articles (first 30%) varies across sites and engagement modes. (Each line: top-300 articles in engagement mode). • The figure clearly demonstrates that the amount of information conveyed in articles’ text differs by site and is associated with different types of user engagement.
15. Predicting Engagement • Target: engagement modes distribution for previously unseen articles. • Model: Dirichlet Regression Model • Features: • Baseline: the length of text; the amount of non-textual elements in an article. • Latent topics: LDA / TUNE (T=50) • Difficulty (sentence length; Flesch-Kincaid score) / Sentiment (VADER) • SIG (the three quadratic terms, first three paragraphs) • Avg. Dwell time / Avg. Scrolling depth /Avg. Engagement
16. Predicting Engagement PCC The regression models were fit using ten-fold cross validation for each site and then averaged across sites.
17. Conclusion • A compact metric of user interaction that carries valuable information about post-click user engagement with a news article. • A novel measure of information gain within the text of news articles. • Empirical evidence linking information gain in text to user engagement as observed in a diversity of news sites, at large scale, and outside lab settings.