Share
Embed
Generative News Recommendation Reproduces Popularity: A Memorization-Based Analysis
Shotaro Ishihara (2026). Generative News Recommendation Reproduces Popularity: A Memorization-Based Analysis. The 11th International Workshop on Application of Big Data for Computational Social Science (ABCSS2026).
Shotaro Ishihara
November 19, 2026
Transcript
31 slides
-
Generative News Recommendation Reproduces Popularity: A Memorization-Based Analysis The 11th
International Workshop on Application of Big Data for Computational Social Science (ABCSS2026 @ IEEE BigData 2026) Shotaro Ishihara (Nikkei Inc.) *Currently resigned [email protected] -
Overview • Fine-tuned Llama 3 on news browsing logs to
analyze string duplication, memorization, and popularity bias in its outputs. • Results suggest that skewed string duplication drives skewed generation via memorization, inducing popularity bias. • Building on this interpretation, we demonstrated that deduplication, a memorization countermeasure, can help mitigate popularity bias. 2 -
Outline • Background: GenRec, Popularity & Memorization • Motivation: Research
Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 3 -
Generative Recommendation (GenRec) Discriminative Generative “Twitter cuts staff to 1/5”
“Twitter cuts staff to 1/5” => => Recommender Candidate 1: score 0.3 Candidate 2: score 0.2 Candidate 3: score 0.1 Recommender “Inside the Twitter Buyout” Directly generates the next article title as text 4 -
Why LLMs for Recommendation? Adapting LLMs to sequential recommendation [1]
=> Generative Recommendation (GenRec) • World knowledge aids cold-start items [3] • Natural language unifies item semantics [4] • Generates human-readable rationales [5] 5 -
Popularity Bias in GenRec • GenRec over-exposes head items [11,
12] • Prompt & text adjustments offer partial relief [12] => However, internal LLM mechanism remains a black box. 6 -
Training Data Memorization in LLMs • Emits verbatim / near-verbatim
strings [23] • Governed by three scaling factors [14]: ◦ String duplication count in training data ◦ Model scale ◦ Prompt length 7 -
Outline • Background: GenRec, Popularity & Memorization • Motivation: Research
Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 8 -
Research Positioning Challenges in Large Language Models Popularity Bias in
GenRec Training Data Memorization … (Studied separately so far) Can popularity bias in GenRec be explained through training data memorization? 9 -
Why News Concentration Matters News shapes civic awareness — skew
self-reinforces: Concentrated Outputs Skewed User Clicks Skewed Logs (Next Targets) Retraining amplifies head monopoly We investigate the practical deployment challenges of GenRec using raw real-world data with unmanipulated occurrence counts, which are often preprocessed in open datasets [26]. 10 -
Premise & Two Hypotheses Premise P Skewed Target Logs Mechanism
H1 String Memorization Outcome H2 Concentrated Generations • H1: Frequent target strings become strongly memorized • H2: Memorized titles are emitted across diverse prompts 11 -
Outline • Background: GenRec, Popularity & Memorization • Motivation: Research
Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 12 -
Nikkei Online Dataset (~20M Sessions) • Unaltered logs (benchmarks often
clip head items) • ~20M Nikkei sessions (Jan 2023, ≥ 3 viewed articles) Session input sequence (≤ 512 tokens): Title a₁ [SEP] Title a₂ [SEP] Title a₃ … Chronological article titles joined by [ARTICLE_SEP] 14 -
Continual Training: Llama3-nikkei-genrec • Backbone: Meta-Llama-3-8B-Instruct • LoRA continual training
(rank = 16, lr = 1e-4, 2 epochs) • Compute: 64 × NVIDIA A100 GPUs (8 cloud nodes) 15 -
Inference & Evaluation Setup Prompt: a₁ [SEP] Greedy (≤50 tok)
Output g vs y Distinct article coverage contracts 5×: Prompts Modal Ref y Generated g 193,815 101,026 20,689 (1/5×) 16 -
Distributional & Memorization Measures • Macro diversity: Concentration (Gini), Entropy,
GS, Types • String memorization (g vs. modal y) [Ishihara 24]: Ref y: “Breaking 90 in Golf: Lesson 4” Gen g: “Breaking 90 in Golf: ah-jeed” Verbatim = exact prefix Approx. = edit similarity 17 -
Outline • Background: GenRec, Popularity & Memorization • Motivation: Research
Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 18 -
Skew in Prompts, References & Generations Both Prompt and Reference
distributions are skewed, yet Generation exhibits even stronger concentration. 19 -
Macro Diversity Metrics (Table I) • Distinct titles shrink 5×
(101k → 20.7k); entropy drops • Higher GS score (0.19 → 0.34) shows semantic narrowing 20 -
H1: Target Frequency & Memorization Target string frequency in training
logs positively correlates with memorization. r = 0.304 / 0.334 (Prompt length: r = 0.02 / −0.12) 21 -
H2: Memorization & Generation Skew More strongly memorized articles appear
far more frequently in generated outputs. r = 0.197 / 0.308 Captures corrupted viral strings. 22 -
Skew Arises via Memorization r = 0.304 / 0.334 H1:
Frequent targets in logs become memorized r = 0.197 / 0.308 H2: Memorized titles are over-emitted 23 -
Top 5 Verbatim Memorization Examples Unrelated prompts (China travel, Twitter
layoffs) emit the same viral title 25,840 times! 24 -
Why Does Many-to-One Collapse Happen? “China travel …” “Twitter layoffs
…” [SEP] Memorized Viral Prefix 25,840× “Market outlook …” • Repeated targets after [SEP] form a strong lexical prior 25 -
Outline • Background: GenRec, Popularity & Memorization • Motivation: Research
Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 26 -
Mitigation: Target-Level Deduplication Original (~20M) a₁ → Viral y* a₂
→ Viral y* a₃ → Viral y* a₄ → Niche y₂ Dedup (193.9k) Keep max 1× per target y a₁ → Viral y* a₄ → Niche y₂ 100× smaller Deduplication is known to curb memorization [15, 16]. 27 -
Deduplication Trade-Off (Table III) • Memorization drops 88% (0.136 →
0.016); types double • Lower concentration & higher entropy curb head skew • Exact hits plunge (2,461 → 81) under 100× less data 28 -
Outline • Background: GenRec, Popularity & Memorization • Motivation: Research
Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 29 -
Conclusion • Fine-tuned Llama 3 on news browsing logs to
analyze string duplication, memorization, and popularity bias in its outputs. • Results suggest that skewed string duplication drives skewed generation via memorization, inducing popularity bias. • Building on this interpretation, we demonstrated that deduplication, a memorization countermeasure, can help mitigate popularity bias. 30 -
Future Directions • Analyze memorization from additional perspectives: ◦ Effects
of model scale, prompt length, etc. • Leverage the built generative recommender beyond popularity bias: ◦ Synthetic data generation, user/article analysis, etc. 31