Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Generative News Recommendation Reproduces Popul...

Generative News Recommendation Reproduces Popularity: A Memorization-Based Analysis

Shotaro Ishihara (2026). Generative News Recommendation Reproduces Popularity: A Memorization-Based Analysis. The 11th International Workshop on Application of Big Data for Computational Social Science (ABCSS2026).

Shotaro Ishihara

November 19, 2026

Transcript

31 slides
  1. Generative News Recommendation Reproduces Popularity: A Memorization-Based Analysis The 11th

    International Workshop on Application of Big Data for Computational Social Science (ABCSS2026 @ IEEE BigData 2026) Shotaro Ishihara (Nikkei Inc.) *Currently resigned [email protected]
  2. Overview • Fine-tuned Llama 3 on news browsing logs to

    analyze string duplication, memorization, and popularity bias in its outputs. • Results suggest that skewed string duplication drives skewed generation via memorization, inducing popularity bias. • Building on this interpretation, we demonstrated that deduplication, a memorization countermeasure, can help mitigate popularity bias. 2
  3. Outline • Background: GenRec, Popularity & Memorization • Motivation: Research

    Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 3
  4. Generative Recommendation (GenRec) Discriminative Generative “Twitter cuts staff to 1/5”

    “Twitter cuts staff to 1/5” => => Recommender Candidate 1: score 0.3 Candidate 2: score 0.2 Candidate 3: score 0.1 Recommender “Inside the Twitter Buyout” Directly generates the next article title as text 4
  5. Why LLMs for Recommendation? Adapting LLMs to sequential recommendation [1]

    => Generative Recommendation (GenRec) • World knowledge aids cold-start items [3] • Natural language unifies item semantics [4] • Generates human-readable rationales [5] 5
  6. Popularity Bias in GenRec • GenRec over-exposes head items [11,

    12] • Prompt & text adjustments offer partial relief [12] => However, internal LLM mechanism remains a black box. 6
  7. Training Data Memorization in LLMs • Emits verbatim / near-verbatim

    strings [23] • Governed by three scaling factors [14]: ◦ String duplication count in training data ◦ Model scale ◦ Prompt length 7
  8. Outline • Background: GenRec, Popularity & Memorization • Motivation: Research

    Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 8
  9. Research Positioning Challenges in Large Language Models Popularity Bias in

    GenRec Training Data Memorization … (Studied separately so far) Can popularity bias in GenRec be explained through training data memorization? 9
  10. Why News Concentration Matters News shapes civic awareness — skew

    self-reinforces: Concentrated Outputs Skewed User Clicks Skewed Logs (Next Targets) Retraining amplifies head monopoly We investigate the practical deployment challenges of GenRec using raw real-world data with unmanipulated occurrence counts, which are often preprocessed in open datasets [26]. 10
  11. Premise & Two Hypotheses Premise P Skewed Target Logs Mechanism

    H1 String Memorization Outcome H2 Concentrated Generations • H1: Frequent target strings become strongly memorized • H2: Memorized titles are emitted across diverse prompts 11
  12. Outline • Background: GenRec, Popularity & Memorization • Motivation: Research

    Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 12
  13. Nikkei Online Dataset (~20M Sessions) • Unaltered logs (benchmarks often

    clip head items) • ~20M Nikkei sessions (Jan 2023, ≥ 3 viewed articles) Session input sequence (≤ 512 tokens): Title a₁ [SEP] Title a₂ [SEP] Title a₃ … Chronological article titles joined by [ARTICLE_SEP] 14
  14. Continual Training: Llama3-nikkei-genrec • Backbone: Meta-Llama-3-8B-Instruct • LoRA continual training

    (rank = 16, lr = 1e-4, 2 epochs) • Compute: 64 × NVIDIA A100 GPUs (8 cloud nodes) 15
  15. Inference & Evaluation Setup Prompt: a₁ [SEP] Greedy (≤50 tok)

    Output g vs y Distinct article coverage contracts 5×: Prompts Modal Ref y Generated g 193,815 101,026 20,689 (1/5×) 16
  16. Distributional & Memorization Measures • Macro diversity: Concentration (Gini), Entropy,

    GS, Types • String memorization (g vs. modal y) [Ishihara 24]: Ref y: “Breaking 90 in Golf: Lesson 4” Gen g: “Breaking 90 in Golf: ah-jeed” Verbatim = exact prefix Approx. = edit similarity 17
  17. Outline • Background: GenRec, Popularity & Memorization • Motivation: Research

    Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 18
  18. Skew in Prompts, References & Generations Both Prompt and Reference

    distributions are skewed, yet Generation exhibits even stronger concentration. 19
  19. Macro Diversity Metrics (Table I) • Distinct titles shrink 5×

    (101k → 20.7k); entropy drops • Higher GS score (0.19 → 0.34) shows semantic narrowing 20
  20. H1: Target Frequency & Memorization Target string frequency in training

    logs positively correlates with memorization. r = 0.304 / 0.334 (Prompt length: r = 0.02 / −0.12) 21
  21. H2: Memorization & Generation Skew More strongly memorized articles appear

    far more frequently in generated outputs. r = 0.197 / 0.308 Captures corrupted viral strings. 22
  22. Skew Arises via Memorization r = 0.304 / 0.334 H1:

    Frequent targets in logs become memorized r = 0.197 / 0.308 H2: Memorized titles are over-emitted 23
  23. Top 5 Verbatim Memorization Examples Unrelated prompts (China travel, Twitter

    layoffs) emit the same viral title 25,840 times! 24
  24. Why Does Many-to-One Collapse Happen? “China travel …” “Twitter layoffs

    …” [SEP] Memorized Viral Prefix 25,840× “Market outlook …” • Repeated targets after [SEP] form a strong lexical prior 25
  25. Outline • Background: GenRec, Popularity & Memorization • Motivation: Research

    Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 26
  26. Mitigation: Target-Level Deduplication Original (~20M) a₁ → Viral y* a₂

    → Viral y* a₃ → Viral y* a₄ → Niche y₂ Dedup (193.9k) Keep max 1× per target y a₁ → Viral y* a₄ → Niche y₂ 100× smaller Deduplication is known to curb memorization [15, 16]. 27
  27. Deduplication Trade-Off (Table III) • Memorization drops 88% (0.136 →

    0.016); types double • Lower concentration & higher entropy curb head skew • Exact hits plunge (2,461 → 81) under 100× less data 28
  28. Outline • Background: GenRec, Popularity & Memorization • Motivation: Research

    Gap & News Context • Setup: Continual Training on ~20M Logs • Results: Skew Driven by Memorization • Mitigation: Target-Level Deduplication • Conclusion & Future Directions 29
  29. Conclusion • Fine-tuned Llama 3 on news browsing logs to

    analyze string duplication, memorization, and popularity bias in its outputs. • Results suggest that skewed string duplication drives skewed generation via memorization, inducing popularity bias. • Building on this interpretation, we demonstrated that deduplication, a memorization countermeasure, can help mitigate popularity bias. 30
  30. Future Directions • Analyze memorization from additional perspectives: ◦ Effects

    of model scale, prompt length, etc. • Leverage the built generative recommender beyond popularity bias: ◦ Synthetic data generation, user/article analysis, etc. 31

More Decks by Shotaro Ishihara

Other Decks in Research