[EMNLP26] RIGEL: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation

Transcript

1. RIGEL: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation Shuitsu Koyama, Kazuki Matsuda, Yuiga Wada, Shinnosuke Hirano, Daichi Yashima, Komei Sugiura Keio University
2. Motivation: Building better automatic evaluation metrics for image and video captions MLLM ”A person uses a yellow tool to carve out a design on a wooden board” Metric • Automatic evaluation metrics are essential to MLLM development • Standard metrics (e.g., BLEU) correlate poorly with human judgments 4/5 -2-
3. Motivation: Building better automatic evaluation metrics for image and video captions 4/5 Video (Pearson 𝑟) MLLM ”A person uses a yellow tool to carve out a design on a wooden board” Metric Our group Polos (CVPR24) RIGEL +14.8 CLIP-S (EMNLP21) Pearl (AAAI26) BERTScore (ICLR20) Image (Kendall 𝜏! ) • Automatic evaluation metrics are essential to MLLM development • Standard metrics (e.g., BLEU) correlate poorly with human judgments -3-
4. Using MLLMs for automatic evaluation Main Question: How should we use MLLMs for classification with a small label set? Existing: Extract label tokens MLLM (score tokens) ... ... 1 Ours: Self-distilled classification head ... 5 tool yellow -4-
5. Using MLLMs for automatic evaluation Main Question: How should we use MLLMs for classification with a small label set? Existing: Extract label tokens Ours: Self-distilled classification head MLLM (score tokens) ... ... 1 ... 5 tool yellow 1 5 -5-
6. Using MLLMs for automatic evaluation Main Question: How should we use MLLMs for classification with a small label set? Existing: Extract label tokens Ours: Self-distilled classification head MLLM (score tokens) ... ... 1 ... 5 tool yellow 1 5 -6-
7. Score smoothing has limited expressive power and ignores non-score logits Head MLLM Score smoothing [Lee+, ACL24] ... ... 1 ... 3.32 5 tool yellow -7-
8. Score smoothing has limited expressive power and ignores non-score logits Head MLLM Score smoothing [Lee+, ACL24] ... ... 1 ... 3.32 5 tool yellow L Limited expressive power -8-
9. Score smoothing has limited expressive power and ignores non-score logits Score smoothing [Lee+, ACL24] Head MLLM L Discards potentially ... ... 1 ... J Does not discard them useful projections of 3.323.32 5 tool yellow score score L Limited expressive power -9-
10. Simply adding a classification head and fine-tuning is insufficient L Simple fine-tuning is insufficient mapping from Scoring head MLLM L Does not carry over the original 1 5 J Models complex mappings between and score labels VATEX-EVAL Head 𝜏" 𝜌 Original 39.7 50.8 Scoring head 35.4 45.4 - 10 - -5.4
11. Phase 1: Distill a scoring head to resolve the vocabulary-label mismatch 1 Phase 1: Self-distillation ❄ J The backbone was frozen to leverage the expressive power of Distillation loss: EMD 🔥 1 J EMD: penalizes 1→5 more than 4→5 L KL: penalizes both equally EMD:0.9 4→5 1→5 KL:2.3 EMD:3.6 KL:2.3 - 11 -
12. Phase 2: Fine-tune the backbone to align with human judgments Phase 2: Align with 1 human judgments 1 from MLLM Phase 1: Distill 🔥 1 1 5 tool yellow Scoring head MLLM J aligns backbone representations Scoring head ❄J stabilizes training MLLM Freezing the scoring head 1 5 CE 1 5 1 5 Human judgment - 12 -
13. Experimental Setup: Constructed the first training dataset for video captioning evaluation L Existing video datasets lack training data [Shi+, CVPR22] [Liu+, EMNLP23] Trained on Vid-Lepus (New) / Spica [Hirano+, AAAI26] J 14k / 333k human judgments J 241 / 2.3k annotators Datasets available !! Vid-Lepus Spica - 13 -
14. Quantitative Results (Video): Large margins over baselines VATEX-EVAL Metrics 𝜏! 𝜌 ActivityNet-Fact Activity Net-FOIL YouCook2-Fact Para Sent Word Para Sent Word 𝑟 𝑟 𝑟 𝑟 𝑟 𝑟 Acc. RIGEL (Ours) 42.1 53.7 63.6 51.9 60.5 55.0 50.3 50.0 96.2 G-VEval 39.4 - 48.8 38.6 43.6 55.8 50.1 44.4 - PAC-S 25.1 32.6 33.2 27.1 38.4 31.2 33.5 33.1 90.1 EMScore 23.2 30.3 25.3 19.0 30.0 33.7 35.3 36.1 89.5 [Tong+, AAAI25] [Sarto+, CVPR23] [Shi+, CVPR22] - 14 -
15. Quantitative Results (Video): Large margins over baselines VATEX-EVAL Para Metrics 𝜏! RIGEL (Ours) G-VEval ActivityNet-Fact 𝜌 Sent Word YouCook2-Fact Para Sent Activity Net-FOIL Word J Better hallucination discrimination power 𝑟 expressive 𝑟 𝑟 𝑟 with𝑟greater 𝑟 Acc. 42.1 53.7 63.6 51.9 60.5 55.0 50.3 50.0 96.2 +5.1 +14.8 +13.3 +16.9 39.4 - 48.8 38.6 43.6 55.8 50.1 44.4 - PAC-S 25.1 32.6 33.2 27.1 38.4 31.2 33.5 33.1 90.1 EMScore 23.2 30.3 25.3 19.0 30.0 33.7 35.3 36.1 89.5 [Tong+, AAAI25] [Sarto+, CVPR23] [Shi+, CVPR22] - 15 -
16. Quantitative Results (Image): Outperformed baselines Nebula Flickr8K-CF Flickr8K-EX Composite Metrics 𝜏! 𝜏" 𝜏! 𝜏" 𝜏! 𝜏" 𝜏! 𝜏" FOIL 1-ref 4-ref Acc. Acc. RIGEL (Ours) 60.6 57.5 40.4 20.9 58.6 59.0 61.2 66.1 99.1 99.2 Pearl 58.4 55.4 38.6 20.0 58.2 58.6 55.8 60.4 96.5 97.2 G-VEval - - 38.2 19.9 60.5 58.7 - - 97.8 98.4 RefPAC-S 54.7 51.9 37.6 19.5 55.5 55.9 53.0 57.3 93.7 94.9 [Hirano+, AAAI26] [Tong+, AAAI25] [Sarto+, CVPR23] - 16 -
17. Qualitative Result: Better performance than GPT-6 Astra Not clearly shown “a girl sitting down and playing a string musical instrument by herself” Human: 4.33/5 RIGEL: 4.15/5 GPT-6 Astra: 1.00/5 L Overpenalization - 17 -
18. Ablation Study: Self-distillation improved performance Nebula SelfDistillation Distillation Loss ✔ ✔ Flickr8K-EX Composite 𝜏! 𝜏" 𝜏! 𝜏" 𝜏! 𝜏" EMD 58.1 55.1 59.5 59.9 59.5 64.3 ー 56.3 KL +1.8 57.7 +1.7 53.4 58.8 59.2 54.7 59.3 59.7 +1.5 58.0 59.0 +1.5 62.8 63.8 - 18 -
19. Ablation Study: EMD outperformed KL, likely because KL cannot account for distances between scores SelfDistillation Distillation Loss ✔ ✔ Nebula Flickr8K-EX Composite 𝜏! 𝜏" 𝜏! 𝜏" 𝜏! 𝜏" EMD 58.1 55.1 59.5 59.9 59.5 64.3 ー 56.3 53.4 58.8 59.2 KL 57.7 54.7 59.3 59.7 +0.5 58.0 59.0 +0.5 62.8 63.8 - 19 -
20. Conclusion: An evaluation metric based on self-distillation, achieving SOTA on 8 benchmarks • RIGEL: MLLM-as-a-Judge based on self-distilled classification head J resolves the vocabulary-label mismatch • Vid-Lepus: The first training dataset for video captioning evaluation J 14.8k human judgments J 241 annotators Code Dataset - 20 -