論文紹介 / Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
第14回最先端NLP勉強会 の発表スライドです．
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, Candace Ross; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5238-5248
提案タスク: Winoground • ⼊⼒として画像・キャプションペアが2つ与えられ，4通りの組合せの中 から正しいペアを⾒つけ出す • 2つのキャプションは同じ単語/形態素を持つが，⽂中の順序は異なる some plants surrounding a lightbulb a lightbulb surrounding some plants 3
13 Linguistic Tag: 1 or 2 Main Preds (292, 108/400) • 述部の数（1つあるいは2つ）による分類． • 述部2つの⽅がより⻑く，複雑な⽂になりやすい there are more [humans] than [balls] there's a [phone] on a [map] the [plant] is eating the [bug] [out]1[swam]2 the person in the red swimcap []2[]1 looking from [above] at a collection of similar objects [below] the [sail] rests below the [water] [gold] for [pan] there are more [hats] than [people] [circular] food on [heart-shaped] wood the [water] is filled with [plastic] 1 Main Predsの例 [it] ran away while [they] pursued the person in a [brown] coat looks back and the person in a [black] coat looks forward the melting white food is [cold] while the brown is [warm] a kid [jumped] then [threw] a basketball the person is [jumping] while the cat is [sitting] a person wearing [yellow] with their feet in the air and a person wearing [stripes] the [computer's] screen is on and the [phone's] screen is off the person with facial hair [cycles] and the other person [runs] the person with green legs is running quite [slowly] and the red legged one runs [faster] a [] person wearing yellow and a person wearing stripes [jumping] 2 Main Predsの例
• comprises examples where the images need to be interpreted non-literally （前置詞句の付与場所が違う，”idiomatic use”など） 14 Visual Tag: Pragmatics (41/400) It starts with ["A”] and ends with ["Z”] It starts with ["Z”] and ends with ["A”]
• 画像が収集元のフォトストックサイトにおける同じシリーズ（登場⼈ 物・背景などが類似）から構成されているもの 15 Visual Tag: Series (31/400) the [masked] wrestler hits the [unmasked] wrestler the [unmasked] wrestler hits the [masked] wrestler
• 記号的な描写を理解する必要のある例（参考︓イラスト系は物体認識器 を使うモデルは⽐較的弱い） 16 Visual Tag: Symbolic (24/400) astronauts in [blue] suits with a [red] planet in the background astronauts in [red] suits with a [blue] planet in the background
• 条件に当てはまる画像・テキストの収集の難しさからか，正例の画像・ テキストペアにやや違和感のあるもの（普通はそのようなキャプショニ ングはしない）も含まれているように感じる 30 議論: やや不⾃然なキャプションについて https://huggingface.co/spaces/CVPR/winoground-explorer 間違ってはいないが，⾃転⾞に乗ってい る⽚⽅をsomeone else扱いは不⾃然 a person spraying water on someone else and a person on a bike a person spraying water on a person on a bike and someone else
[1] Alec Radford et al.: Learning Transferable Visual Models From Natural Language Supervision. ICML 2021: 8748-8763 [2] Amanpreet Singh et al.: FLAVA: A Foundational Language And Vision Alignment Model. CoRR abs/2112.04482 (2021) [3] Yen-Chun Chen et al.: UNITER: UNiversal Image-TExt Representation Learning. ECCV (30) 2020: 104-120 [4] Zhe Gan et al.: Large-Scale Adversarial Training for Vision-and-Language Representation Learning. NeurIPS 2020 [5] Pengchuan Zhang et al.: VinVL: Revisiting Visual Representations in Vision-Language Models. CVPR 2021: 5579-5588 [6] Wonjae Kim et al.: ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision. ICML 2021: 5583-5594 [7] Liunian Harold Li et al.: VisualBERT: A Simple and Performant Baseline for Vision and Language. CoRR abs/1908.03557 (2019) [8] Hao Tan et al.: LXMERT: Learning Cross-Modality Encoder Representations from Transformers. EMNLP/IJCNLP (1) 2019: 5099-5110 [9] Ronghang Hu et al.: UniT: Multimodal Multitask Learning with a Unified Transformer. ICCV 2021: 1419-1429 [10] Jiasen Lu et al.: ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision- and-Language Tasks. NeurIPS 2019: 13-23 32 参考⽂献（V&Lモデル）
kyoun
kei18
isayan
masatoto
hide2kano
rina_glutton
yuyay
shimacos
yosukefk
haraduka
xtalent
gpeyre
sensetime_japan
paigeccino
smashingmag
jonrohan
maggiecrowley
chriscoyier
zakiyama
jennybc
bermonpainter
roundedbygravity
holman
panda_program
sstephenson