19 ファインチューニング Transformerエンコーダ [SEP] How many children are in the bed ? 2 • 画像キャプションで事前学習したモデルを基に， QAデータを使って画像と質問⽂から回答を分類するタスク としてファインチューニング 質問⽂ 回答（クロスエント ロピー損失で学習） 画像（オブジェクト系列）
• インフォグラフィックに対するVQA • これまでの物に⽐べてグラフやアイコンに対する理解が重要 となる 33 Infographics VQA [Robust Reading Challenge, 2021] Q. How many females are affected by diabetes A. 3.6% Q. What percentage of cases can be prevented A. 60% Q. What could lead to blindness or stroke A. diabetes https://rrc.cvc.uab.es/?ch=17
CLIPで商品ページ検索を試してみた Lenobo Chromebook MacBook Air M1 USB-C Adapter for MacBook Air M1 What is the screen size of chromebook? What is the CPU chip of the latest macbook 13-inch? Is there a usb adapter that will work with MacBook Air M1? 0.32 0.29 0.27 0.26 0.31 0.28 0.25 0.32 0.32 モデルはCLIP ViT-B/32 を利⽤ https://github.com/o penai/CLIP 63
• Vicente Ordonez, Girish Kulkarni, Tamara L. Berg: Im2Text: Describing Images Using 1 Million Captioned Photographs. NIPS 2011: 1143-1151 • Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C. Lawrence Zitnick: Microsoft COCO: Common Objects in Context. ECCV (5) 2014: 740-755 • Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, C. Lawrence Zitnick: Microsoft COCO Captions: Data Collection and Evaluation Server. CoRR abs/1504.00325 (2015) • Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, Li Fei-Fei: Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. Int. J. Comput. Vis. 123(1): 32-73 (2017) • Piyush Sharma, Nan Ding, Sebastian Goodman, Radu Soricut: • Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. ACL (1) 2018: 2556-2565 • Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei A. F. Florêncio, Lijuan Wang, Cha Zhang, Lei Zhang, Jiebo Luo: TAP: Text-Aware Pre-training for Text-VQA and Text- Caption. CoRR abs/2012.04638 (2020) • Soravit Changpinyo, Piyush Sharma, Nan Ding, Radu Soricut: Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. CoRR abs/2102.08981 (2021) • Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, Marc Najork: WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning. CoRR abs/2103.01913 (2021) ⼤規模画像キャプションデータ 67
• Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh: Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017: 6325-6334 • Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay, Dhruv Batra, Devi Parikh: Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. Int. J. Comput. Vis. 127(4): 398-414 (2019) • Peter Young, Alice Lai, Micah Hodosh, Julia Hockenmaier: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguistics 2: 67-78 (2014) • Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C. Lawrence Zitnick: Microsoft COCO: Common Objects in Context. ECCV (5) 2014: 740-755 • Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, C. Lawrence Zitnick: Microsoft COCO Captions: Data Collection and Evaluation Server. CoRR abs/1504.00325 (2015) 68 参考⽂献︓VQA, IR
• Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang: Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. CVPR 2018: 6077-6086 • Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, Devi Parikh: Pythia v0.1: the Winning Entry to the VQA Challenge 2018. CoRR abs/1807.09956 (2018) • Duy-Kien Nguyen, Vedanuj Goswami, Xinlei Chen: MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond. ILCR 2021 • Ang Li, Allan Jabri, Armand Joulin, Laurens van der Maaten: Learning Visual N-Grams from Web Data. ICCV 2017: 4193-4202 71 参考⽂献︓⾮事前学習のVQA/IRモデル
• Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samuel White, Tom Yeh: VizWiz: nearly real-time answers to visual questions. UIST 2010: 333-342 • Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, Jeffrey P. Bigham: VizWiz Grand Challenge: Answering Visual Questions From Blind People. CVPR 2018: 3608-3617 • Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, Marcus Rohrbach: Towards VQA Models That Can Read. CVPR 2019: 8317-8326 • Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, C. V. Jawahar: DocVQA: A Dataset for VQA on Document Images. WACV 2021 • Lu Chen, Xingyu Chen, Zihan Zhao, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, Kai Yu: WebSRC: A Dataset for Web-Based Structural Reading Comprehension. CoRR abs/2101.09465 (2021) • Ryota Tanaka, Kyosuke Nishida, Sen Yoshida: VisualMRC: Machine Reading Comprehension on Document Images. AAAI 2021 • Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, Amanpreet Singh: TextCaps: A Dataset for Image Captioning with Reading Comprehension. ECCV (2) 2020: 742-758 • Tsu-Jui Fu, William Yang Wang, Daniel J. McDuff, Yale Song: DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents. CoRR abs/2101.11796 (2021) • Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, Zhiwei Guan: Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements. EMNLP (1) 2020: 5495-5510 72 参考⽂献︓視覚情報に含まれる⾔語情報 （データセット）
• Xiaoxue Chen, Lianwen Jin, Yuanzhi Zhu, Canjie Luo, Tianwei Wang: Text Recognition in the Wild: A Survey. CoRR abs/2005.03492 (2020) • Ronghang Hu, Amanpreet Singh, Trevor Darrell, Marcus Rohrbach: Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA. CVPR 2020: 9989-9999 • Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou: LayoutLM: Pre- training of Text and Layout for Document Image Understanding. KDD 2020: 1192-1200 • Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei A. F. Florêncio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou: LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding. CoRR abs/2012.14740 (2020) 73 参考⽂献︓視覚情報に含まれる⾔語情報 （OCR組込モデル）
• Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021. • Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis E. H. Tay, Jiashi Feng, Shuicheng Yan: Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet. CoRR abs/2101.11986 (2021) • OpenAI Blog, DALL·E: Creating Images from Text. January 2021. https://openai.com/blog/dall-e/ • Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, Ilya Sutskever: Zero-Shot Text-to-Image Generation. CoRR abs/2102.12092 (2021) • OpenAI Blog, CLIP: Connecting Text and Images. January 2021. https://openai.com/blog/clip/ • Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever: Learning Transferable Visual Models From Natural Language Supervision. CoRR abs/2103.00020 (2021) • OpenAI Blog, Multimodal Neurons in Artificial Neural Networks. March 2021. https://openai.com/blog/multimodal-neurons/ https://distill.pub/2021/multimodal-neurons/ 74 参考⽂献︓ ViT / DALL·E / CLIP
kyoun
sh01k
kitachan_black
tosseto
javiergs
unzaih
otanet
yamdan
hkefka385
yowata
hiroshinakagawa
takuma_okamoto
yumulab
sonatard
reverentgeek
sferik
danielanewman
aki_iinuma
rmw
cassininazir
bkeepers
speakerdeck
thekraken
addyosmani