Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
Search
onizuka laboratory
December 18, 2018
Research
0
65
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
弊研究室で行なったEMNLP2018読み会の発表資料です。
onizuka laboratory
December 18, 2018
Tweet
Share
More Decks by onizuka laboratory
See All by onizuka laboratory
Phrase-Based & Neural Unsupervised Machine Translation
onilab
0
110
Card-660: A Reliable Evaluation Framework for Rare Word Representation Models
onilab
0
31
A Word-Complexity Lexicon and A Neural Readability Ranking Model for Lexical Simplification
onilab
0
98
Integrating Transformer and Paraphrase Rules for Sentence Simplification
onilab
0
56
An Auto-Encoder Matching Model for Learning Utterance-Level Semantic Dependency in Dialogue Generation
onilab
0
48
Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints
onilab
0
98
Modeling Multi-turn Conversation with Deep Utterance Aggregation
onilab
0
91
Learning Semantic Sentence Embeddings using Pair-wise Discriminator
onilab
0
110
SGM: Sequence Generation Model for Multi-Label Classification
onilab
0
67
Other Decks in Research
See All in Research
脳卒中患者・家族からみた循環器病対策推進基本計画の進捗に関する調査
japanstrokeassociation
0
530
センサデータを活用した 肌質改善への支援システムに関する研究
comfortdesignlab
0
150
訓練データ作成のためのCloudCompareを利用した点群の手動ラベリング
kentaitakura
0
540
説明可能AI:代表的手法と最近の動向
yuyay
1
600
ICLR2024 LLMエージェントの研究動向
masatoto
6
1.9k
LLMマルチエージェントを俯瞰する
masatoto
26
16k
メタ動画データセットによる動作認識の現状と可能性
yuyay
0
180
20240127_熊本から今いちど真面目に都市交通~めざせ「車1割削減、渋滞半減、公共交通2倍」~ 全国路面電車サミット2024宇都宮
trafficbrain
1
660
LiDARセキュリティ最前線
kentaroy47
0
280
Breaking Tradeoffs: Extremely Scalable Multi-Agent Pathfinding Algorithms
kei18
0
150
Azure Arc-enabled Serversを利用した ハイブリッド・マルチクラウド環境の管理 / Managing Hybrid Multi-cloud Environments with Azure Arc-enabled Servers
nttcom
0
210
サウナでのプロジェクションマッピングの可能性の検討 / EC71koizumi
yumulab
0
170
Featured
See All Featured
Building Effective Engineering Teams - LeadDev
addyosmani
28
1.8k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
352
28k
Unsuck your backbone
ammeep
663
57k
The Brand Is Dead. Long Live the Brand.
mthomps
49
29k
Music & Morning Musume
bryan
41
5.6k
Debugging Ruby Performance
tmm1
70
11k
Designing the Hi-DPI Web
ddemaree
276
33k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
121
39k
Building Applications with DynamoDB
mza
88
5.6k
The Illustrated Children's Guide to Kubernetes
chrisshort
31
46k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.1k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
659
120k
Transcript
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes
and Captions Q. Li, J. Fu, D. Yu et al. EMNLP 2018 20181218
VQA CNN RNN ;* end-to-end !" → <2AE+H
C0 A9#5 2 4G' C0 <2I!".(AE)= >'1 %FI6?B,?:$D/-8 3@ VQA end-to-end &7 2 4G' <2AE)= 1
Visual Q&A Q: where is
the man swinging the racket? A: tennis court 2
Visual Q&A Q: what kind
of drink is in the glass? A: water 3
Visual Q&A Q: what is
walking next to the bus? A: cow 4
Visual Q&A Q: does the
man need a haircut? A: yes 5
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and
Captions 6
"# where is the man swinging
the racket? yes no water tennis court ⋮ CNN RNN $ ! 7
7# 9+ D >(. <& D (CB D *
"6* D -2,4 end-to-end 2 3A $% @8181;?':/4-2 0),4 =C !581;?B 8
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and
Captions 9
( ,# &+ Ø#'! Ø#(" ) ,( %*&$ ! Ø'
ver. Ø( ver. Ø ver. ( ) 10
11
@,→H3L0 !%$ %)=1M ;F$ % @, I87 H3L0 ResNet152
'#(' .? ØBAK6 ED "&'&( .?9 /$ % -NC> .? G+ ;F$ % cos N*2 J17 H3L0 cos N*2.? 5:4< H3)= 12
/ →3*) / ResNet152 LSTM 1 .%0
1', (e.g. BLEU) 4") .%) 2$5! cos 6# (+&- / 3*).% 13
7*='?8.-?9(-→64 #2(> 8.-9(- LSTM !" Ø LSTM &7!!)<%/
;5 $5') 3: softmax ,0+1 64#2 14
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and
Captions 15
-VQA-real Ø1 & 3 ,1 10
)', *!#-min(#)*+,-. /010-/ 2),2 ,-.345 6 , 1) Ø%)' 10 " 3 ( $) + * 16
VQA vs.
17
vs.
18
VQA *!-# +$ .% -#"0,)( 0'/ Ø& -#" Ø
-#" ØNULL 1 -# $ 19
* '" Ø+! & )%
Ø+! & )% Ø+! & )% Ø+! & )% $ ,# - +! ( VQA & 20
tennis, ball, man, racket, hit, court, play, player,
swing, hold a man holding a tennis racket on a tennis court. tennis court & Q: where is the man swinging the racket? A: tennis court 21
bicycle, man, sit, eat, bike, look, outside, food,
person, table a man sitting at a table with a plate of food. beer & Q: what kind of drink is in the glass? A: water 22
street, bus, cow, city, walk, car, drive, stand,
road, white a cow that is walking in the street. car & Q: what is walking next to the bus? A: cow 23
woman, bear, teddy, hold, sit, glass, animal, large,
lady a woman holding a sandwich in her hands. yes & Q: does the man need a haircut? A: yes 24
30%
65% yes/no 80% 25
VQA 26
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and
Captions 27
>1=9 5 2 3B# VQA #2 Ø7!6*A-"?&.% / Ø7!
<+; 0',4:(C $ 8 VQA =@ ) = 28