NAACL HLT 2016 ʢBest Paperʣ ࿦จಡΈձ@य़೔ΤϦΞɹ2016೥7݄16೔(౔) ಡΜͩਓɿhimkt * εϥΠυதͷਤ͸͢΂ͯ࿦จ͔ΒҾ༻

Overview • ෳ਺ͷ࣭໰Ԡ౴λεΫʹରԠͰ͖ΔϞσϧͷఏҊ • ը૾ • ߏ଄Խ͞Εͨ஌ࣝϕʔε • ࣭໰จΛߏจղੳͯ͠ରԠ͢ΔωοτϫʔΫΛಈతʹ ߏங͢ΔʢDynamic Neural module networkʣ • ύϥϝʔλͷֶशʹ͸ڧԽֶशΛ࢖͍ͬͯΔ

Overview 1 2 3

Overview - 1. Network Layout • ࣭໰จΛ܎Γड͚ղੳʢStanford Dependency Parserʣ • ܎Γड͚݁Ռʹ΋ͱ͍ͮͯऔΓ͏ΔωοτϫʔΫߏ଄ͷ ީิΛྻڍ • ࣭໰จΛॴ༩ͱͨ͠ࡍͷωοτϫʔΫʹؔ͢Δ৚݅෇͖ ֬཰ΛධՁͯ͠ωοτϫʔΫΛܾఆ

Overview - 2. Module inventory 1 2 3

Module inventory • 6छྨͷϞδϡʔϧͱݺ͹ΕΔؔ਺ • Attention͔LabelΛग़ྗ͢Δ • Attention: pixels • Label: true/false or lexicon (e.g. “bird”) • ֤Ϟδϡʔϧ͸ग़ྗͱҾ਺ʹؔͯ͠ʮܕʯ੍໿Λ࣋ͭ • Lookup :: input -> Attention • Find :: input -> Attention • Relate :: Attention -> Attention • And :: Attention* -> Attention • Describe :: Attention -> Labels • Exists :: Attention -> Labels

Attention • find :: input -> Attention • ը૾ͷҰ෦ʢpixelͷू߹ʣΛग़ྗ

Overview - 3. Produce an answer 1 2 3

Produce an answer • What color is the bird? -> (describe[color] find[bird]) -> black and white (lexicon) • Are there any states? -> (exists find[state]) -> true

Components • Layout model • ωοτϫʔΫߏ଄Λਪఆ͢Δ • Execution model • ճ౴Λੜ੒͢Δ • Training • ;ͨͭͷύϥϝʔλΛಉ࣌ʹֶश • ڧԽֶश p(z|x; l ) pz (y|w; e )

Layout Model • ৚͖݅ͭ֬཰͸ιϑτϚοΫεͷग़ྗ • ͨͩ͠ɼ • ɹɹɹɹɹɹɹ͸ύϥϝʔλ • ɹɹɹ͸LSTMͷग़ྗ • ɹɹɹ͸ɹ ʢi൪໨ͷީิͷωοτϫʔΫʣͷ embedding? ʢfeature vectorʣ p(zi |x; l) = es(zi |x) n j=1 es(zj |x) s(zi |x) = aT (Bhq (x) + Cf(zi ) + d) l = (a, B, C, d) hq (x) f(zi ) zi

Execution Model • ճ౴Λੜ੒͢ΔϞσϧ • ࣗ਎ͷೖྗ͕Θ͔͍ͬͯΔͱ͖ ɹͱॻ͚Δ pz(y|z) = z w y ( z w )y = m(h1, h2) and(find, relate(lookup))

Training • ڧԽֶश • ɹΛɹɹɹɹɹ͔ΒαϯϓϦϯά • ωοτϫʔΫ͕ܾఆͨ͠ΒɹɹɹɹɹɹɹΛ
 ௚઀࠷େԽͯ͠ɹɹΛߋ৽ • Policy Gradient MethodʹΑΓɹ Λߋ৽ • ޯ഑ɿɹɹɹɹɹɹɹɹɹɹɹɹɹɹʢɹ͸ใुʣ z p(z|x; l ) log p(y|z, x, e ) e l J( l ) = E log p(z|x; l ) · r r J( l ) = E log p(z|x; l ) log p(y|z, w; e )

Experimental result • VisualQAʢTable 1ʣͱGeoQAʢTable 2ʣͰstate-of-the-art • VisualQA: images • GeoQAɿstructured domains • ෳ਺ͷ࣭໰Ԡ౴λεΫʹରԠͰ͖Δ͜ͱ͕ূ໌͞Εͨ