Confidence Modeling for 
 Neural Semantic Parsing จݙ঺հɹ ௕Ԭٕज़Պֶେֶɹࣗવݴޠॲཧݚڀࣨ ҴԬɹເਓ

Literature Confidence Modeling for Neural Semantic Parsing Li Dong† and Chris Quirk‡ and Mirella Lapata† †School of Informatics, University of Edinburgh
 ‡Microsoft Research, Redmond Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers), pages 743–753, 2018. !2

Abstract • Neural Semantic Parsing (seq2seq) ʹ͓͚Δ֬৴౓ϞσϦϯά • ೖྗͷͲ͕͜ෆ͔֬͞ͷཁҼʹͳ͍ͬͯΔ͔Λࣝผ • ࣄޙ֬཰ɺΞςϯγϣϯʹґଘ͢Δख๏ΑΓ΋༏ल !3

Introduction • Neural Semantic Parsing͸ظ଴Ͱ͖Δ݁ՌΛग़͢ҰํͰ
 ग़ྗͷݪҼ͕ղऍͮ͠Β͍ϒϥοΫϘοΫεͱͯ͠ಈ࡞ • Ϟσϧͷ༧ଌʹର͢Δ֬৴౓ͷਪఆʹΑͬͯ༗ҙٛͳ
 ϑΟʔυόοΫ͕ՄೳʹͳΔͷͰ͸ͳ͍͔ • ֬৴౓ͷείΞϦϯάख๏͸ࣄޙ֬཰ p(y|x) ͕Α͘࢖༻͞ΕΔ
 → ઢܗϞσϧͰ͸༗ޮ͕ͩχϡʔϥϧϞσϧͰ͸ྑ͘ͳ͍ !4

Neural Semantic Parsing • In: Natural Language
 Out: Logical form • Seq2seq with LSTM • Attention mechanism • Maximize the likelihood • Beam Search !5 !5

Confidence Estimation • ೖྗqͱ༧ଌͨ͠ҙຯදݱa͔Β֬৴౓s(q, a) ∈ (0, 1)Λ༧ଌ • ֬৴౓ͷ൑அʹ͸ʮԿΛ஌Βͳ͍͔ʯΛਪఆ͢Δඞཁ͕͋Δ • Ϟσϧͷෆ͔֬͞ɺσʔλͷෆ͔֬͞ɺೖྗͷෆ͔֬͞Λجʹ
 ࡞ΒΕΔࢦඪ͔Β֬৴౓ΛճؼϞσϧʹΑͬͯٻΊΔ !6

Model Uncertainty • Ϟσϧͷύϥϝʔλ΍ߏ଄ʹΑΔෆ͔֬͞Ͱ֬৴౓͕௿Լ
 ← ྫ͑͹܇࿅σʔλʹؚ·ΕΔϊΠζ΍֬཰తֶशΞϧΰϦζϜ • Dropout Perturbation, Gaussian Noise, Posterior Probability͔Β
 ࢦඪΛ࡞੒͠ɺෆ͔֬͞Λ༧ଌ !7

Dropout Perturbation • DropoutΛςετ࣌ʹ࢖༻
 (ਤதͷi, ii, iii, ivͷՕॴ) • จϨϕϧͰͷࢦඪɿ • τʔΫϯϨϕϧͰͷࢦඪɿ • ɹɹ͸ઁಈͤ͞Δύϥϝʔλɹ݁ՌΛूΊͯ෼ࢄΛܭࢉ !8

Gaussian Noise • Gaussian NoiseΛϕΫτϧ΁Ճ͑ͯDropoutͱಉ༷ʹ෼ࢄΛܭࢉ
 ← Dropout͸ϕϧψʔΠ෼෍ɺ͜Ε͸Ψ΢ε෼෍ʹै͏ϊΠζ • ϊΠζͷՃ͑ํ͸ҎԼͷ2ͭ (v͸ݩͷϕΫτϧ, g͸Gaussian Noise) !9

Posterior Probability • ࣄޙ֬཰ p(a | q)ΛจϨϕϧͰͷࢦඪʹ࢖༻ • τʔΫϯϨϕϧͰ͸ҎԼͷ2ͭΛࢦඪʹ࢖༻ • ɹɹɹɹɹɹɹɹɹɹɹɹɿ࠷΋ෆ͔֬ͳ୯ޠʹண໨ • ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɿτʔΫϯຖͷperplexity !10

Data Uncertainty • ܇࿅σʔλͷΧόϨοδ͸ෆ͔֬͞ʹӨڹΛ༩͑Δ • ܇࿅σʔλͰݴޠϞσϧΛֶशͤ͞ɺೖྗͷݴޠϞσϧ֬཰Λ
 ࢦඪʹ༻͍Δ • ೖྗͷະ஌ޠτʔΫϯ਺Λࢦඪʹ༻͍Δ !11

Input Uncertainty • Ϟσϧ͕׬ᘳͰ΋ೖྗ͕ᐆດͩͱෆ͔֬͞͸ൃੜ͢Δ
 (e.g. 9 o’clock -> flight_time(9am) or flight_time(9pm) ) • ্Ґީิͷ֬཰ͷ෼ࢄΛ༻͍Δ • ΤϯτϩϐʔΛ༻͍Δ
 ← a’͸αϯϓϦϯάۙࣅ !12

Confidence Storing • ͜ΕΒͷ༷ʑͳࢦඪΛ༻͍ͯ֬৴౓ͷείΞϦϯάΛߦ͏ • ޯ഑ϒʔεςΟϯάϞσϧʹ֤ࢦඪΛ༩ֶ͑ͯशͤ͞Δ
 ग़ྗ͕0ʙ1ʹͳΔΑ͏ϩδεςΟοΫؔ਺Ͱϥοϓ • ޯ഑ϒʔεςΟϯάϞσϧ͸ҎԼͷղઆهࣄ͕෼͔Γ΍͍͢
 (ʮGradient Boosting ͱ XGBoostʯ:
 ɹ https://zaburo-ch.github.io/post/xgboost/ ) !13

Uncertainty Interpretation • Ͳͷೖྗ͕ෆ͔֬͞ʹ࡞༻͍ͯ͠Δ͔Λಛఆ
 → ͦͷೖྗΛಛผͳέʔεͱͯ͠ѻ͏౳͕ग़དྷΔ • ༧ଌ͔ΒೖྗτʔΫϯؒ·ͰΛٯ఻ൖ
 → ֤τʔΫϯͷෆ͔֬͞΁ͷد༩͕Θ͔Δ !14

Experiments (Datasets) • IFTTT σʔληοτ (train-dev-test : 77,495 - 5,171 - 4,294) • DJANGO σʔληοτ (train-dev-test : 16,000 - 1,000 - 1,805)
 !15

Experiments (Settings) • Dropout Perturbation
 Dropout rate͸0.1ɺ30ճ࣮ߦͯ͠෼ࢄΛܭࢉ • Gaussian Noise
 ඪ४ภࠩΛ0.05ʹઃఆ • Probability of Input
 ݴޠϞσϧͱͯ͠KenLMΛ࢖༻ • Input Uncertainty
 10-best ͷީิ͔Β෼ࢄΛܭࢉ !16

Experiments (Results) • Model Uncertainty͕࠷΋ޮՌత • Data Uncertainty͸Өڹ͕খ͍͞
 → In-domainͰ͋ΔͨΊ !17

Experiments (Results) !18

Experiments (Results) • Model Uncertaintyͷ
 ࢦඪ͕ॏཁ • ಛʹIFTTT͸#UNKͱ
 Var͕ॏཁ !19

Experiments (Results) !20

Experiments (Results) • ϊΠζΛՃ͑ͨτʔΫϯྻͱ
 ٯ఻ൖͰಘͨτʔΫϯྻͷ
 ΦʔόʔϥοϓͰධՁ • Attentionͱൺֱͯ͠ߴ͍ • K=4ʹ͓͍ͯ͸໿80%͕Ұக !21

Experiments (Results) !22

Conclusions • Neural Semantic ParsingͷͨΊͷ֬৴౓ਪఆϞσϧΛఏࣔ • ෆ͔֬͞ΛೖྗτʔΫϯϨϕϧͰղऍ͢Δํ๏Λఏࣔ • IFTTT, DJANGOσʔληοτʹ͓͍ͯ༗ޮੑΛ֬ೝ • ఏҊϞσϧ͸Seq2seqΛ࠾༻͢Δ༷ʑͳλεΫͰద༻Մೳ • Neural Semantic ParsingͷActive Learningʹ͓͍ͯར༻Ͱ͖Δ !23