Pro Yearly is on sale from $80 to $50! »

IALP2019: Confidence Modeling for Neural Machine Translation

9e650916f36300d64c9c61eeb4ab697e?s=47 Taichi Aida
November 15, 2019

IALP2019: Confidence Modeling for Neural Machine Translation

IALP2019 Presentation slide
"Confidence Modeling for Neural Machine Translation"
Taichi Aida (Undergraduate student, Nagaoka University of Technology)
Kazuhide Yamamoto (Associate Professor, Nagaoka University of Technology)

9e650916f36300d64c9c61eeb4ab697e?s=128

Taichi Aida

November 15, 2019
Tweet

Transcript

  1. Confidence Modeling for Neural Machine Translation Taichi Aida, Kazuhide Yamamoto

    Nagaoka University of Technology IALP2019
  2. Introduction - Neural Machine Translation (NMT) system is widely used

    - NMT system outputs are wide range of quality - there may be mistranslations... 2 ➢Can we estimate the quality of sentence in the process of generation?
  3. Introduction - Our goal is outputting only high- quality translations

    - NOT output low-quality translations ➔input sentences > output sentences - Output translations are reliable - Helps for translators 3
  4. Methods 4 Figure 1. Overview of proposed method

  5. Methods 5 Figure 1. Overview of proposed method

  6. Methods 6 Figure 1. Overview of proposed method

  7. Methods - Indices - Sentence log-likelihood - Average variance 7

  8. Sentence log-likelihood 8

  9. Sentence log-likelihood 9 - Taking the sum of log-probabilities of

    all output words in a sentence
  10. Average variance 10

  11. Average variance 11 - Using top-5 candidates to calculate variance

    from top in each part of sentence
  12. Experiments 1. Appropriateness of indices - correlation with BLEU -

    (The Pearson correlation coefficient) 2. Using threshold - Changing the threshold… - number of output sentences - average BLEU in output sentences 12
  13. Experiments - Model - Transformer (fairseq) - Data - ASPEC-JE

    (translating scientific papers) 13 Train Validation Test 1,000,000 1,790 1,812
  14. 1. Appropriateness of indices - correlation with BLEU Results 14

    Indices ρ(index, BLEU) Sentence log-likelihood 0.308 Average variance 0.268
  15. Results 2. Using threshold - Threshold = 0.0 - 1812

    sentences - BLEU 22.11 - Threshold = 0.9 - 13 sentences - BLEU 43.45 15 Figure 5(a) Sentence log-likelihood
  16. Results 2. Using threshold - Threshold = 0.0 - 1812

    sentences - BLEU 22.11 - Threshold = 0.2 - 98 sentences - BLEU 33.12 16 Figure 5(b) Average variance
  17. Conclusion ➢We proposed calculating a translation confidence from NMT features

    ◦ sentence log-likelihood ◦ average variance ➢It can limit low-quality translations in the process of generation 17
  18. Thank you for Listening! 18