IALP2019: Confidence Modeling for Neural Machine Translation

IALP2019 Presentation slide
"Confidence Modeling for Neural Machine Translation"
Taichi Aida (Undergraduate student, Nagaoka University of Technology)
Kazuhide Yamamoto (Associate Professor, Nagaoka University of Technology)

1. Confidence Modeling for Neural Machine Translation Taichi Aida, Kazuhide Yamamoto Nagaoka University of Technology IALP2019
2. Introduction - Neural Machine Translation (NMT) system is widely used - NMT system outputs are wide range of quality - there may be mistranslations... 2 ➢Can we estimate the quality of sentence in the process of generation?
3. Introduction - Our goal is outputting only high- quality translations - NOT output low-quality translations ➔input sentences > output sentences - Output translations are reliable - Helps for translators 3
4. Methods 4 Figure 1. Overview of proposed method
5. Methods 5 Figure 1. Overview of proposed method
6. Methods 6 Figure 1. Overview of proposed method
7. Methods - Indices - Sentence log-likelihood - Average variance 7
8. Sentence log-likelihood 8
9. Sentence log-likelihood 9 - Taking the sum of log-probabilities of all output words in a sentence
10. Average variance 10
11. Average variance 11 - Using top-5 candidates to calculate variance from top in each part of sentence
12. Experiments 1. Appropriateness of indices - correlation with BLEU - (The Pearson correlation coefficient) 2. Using threshold - Changing the threshold… - number of output sentences - average BLEU in output sentences 12
13. Experiments - Model - Transformer (fairseq) - Data - ASPEC-JE (translating scientific papers) 13 Train Validation Test 1,000,000 1,790 1,812
14. 1. Appropriateness of indices - correlation with BLEU Results 14 Indices ρ(index, BLEU) Sentence log-likelihood 0.308 Average variance 0.268
15. Results 2. Using threshold - Threshold = 0.0 - 1812 sentences - BLEU 22.11 - Threshold = 0.9 - 13 sentences - BLEU 43.45 15 Figure 5(a) Sentence log-likelihood
16. Results 2. Using threshold - Threshold = 0.0 - 1812 sentences - BLEU 22.11 - Threshold = 0.2 - 98 sentences - BLEU 33.12 16 Figure 5(b) Average variance
17. Conclusion ➢We proposed calculating a translation confidence from NMT features ◦ sentence log-likelihood ◦ average variance ➢It can limit low-quality translations in the process of generation 17
18. Thank you for Listening! 18