UKARA 1.0 Challenge Track 1

Our results from UKARA 1.0 Challenge Track 1, an NLP challenge to build an automatic short-answer scoring.

Transcript

1. Ali Akbar Septiandri Yosef Ardhito Winatmoko Pesimis Positif
4. • • •
5. Task A Task B # Positive Train 191 (71%) 168 (55%) # Negative Train 77 (29%) 137 (45%) Avg. # Char 87.23 97.33 # Dev 215 244 # Test 855 974
7. Labelled as 1 (correct) but does not fit the criteria – Task A
10. Original Label Corrected Label Frequency Task A 0 1 10 1 0 4 Task B 0 1 46 1 0 13
11. PREPROCESSING FEATURE EXTRACTION CLASSIFICATION
12. • Tokenizer + lemmatizer • Unigram / TF-IDF ◦ • Latent Semantic Analysis (LSA) ◦ ◦ • ML algorithms ◦ ◦ ◦ • Evaluation metric ◦
13. SEPARATE MODELS
14. • Typo corrector • hyperopt • Machine learning ◦ ◦ ◦ ◦ • ensemble models
16. (0.879±0.014) (0.764±0.035)
17. Train A Train B Dev Test Best 0.879 0.764 0.810 0.812 Ensemble+Original 0.885 0.764 0.799 0.801 Ensemble+Updated 0.898 0.831 0.810 0.803
23. ◂ ◂ ◂ ◂ ◂ ◂