UKARA 1.0 Challenge Track 1

Dfda3ce33093a2ce23246410c5087a92?s=47 Ali Akbar S.
October 14, 2019

UKARA 1.0 Challenge Track 1

Our results from UKARA 1.0 Challenge Track 1, an NLP challenge to build an automatic short-answer scoring.

Dfda3ce33093a2ce23246410c5087a92?s=128

Ali Akbar S.

October 14, 2019
Tweet

Transcript

  1. Ali Akbar Septiandri Yosef Ardhito Winatmoko Pesimis Positif

  2. None
  3. None
  4. • • •

  5. Task A Task B # Positive Train 191 (71%) 168

    (55%) # Negative Train 77 (29%) 137 (45%) Avg. # Char 87.23 97.33 # Dev 215 244 # Test 855 974
  6. None
  7. Labelled as 1 (correct) but does not fit the criteria

    – Task A
  8. None
  9. None
  10. Original Label Corrected Label Frequency Task A 0 1 10

    1 0 4 Task B 0 1 46 1 0 13
  11. PREPROCESSING FEATURE EXTRACTION CLASSIFICATION

  12. • Tokenizer + lemmatizer • Unigram / TF-IDF ◦ •

    Latent Semantic Analysis (LSA) ◦ ◦ • ML algorithms ◦ ◦ ◦ • Evaluation metric ◦
  13. SEPARATE MODELS

  14. • Typo corrector • hyperopt • Machine learning ◦ ◦

    ◦ ◦ • ensemble models
  15. None
  16. (0.879±0.014) (0.764±0.035)

  17. Train A Train B Dev Test Best 0.879 0.764 0.810

    0.812 Ensemble+Original 0.885 0.764 0.799 0.801 Ensemble+Updated 0.898 0.831 0.810 0.803
  18. None
  19. None
  20. None
  21. None
  22. None
  23. ◂ ◂ ◂ ◂ ◂ ◂