Save 37% off PRO during our Black Friday Sale! »

[ICCE2021] Prior Knowledge on the Dynamics of Skill Acquisition Improves Deep Knowledge Tracing

15e9926baa86af3bb05a9027f6927999?s=47 qqhann
November 24, 2021

[ICCE2021] Prior Knowledge on the Dynamics of Skill Acquisition Improves Deep Knowledge Tracing

Proceedings p.265 https://icce2021.apsce.net/proceedings/volume1/
GitHub: https://github.com/qqhann/KnowledgeTracing

Abstract: Knowledge tracing (KT) is the task of modeling how students' academic skills change over time. Given a sequence of a student's learning history, one goal of KT is to predict how well he/she will perform in the next interaction. Unlike in BKT (Bayesian knowledge tracing), the models in DKT (Deep knowledge tracing) cannot be improved simply by introducing elaborate prior knowledge about the task domain. Instead, we need to observe how trained models behave and identify their shortcomings. In this paper, we examine a problem in existing models that have not been discussed previously: the inverted prediction problem, in which the model occasionally gives predictions that are opposite to a student's actual performance development. Specifically, given an input sequence where a student has solved several problems correctly in a row, the model will occasionally estimate his/her skills to be lower than when he/she could not solve them. To tackle this problem, we propose pre-training regularization, which incorporates prior knowledge by supplying synthetic sequences to the neural network before training it with real data. We provide regular, simplistic synthetic data to a sequence- processing neural network as a specific implementation of pre-training regularization. This method solves the inverted prediction problem and improves the performance of the model in terms of AUC. We observed its effect qualitatively and introduced a quantitative measure to assess the improvement also. For ASSISTments 2009, ASSISTments 2015, and Statics 2011, improvements in AUC scores were 0.2 ~ 0.7 %, which are significant considering the scores are already high (around 70~80%). We developed an open-source framework for DKT with pre- training regularization. It also contains user-friendly hyperparameter optimization functionality.

15e9926baa86af3bb05a9027f6927999?s=128

qqhann

November 24, 2021
Tweet

Transcript

  1. 2JVTIJ1"/5BSP5&;6," 6OJWFSTJUZPG5TVLVCB 1SJPS,OPXMFEHFPOUIF%ZOBNJDTPG4LJMM "DRVJTJUJPO*NQSPWFT%FFQ,OPXMFEHF 5SBDJOH

  2. #BDLHSPVOE

  3. #BDLHSPVOE  • *OF-FBSOJOHTZTUFNT FTUJNBUJOH TUVEFOUT`BDBEFNJDBCJMJUJFT JNQSPWFUIFJSMFBSOJOHFYQFSJFODF

  4. ,OPXMFEHF5SBDJOH $PSCFUU   • "UBTLUPNPEFMIPXUIFTUVEFOU`TBDBEFNJDBCJMJUZ DIBOHFTPWFSUJNF • ,5BJNTUPFTUJNBUFJGIFTIFXJMMBOTXFSUIFOFYU RVFTUJPODPSSFDUMZ

    q = 13 a = q = 15 a = q = 41 a = ? … q = 13 a = q = 38 a = 4FRVFODF
  5. % %BUBTFU"44*45NFOUTDPSSFDUFE  • &BDIRVFTUJPOJTMBCFMFECZBTLJMM ,$ LOPXMFEHF DPODFQU UIBUJTSFRVJSFEUPTPMWFJU •

    %BUBTFUTDPOUBJOTRVFTUJPOBOTXFSJOHMPHT 4LJMM*% $POUFOU q = 13 a = q = 15 a = q = 41 a = … q = 13 a = q = 38 a = $POUFOUGSPN "44*45NFOUTDPSSFDUFE %BUBTFU 4FRVFODF 4FRVFODF 4FRVFODF
  6. %FFQ,OPXMFEHF5SBDJOH 1JFDI   *OQVU JODMVEFTJOGPSNBUJPOPGRVFTUJPO*% BOE UIFBOTXFSSFTVMU XIFSF JTUIFOVNCFSPGTLJMMT

    xt qt ∈ {0,…, m} at ∈ {0,1} m qt a = 0 = qt a = 1 =
  7. *OWFSUFEQSFEJDUJPOQSPCMFN   • &YJTUJOH%,5NPEFMTPDDBTJPOBMMZQSFEJDUFEQFSGPSNBODFVOGBJSMZ   5SBJOFE%,5 5SBJOFE%,5

  8. *OWFSUFEQSFEJDUJPOQSPCMFN   1SFEJDUFE1FSGPSNBODF 1SFEJDUFE1FSGPSNBODF 1SFEJDUFE1FSGPSNBODF • 8FJOWFTUJHBUFEUSBJOFE%,5NPEFMTXJUIEVNNZEBUBTFRVFODF 
 

    JG BOE PUIFSXJTF sq k = ((q, a1), (q, a2), …, (q, aT)) at = 1 t > T − k at = 0
  9. *OWFSUFEQSFEJDUJPOQSPCMFN   • 8FJOWFTUJHBUFEUSBJOFE%,5NPEFMTXJUIEVNNZEBUBTFRVFODF 
  JG BOE PUIFSXJTF

    sq k = ((q, a1), (q, a2), …, (q, aT)) at = 1 t > T − k at = 0  1SFEJDUFE1FSGPSNBODF 0.0 s0 s20 s10 s0 s20 s10 s0 s20 s10 0.5 1.0 0.0 0.5 1.0 0.0 0.5 1.0 1SFEJDUJPODVSWF B  q = KC30 1SFEJDUJPODVSWF C  q = KC83 1SFEJDUJPODVSWF D  q = KC98
  10. .FUIPET

  11. 2VBOUJ fi DBUJPOPGUIFJOWFSUFEQSFEJDUJPOQSPCMFN   • $MFBSBOESPVHIJOEJDBUPS r1 = 1

    Q ∑ rq 1 = 1 Q ∑Q q=1 1 { ̂ y(sq T , q)> ̂ y(sq 0 , q)} rq=KC98 1 = 0 rq=KC83 1 = 1 rq=KC30 1 = 1  1SFEJDUFE1FSGPSNBODF 0.0 s0 s20 s10 s0 s20 s10 s0 s20 s10 0.5 1.0 0.0 0.5 1.0 0.0 0.5 1.0 1SFEJDUJPODVSWF B  q = KC30 1SFEJDUJPODVSWF C  q = KC83 1SFEJDUJPODVSWF D  q = KC98
  12. 2VBOUJ fi DBUJPOPGUIFJOWFSUFEQSFEJDUJPOQSPCMFN   • &YQSFTTEFHSFF r2 = 1

    Q ∑ rq 2 = 1 Q ∑Q q=1 NDCG ( ̂ y (sq 0 , q), …, ̂ y (sq T , q), (0,…, T)) rq=KC98 2 = 0.51 rq=KC83 2 = 0.74 rq=KC30 2 = 0.83  1SFEJDUFE1FSGPSNBODF 0.0 s0 s20 s10 s0 s20 s10 s0 s20 s10 0.5 1.0 0.0 0.5 1.0 0.0 0.5 1.0 1SFEJDUJPODVSWF B  q = KC30 1SFEJDUJPODVSWF C  q = KC83 1SFEJDUJPODVSWF D  q = KC98
  13. 1SFUSBJOJOHSFHVMBSJ[BUJPO  • "MMXSPOHBOEBMMDPSSFDUTZOUIFUJDEBUBGPSFBDITLJMMT • 1SJPSLOPXMFEHFUIBU   X′ 

    wro = {(qi ,0), …, (qi ,0)}, y′  wro = (qi ,0) X′  cor = {(qi ,1), …, (qi ,1)}, y′  cor = (qi ,1) qi a = qi a = qi a = qi a = qi a = qi a = qi a = qi a = … …
  14. 1SFUSBJOJOHSFHVMBSJ[BUJPO  qi a = qi a = qi a

    = qi a = qi a = qi a = qi a = qi a = q1 a = q1 a = q2 a = q2 a = qt a = qt+1 a = qt+1 a = qt a = 4ZOUIFUJDEBUB 3FBMEBUB … … … … 1SFUSBJOJOH 5SBJOJOH
  15. 3FTVMUT

  16. 1FSGPSNBODFUBCMF 

  17. 1SFEJDUJPODVSWFT  • .BOZTLJMMTXJUIXPSTU CFDBNFNPOPUPOJDBMMZJODSFBTJOH r1 , r2 QSFUSBJO QSFUSBJO

    1SFEJDUFE1FSGPSNBODF
  18. /%$(EJTUSJCVUJPO  • /FBSPQUJNBMTLJMMT /%$(TDPSF JODSFBTFEJOBMMEBUBTFUT rq 2 ≈ 1

    'SFRVFODZ /%$(TDPSF /%$(TDPSF /%$(TDPSF /%$(TDPSF "TTJTUNFOU "TTJTUNFOU 4ZOUIFUJD 4UBUJDT QSFUSBJO QSFUSBJO
  19. -FBOJOHDVSWF  • )JHIFSJOJUJBM"6$BGUFSQSFUSBJOJOH • 'FXFSJUFSBUJPOTMFBEUPFBSMZTUPQQJOHBUIJHIFS fi OBM"6$ "6$ &QPDI

    &QPDI &QPDI &QPDI "TTJTUNFOU "TTJTUNFOU 4ZOUIFUJD 4UBUJDT QSFUSBJO QSFUSBJO
  20. & ff FDUPGTFRVFODFMFOHUI  • -POHFSTFRVFODFTBSFNPSFMJLFMZUPTV ff FSGSPN JOWFSUFEQSFEJDUJPOQSPCMFN QSFUSBJO

    QSFUSBJO r2 r1
  21. $PODMVTJPO

  22. $PODMVTJPO  • 1PJOUFEPVUUIFJOWFSUFEQSFEJDUJPOQSPCMFN • 1SPQPTFEBRVBOUJUBUJWFXBZPGNFBTVSJOHJUTTFWFSJUZ • *OUSPEVDFEQSFUSBJOJOHUPPWFSDPNFUIFQSPCMFN • &NQJSJDBMMZWFSJ

    fi FEJUTF ff FDUJWFOFTT
  23. 5IBOLZPV

  24. 3FGFSFODFT • <>$PSCFUU "5BOE"OEFSTPO +3,OPXMFEHFUSBDJOH.PEFMJOHUIFBDRVJTJUJPO PGQSPDFEVSBMLOPXMFEHF 6TFS.PEFMJOHBOE6TFSBEBQUFE*OUFSBDUJPO 7PM /P 

    QQr   • <>1JFDI $ #BTTFO + )VBOH + (BOHVMJ 4 4BIBNJ . (VJCBT -+BOE4PIM %JDLTUFJO +%FFQLOPXMFEHFUSBDJOH "EWBODFTJO/FVSBM*OGPSNBUJPO1SPDFTTJOH 4ZTUFNT QQr   • <>:FVOH $,BOE:FVOH %:"EESFTTJOHUXPQSPCMFNTJOEFFQLOPXMFEHF USBDJOHWJBQSFEJDUJPODPOTJTUFOUSFHVMBSJ[BUJPO BS9JWQSFQSJOUBS9JW