Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Explaining Human Cognition through Deep Learning

Amar
April 28, 2018
54

Explaining Human Cognition through Deep Learning

Amar

April 28, 2018
Tweet

Transcript

  1. What do we expect when we teach? • Some teachers

    believe their students should “really understand” • Others desire their students to “internalize knowledge” • Still others want their students to “grasp the core or essence” or “comprehend” • Specifically, what does a student do who "really understands" which he does not do when he does not understand? There was no common vocabulary among teachers to discuss their curriculum outcomes with other teachers.
  2. • Benjamin Bloom had observed that the curriculum planned by

    the teachers only catered to students low level of cognition. • In order for students to master or excel at a subject, they need to be taught lessons that would exercise their higher level of cognition. • Benjamin Bloom along with other fellow scientist’s developed a taxonomy • as a means for teachers to have a common vocabulary to define the curriculum outcomes in the cognitive area
  3. Bloom’s Taxonomy - The Cognitive Domain (1956) • Following taxonomy

    was proposed which categorizes various cognitive behaviors that were believed to be important in the process of learning
  4. • Taxonomy contained the categories ordered from simple to complex

    and from concrete to abstract. • The taxonomy was assumed to have a cumulative hierarchy; • mastery of a simple category was a prerequisite to the mastery of the next more complex category.
  5. • Taxonomy contained the categories ordered from simple to complex

    and from concrete to abstract. • The taxonomy allowed for some overlaps across the categories relaxing the cumulative hierarchical assumption • mastery of a simple category was not necessarily a prerequisite to the mastery of the next more complex category.
  6. Research Question • How do you validate these assumptions? •

    Are these assumptions even true? • Is Human Cognition so linear? • Is “Understanding” really a pre-requisite for “Applying”?
  7. Challenges • What is Cognition? How do you model it?

    • Need to design experiments – Need human subject • Too expensive • Ethical?
  8. Approach • From the student interactions and data – Learn

    computational model of student Cognition • Use this model as a virtual student – to simulate student behavior • Design and conduct experiments on this virtual student
  9. Computational Student Model • Need a sophisticated Knowledge Tracing Model

    • Knowledge Tracing Techniques • Bayesian Knowledge Tracing (BKT) • Performance Factor Analysis (PFA) • Item Response Theory (IRT) • None of these available Knowledge Tracing Techniques can capture • Dynamic relationships between different cognitive skills • Lack flexibility and predictive power
  10. How RNN Works? • Maintains an extensive hidden state (of

    knowledge) • Hidden state is a function of itself and input • Hidden state evolves with student’s performance • LSTMs: maintain a memory like students do • Both short term and long term
  11. How RNN/LSTM Works • Models the progression of student’s performance

    • Models the knowledge acquisition process of the student • Even trying to consider the knowledge retention, long term memory, short term memory and practice effects • Explores and figures out the links and knowledge dependencies between different skills • The skills may be very distant and unrelated • Skill Discovery
  12. funtoot • Intelligent Tutoring System • Learning by Answering Problems

    • Grade 2 to 9 • More than 100,000 students • More than 100 schools across India • Subjects: Mathematics and Science • Indian Boards of Education: CBSE, KASB, ICSE, …
  13. Funtoot: Knowledge Graph LG 1 LG 2 LG 3 Rules

    of Congruency Applications of Congruency contains contains Math Triangles Congruency contains contains depends on induces Subject Concept Sub-concept Sub-sub- concept Learning Gaps
  14. Data • 74 topics • 682 SSCs • 536 SSCs

    (having at least two btlos) • Btlo – (Revised) Bloom’s Taxonomy Learning Objective • 1,03,593 students • 10,158 problems • 41.7 million data points • 29.7 million solved (71 %) • 22.7 million solved-fast (54%)
  15. Deep Knowledge Tracing • We have 536 DKT models and

    btlos are used as features. • The input vector is a series of btlos (tagged to the problems attempted by the student) and their corresponding outcomes of the problem attempts: solved and solved-fast in our case. • For instance, if an ssc consists of three btlos, the input consists of three neurons per btlo, one representing whether the interaction belonged to that btlo, one for solved and one for solved-fast. • The model outputs the probability that the problem will be solved and solved-fast for each btlo, amounting to two output neurons per btlo.
  16. A hypothetical Example • An SSC – Remember, Understand, Apply

    Input Question Skill Response Output 0, 0, 0, 0, 0, 0 Q1 Understand 1 X, 1, X 0, 0, 1, 1, 0, 0 Q2 Remember 0 0, X, X 1, 0, 0, 0, 0, 0 Q3 Apply 0 X, X, 0 0, 0, 0, 0, 1, 0 Q4 Remember 1 1, X, X 1, 1, 0, 0, 0, 0 Q5 Understand 1 X, 1, X
  17. AUC

  18. Experiments: Initial Correlations Understand Apply Analyse Evaluate Create Remember 0.39

    (0.0) 0.14 (0.03) 0.1 (0.22) -0.24 (0.15) -0.58 (0.61) Understand 0.34 (0.0) 0.28 (0.0) 0.16 (0.21) 0.83 (0.02) Apply 0.22 (0.0) -0.07 (0.6) -0.23 (0.58) Analyse 0.36 (0.0) 0.7 (0.86) Evaluate 0.68 (0.09)
  19. Experiments: Final correlations Understand Apply Analyse Evaluate Create Remember 0.2

    (0.0) 0.08 (0.19) 0.04 (0.64) -0.21 (0.2) -1.0 (0.02) Understand 0.18 (0.0) 0.11 (0.08) 0.27 (0.03) 0.78 (0.04) Apply 0.14 (0.04) 0.25 (0.05) 0.77 (0.03) Analyse 0.11 (0.39) 0.49 (0.22) Evaluate 0.74 (0.06)
  20. Experiments • Effect of simple skill i on complex skill

    j • For an SSC having Remember, Understand, Apply and Analyse, • is Remember, Understand and Apply • is Remember and Understand
  21. Formula 1 - Results U Ap An E C R

    0.1 0.04 0.03 0.01 -0.08 U 0.05 0.03 0.02 0.06 Ap 0.07 0.03 0.08 An 0.05 0.1 E 0.14
  22. Formula 2 - Results U Ap An E C R

    - - - - - U 0.07 0.07 0.02 0.02 Ap 0.09 0.04 0.08 An 0.07 0.09 E 0.1
  23. Formula 3 - Results U Ap An E C R

    - - - - - U 0.03 0.04 0.01 0.1 Ap 0.04 0.02 0.06 An 0.02 0.02 E 0.07
  24. Conclusion • Revised Bloom’s Taxonomy does have a hierarchy when

    the skills are judged on their relative (median) complexity. • Clearly observed in the lower learning objectives - Remember, Understand and Apply • But, it is not a strict hierarchy in the sense that it allows for overlaps even among the non-adjacent skills, ensuring higher order skills do not subsume lower order skills.
  25. Critical Finding: Understand • Understand seems to correlate with almost

    all the higher order skills. Understand might be force fitted and seems out of place. • Understand was not included in the Original Taxonomy for the very similar reason that, Understand for all the practical purposes actually means anything from Comprehension to Synthesis. • Despite of that, Understand was included in the Revised Taxonomy considering its widespread usage as a synonym of Comprehension.
  26. References • Lalwani, A., Agrawal, S.: Validating revised bloom's taxonomy

    using deep knowledge tracing. 19th International Conference on Artificial Intelligence in Education (June 2018), (to appear)