Explaining Human Cognition through Deep Learning Amar Lalwani

What do we expect when we teach? • Some teachers believe their students should “really understand” • Others desire their students to “internalize knowledge” • Still others want their students to “grasp the core or essence” or “comprehend” • Specifically, what does a student do who "really understands" which he does not do when he does not understand? There was no common vocabulary among teachers to discuss their curriculum outcomes with other teachers.

• Benjamin Bloom had observed that the curriculum planned by the teachers only catered to students low level of cognition. • In order for students to master or excel at a subject, they need to be taught lessons that would exercise their higher level of cognition. • Benjamin Bloom along with other fellow scientist’s developed a taxonomy • as a means for teachers to have a common vocabulary to define the curriculum outcomes in the cognitive area

Bloom’s Taxonomy - The Cognitive Domain (1956) • Following taxonomy was proposed which categorizes various cognitive behaviors that were believed to be important in the process of learning

• Taxonomy contained the categories ordered from simple to complex and from concrete to abstract. • The taxonomy was assumed to have a cumulative hierarchy; • mastery of a simple category was a prerequisite to the mastery of the next more complex category.

Revised Bloom’s Taxonomy - The Cognitive Domain (2001)

• Taxonomy contained the categories ordered from simple to complex and from concrete to abstract. • The taxonomy allowed for some overlaps across the categories relaxing the cumulative hierarchical assumption • mastery of a simple category was not necessarily a prerequisite to the mastery of the next more complex category.

Research Question • How do you validate these assumptions? • Are these assumptions even true? • Is Human Cognition so linear? • Is “Understanding” really a pre-requisite for “Applying”?

Challenges • What is Cognition? How do you model it? • Need to design experiments – Need human subject • Too expensive • Ethical?

Approach • From the student interactions and data – Learn computational model of student Cognition • Use this model as a virtual student – to simulate student behavior • Design and conduct experiments on this virtual student

Computational Student Model • Need a sophisticated Knowledge Tracing Model • Knowledge Tracing Techniques • Bayesian Knowledge Tracing (BKT) • Performance Factor Analysis (PFA) • Item Response Theory (IRT) • None of these available Knowledge Tracing Techniques can capture • Dynamic relationships between different cognitive skills • Lack flexibility and predictive power

Deep Knowledge Tracing (DKT) • Recurrent Neural Networks (RNNs) • LSTMs (Long Short Term Memory Networks)

How RNN Works? • Maintains an extensive hidden state (of knowledge) • Hidden state is a function of itself and input • Hidden state evolves with student’s performance • LSTMs: maintain a memory like students do • Both short term and long term

How RNN/LSTM Works • Models the progression of student’s performance • Models the knowledge acquisition process of the student • Even trying to consider the knowledge retention, long term memory, short term memory and practice effects • Explores and figures out the links and knowledge dependencies between different skills • The skills may be very distant and unrelated • Skill Discovery

funtoot • Intelligent Tutoring System • Learning by Answering Problems • Grade 2 to 9 • More than 100,000 students • More than 100 schools across India • Subjects: Mathematics and Science • Indian Boards of Education: CBSE, KASB, ICSE, …

Funtoot: Knowledge Graph LG 1 LG 2 LG 3 Rules of Congruency Applications of Congruency contains contains Math Triangles Congruency contains contains depends on induces Subject Concept Sub-concept Sub-sub- concept Learning Gaps

Data • 74 topics • 682 SSCs • 536 SSCs (having at least two btlos) • Btlo – (Revised) Bloom’s Taxonomy Learning Objective • 1,03,593 students • 10,158 problems • 41.7 million data points • 29.7 million solved (71 %) • 22.7 million solved-fast (54%)

BTLO Distribution

Deep Knowledge Tracing • We have 536 DKT models and btlos are used as features. • The input vector is a series of btlos (tagged to the problems attempted by the student) and their corresponding outcomes of the problem attempts: solved and solved-fast in our case. • For instance, if an ssc consists of three btlos, the input consists of three neurons per btlo, one representing whether the interaction belonged to that btlo, one for solved and one for solved-fast. • The model outputs the probability that the problem will be solved and solved-fast for each btlo, amounting to two output neurons per btlo.

A hypothetical Example • An SSC – Remember, Understand, Apply Input Question Skill Response Output 0, 0, 0, 0, 0, 0 Q1 Understand 1 X, 1, X 0, 0, 1, 1, 0, 0 Q2 Remember 0 0, X, X 1, 0, 0, 0, 0, 0 Q3 Apply 0 X, X, 0 0, 0, 0, 0, 1, 0 Q4 Remember 1 1, X, X 1, 1, 0, 0, 0, 0 Q5 Understand 1 X, 1, X

AUC

Experiments: Initial Correlations Understand Apply Analyse Evaluate Create Remember 0.39 (0.0) 0.14 (0.03) 0.1 (0.22) -0.24 (0.15) -0.58 (0.61) Understand 0.34 (0.0) 0.28 (0.0) 0.16 (0.21) 0.83 (0.02) Apply 0.22 (0.0) -0.07 (0.6) -0.23 (0.58) Analyse 0.36 (0.0) 0.7 (0.86) Evaluate 0.68 (0.09)

Experiments: Final correlations Understand Apply Analyse Evaluate Create Remember 0.2 (0.0) 0.08 (0.19) 0.04 (0.64) -0.21 (0.2) -1.0 (0.02) Understand 0.18 (0.0) 0.11 (0.08) 0.27 (0.03) 0.78 (0.04) Apply 0.14 (0.04) 0.25 (0.05) 0.77 (0.03) Analyse 0.11 (0.39) 0.49 (0.22) Evaluate 0.74 (0.06)

Experiments • Effect of simple skill i on complex skill j • For an SSC having Remember, Understand, Apply and Analyse, • is Remember, Understand and Apply • is Remember and Understand

Formula 1 - Results U Ap An E C R 0.1 0.04 0.03 0.01 -0.08 U 0.05 0.03 0.02 0.06 Ap 0.07 0.03 0.08 An 0.05 0.1 E 0.14

Formula 2 - Results U Ap An E C R - - - - - U 0.07 0.07 0.02 0.02 Ap 0.09 0.04 0.08 An 0.07 0.09 E 0.1

Formula 3 - Results U Ap An E C R - - - - - U 0.03 0.04 0.01 0.1 Ap 0.04 0.02 0.06 An 0.02 0.02 E 0.07

Conclusion • Revised Bloom’s Taxonomy does have a hierarchy when the skills are judged on their relative (median) complexity. • Clearly observed in the lower learning objectives - Remember, Understand and Apply • But, it is not a strict hierarchy in the sense that it allows for overlaps even among the non-adjacent skills, ensuring higher order skills do not subsume lower order skills.

Critical Finding: Understand • Understand seems to correlate with almost all the higher order skills. Understand might be force fitted and seems out of place. • Understand was not included in the Original Taxonomy for the very similar reason that, Understand for all the practical purposes actually means anything from Comprehension to Synthesis. • Despite of that, Understand was included in the Revised Taxonomy considering its widespread usage as a synonym of Comprehension.

References • Lalwani, A., Agrawal, S.: Validating revised bloom's taxonomy using deep knowledge tracing. 19th International Conference on Artificial Intelligence in Education (June 2018), (to appear)