Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What can student results tell us about school performance?

6ee4ce46deafee68da8ae793f7f10add?s=47 Dave Kinkead
December 09, 2015

What can student results tell us about school performance?

It seems likely that most people believe schooling somehow affects student ability. After all, we as a society invest significant amounts of time and money in various endeavours trying to measure exactly this. Yet these endeavours face an epistemic challenge. Because we can't measure the causal impact of schools directly, we can't know this causal impact with certainty. Instead, we infer the causal impact of schools on student ability by way of proxy measures such as student results. If student results improve, then we can infer that some aspect of schooling caused this. Perhaps.

How warranted is this inference from student results to school performance? With the aid of computer simulation, I investigate the robustness of this inference mechanism in a variety of common scenarios. Simulation allows us to stipulate causal mechanisms that cannot be observed in the real word and measure how well our empiric inferences map actual causes. I show that when selectivity, either by student or school, is present, the inference mechanism from student results to school performance is very poor. And if our causal inferences fail when causes are known, they must also fail when causes are not known.

6ee4ce46deafee68da8ae793f7f10add?s=128

Dave Kinkead

December 09, 2015
Tweet

Transcript

  1. WHAT CAN STUDENT RESULTS TELL US ABOUT SCHOOL PERFORMANCE? Dave

    Kinkead, University of Queensland
  2. WHICH SCHOOL DO WE CHOOSE? #PESACONF2015 #schoolperformance @davekinkead

  3. MEASURING SCHOOL PERFORMANCE IS IMPORTANT #PESACONF2015 #schoolperformance @davekinkead

  4. MEASURING SCHOOL PERFORMANCE IS DIFFICULT #PESACONF2015 #schoolperformance @davekinkead

  5. STUDENT RESULTS SCHOOL PERFORMANCE

  6. HOW ROBUST IS THIS INFERENCE? #PESACONF2015 #schoolperformance @davekinkead

  7. This is an argument for skepticism

  8. STUDENT RESULTS SCHOOL PERFORMANCE Assess the inference Simulate the process

  9. WHY COMPUTER SIMULATION? ➤ No direct empiric access to causal

    processes ➤ Stipulate causal relationships ➤ Specify ideal conditions ➤ Compare against known processes ➤ Act as a constrain to possible theories #PESACONF2015 #schoolperformance @davekinkead
  10. “ If an inference isn’t reliable under ideal settings, then

    it can’t be reliable under non-ideal settings.
  11. THE MODEL

  12. STUDENT & SCHOOLS ➤ Students have randomly assigned ‘ability’ (0.0

    - 1.0) ➤ Schools are collections of 1000 students ➤ Schools have stipulated ‘impact’ (0.0 - 1.0) ➤ School impact exclusively affects student ability ➤ School ‘performance’ perfectly tracks aggregate student ability #PESACONF2015 #schoolperformance @davekinkead
  13. SIMULATION PROCESS ➤ Students are randomly assigned to schools ➤

    In each tick… ➤ Schools ‘impact’ student ‘ability’ ➤ 20% of students graduate (leave the simulation) ➤ 20% of students enrol (with random ability) ➤ School ‘performance’ measured by aggregate student ability #PESACONF2015 #schoolperformance @davekinkead
  14. ENROLMENT ➤ The simulation has stipulated ‘selectivity’ (0.0 - 1.0)

    ➤ Students (selectivity %) will choose the top performing school ➤ Schools will accept the best students to maintain 1000 students each ➤ The remainder will be randomly enrolled. #PESACONF2015 #schoolperformance @davekinkead
  15. SANITY CHECKS ➤ Two schools of 1000 students ➤ No

    causal impact ➤ No selectivity ➤ dave.kinkead.com.au/school-performance/#sanity-check-1 ➤ With causal impact ➤ dave.kinkead.com.au/school-performance/#sanity-check-2 ➤ Student result to school performance inference is warranted
  16. SHIFTING AVERAGES ➤ Two schools ➤ No causal impact ➤

    50% selectivity ➤ http://dave.kinkead.com.au/school-performance/ #shifting-averages ➤ One school always ‘out-performs’ despite schools being identical ➤ Student results to school performance inference is NOT warranted #PESACONF2015 #schoolperformance @davekinkead
  17. DIFFERENTIAL PERFORMANCE ➤ Two schools ➤ Different causal impact ➤

    No selectivity ➤ http://dave.kinkead.com.au/school-performance/ #relative-1 ➤ Student results to school performance inference is warranted ➤ With selectivity ➤ http://dave.kinkead.com.au/school-performance/ #relative-2 ➤ Student results to school performance inference is NOT warranted #PESACONF2015 #schoolperformance @davekinkead
  18. WHAT CAN STUDENT RESULTS TELL US ABOUT SCHOOL PERFORMANCE? #PESACONF2015

    #schoolperformance @davekinkead
  19. “ If an inference isn’t reliable under ideal settings, then

    it can’t be reliable under non-ideal settings.
  20. The inference from student results to school performance is NOT

    reliable, even in ideal settings, whenever selection is present.
  21. #PESACONF2015 #schoolperformance @davekinkead ➤ Which measures of school performance are

    vulnerable? ➤ Static, aggregative measures. ➤ To what extent is selection present in reality? ➤ Private vs public ➤ Student vs School WHAT NOW?
  22. dave.kinkead.com.au/school-performance @davekinkead d.kinkead@uq.edu.au Questions?