Longitudinal Data and Higher Education Accountability

Initially presented at the Community College Leadership and Policy Brown Bag Seminar at Iowa State University in Ames, Iowa on September 27, 2009. Recording of presentation here: http://www.cclp.hs.iastate.edu/bb_schenk.html

Transcript

1. LONGITUDINAL DATA AND HIGHER EDUCATION ACCOUNTABILITY Tom Schenk Jr., Consultant
2. National Student Clearinghouse Student Records Student Courses Student Awards Employee Records Faculty Positions Unemployment Insurance Iowa College Student Aid Commission AS-28 [Program Information] K-12 [Project EASIER] [Iowa Testing Services] [CTE Plus]
3. PROJECT LEAD THE WAY EVALUATION
4. 11th Grade ITED: Math 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20% 0 10 20 30 40 50 60 70 80 90 Nonparticipants Average = 65 Participants Average = 79 4
5. 11th Grade ITED: Science 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20% 0 10 20 30 40 50 60 70 80 90 5
6. Project Lead The Way 6
7. Project Lead The Way 7
8. Selection Bias 8 Socio-demographic Background Cognitive ability School-level factors }
9. Propensity Score Matching 9 y = φ(x1 ,…,xn ) Binary outcome Logit Function
10. Propensity Bins 10 0% - 25% 25.1% - 50% 50.1% - 75% 75.1% - 100%
11. Subsequent Analysis 11 z = β1 x1 ,…, βn xn , βy y Outcome Propensity Bins
12. Enrollment by Gender 84% 16% 49% 51% Participants Nonparticipants Female Male 12
13. Percentage of Whites by Cohort 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% 2008 2009 2010 2011 Participants Nonparticipants 13
14. 8th Grade ITBS: Math 0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 0 10 20 30 40 50 60 70 80 90 Nonparticipants Average = 58 Participants Average = 80 14
15. 8th Grade ITBS: Science 0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 0 10 20 30 40 50 60 70 80 90 Nonparticipants Average = 61 Participants Average = 79 15
16. Evidence of Selection Bias Participants are more likely to be male, by a wide margin. Participation by gender holds constant around 90 percent. Students are in the 80th percentile in math and science. 16
17. 17 LABOR SUPPLY
18. Wages for employees in most sectors. Provided on a quarterly basis. Includes multiple jobs. Excludes workers in some sectors (e.g., military). Does not include hourly wages or full-time status. Only includes employees within Iowa. Unemployment Insurance 18
19. Matching Wage Data 19 • Educational data MIS • Remove students NSC • Match wage data IWD
20. Human Capital Theory 20 Wages Completers Direct Costs Time Leavers
21. Returns on Investment 21 Σ yi - xi (1+r)t t=1 T - C
22. Returns on Investment Track wages over a set period of time, T. Find the difference between wages between completers and leavers on the student level. Find the cost of tuition. If you assert an interest rate, r, then solve to find Net Present Value. If the interest rate if left unknown, then solve to find the Internal Rate of Return. 22
23. Methodology Assemble a cohort of graduates (completers) and those who left college without a degree (leavers). Stagger the cohorts so completers are finishing their final year in college as leavers are in their first year in the workforce. Exclude students who are found in any postsecondary institution. Estimate the tuition expenses for completers in their final year. Cohorts from 2002 and 2006. 23
24. Wages by Year 24 Net Present Value for Completers was $3,131. Internal Rate of Return was 6 percent.
25. Returns by Cluster 25 NPV
26. Returns by Select Clusters 26
28. In-state Retention Rate 28
29. In-state Out-of-state Working School Working School
30. HIGHER EDUCATION ACCOUNTABILITY
31. Slightly over 52 percent of first-time, full-time students entering in 2006 either transferred or graduated within three years (by 2008). Success Rate 31 0% 10% 20% 30% 40% 50% 60% 2005 2006
33. The three year graduation rate for first-time, full-time students rose slightly to 39.1 percent for the 2006 cohort. The graduation rate has fluctuated, but stayed ahead of the national graduation rate (33 percent). Graduation Rate 33 0% 10% 20% 30% 40% 50% 2001 2002 2003 2004 2005 2006
34. Time-to-degree rose slightly to 2.4 years for the 2004 cohort from 2.3 years. Time to Degree 34 0 1 2 2003 2004
35. Accountability Systems Measuring the effectiveness of institutions through student outcomes. Desirable qualities of accountability measures: 1. Specifically measures the effectiveness of the institution, not other factors. 2. Measures improvement. 3. Flexible to accommodate a variety of outcomes. The current traditional measures cannot capture these elements. 35
36. Accountability Systems Issues with graduation/success rate: 1. Denominator debate. 2. Selection bias. 3. Positive feedback mechanism. Solutions: Use the same methods that assist with Project Lead The Way evaluations. 36
37. 50% Estimating Success 37 50% 50% 50%
38. Simulated Example 38
39. OBTAINING DATA
40. Obtaining Data Contact your major professor and/or the Department of Education. Education data can be obtained from the Department of Education, other data (e.g., UI records) will need special permission. Data is distributed de-identified and must be returned or destroyed at the completion of the research study. Obtaining data is relatively low-cost, nominal fee (<$100) for a secure flash drive. 40
41. Tom Schenk Jr. Consultant – Institutional Effectiveness & Accountability Iowa Department of Education Phone: 515-281-3753 E-mail: tom.schenk@iowa.gov