Iowa Project Lead The Way: Implications for Research, Policy, & Practice

Transcript

1. IOWA PROJECT LEAD THE WAY IMPLICATIONS FOR RESEARCH, POLICY, &

   PRACTICE Tom Schenk, Jr., Iowa Department of Education David Rethwisch, University of Iowa Soko Starobin, Iowa State University Melissa Chapman Haynes, PDA, Inc. & Consultant Frankie Santos Laanan, Iowa State University Presented at CESMEE Brown Bag Seminar October 5, 2011

2. BACKGROUND  Expanding the pool of engineers and scientists has

   been a focus for the U.S. higher education in the past decades.  Project Lead The Way (PLTW)  Aims to provides middle school and high school students with a seamless path to college and career success in STEM-related fields.  Aims to expand the proportion of students who persist in STEM fields. 2

3. PROJECT LEAD THE WAY  Sequence of year-long pre-engineering courses

   for secondary students.  Problem-based/Project-based Learning (PBL) approach to increase engagement & performance.  Provides “dual-credit” for students in either CTE or A&S areas depending upon performance. 3

4. MIDDLE SCHOOL PROGRAM: GATEWAY TO TECHNOLOGY  Automation and Robotics

   (9 wks)  Design and Modeling (9 wks)  Energy and the Environment (9 wks)  The Magic of Electrons (9 wks)  The Science of Technology (9 wks)  Flight and Space (9 wks) 4

5. HIGH SCHOOL COURSE PROGRAM: PATHWAY TO ENGINEERING 5 Foundation: ---------------------------------------------------------------------------------------------------------------------------------

   Specialization: --------------------------------------------------------------------------------------------------------------------------------- Capstone: Digital Electronics Biotechnical Engineering Computer Integrated Manufacturing Civil Engineering and Architecture Aerospace Engineering Introduction to Engineering Design Principles Of Engineering Engineering Design and Development Note: Course program requires college prep mathematics each year.

6. 3 PHASE PROFESSIONAL DEVELOPMENT 6 Ready for core training Ready

   for teaching • Gateway To Technology (Middle School) • Introduction To Engineering Design • Principles of Engineering • Biotechnical Engineering • Digital Electronics • Computer Integrated Manufacturing • Civil Engineering and Architecture • Aerospace Engineering • Engineering Design and Development Core Training-Summer Institute Self- Assessment and Pre-Core Training PLTW Continuous Training Master Teacher Virtual Academy University Based PD Level II Training

7. SUMMER TRAINING  Iowa State University  Gateway to Technology

    Introduction to Engineering Design  Digital Electronics  University of Iowa  Principles of Engineering  Biotechnology Engineering  Civil Engineering & Architecture 7

8. NUMBERS  188 high school and community college teachers 

   126 High schools or centers  38 Middle school teachers  24 Middle schools  2067 High school students (2008/9) 8

9. DATA SOURCES & METHODS  A part of a large-scale,

   statewide research project  PLTW and non-PLTW students who graduated from high schools in Iowa in 2009  Data sources 1: Iowa Dept. of Ed. K-12 Data; 2: Community College MIS 3: Regent University Partnership; 4: National Student Clearinghouse 9

10. HYPOTHETICAL MODEL Demographic Characteristics Ethnicity Gender Free/Reduced Lunch Gifted/Talented Academic

    Backgrounds ITED Math & Science Scores Grades of HS Courses Status of PLTW Participation Transition to Higher Ed No College 2-year College 4-year College 10 Selection Bias

11. IMBALANCE – DEMOGRAPHICS 20 40 60 80 100 Non-PLTW PLTW

    Caucasian 20 40 60 80 100 Non-PLTW PLTW Male 20 40 60 80 100 Non-PLTW PLTW Free/Reduced Lunch 20 40 60 80 100 Non-PLTW PLTW IEP 20 40 60 80 100 Non-PLTW PLTW Section 504 20 40 60 80 100 Non-PLTW PLTW Gifted & Talented 11

12. 8TH GRADE MATHEMATICS SCORES 12

13. 8TH GRADE SCIENCE SCORES 13

14. SCHOOL DISTRICT SELECTION BIAS 14 O denotes PLTW district; X

    denotes non-PLTW districts

15. PROPENSITY SCORES  τ = PLTW participation  Race/Ethnicity 

    Free/Reduced Lunch  IEP / Section 504  Gifted & Talented  8th Grade ITBS subtest in Math, Science, & Reading 15 ρ(τ)=τ = φ(α + βX + ε)

16. PREDICTING ENTRY: Coefficient z-statistic ITBS, 8th Grade Standard Score Reading

    -0.003 -1.86 Math 0.015 8.69 Science 0.008 4.90 Demographics Black -0.6 -1.45 Asian 0.0 -0.11 Hispanic -0.3 -0.71 Male 1.8 21.72 Economic Status Proxy Free Lunch -0.2 -2.47 Reduced Lunch 0.0 -0.14 Homeless -0.4 -0.69 Special Populations IEP -1.0 -5.77 Section 504 -0.5 -1.42 Gifted & Talented 0.3 4.28 16

17. NEAREST NEIGHBOR MATCHING 17 0 0.1 0.2 0.3 0.4 0.5

    0.6 Estimated propensity E[p(X)]: PLTW: solid, non-PLTW: dashed Treated Units from lowest to highest estimated propensity score

18. MATCHING METHODS 18 Participants Non-participants dNN,i = ║xτ,i – x

    τ’,i ║ D = Σ di Local minima dG,i = wi ║xτ,i – x τ’,i ║ D = Σ di Global minima NEAREST NEIGHBOR GENETIC ALGORITHMS w = {w1 ,…,wn }

19. MATCHING TRADE-OFF 19

20. VARIATIONS IN MATCHING ALGORITHMS Matching Methodology Prior to Matching Nearest

    Neighbor with Exact Matching on Districts Genetic Algorithm One-to-One Genetic Algorithm One-to-Two Propensity Score Distance 0.953 0.036 -0.001 0.002 Cohort -0.067 -0.075 0 0.001 ITBS, 8th Grade Reading 0.597 0.106 -0.002 -0.005 Mathematics 0.891 0.103 0.001 0.004 Science 0.719 0.148 0.002 0.007 Demographics White 0.353 -0.044 0 0 African American -0.386 0.034 0 0 Asian/Pacific Islander 0.027 -0.024 0 0 Hispanic -0.224 0.05 0 0 American Indian -0.097 0.052 0 0 Male 0.972 -0.051 0 0.002 Economic Status Proxy Free Lunch -0.405 -0.035 0 0 Reduced Lunch -0.064 0.011 0 0.003 Homeless -0.068 0.026 0 0 Special Populations IEP -0.563 -0.014 0 0.004 Section 504 Plan -0.086 0.052 0 0 Gifted & Talented 0.271 0.038 0.003 0 School District -0.049 0 NA NA Sample Size (ni ) 15,660 1,500 1,477 2,111 20

21. MATCHED TREATMENT AND CONTROL GROUP [GENETIC 1-2 MATCHING] Variable Weighted

    Mean Weighted Standard Deviation PLTW Participant 0.38 0.49 Demographics White 0.90 0.30 Black 0.03 0.17 Asian 0.04 0.18 Hispanic 0.03 0.17 American Indian 0.00 0.06 Male 0.85 0.36 Economic Status Proxy Free Lunch 0.11 0.31 Reduced Lunch 0.04 0.21 Homeless Status 0.00 0.06 Special Populations IEP 0.02 0.14 Section 504 0.01 0.09 Gifted Talented 0.23 0.42 11th Grade scores Reading 306.28 37.31 Mathematics 315.06 31.14 Science 320.28 36.43 8th Grade Scores Reading 277.48 30.72 Mathematics 283.72 26.66 Science 288.92 29.88 21

22. LITERATURE  Few published studies of the PLTW curriculum 

    PLTW students did significantly better on a mathematics and science assessment than career/technical students (Bottoms & Anthony, 2005)  Students matched only on race and gender  Tran & Nathan reported PLTW had no measurable impact on science scores and participants actually scored lower in mathematics compared to similar students  Study only included ca. 30 PLTW students 22

23. 23 PLTW IMPACT ON MATHEMATICS SCORES, 8TH TO 11TH GRADE

24. PLTW IMPACT ON SCIENCE SCORES, 8TH TO 11TH GRADE 24

25. PLTW IMPACT ON 11TH GRADE ITED MATH AND SCIENCE SCORES

    25 Test Score Change McFadden R2 t-score ITED Math Scores 5.2 6.1% 2.40 ITED Science Scores 5.2 4.7% 2.01

26. EX-POST – COURSE TAKING IN HS 0 10 20 30

    40 50 60 70 80 90 100 Non-PLTW PLTW 26

27. ESTIMATED IMPACT OF PLTW ON MATHEMATICS AND SCIENCE SCORES WITH

    COURSE COVARIATES 27

28. REMAINING QUESTIONS  Additional assessment instruments needed  Can we

    separate PLTW curriculum and other courses taken?  Assess problem-solving skills  Mechanical aptitude test  Student self-efficacy 28

29. HIGH SCHOOL GRADUATION AGENDA  Actual high school graduation rate

    is about 77% in United States (Heckman & LaFontaine, 2007).  High school dropouts/stop outs/non-graduates causes a loss of personal economic benefit around $60,000 (Rouse, 2005).  Dropouts generate less in tax revenues and create negative social externalities through greater likelihood of crime (Moretti, 2005).  Iowa’s graduation rate for the Class of 2009 was 87% (Iowa Department of Education, 2010). 29

30. DEFINING GRADUATION  Students who graduated within an Iowa school

    district within four years.  Includes students who transferred to another school district within Iowa.  Excludes students who may left Iowa (counted as non-graduate). 30

31. HIGH SCHOOL GRADUATION: CLASS OF 2009 [MATCHED] Control PLTW Participants

    (N = 623) (N = 416) Mean Standard Deviation Mean Standard Deviation Demographics Male 0.82 0.39 0.85 0.36 American Indian 0.00 0.04 0.00 0.00 Asian 0.02 0.14 0.02 0.14 Black 0.03 0.16 0.02 0.15 Hispanic 0.03 0.17 0.03 0.17 White 0.92 0.27 0.93 0.26 Economic Status Proxy Free Lunch 0.09 0.28 0.08 0.27 Reduced Lunch 0.07 0.25 0.06 0.24 Homeless 0.00 0.04 0.00 0.05 Special Populations Section 504 0.00 0.04 0.00 0.05 Gifted & Talented 0.18 0.39 0.20 0.40 IEP 0.04 0.20 0.04 0.19 Testing - 8th Grade ITBS Reading - Standard Score 274.32 32.76 275.25 31.86 Reading - Percentile Rank 71.30 22.72 72.18 22.07 Math - Standard Score 278.36 26.91 280.90 26.07 Math - Percentile Rank 75.95 20.27 78.00 19.25 Science - National Standard Score 285.86 30.24 288.89 30.39 Science - Percentile Rank 76.93 19.28 78.77 19.00 High School Graduation Graduated within 4 years 0.92 0.30 0.95 0.21 31

32. HIERARCHICAL LINEAR LOGIT REGRESSION 32 γ = φ(αj +υj +βj

    X+Δj τ+εij )  τ PLTW participation  Δj estimated impact of PLTW on graduation.  αj intercept for jth school district  υj error term for the intercept at the jth school district  Covariates include: Race/Ethnicity; Free/Reduced Lunch IEP / Section 504; Gifted & Talented; 8th Grade ITBS subtest in Math, Science, & Reading

33. HIGH SCHOOL GRADUATION RESULTS Genetic One-to-Two Genetic One-to-One Nearest Neighbor

    Estimate z-statistic Estimate t-statistic Estimate t-statistic Intercept (α) 0.00 0.00 -0.65 -0.41 -2.67 -1.90 PLTW Participant 0.64 1.91 0.79 2.39 0.59 2.20 Demographics: African-American -0.53 -1.07 -0.84 -1.49 -0.38 -0.54 Asian/Pacific Islander -0.69 -1.21 -0.73 -0.90 0.09 0.09 Hispanic -0.35 -0.60 -0.65 -1.00 -0.32 -0.51 American Indian 14.49 0.01 12.47 0.01 12.51 0.01 Male -1.37 -3.75 -2.20 -3.36 -1.18 -2.41 Economic Status Proxy Free Lunch Eligible -1.30 -4.97 -1.58 -4.80 -0.84 -2.40 Reduced Lunch Eligible -1.09 -3.30 -0.89 -1.92 -1.03 -2.57 Homeless -0.47 -1.17 -1.69 -1.09 13.34 0.01 Special Populations IEP -0.11 -0.09 -0.35 -0.64 0.05 0.12 Section 504 Plan 0.46 1.13 -2.50 -2.15 -0.63 -0.53 Gifted and Talented 13.98 0.01 0.73 1.25 1.05 1.40 8th Grade Testing Math 0.02 2.84 0.02 2.59 0.02 2.61 Science 0.01 2.00 0.01 1.16 0.01 1.05 Reading -0.01 -2.37 -0.01 -0.94 0.00 0.14 Additional Testing Controls: Midyear -0.31 -0.88 -0.58 -1.50 -0.36 -1.25 Spring -0.42 -0.80 -0.46 -0.77 -0.64 -1.41 33

34. INTERPRETING EFFECTS OF PLTW 34 Genetic 1-to-2 Genetic 1-to-1 Nearest

    Neighbor Coefficient 0.64 0.79 0.59 Odds Ratio 1.90 2.20 1.81 Marginal Effects 0.03 0.02 0.01 t-statistic 1.91 2.39 2.20 Interpreting logit regressions are complicated by the non-linear logit function (φ). Odds ratios show that the odds to graduate are between 90 and 120 higher for PLTW participants. The marginal effect shows this is an increase between 1 to 3 percent higher.

35. NONLINEAR INTERPRETATION 35 0 0.1 0.2 0.3 0.4 0.5 0.6

    0.7 0.8 0.9 1

36. MARGINAL EFFECTS ON HIGH SCHOOL GRADUATION BY ACHIEVEMENT TESTS 36

    Science Mathematics Maximum 0.01 0.01 3rd Quartile 0.02 0.02 Median 0.03 0.03 1st Quartile 0.04 0.04 Minimum 0.10 0.10 By adjusting for 8th grade test scores by quartile, but holding everything else constant, the marginal effect grows larger for students who were in the lower quartiles relative to other PLTW participants.

37. MARGINAL EFFECTS ON HIGH SCHOOL GRADUATION BY DEMOGRAPHICS 37 Males

    Females African Americans Asian/Pacific Islanders Hispanic Marginal Effect 0.03 0.01 0.04 0.05 0.04 By adjusting for demographics, while holding other variables constant, the impact on PLTW is stronger for males than females, while the impact by racial subgroup is consistent.

38. BACKGROUND  The United States is facing a crisis with

    regard to maintain the global leadership and competitiveness in science and technology.  In order to maintain the leadership and competitiveness, we must grow a strong, talented and innovative science and technology workforce.  Expanding the pool of engineers and scientists has been a focus for the U.S. higher education in the past decades. Specifically, the critical role of community college education in STEM fields has been recognized in the past several years. 38

39. THEORETICAL FRAMEWORK  Student Learning and Cognitive Development (Pascarella, 1985)

     student background and precollege characteristics  structural and organization characteristics of the institution  institutional environment  interactions with socializing agents  quality of student effort 39

40. THEORETICAL FRAMEWORK  Conceptualization of Student Retention (Hagedorn & Cepeda,

    2004; Hagedorn, Moon, Cypers, Maxwell & Lester, 2006; Hagedorn, Cypers & Lester, 2008)  The multiple varieties in retention (institutional retention; system retention; retention within a major or discipline; retention within the course)  A transfer student who leaves one institution to attend another should be included in retention calculation (system retention).  students’ academic success lead to a likelihood of transfer to a postsecondary education institution and have positive influence on student retention in college. 40

41. HYPOTHETICAL MODEL Demographic Characteristics Ethnicity Gender Free/Reduced Lunch Gifted/Talented Academic

    Backgrounds ITED Math & Science Scores Grades of HS Courses Status of PLTW Participation Transition to Higher Ed No College 2-year College 4-year College 41 Selection Bias

42. RESULTS – TRANSITION TO HE Postsecondary Enrollment PLTW Participation Total

    Non-PLTW PLTW n % n % 4-year 4,132 28.74 295 33.33 4,427 2-year 3,299 22.95 336 37.97 3,635 No College 6,944 48.31 254 28.70 7,198 Total 14,375 100.00 885 100.00 15,260 Table 1. Transition of PLTW and Non-PLTW Students to Higher Education 42

43. RESULTS – MULTINOMIAL REGRESSION Nearest Neighbor Genetic 1-to-1 Genetic 1-to-2

    2-year Entry Odds Ratio 1.64 1.62 1.57 t-statistic 2.29 2.12 2.30 4-year Entry Odds Ratio 1.08 1.04 0.94 t-statistic 0.33 0.17 -0.30  Propensity Score Matching  Nearest Neighbor  Genetic One treatment-to-One Control  Genetic One treatment-to-Two Control Table 2. Odds Ratio and t-Statistics of PLTW Participation by Methods of Propensity Score Matching (No College is Reference) 43

44. MULTINOMIAL REGRESSION Pr(2-year College) / Pr(No College) Pr(4-year College) /

    Pr(No College) Odds Ratio t-value Odds Ratio t-value PLTW 1.57 2.30 0.94 -0.30 Black 1.27 0.43 0.84 -0.24 Asian 1.14 0.23 0.72 -0.49 Hispanic 1.10 0.16 3.79 2.11 American Indian 7.79E-07 -8.75E+07 3.59E-07 -1.05E+08 Male 0.58 -2.15 0.64 -1.74 FreeLunch 0.48 -2.56 0.35 -2.90 Reduced Lunch 0.88 -0.35 0.50 -1.54 IEP 1.08 0.15 0.74 -2.45 Section504 2.77 1.53 0.18 -10.41 Gifted/Talented 0.82 -0.62 0.76 -0.99 Homeless Status 0.82 -3.21 0.67 -8.42 8th Grade ITS_Nat_Standard_Read 1.00 0.68 1.02 3.08 8th Grade ITS_Nat_Standard_Math 1.00 0.44 1.01 2.19 8th Grade ITS_Nat_Standard_Science 1.00 -0.83 1.00 -0.64 Course_Science_EarthSciences_Cumulative 1.38 2.39 1.07 0.45 Course_Science_Biology_Cumulative 1.04 0.28 1.01 0.07 Course_Science_Chemistry_Cumulative 1.47 2.06 2.16 3.63 Course_Science_Physics_Cumulative 1.02 0.11 1.64 2.74 Course_Science_SciEngTech_Cumulative 1.38 1.01 2.25 2.34 Course_Math_Geometry_Cumulative 0.94 -0.33 0.82 -0.86 Course_Math_Algebra1_Cumulative 1.06 0.35 0.86 -0.78 Course_Math_Algebra2_Cumulative 0.61 -2.65 1.09 0.42 Course_Math_Alg3Trig_Cumulative 1.69 2.25 2.15 3.30 Course_Math_Precalculus_Cumulative 1.23 0.72 1.49 1.45 Course_Math_Calculus_Cumulative 2.53 3.07 4.87 5.44 Course_Math_ProbStat_Cumulative 1.05 0.21 0.72 -1.23 Course_Math_IBMath_Cumulative 0.78 -0.66 2.01 2.20 Course_Math_BusinessTechnical_Cumulative 0.90 -0.40 0.50 -1.30 Course_Math_Other_Cumulative 0.46 -2.09 1.07 0.22 EASIER_GraduationStatusN 0.18 -4.23 0.02 -9.69 EASIER_GraduationStatusY 1.98 2.01 0.15 -5.14 44

45. CONCLUSION  PLTW were more likely to be high ability,

    white, and male – a subset of the population already likely to enter STEM fields. (Sax, Jacobs, & Riggers, 2010)  A higher percentage of PLTW students transitioned to higher education immediate after graduation.  PLTW seems to “cause” an increase in students attending community college, but not 4-year universities.  PLTW students were 57 percent more likely to transition to 2-year colleges compared to not attending any types of higher education institutions than non-PLTW students. 45

46. IMPLICATIONS FOR FUTURE RESEARCH  Examine students’ transition (transfer) from

    2-year to 4-year institutions.  Examine the effectiveness of PBL, which includes the element of professional development for teachers.  Examine students’ choice of major at 2-year institutions (STEM or non-STEM, or selected STEM disciplines: multinominal outcomes).  Examine the influence of educational credentials (e.g., AA, AAS, etc.) obtained at 2-year institutions on transfer and majoring in STEM. 46

47. [SPACE FOR IMPLICATIONS]  Implications for PLTW.  Implications for

    evaluation in education. 47