Efficiently Exploring Multilevel Data with Recursive Partitioning: A Dissertation Proposal

Transcript

1. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Efficiently Exploring Multilevel Data with Recursive Partitioning Daniel P. Martin University of Virginia Department of Psychology February 2, 2015 Daniel P. Martin 1 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
2. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Acknowledgements My Committee - Brian Nosek - Sara Rimm-Kaufman - Timo von Oertzen - Vivian Wong Implicit Social Cognition Lab - Jordan Axt - Charlie Ebersole - Becca Frazier - Anup Gampa - Kelly Hoffman - Calvin Lai - Erin Westgate Math Psy Lab - Bommae Kim - Joey Meyer - Siny Tsang Social Development Lab - Holland Banse - Michelle Ko - Micela Leis - Carol Paxton - Lia Sandilos - Julie Thomas IES Training Grant Daniel P. Martin 2 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
3. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Outline Introduction Recursive Partitioning Previous Research Multilevel Issues The Proposal Daniel P. Martin 3 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
4. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Once upon a time statisticians only explored. Then they learned to confirm exactly - to confirm a few things exactly, each under very specific circumstances. As they emphasized exact confirmation, their techniques inevitably became less flexible. The connection of the most used techniques with past insights was weakened. Anything to which a confirmatory procedure was not explicitly attached was decried as “mere descriptive statistics”, no matter how much we had learned from it. -John Tukey, 1977 Daniel P. Martin 4 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
5. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal The term “exploratory” is considered by many as less than an approach to data analysis and more a confession of guilt. -Jack McArdle, 2014 Daniel P. Martin 5 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
6. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Exploratory vs. Confirmatory: A Continuum Source: Wagenmakers et al. (2012) Daniel P. Martin 6 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
7. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal The Answer For Some Source: The Open Science Framework: osf.io Daniel P. Martin 7 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
8. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal The Answer For All? Daniel P. Martin 8 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
9. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal What Comes After Confirmatory Tests? Try data-driven exploration with NHST Daniel P. Martin 9 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
10. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal What Comes After Confirmatory Tests? Move on to more interesting projects Daniel P. Martin 10 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
11. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Need: Efficient Exploration Daniel P. Martin 11 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
12. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Need: Efficient Exploration Daniel P. Martin 12 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
13. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Need: Efficient Exploration Daniel P. Martin 13 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
14. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Need: Efficient Exploration Daniel P. Martin 14 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
15. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Need: Efficient Exploration Daniel P. Martin 15 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
16. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal An Introduction to Decision Trees 0 25 50 75 100 5000 10000 15000 20000 Out of State Tuition Percent of Students in Top 10% of High School Graduation Rate: Poor Moderate Good Great Daniel P. Martin 16 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
17. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal An Introduction to Decision Trees 0 25 50 75 100 5000 10000 15000 20000 Out of State Tuition Percent of Students in Top 10% of High School Graduation Rate: Poor Moderate Good Great RSStotal ? < RSSpart1 + RSSpart2 Daniel P. Martin 16 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
18. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal An Introduction to Decision Trees 0 25 50 75 100 5000 10000 15000 20000 Out of State Tuition Percent of Students in Top 10% of High School Graduation Rate: Poor Moderate Good Great 229 ? < 19 + 195 = 215 Daniel P. Martin 17 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
19. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal An Introduction to Decision Trees 0 25 50 75 100 5000 10000 15000 20000 Out of State Tuition Percent of Students in Top 10% of High School Graduation Rate: Poor Moderate Good Great 229 ? < 104 + 63 = 168 Daniel P. Martin 18 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
20. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal An Introduction to Decision Trees 0 25 50 75 100 5000 10000 15000 20000 Out of State Tuition Percent of Students in Top 10% of High School Graduation Rate: Poor Moderate Good Great Outstate 1 < 10572 ≥ 10572 Outstate 2 < 8558 ≥ 8558 Top10perc 3 < 30.5 ≥ 30.5 Node 4 (n = 232) 0 20 40 60 80 100 120 Node 5 (n = 49) 0 20 40 60 80 100 120 Node 6 (n = 151) 0 20 40 60 80 100 120 Outstate 7 < 16791 ≥ 16791 Top10perc 8 < 17.5 ≥ 17.5 Node 9 (n = 46) 0 20 40 60 80 100 120 Node 10 (n = 225) 0 20 40 60 80 100 120 Top10perc 11 < 70.5 ≥ 70.5 Node 12 (n = 50) 0 20 40 60 80 100 120 Node 13 (n = 24) 0 20 40 60 80 100 120 Daniel P. Martin 19 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
21. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Detour: The Bias-Variance Tradeoff training, testing, and cross-validation Source: An Introduction to Statistical Learning Daniel P. Martin 20 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
22. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Detour: The Bias-Variance Tradeoff training, testing, and cross-validation Daniel P. Martin 21 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
23. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Decision Tree Pseudocode CART: Breiman et al. (1984) 1. Search all variables for splits in a greedy, top-down manner Daniel P. Martin 22 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
24. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Decision Tree Pseudocode CART: Breiman et al. (1984) 1. Search all variables for splits in a greedy, top-down manner 2. Identify the best split by some criterion Daniel P. Martin 22 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
25. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Decision Tree Pseudocode CART: Breiman et al. (1984) 1. Search all variables for splits in a greedy, top-down manner 2. Identify the best split by some criterion 3. Split the sample on this threshold, resulting in two child nodes Daniel P. Martin 22 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
26. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Decision Tree Pseudocode CART: Breiman et al. (1984) 1. Search all variables for splits in a greedy, top-down manner 2. Identify the best split by some criterion 3. Split the sample on this threshold, resulting in two child nodes 4. Repeat steps 1-3 on the resulting nodes until some stopping criterion is reached Daniel P. Martin 22 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
27. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Decision Tree Pseudocode CART: Breiman et al. (1984) 1. Search all variables for splits in a greedy, top-down manner 2. Identify the best split by some criterion 3. Split the sample on this threshold, resulting in two child nodes 4. Repeat steps 1-3 on the resulting nodes until some stopping criterion is reached 5. Prune tree using cross-validation Daniel P. Martin 22 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
28. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Decision Tree Pseudocode Conditional Inference Trees (Hothorn et al., 2006) 1. Test global null hypothesis that variables are independent of the outcome Daniel P. Martin 23 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
29. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Decision Tree Pseudocode Conditional Inference Trees (Hothorn et al., 2006) 1. Test global null hypothesis that variables are independent of the outcome 2. If rejected, identify the best variable to split on in a permutation test framework Daniel P. Martin 23 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
30. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Decision Tree Pseudocode Conditional Inference Trees (Hothorn et al., 2006) 1. Test global null hypothesis that variables are independent of the outcome 2. If rejected, identify the best variable to split on in a permutation test framework 3. Identify the best split and create two child nodes Daniel P. Martin 23 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
31. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Decision Tree Pseudocode Conditional Inference Trees (Hothorn et al., 2006) 1. Test global null hypothesis that variables are independent of the outcome 2. If rejected, identify the best variable to split on in a permutation test framework 3. Identify the best split and create two child nodes 4. Repeat steps 1-3 on the resulting child nodes until global null hypothesis is no longer rejected Daniel P. Martin 23 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
32. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Recap: Pros and Cons of Decision Trees Pros: intuitive, easy to explain and visualize can handle continuous or categorical outcomes non-parametric, robust to outliers no model specification required Daniel P. Martin 24 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
33. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Recap: Pros and Cons of Decision Trees Pros: intuitive, easy to explain and visualize can handle continuous or categorical outcomes non-parametric, robust to outliers no model specification required Cons: biased toward variables with many possible splits (CART) more computationally expensive (conditional inference) typically outperformed by regression techniques prone to overfitting Daniel P. Martin 24 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
34. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Creating Ensembles of Trees O < 10568 ≥ 10568 T < 21.5 ≥ 21.5 O < 7624 ≥ 7624 T < 11.5 ≥ 11.5 terminal node terminal node terminal node O < 9020 ≥ 9020 T < 31.5 ≥ 31.5 terminal node terminal node terminal node O < 17913 ≥ 17913 T < 6.5 ≥ 6.5 terminal node terminal node terminal node O < 10509 ≥ 10509 O < 6394 ≥ 6394 T < 27 ≥ 27 terminal node terminal node T < 5.5 ≥ 5.5 terminal node T ≥ 9.5 < 9.5 T < 19.5 ≥ 19.5 T ≥ 14.5 < 14.5 terminal node O < 9741 ≥ 9741 terminal node terminal node terminal node terminal node T < 45.5 ≥ 45.5 T < 17.5≥ 17.5 terminal node terminal node terminal node O < 9380 ≥ 9380 T < 32.5 ≥ 32.5 O < 7397 ≥ 7397 terminal node terminal node terminal node T < 61 ≥ 61 T < 24.5 ≥ 24.5 terminal node terminal node terminal node O < 11454 ≥ 11454 O < 7800 ≥ 7800 T < 33.5 ≥ 33.5 terminal node terminal node T < 21.5 ≥ 21.5 O ≥ 7852 < 7852 O < 8297≥ 8297 terminal node terminal node terminal node terminal node O < 17196 ≥ 17196 T < 14.5 ≥ 14.5 terminal node T < 72.5 ≥ 72.5 terminal node terminal node terminal node Daniel P. Martin 25 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
35. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Creating Ensembles of Trees why it works Source: Scott Fortmann-Roe Daniel P. Martin 26 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
36. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Interpreting the Black Box 1. Variable Importance 2. Partial Dependence Plots Daniel P. Martin 27 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
37. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Not a “Magic” Solution Daniel P. Martin 28 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
38. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Not a “Magic” Solution Random forests make no general assumptions regarding independence, and thus have the potential to be used for multilevel EDA with little added complexity Daniel P. Martin 28 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
39. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Previous Research Applying recursive partitioning to multilevel data is not new RE-EM and MERT techniques can be inflexible ignore problems commonly found in multilevel data Daniel P. Martin 29 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
40. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Previous Research Applying recursive partitioning to multilevel data is not new RE-EM and MERT techniques can be inflexible ignore problems commonly found in multilevel data Why not just use “traditional” recursive partitioning methods? Daniel P. Martin 29 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
41. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Proof of Concept 0 20 40 T1 T2 T3 T4 T5 Time Score Daniel P. Martin 30 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
42. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Proof of Concept High SES: (10, 4) Low SES: (8, 0.5) 0 20 40 T1 T2 T3 T4 T5 T1 T2 T3 T4 T5 Time Score Daniel P. Martin 31 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
43. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Proof of Concept V4 V5 V8 V6 V7 V2 V1 V3 Time SES 2000 4000 6000 8000 Increase in Node Purity Variable Variable Importance High SES Low SES −10 0 10 20 30 40 50 T1 T2 T3 T4 T5 Time Mean Prediction Partial Dependence Daniel P. Martin 32 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
44. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Issue 1: CART biased variable selection single level (N = 1000) Categorical (k = 2) Categorical (k = 3) Ordinal (k = 5) Ordinal (k = 7) Categorical (k = 10) Continuous 50 100 150 200 250 Increase in Node Purity Variable Variable Importance Daniel P. Martin 33 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
45. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Issue 1: CART biased variable selection multilevel (N = 1000, L2/L1 = 100/10) L2 Categorical (k = 2) L1 Categorical (k = 2) L2 Ordinal (k = 5) L1 Ordinal (k = 5) L2 Continuous L1 Continuous 50 100 150 200 250 Increase in Node Purity Variable Variable Importance Daniel P. Martin 34 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
46. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Issue 2: The Breakdown of Conditional Inference...? CART: MSEtrain = 37.09 MSEtest = 42.85 Daniel P. Martin 35 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
47. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Issue 2: The Breakdown of Conditional Inference...? CART: MSEtrain = 37.09 MSEtest = 42.85 Conditional Inference: MSEtrain = 35.58 MSEtest = 43.97 Daniel P. Martin 35 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
48. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Issue 2: The Breakdown of Conditional Inference...? SES p < 0.001 1 High SES: (10, 4)Low SES: (8, 0.5) Time p < 0.001 2 ≤ T3 > T3 Time p < 0.001 3 ≤ T2 > T2 Time p < 0.001 4 ≤ T1 > T1 Node 5 (n = 50) −10 0 10 20 30 40 Node 6 (n = 50) −10 0 10 20 30 40 Node 7 (n = 50) −10 0 10 20 30 40 Node 8 (n = 100) −10 0 10 20 30 40 Node 9 (n = 250) −10 0 10 20 30 40 Daniel P. Martin 36 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
49. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Issue 3: Underestimation of OOB error Pnotselected = (1 − 1 n )n limn→∞ P = 1 e ≈ 0.368 Daniel P. Martin 37 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
50. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Issue 3: Underestimation of OOB error MSEtest = 48.84 MSEOOB = 23.95 Daniel P. Martin 38 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
51. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Open Questions Under varying levels of non-independence: Can CART still rank variables correctly despite potential bias? Daniel P. Martin 39 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
52. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Open Questions Under varying levels of non-independence: Can CART still rank variables correctly despite potential bias? When do conditional inference methods begin to break down? Daniel P. Martin 39 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
53. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Open Questions Under varying levels of non-independence: Can CART still rank variables correctly despite potential bias? When do conditional inference methods begin to break down? Can forests yield reliable variable importances with underestimated OOB error? Daniel P. Martin 39 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
54. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Open Questions Under varying levels of non-independence: Can CART still rank variables correctly despite potential bias? When do conditional inference methods begin to break down? Can forests yield reliable variable importances with underestimated OOB error? How do these methods perform with missing data present? Daniel P. Martin 39 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
55. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Open Questions Under varying levels of non-independence: Can CART still rank variables correctly despite potential bias? When do conditional inference methods begin to break down? Can forests yield reliable variable importances with underestimated OOB error? How do these methods perform with missing data present? Are recursive partitioning methods feasible/useful for multilevel EDA? Daniel P. Martin 39 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
56. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal The Proposal: Simulation Simulation Parameter Number of Categories Category Values Number of levels 1 2 Nature of outcome 1 continuous Sample size (L1/L2) 4 5/20, 15/20, 15/100, 100/50 Intra-class correlation 3 0, 0.15, 0.30 Missingness (MCAR) 3 0, 10%, 30% Covariate level 2 level-1 only, both levels 96 (4 × 3 × 4 × 2) conditions 200 repetitions 19,200 total data sets Daniel P. Martin 40 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
57. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal The Proposal: Application Data Set 1: Math Achievement (Bliese, 2013) 7,185 students nested in 160 schools predictors include students per school, whether the school is public/Catholic, and student demographics large sample size, no missing data, and a continuous outcome Daniel P. Martin 41 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
58. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal The Proposal: Application Data Set 2: Fidelity of Implementation (Rimm-Kaufman et al., 2014) 139 teachers in 24 schools predictors include baseline teacher self-efficacy in instruction and classroom management, and their attitudes toward their principal, schools programs, and teaching in general small sample, moderate rates of missingness, and a continuous outcome Daniel P. Martin 42 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
59. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal The Proposal: Application Data Set 3: Student Achievement (Allen et al., 2011) 960 students and 68 teachers classes split across Math, English, History, and Science predictors include student-reports (motivation, engagement, etc.), teacher-reports (opinions of professional development, student learning, etc.), and teacher instruction quality larger sample size, minimal missingness, longitudinal predictors, and a categorical outcome Daniel P. Martin 43 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
60. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal The Proposal: Dissemination Will conduct a small workshop followed by a group interview with prepared questions for qualitative feedback to help construct an R package and tutorial sample R package (IAT) sample R markdown tutorial Daniel P. Martin 44 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
61. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal The Proposal: Timeline Daniel P. Martin 45 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
62. Introduction Recursive Partitioning Previous Research Multilevel Issues Proposal Thank you Daniel P. Martin 46 of 46 Efficiently Exploring Multilevel Data with Recursive Partitioning
63. Additional Material Possible Simulated Variable Relationships With any simulation study, there will be some tinkering involved Meaningful continuous variable with non-linear relationship Meaningful categorical variable (L1 or L2) Meaningful interaction (L1 only or cross-level) Null continuous variables Null ordinal variables (k = 5) Null categorical variables (k = 2 - 5) Daniel P. Martin 1 of 10 Efficiently Exploring Multilevel Data with Recursive Partitioning
64. Additional Material Math Behind CART Categorical outcome: Let ˆ pjk correspond to the proportion of observations in the training set in the jth subsection that are from the kth class. Classification error = 1 − maxk(ˆ pjk) Gini = K k=1 ˆ pjk(1 − ˆ pjk) Entropy = − K k=1 ˆ pjklog(ˆ pjk) Daniel P. Martin 2 of 10 Efficiently Exploring Multilevel Data with Recursive Partitioning
65. Additional Material Math Behind Conditional Inference Step 1: Global Hypothesis Test H0 = p j=1 Hj 0 , where Hj 0 : D(Y |Xj ) = D(Y ) Is tested with: Tj (Ln, w) = vec n i=1 wi gj (Xj,i )h(Yi )T ∈ Rpj q and standardized by: c(t, µ, Σ) = maxk=1,...,pq (t−µ)k √ Σkk Daniel P. Martin 3 of 10 Efficiently Exploring Multilevel Data with Recursive Partitioning
66. Additional Material Math Behind Conditional Inference Step 2: Variable Selection If the global test is rejected, then the variable with the lowest p-value calculated in Step 1 is selected for splitting. Daniel P. Martin 4 of 10 Efficiently Exploring Multilevel Data with Recursive Partitioning
67. Additional Material Math Behind Conditional Inference Step 3: Split Point Selection First use a modified test statistic: TA j∗ (Ln, w) = vec n i=1 wi I(Xj∗,i ∈ A)h(Yi )T ∈ Rq Then, find the split point that yields the largest standardized value: A∗ = argmax A c(tA j∗ , µA j∗ , ΣA j∗ ) Daniel P. Martin 5 of 10 Efficiently Exploring Multilevel Data with Recursive Partitioning
68. Additional Material Math Behind Conditional Inference Step 4: Repeat The weights within each node are re-normalized, and these steps are repeated until the global null hypothesis is no longer rejected in each subsection. Daniel P. Martin 6 of 10 Efficiently Exploring Multilevel Data with Recursive Partitioning
69. Additional Material Math Behind Partial Dependence The partial dependence of the S predictors on the predictive function f (X) is: fS (XS ) = EXC [f (XS , XC )] and can be estimated by ¯ fS (XS ) = 1 N N i=1 [f (XS , XCi )] where {xC1, xC2, . . . , xCN} are the values of XC ocurring over all observations in the training data. Daniel P. Martin 7 of 10 Efficiently Exploring Multilevel Data with Recursive Partitioning
70. Additional Material Imputation By Proximity The proximities form an NxN matrix, and calculated as follows: 1. Put both training and oob data into a given tree 2. If two cases are in the same terminal node, increase their proximity by one 3. Normalize the proximities by dividing by the number of trees For missing data: 1. Impute missing values using medians and levels of the highest frequency 2. Run a random forest model 3. Update missing values to a weighted mean of the observations or category with the largest average proximity 4. Repeat 5-10 times Daniel P. Martin 8 of 10 Efficiently Exploring Multilevel Data with Recursive Partitioning
71. Additional Material Bias-Variance Decomposition Proof E[(y − ˆ f (x))2] = E[(f (x) − E[ˆ f (x)])2] + E[(ˆ f (x) − E[ˆ f (x)])2] + E[ 2] = Bias(ˆ f (x))2 + Variance(ˆ f (x)) + σ2 MSE is composed of three terms: squared bias - error caused by parsimonious assumptions variance - how much the approximation moves around the mean irreducible error - noise in the true model that cannot be captured Daniel P. Martin 9 of 10 Efficiently Exploring Multilevel Data with Recursive Partitioning
72. Additional Material Bias-Variance Decomposition Note that any random variable X with mean µ, we have E[X2] = E[(X − µ)2] + µ2. Also, E[ ] = 0 and f is deterministic. MSE = E[(y − ˆ f 2] = E[y2] + E[ ˆ f 2] − 2E[yˆ f ] = E[(y − f )2] + f 2 + E[(ˆ f − E[ˆ f ])2] + E[ˆ f 2] − 2E[(f + )ˆ f ] = σ2 + f 2 + Var[ˆ f ] + E[ˆ f 2] − 2f E[ˆ f ] − 2E[ f ] = σ2 + (f − E[ˆ f ])2 + Var[ˆ f ] = σ2 + Bias(ˆ f (x))2 + Var(ˆ f (x)) Daniel P. Martin 10 of 10 Efficiently Exploring Multilevel Data with Recursive Partitioning