Efficiently Exploring (Multilevel) Data: Lessons from the Data Mining Community

Transcript

1. Introduction Decision Trees Forests Multilevel Difficulties Efficiently Exploring (Multilevel) Data Lessons from the Data Mining Community Daniel P. Martin University of Virginia Design and Data Analysis October 30, 2014 Daniel P. Martin 1 of 37 Efficiently Exploring (Multilevel) Data
2. Introduction Decision Trees Forests Multilevel Difficulties Acknowledgements Timo, Sara, Brian, & Vivian IES VEST fellowship the quant area! Daniel P. Martin 2 of 37 Efficiently Exploring (Multilevel) Data
3. Introduction Decision Trees Forests Multilevel Difficulties Outline Why exploratory research? Introduction to recursive partitioning Walkthrough of an applied example Challenges with multilevel data Daniel P. Martin 3 of 37 Efficiently Exploring (Multilevel) Data
4. Introduction Decision Trees Forests Multilevel Difficulties The term “exploratory” is considered by many as less than an approach to data analysis and more a confession of guilt. -Jack McArdle, 2014 Daniel P. Martin 4 of 37 Efficiently Exploring (Multilevel) Data
5. Introduction Decision Trees Forests Multilevel Difficulties Exploratory vs. Confirmatory Exploratory Findings, Confirmatory Conclusions = BAD Source: Wagenmakers et al. (2012) Daniel P. Martin 5 of 37 Efficiently Exploring (Multilevel) Data
6. Introduction Decision Trees Forests Multilevel Difficulties The Answer Follow the 3 R’s: replicate, replicate, replicate Daniel P. Martin 6 of 37 Efficiently Exploring (Multilevel) Data
7. Introduction Decision Trees Forests Multilevel Difficulties The Answer...? Daniel P. Martin 7 of 37 Efficiently Exploring (Multilevel) Data
8. Introduction Decision Trees Forests Multilevel Difficulties A Proposed Solution: Predictive Modeling Needs: can parse a large number of variables requires minimal model specification Daniel P. Martin 8 of 37 Efficiently Exploring (Multilevel) Data
9. Introduction Decision Trees Forests Multilevel Difficulties A Proposed Solution: Predictive Modeling Needs: can parse a large number of variables requires minimal model specification handles non-independence Daniel P. Martin 8 of 37 Efficiently Exploring (Multilevel) Data
10. Introduction Decision Trees Forests Multilevel Difficulties A Proposed Solution: Predictive Modeling Needs: can parse a large number of variables requires minimal model specification handles non-independence One Solution: recursive partitioning Daniel P. Martin 8 of 37 Efficiently Exploring (Multilevel) Data
11. Introduction Decision Trees Forests Multilevel Difficulties An Introduction to Decision Trees 0 50 100 150 200 0 5 10 15 20 25 Years Hits salary Lowest Low High Highest Daniel P. Martin 9 of 37 Efficiently Exploring (Multilevel) Data
12. Introduction Decision Trees Forests Multilevel Difficulties An Introduction to Decision Trees 0 50 100 150 200 0 5 10 15 20 25 Years Hits salary Lowest Low High Highest Daniel P. Martin 10 of 37 Efficiently Exploring (Multilevel) Data
13. Introduction Decision Trees Forests Multilevel Difficulties An Introduction to Decision Trees 0 50 100 150 200 0 5 10 15 20 25 Years Hits salary Lowest Low High Highest Daniel P. Martin 11 of 37 Efficiently Exploring (Multilevel) Data
14. Introduction Decision Trees Forests Multilevel Difficulties An Introduction to Decision Trees Daniel P. Martin 12 of 37 Efficiently Exploring (Multilevel) Data
15. Introduction Decision Trees Forests Multilevel Difficulties CART: Breiman (1984) pseudocode The general algorithm: 1. Search all variables for splits in a greedy, top-down manner Daniel P. Martin 13 of 37 Efficiently Exploring (Multilevel) Data
16. Introduction Decision Trees Forests Multilevel Difficulties CART: Breiman (1984) pseudocode The general algorithm: 1. Search all variables for splits in a greedy, top-down manner 2. Identify the best split by some criterion Daniel P. Martin 13 of 37 Efficiently Exploring (Multilevel) Data
17. Introduction Decision Trees Forests Multilevel Difficulties CART: Breiman (1984) pseudocode The general algorithm: 1. Search all variables for splits in a greedy, top-down manner 2. Identify the best split by some criterion 3. Split the sample on this threshold, resulting in two child nodes Daniel P. Martin 13 of 37 Efficiently Exploring (Multilevel) Data
18. Introduction Decision Trees Forests Multilevel Difficulties CART: Breiman (1984) pseudocode The general algorithm: 1. Search all variables for splits in a greedy, top-down manner 2. Identify the best split by some criterion 3. Split the sample on this threshold, resulting in two child nodes 4. Repeat steps 1-3 on the resulting sub-groups (nodes) until some stopping criterion is reached Daniel P. Martin 13 of 37 Efficiently Exploring (Multilevel) Data
19. Introduction Decision Trees Forests Multilevel Difficulties Detour: The Bias-Variance Tradeoff training, testing, and cross-validation Source: An Introduction to Statistical Learning Daniel P. Martin 14 of 37 Efficiently Exploring (Multilevel) Data
20. Introduction Decision Trees Forests Multilevel Difficulties Detour: The Bias-Variance Tradeoff training, testing, and cross-validation Daniel P. Martin 15 of 37 Efficiently Exploring (Multilevel) Data
21. Introduction Decision Trees Forests Multilevel Difficulties CART: Breiman (1984) pseudocode The general algorithm: 1. Search all variables for splits in a greedy, top-down manner 2. Identify the best split by some criterion 3. Split the sample on this threshold, resulting in two child nodes 4. Repeat steps 1-3 on the resulting sub-groups (nodes) until some stopping criterion is reached Daniel P. Martin 16 of 37 Efficiently Exploring (Multilevel) Data
22. Introduction Decision Trees Forests Multilevel Difficulties CART: Breiman (1984) pseudocode The general algorithm: 1. Search all variables for splits in a greedy, top-down manner 2. Identify the best split by some criterion 3. Split the sample on this threshold, resulting in two child nodes 4. Repeat steps 1-3 on the resulting sub-groups (nodes) until some stopping criterion is reached 5. Prune tree using cross-validation Daniel P. Martin 16 of 37 Efficiently Exploring (Multilevel) Data
23. Introduction Decision Trees Forests Multilevel Difficulties CART: Breiman (1984) in action! CHmRun 1 < 91 ≥ 91 CAtBat 2 < 1452 ≥ 1452 CHits 3 < 182 ≥ 182 Node 4 (n = 30) 0 500 1000 1500 2000 2500 Node 5 (n = 20) 0 500 1000 1500 2000 2500 Hits 6 < 103.5 ≥ 103.5 Node 7 (n = 18) 0 500 1000 1500 2000 2500 Node 8 (n = 21) 0 500 1000 1500 2000 2500 Runs 9 < 59.5 ≥ 59.5 Node 10 (n = 18) 0 500 1000 1500 2000 2500 CHmRun 11 < 256.5 ≥ 256.5 Node 12 (n = 17) 0 500 1000 1500 2000 2500 Node 13 (n = 8) 0 500 1000 1500 2000 2500 Daniel P. Martin 17 of 37 Efficiently Exploring (Multilevel) Data
24. Introduction Decision Trees Forests Multilevel Difficulties CART: Breiman (1984) in action! CHmRun 1 < 91 ≥ 91 CAtBat 2 < 1452 ≥ 1452 Node 3 (n = 50) 0 500 1000 1500 2000 2500 Node 4 (n = 39) 0 500 1000 1500 2000 2500 Runs 5 < 59.5 ≥ 59.5 Node 6 (n = 18) 0 500 1000 1500 2000 2500 Node 7 (n = 25) 0 500 1000 1500 2000 2500 Daniel P. Martin 18 of 37 Efficiently Exploring (Multilevel) Data
25. Introduction Decision Trees Forests Multilevel Difficulties CART: Breiman (1984) in action! CHmRun 1 < 91 ≥ 91 CAtBat 2 < 1452 ≥ 1452 Node 3 (n = 50) 0 500 1000 1500 2000 2500 Node 4 (n = 39) 0 500 1000 1500 2000 2500 Runs 5 < 59.5 ≥ 59.5 Node 6 (n = 18) 0 500 1000 1500 2000 2500 Node 7 (n = 25) 0 500 1000 1500 2000 2500 MSEtest =123 Daniel P. Martin 18 of 37 Efficiently Exploring (Multilevel) Data
26. Introduction Decision Trees Forests Multilevel Difficulties CART: Breiman (1984) pros/cons Pros: intuitive, easy to explain and visualize can handle continuous or categorical outcomes non-parametric, robust to outliers no model specification required Daniel P. Martin 19 of 37 Efficiently Exploring (Multilevel) Data
27. Introduction Decision Trees Forests Multilevel Difficulties CART: Breiman (1984) pros/cons Pros: intuitive, easy to explain and visualize can handle continuous or categorical outcomes non-parametric, robust to outliers no model specification required Cons: prone to overfitting biased toward variables with many possible splits typically outperformed by regression techniques Daniel P. Martin 19 of 37 Efficiently Exploring (Multilevel) Data
28. Introduction Decision Trees Forests Multilevel Difficulties Conditional Inference: Hothorn, Hornik, & Zeileis (2006) pseudocode The general algorithm: 1. Test global null hypothesis that variables are independent of the outcome Daniel P. Martin 20 of 37 Efficiently Exploring (Multilevel) Data
29. Introduction Decision Trees Forests Multilevel Difficulties Conditional Inference: Hothorn, Hornik, & Zeileis (2006) pseudocode The general algorithm: 1. Test global null hypothesis that variables are independent of the outcome 2. If rejected, identify the best variable to split on in a permutation test framework Daniel P. Martin 20 of 37 Efficiently Exploring (Multilevel) Data
30. Introduction Decision Trees Forests Multilevel Difficulties Conditional Inference: Hothorn, Hornik, & Zeileis (2006) pseudocode The general algorithm: 1. Test global null hypothesis that variables are independent of the outcome 2. If rejected, identify the best variable to split on in a permutation test framework 3. Identify the best split and create two child nodes Daniel P. Martin 20 of 37 Efficiently Exploring (Multilevel) Data
31. Introduction Decision Trees Forests Multilevel Difficulties Conditional Inference: Hothorn, Hornik, & Zeileis (2006) pseudocode The general algorithm: 1. Test global null hypothesis that variables are independent of the outcome 2. If rejected, identify the best variable to split on in a permutation test framework 3. Identify the best split and create two child nodes 4. Repeat steps 1-3 on the resulting sub-groups (nodes) until global null hypothesis is no longer rejected Daniel P. Martin 20 of 37 Efficiently Exploring (Multilevel) Data
32. Introduction Decision Trees Forests Multilevel Difficulties Conditional Inference: Hothorn, Hornik, & Zeileis (2006) in action! CHmRun p < 0.001 1 ≤ 90 > 90 CHits p < 0.001 2 ≤ 377 > 377 CAtBat p < 0.001 3 ≤ 815 > 815 CRuns p < 0.001 4 ≤ 61 > 61 Node 5 (n = 18) 0 500 1000 1500 2000 2500 Node 6 (n = 16) 0 500 1000 1500 2000 2500 Node 7 (n = 18) 0 500 1000 1500 2000 2500 Node 8 (n = 37) 0 500 1000 1500 2000 2500 Runs p = 0.002 9 ≤ 58 > 58 Node 10 (n = 18) 0 500 1000 1500 2000 2500 Node 11 (n = 25) 0 500 1000 1500 2000 2500 MSEtest =128 Daniel P. Martin 21 of 37 Efficiently Exploring (Multilevel) Data
33. Introduction Decision Trees Forests Multilevel Difficulties An Introduction to Decision Trees conditional inference pros/cons Pros: same benefits as CART unbiased in selecting splits no pruning needed Daniel P. Martin 22 of 37 Efficiently Exploring (Multilevel) Data
34. Introduction Decision Trees Forests Multilevel Difficulties An Introduction to Decision Trees conditional inference pros/cons Pros: same benefits as CART unbiased in selecting splits no pruning needed Cons: prone to overfitting less efficient than CART iid assumptions regarding permutation tests typically outperformed by regression techniques Daniel P. Martin 22 of 37 Efficiently Exploring (Multilevel) Data
35. Introduction Decision Trees Forests Multilevel Difficulties Creating Ensembles of Trees the basic idea Daniel P. Martin 23 of 37 Efficiently Exploring (Multilevel) Data
36. Introduction Decision Trees Forests Multilevel Difficulties Creating Ensembles of Trees why it works Source: Scott Fortmann-Roe Daniel P. Martin 24 of 37 Efficiently Exploring (Multilevel) Data
37. Introduction Decision Trees Forests Multilevel Difficulties Random Forests: Breiman (2001) in action! NewLeague League Division Assists Errors Years Runs HmRun AtBat Walks CWalks PutOuts Hits RBI CAtBat CRuns CHits CRBI CHmRun 0e+00 1e+06 2e+06 3e+06 4e+06 Variable Importance IncNodePurity Daniel P. Martin 25 of 37 Efficiently Exploring (Multilevel) Data
38. Introduction Decision Trees Forests Multilevel Difficulties Conditional Inference Forests: Strobl et al. (2007) in action! NewLeague League Errors Division Assists HmRun PutOuts Years CWalks Walks AtBat Runs CAtBat RBI Hits CHits CRuns CHmRun CRBI 0 10000 20000 30000 Importance Variable Daniel P. Martin 26 of 37 Efficiently Exploring (Multilevel) Data
39. Introduction Decision Trees Forests Multilevel Difficulties Visualizing Random Forests - Unpacking the Black Box CART: Top 5 Predictors CHmRun CRBI CHits CRuns CAtBat 0 500 1000 1500 0 500 1000 Actual Predicted 0 200 400 0 500 1000 1500 0 1000 2000 3000 4000 0 500 1000 1500 2000 0 5000 10000 Value Salary Daniel P. Martin 27 of 37 Efficiently Exploring (Multilevel) Data
40. Introduction Decision Trees Forests Multilevel Difficulties Visualizing Random Forests - Unpacking the Black Box Conditional Inference: Top 5 Predictors CRBI CHmRun CRuns CHits Hits 0 500 1000 1500 0 250 500 750 1000 1250 Actual Predicted 0 500 1000 1500 0 200 400 0 500 1000 1500 2000 0 1000 2000 3000 4000 0 50 100 150 200 Value Salary Daniel P. Martin 28 of 37 Efficiently Exploring (Multilevel) Data
41. Introduction Decision Trees Forests Multilevel Difficulties Visualizing Random Forests - Unpacking the Black Box CART: Bottom 5 Predictors Errors Assists Division League NewLeague 0 500 1000 0 300 600 Actual Predicted 0 10 20 30 0 100 200 300 400 500 1.00 1.25 1.50 1.75 2.00 1.00 1.25 1.50 1.75 2.00 1.00 1.25 1.50 1.75 2.00 Value Salary Daniel P. Martin 29 of 37 Efficiently Exploring (Multilevel) Data
42. Introduction Decision Trees Forests Multilevel Difficulties Comparing Forests to Trees MSEcart = 123 MSEctree =128 Daniel P. Martin 30 of 37 Efficiently Exploring (Multilevel) Data
43. Introduction Decision Trees Forests Multilevel Difficulties Comparing Forests to Trees MSEcart = 123 MSEctree =128 MSErforest = 90 MSEcforest = 94 Daniel P. Martin 30 of 37 Efficiently Exploring (Multilevel) Data
44. Introduction Decision Trees Forests Multilevel Difficulties Issues in Multilevel Applications cart vs. conditional inference MEANSES 1 < −0.25 ≥ −0.25 Node 2 (n = 873) −5 0 5 10 15 20 25 SES 3 < 0.31 ≥ 0.31 Node 4 (n = 1420) −5 0 5 10 15 20 25 Node 5 (n = 1131) −5 0 5 10 15 20 25 MSEcart = 41 Daniel P. Martin 31 of 37 Efficiently Exploring (Multilevel) Data
45. Introduction Decision Trees Forests Multilevel Difficulties Issues in Multilevel Applications cart vs. conditional inference SES p < 0.001 1 ≤ −0.32 > −0.32 MEANSES p < 0.001 2 ≤ −0.28 > −0.28 SES p < 0.001 3 ≤ −1.72 > −1.72 Node 4 (n = 45) −5 0 5 10 15 20 25 MEANSES p = 0.004 5 ≤ −0.7 > −0.7 Node 6 (n = 146) −5 0 5 10 15 20 25 Sex p = 0.005 7 Male Female Node 8 (n = 197) −5 0 5 10 15 20 25 Node 9 (n = 183) −5 0 5 10 15 20 25 SES p < 0.001 10 ≤ −0.94 > −0.94 Node 11 (n = 158) −5 0 5 10 15 20 25 Minority p = 0.027 12 No Yes Sex p = 0.041 13 Male Female Node 14 (n = 182) −5 0 5 10 15 20 25 Node 15 (n = 196) −5 0 5 10 15 20 25 Node 16 (n = 111) −5 0 5 10 15 20 25 MEANSES p < 0.001 17 ≤ −0.28 > −0.28 Minority p < 0.001 18 No Yes Node 19 (n = 175) −5 0 5 10 15 20 25 Node 20 (n = 127) −5 0 5 10 15 20 25 Minority p < 0.001 21 No Yes SES p < 0.001 22 ≤ 0.3 > 0.3 Node 23 (n = 639) −5 0 5 10 15 20 25 SES p = 0.004 24 ≤ 0.84 > 0.84 Node 25 (n = 552) −5 0 5 10 15 20 25 Node 26 (n = 444) −5 0 5 10 15 20 25 Node 27 (n = 269) −5 0 5 10 15 20 25 MSEctree = 39 Daniel P. Martin 32 of 37 Efficiently Exploring (Multilevel) Data
46. Introduction Decision Trees Forests Multilevel Difficulties Issues in Multilevel Applications out-of-bag error Sex Minority MEANSES SES 0 2000 4000 6000 8000 10000 12000 train_rforest IncNodePurity Daniel P. Martin 33 of 37 Efficiently Exploring (Multilevel) Data
47. Introduction Decision Trees Forests Multilevel Difficulties Issues in Multilevel Applications out-of-bag error Daniel P. Martin 34 of 37 Efficiently Exploring (Multilevel) Data
48. Introduction Decision Trees Forests Multilevel Difficulties Issues in Multilevel Applications out-of-bag error Pnotselected = (1 − 1 n )n limn→∞ P = 1 e ≈ 0.368 Daniel P. Martin 34 of 37 Efficiently Exploring (Multilevel) Data
49. Introduction Decision Trees Forests Multilevel Difficulties Issues in Multilevel Applications out-of-bag error MSEtest = 40.2295 MSEOOB = 37.7666 Daniel P. Martin 35 of 37 Efficiently Exploring (Multilevel) Data
50. Introduction Decision Trees Forests Multilevel Difficulties Issues in Multilevel Applications out-of-bag error MSEtest = 40.2295 MSEOOB = 37.7666 MSEtest −MSEOOB MSEtest = 0.06 Daniel P. Martin 35 of 37 Efficiently Exploring (Multilevel) Data
51. Introduction Decision Trees Forests Multilevel Difficulties Open Questions Under varying levels of non-independence: When do trees begin to break down? Daniel P. Martin 36 of 37 Efficiently Exploring (Multilevel) Data
52. Introduction Decision Trees Forests Multilevel Difficulties Open Questions Under varying levels of non-independence: When do trees begin to break down? How do conditional inference forests perform? Daniel P. Martin 36 of 37 Efficiently Exploring (Multilevel) Data
53. Introduction Decision Trees Forests Multilevel Difficulties Open Questions Under varying levels of non-independence: When do trees begin to break down? How do conditional inference forests perform? Can random forests yield good variable importances with OOB error? Daniel P. Martin 36 of 37 Efficiently Exploring (Multilevel) Data
54. Introduction Decision Trees Forests Multilevel Difficulties Open Questions Under varying levels of non-independence: When do trees begin to break down? How do conditional inference forests perform? Can random forests yield good variable importances with OOB error? How do these methods perform with missing data present? Daniel P. Martin 36 of 37 Efficiently Exploring (Multilevel) Data
55. Introduction Decision Trees Forests Multilevel Difficulties Open Questions Under varying levels of non-independence: When do trees begin to break down? How do conditional inference forests perform? Can random forests yield good variable importances with OOB error? How do these methods perform with missing data present? Stay tuned! Daniel P. Martin 36 of 37 Efficiently Exploring (Multilevel) Data
56. Introduction Decision Trees Forests Multilevel Difficulties Thank you Daniel P. Martin 37 of 37 Efficiently Exploring (Multilevel) Data