Bootstrap and Jackknife Statistical Techniques Dr. Abbie Stevens Michigan State University & University of Michigan [email protected] @abigailStev github.com/abigailstev 

Outline • Overview • Jackknife • Bootstrap • Compare/contrast with examples • Tutorial for trying them out and practicing IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 2 

Brute-force vs shiny and new IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 3 

Brute-force vs shiny and new • Bootstrap and jackknife are not fancy, and not new • Random re-sampling techniques for estimating error/accuracy (variance, bias) of data (samples) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 4 

Brute-force vs shiny and new • Bootstrap and jackknife are not fancy, and not new • Random re-sampling techniques for estimating error/accuracy (variance, bias) of data (samples) • Building and adding to your toolbox for problem-solving • Important to select the right tool for the job! • Commonly used in, e.g., biostatistics, industry data science, game science IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 5 

Brute-force vs shiny and new • Bootstrap and jackknife are not fancy, and not new • Random re-sampling techniques for estimating error/accuracy (variance, bias) of data (samples) • Building and adding to your toolbox for problem-solving • Important to select the right tool for the job! • Commonly used in, e.g., biostatistics, industry data science, game science • Sometimes you get stuck, and this stuff usually works, very well IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 6 

When to use them? • Check the stability (variance) and goodness of your choice of model and model complexity • Slight correlations between data points, don’t know the extent, doesn’t affect the underlying physics • Can’t (or don’t want to) calculate covariance matrix coefficients and derive error • Don’t know the statistical distribution of the parameter space, esp. if no full physical model to fit IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 7 

When to use them? • Check the stability (variance) and goodness of your choice of model and model complexity • Slight correlations between data points, don’t know the extent, doesn’t affect the underlying physics • Can’t (or don’t want to) calculate covariance matrix coefficients and derive error • Don’t know the statistical distribution of the parameter space, esp. if no full physical model to fit IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 8 When not to use them? • Very large data sets • Complex operators that take a long time to evaluate (unless you have a lot of compute power at your disposal!) • Use caution with non-linear and/or non-smooth or discrete operations, like the median, or distributions with long, asymmetric tails 

Jackknife basics • Name comes from a simple, useful little pocket knife • Developed by Maurice Quenouille and later John Tukey in the 1950s • M. Quenouille, “Notes on Bias in Estimation,” Biometrika, 1956 • J. Tukey, “Bias and Confidence in Not-quite Large Samples,” Annals of Mathematical Statistics, 1958 IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 9 Image credit: Tochanchai, Shutterstock via Collins Dictionary 

Jackknife basics • Name comes from a simple, useful little pocket knife • Developed by Maurice Quenouille and later John Tukey in the 1950s • M. Quenouille, “Notes on Bias in Estimation,” Biometrika, 1956 • J. Tukey, “Bias and Confidence in Not-quite Large Samples,” Annals of Mathematical Statistics, 1958 • Using an “estimator” to resample observations to estimate, e.g., the variance and bias • Sample mean is often the “estimator” for the population mean • Leaving out a single sample when re-sampling IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 10 Image credit: Tochanchai, Shutterstock via Collins Dictionary 

Jackknife steps 1. Start with your dataset of length n and the function F that you want to compute on it. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 11 

Jackknife steps 1. Start with your dataset of length n and the function F that you want to compute on it. 2. Re-sample your data without replacement n-1 times. • Without replacement and n-1 times are very important! IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 12 

Jackknife steps 1. Start with your dataset of length n and the function F that you want to compute on it. 2. Re-sample your data without replacement n-1 times. • Without replacement and n-1 times are very important! 3. Evaluate the function F on the resample. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 13 

Jackknife steps 1. Start with your dataset of length n and the function F that you want to compute on it. 2. Re-sample your data without replacement n-1 times. • Without replacement and n-1 times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n times, such that each data point has been left out once in all of those jackknife iterations. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 14 

Jackknife steps 1. Start with your dataset of length n and the function F that you want to compute on it. 2. Re-sample your data without replacement n-1 times. • Without replacement and n-1 times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n times, such that each data point has been left out once in all of those jackknife iterations. 5. Now you have n values of F from each jackknife iteration. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 15 

Jackknife steps 1. Start with your dataset of length n and the function F that you want to compute on it. 2. Re-sample your data without replacement n-1 times. • Without replacement and n-1 times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n times, such that each data point has been left out once in all of those jackknife iterations. 5. Now you have n values of F from each jackknife iteration. 6. Calculate the variance *(n-1) and standard error on the distribution of F resample values. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 16 

Jackknife example 1. Have some dataset x = (x1 , x2 , x3 , x4 , …, xn ) and function F = mean(x) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 17 

Jackknife example 1. Have some dataset x = (x1 , x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 18 

Jackknife example 1. Have some dataset x = (x1 , x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) 4. Repeat jackknife iterations! xb = (x1 , x3 , x4 , …, xn ), Fb = mean(xb ) xc = (x1 , x2 , x4 , …, xn ), Fc = mean(xc ) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 19 

Jackknife example 1. Have some dataset x = (x1 , x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) 4. Repeat jackknife iterations! xb = (x1 , x3 , x4 , …, xn ), Fb = mean(xb ) xc = (x1 , x2 , x4 , …, xn ), Fc = mean(xc ) 5. Now we have F from the original dataset, and Fa , Fb , Fc , …, Fn IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 20 

Jackknife example 1. Have some dataset x = (x1 , x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) 4. Repeat jackknife iterations! xb = (x1 , x3 , x4 , …, xn ), Fb = mean(xb ) xc = (x1 , x2 , x4 , …, xn ), Fc = mean(xc ) 5. Now we have F from the original dataset, and Fa , Fb , Fc , …, Fn 6. Calculate σ2 = (n-1) * Variance (Fa , Fb , Fc , …, Fn ) and σ (gives a very conservative overestimate of error) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 21 Source: Efron & Stein 1981, “The Jackknife Estimate of Variance” 

Variations on the jackknife Half-sample method • Sample n/2 points for each iteration, evaluate function, do this n times IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 22 

Variations on the jackknife Half-sample method • Sample n/2 points for each iteration, evaluate function, do this n times Delete-k jackknife • Resample n-k points for k < n instead of n-1 points for each jackknife iteration, evaluate, etc. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 23 

Bootstrap basics IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 24 Image credit: vintagewesternwear.com • Name comes from the very useful but unglamorous tab used to pull on boots (see also: “pull up by your bootstraps”) • More generalized version of the jackknife • Efron & Tibshirani 1993, “An Introduction to the Bootstrap” • Efron 2003, “Second Thoughts on the Bootstrap” 

Bootstrap basics IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 25 Image credit: vintagewesternwear.com • Name comes from the very useful but unglamorous tab used to pull on boots (see also: “pull up by your bootstraps”) • More generalized version of the jackknife • Efron & Tibshirani 1993, “An Introduction to the Bootstrap” • Efron 2003, “Second Thoughts on the Bootstrap” • Like Monte Carlo from the original data, with uniform probability of selecting an individual point • Resampling with replacement for each iteration, many times • Every time you run will give slightly different results, because of random element in resample selection 

1. Start with your dataset of length n and the function F that you want to compute on it. Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 26 

1. Start with your dataset of length n and the function F that you want to compute on it. 2. Re-sample your data with replacement n times. • With replacement and n times are very important! Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 27 

1. Start with your dataset of length n and the function F that you want to compute on it. 2. Re-sample your data with replacement n times. • With replacement and n times are very important! 3. Evaluate the function F on the resample. Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 28 

1. Start with your dataset of length n and the function F that you want to compute on it. 2. Re-sample your data with replacement n times. • With replacement and n times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient, especially if n>100, but if you have compute power, n[log(n)]2 is robust. Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 29 

1. Start with your dataset of length n and the function F that you want to compute on it. 2. Re-sample your data with replacement n times. • With replacement and n times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient, especially if n>100, but if you have compute power, n[log(n)]2 is robust. 5. Now you have n[log(n)]2 values of F from each bootstrap iteration. Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 30 

1. Start with your dataset of length n and the function F that you want to compute on it. 2. Re-sample your data with replacement n times. • With replacement and n times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient, especially if n>100, but if you have compute power, n[log(n)]2 is robust. 5. Now you have n[log(n)]2 values of F from each bootstrap iteration. 6. Calculate the variance and standard error on the distribution of F resample values. Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 31 

Bootstrap example 1. Have some dataset x = (x1 , x2 , x3 , x4 , …, xn ) and function F = mean(x) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 32 If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife. 

Bootstrap example 1. Have some dataset x = (x1 , x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x4 , x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 33 If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife. 

Bootstrap example 1. Have some dataset x = (x1 , x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x4 , x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) 4. Repeat bootstrap iterations many times! xb = (x1 , x1 , x4 , …, xn ), Fb = mean(xb ) , xc = (x3 , x2 , x5 , …, xn ), Fc = mean(xc ) , etc. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 34 If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife. 

Bootstrap example 1. Have some dataset x = (x1 , x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x4 , x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) 4. Repeat bootstrap iterations many times! xb = (x1 , x1 , x4 , …, xn ), Fb = mean(xb ) , xc = (x3 , x2 , x5 , …, xn ), Fc = mean(xc ) , etc. 5. Now we have F from the original dataset, and Fa , Fb , Fc , …, Fn 6. Calculate σ2 = Variance (Fa , Fb , Fc , …, Fn ) and σ IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 35 If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife. 

Variations on the bootstrap Block bootstrap • Replicates correlation in the data by sampling blocks or subsets of data IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 36 

Variations on the bootstrap Block bootstrap • Replicates correlation in the data by sampling blocks or subsets of data Paired bootstrap • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity in errors (variance isn’t the same for each point, e.g., different telescopes or different observing modes) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 37 

Variations on the bootstrap Block bootstrap • Replicates correlation in the data by sampling blocks or subsets of data Paired bootstrap • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity in errors (variance isn’t the same for each point, e.g., different telescopes or different observing modes) Parametric bootstrap (esp. for simulations) • Get a functional approximation (like least-squares or MLE fit) of the original dataset, then take the random samples from the functional approximation IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 38 

Variations on the bootstrap Block bootstrap • Replicates correlation in the data by sampling blocks or subsets of data Paired bootstrap • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity in errors (variance isn’t the same for each point, e.g., different telescopes or different observing modes) Parametric bootstrap (esp. for simulations) • Get a functional approximation (like least-squares or MLE fit) of the original dataset, then take the random samples from the functional approximation Bayesian bootstrap • Non-uniform chance of sampling a point, can estimate the posterior distribution IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 39 

Variations on the bootstrap Block bootstrap • Replicates correlation in the data by sampling blocks or subsets of data Paired bootstrap • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity in errors (variance isn’t the same for each point, e.g., different telescopes or different observing modes) Parametric bootstrap (esp. for simulations) • Get a functional approximation (like least-squares or MLE fit) of the original dataset, then take the random samples from the functional approximation Bayesian bootstrap • Non-uniform chance of sampling a point, can estimate the posterior distribution Gaussian process regression bootstrap, wild bootstrap IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 40 

Examples in the astronomy literature Bahramian et al. 2017: characteristics of the X-ray binary 47 Tuc X-9 • Found a 28-minute periodic modulation in Chandra data, identified as the orbital period ØConfirmed the system as an ultracompact X-ray binary with a white dwarf companion • Used bootstrapping to get errors on the orbital period ØThink of it like simulating many more observations with the same properties IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 41 

Examples in the astronomy literature Holwerda et al. 2009: dusty disk in an occulting galaxy pair • Found spiral arms in an extended dusty disk, wanted to measure size and optical depth of the arms, estimate extent of dust and dust extinction ØComparing analysis techniques for this high-quality serendipitous observation, for use on likely lower-quality future observations • Bootstrapped extinction values in three filters to compare mean and standard deviation for their three analysis methods IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 42 

Bootstrapping saved my PhD thesis! • Phase-resolved spectra weren’t strictly independent in time for a given energy bin, but this correlation is non-physical (arises because of the analysis) • Wanted to fit the energy spectra, get errors on parameters that make sense • 198 segments of data that were being averaged together. Selected with replacement 5537 times, made a new average each time, fitting, got spectral parameters. Got robust errors! Took a week to run, though. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 43 

Tutorial time • In the Jupyter hub: tutorials/abigail_bootjack/boot-jack-workbook.ipynb IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 44 Sources: • A.C. Cameron & P.K. Trivedi 2006, “Bootstrap Methods” (UC Davis lecture notes) • E. Feigelson & G.J. Babu 2012, “Statistical Challenges in Modern Astronomy” (book chapter) • S. Sawyer 2005, “Resampling Data: Using a Statistical Jackknife” (Washington University lecture notes) • S. Sinharay 2010, “An Overview of Statistics in Education” (book chapter) • The Wikipedia pages for “Bootstrapping (statistics)” and “Jackknife method (statistics)” are pretty good starting points