Share
Embed
SOSTAT 2021: Bootstrap & Jackknife Statistical Techniques
Introduction lecture to bootstrap and jackknife statistical techniques for the 2nd Severo Ochoa School on Statistics, Data Mining, and Machine Learning in Granada, Spain: https://www.granadacongresos.com/sostat2021
Tutorial in Jupyter notebooks: https://github.com/abigailStev/bootjack-tutorial
More on Dr. Abbie Stevens: https://abigailstevens.com/
Dr. Abbie Stevens
November 30, 2021
Transcript
44 slides
-
Bootstrap and Jackknife Statistical Techniques Dr. Abbie Stevens Michigan State
University & University of Michigan [email protected] @abigailStev github.com/abigailstev -
Outline • Overview • Jackknife • Bootstrap • Compare/contrast with
examples • Tutorial for trying them out and practicing IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 2 -
Brute-force vs shiny and new • Bootstrap and jackknife are
not fancy, and not new • Random re-sampling techniques for estimating error/accuracy (variance, bias) of data (samples) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 4 -
Brute-force vs shiny and new • Bootstrap and jackknife are
not fancy, and not new • Random re-sampling techniques for estimating error/accuracy (variance, bias) of data (samples) • Building and adding to your toolbox for problem-solving • Important to select the right tool for the job! • Commonly used in, e.g., biostatistics, industry data science, game science IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 5 -
Brute-force vs shiny and new • Bootstrap and jackknife are
not fancy, and not new • Random re-sampling techniques for estimating error/accuracy (variance, bias) of data (samples) • Building and adding to your toolbox for problem-solving • Important to select the right tool for the job! • Commonly used in, e.g., biostatistics, industry data science, game science • Sometimes you get stuck, and this stuff usually works, very well IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 6 -
When to use them? • Check the stability (variance) and
goodness of your choice of model and model complexity • Slight correlations between data points, don’t know the extent, doesn’t affect the underlying physics • Can’t (or don’t want to) calculate covariance matrix coefficients and derive error • Don’t know the statistical distribution of the parameter space, esp. if no full physical model to fit IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 7 -
When to use them? • Check the stability (variance) and
goodness of your choice of model and model complexity • Slight correlations between data points, don’t know the extent, doesn’t affect the underlying physics • Can’t (or don’t want to) calculate covariance matrix coefficients and derive error • Don’t know the statistical distribution of the parameter space, esp. if no full physical model to fit IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 8 When not to use them? • Very large data sets • Complex operators that take a long time to evaluate (unless you have a lot of compute power at your disposal!) • Use caution with non-linear and/or non-smooth or discrete operations, like the median, or distributions with long, asymmetric tails -
Jackknife basics • Name comes from a simple, useful little
pocket knife • Developed by Maurice Quenouille and later John Tukey in the 1950s • M. Quenouille, “Notes on Bias in Estimation,” Biometrika, 1956 • J. Tukey, “Bias and Confidence in Not-quite Large Samples,” Annals of Mathematical Statistics, 1958 IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 9 Image credit: Tochanchai, Shutterstock via Collins Dictionary -
Jackknife basics • Name comes from a simple, useful little
pocket knife • Developed by Maurice Quenouille and later John Tukey in the 1950s • M. Quenouille, “Notes on Bias in Estimation,” Biometrika, 1956 • J. Tukey, “Bias and Confidence in Not-quite Large Samples,” Annals of Mathematical Statistics, 1958 • Using an “estimator” to resample observations to estimate, e.g., the variance and bias • Sample mean is often the “estimator” for the population mean • Leaving out a single sample when re-sampling IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 10 Image credit: Tochanchai, Shutterstock via Collins Dictionary -
Jackknife steps 1. Start with your dataset of length n
and the function F that you want to compute on it. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 11 -
Jackknife steps 1. Start with your dataset of length n
and the function F that you want to compute on it. 2. Re-sample your data without replacement n-1 times. • Without replacement and n-1 times are very important! IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 12 -
Jackknife steps 1. Start with your dataset of length n
and the function F that you want to compute on it. 2. Re-sample your data without replacement n-1 times. • Without replacement and n-1 times are very important! 3. Evaluate the function F on the resample. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 13 -
Jackknife steps 1. Start with your dataset of length n
and the function F that you want to compute on it. 2. Re-sample your data without replacement n-1 times. • Without replacement and n-1 times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n times, such that each data point has been left out once in all of those jackknife iterations. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 14 -
Jackknife steps 1. Start with your dataset of length n
and the function F that you want to compute on it. 2. Re-sample your data without replacement n-1 times. • Without replacement and n-1 times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n times, such that each data point has been left out once in all of those jackknife iterations. 5. Now you have n values of F from each jackknife iteration. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 15 -
Jackknife steps 1. Start with your dataset of length n
and the function F that you want to compute on it. 2. Re-sample your data without replacement n-1 times. • Without replacement and n-1 times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n times, such that each data point has been left out once in all of those jackknife iterations. 5. Now you have n values of F from each jackknife iteration. 6. Calculate the variance *(n-1) and standard error on the distribution of F resample values. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 16 -
Jackknife example 1. Have some dataset x = (x1 ,
x2 , x3 , x4 , …, xn ) and function F = mean(x) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 17 -
Jackknife example 1. Have some dataset x = (x1 ,
x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 18 -
Jackknife example 1. Have some dataset x = (x1 ,
x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) 4. Repeat jackknife iterations! xb = (x1 , x3 , x4 , …, xn ), Fb = mean(xb ) xc = (x1 , x2 , x4 , …, xn ), Fc = mean(xc ) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 19 -
Jackknife example 1. Have some dataset x = (x1 ,
x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) 4. Repeat jackknife iterations! xb = (x1 , x3 , x4 , …, xn ), Fb = mean(xb ) xc = (x1 , x2 , x4 , …, xn ), Fc = mean(xc ) 5. Now we have F from the original dataset, and Fa , Fb , Fc , …, Fn IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 20 -
Jackknife example 1. Have some dataset x = (x1 ,
x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) 4. Repeat jackknife iterations! xb = (x1 , x3 , x4 , …, xn ), Fb = mean(xb ) xc = (x1 , x2 , x4 , …, xn ), Fc = mean(xc ) 5. Now we have F from the original dataset, and Fa , Fb , Fc , …, Fn 6. Calculate σ2 = (n-1) * Variance (Fa , Fb , Fc , …, Fn ) and σ (gives a very conservative overestimate of error) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 21 Source: Efron & Stein 1981, “The Jackknife Estimate of Variance” -
Variations on the jackknife Half-sample method • Sample n/2 points
for each iteration, evaluate function, do this n times IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 22 -
Variations on the jackknife Half-sample method • Sample n/2 points
for each iteration, evaluate function, do this n times Delete-k jackknife • Resample n-k points for k < n instead of n-1 points for each jackknife iteration, evaluate, etc. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 23 -
Bootstrap basics IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich
24 Image credit: vintagewesternwear.com • Name comes from the very useful but unglamorous tab used to pull on boots (see also: “pull up by your bootstraps”) • More generalized version of the jackknife • Efron & Tibshirani 1993, “An Introduction to the Bootstrap” • Efron 2003, “Second Thoughts on the Bootstrap” -
Bootstrap basics IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich
25 Image credit: vintagewesternwear.com • Name comes from the very useful but unglamorous tab used to pull on boots (see also: “pull up by your bootstraps”) • More generalized version of the jackknife • Efron & Tibshirani 1993, “An Introduction to the Bootstrap” • Efron 2003, “Second Thoughts on the Bootstrap” • Like Monte Carlo from the original data, with uniform probability of selecting an individual point • Resampling with replacement for each iteration, many times • Every time you run will give slightly different results, because of random element in resample selection -
1. Start with your dataset of length n and the
function F that you want to compute on it. Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 26 -
1. Start with your dataset of length n and the
function F that you want to compute on it. 2. Re-sample your data with replacement n times. • With replacement and n times are very important! Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 27 -
1. Start with your dataset of length n and the
function F that you want to compute on it. 2. Re-sample your data with replacement n times. • With replacement and n times are very important! 3. Evaluate the function F on the resample. Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 28 -
1. Start with your dataset of length n and the
function F that you want to compute on it. 2. Re-sample your data with replacement n times. • With replacement and n times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient, especially if n>100, but if you have compute power, n[log(n)]2 is robust. Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 29 -
1. Start with your dataset of length n and the
function F that you want to compute on it. 2. Re-sample your data with replacement n times. • With replacement and n times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient, especially if n>100, but if you have compute power, n[log(n)]2 is robust. 5. Now you have n[log(n)]2 values of F from each bootstrap iteration. Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 30 -
1. Start with your dataset of length n and the
function F that you want to compute on it. 2. Re-sample your data with replacement n times. • With replacement and n times are very important! 3. Evaluate the function F on the resample. 4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient, especially if n>100, but if you have compute power, n[log(n)]2 is robust. 5. Now you have n[log(n)]2 values of F from each bootstrap iteration. 6. Calculate the variance and standard error on the distribution of F resample values. Bootstrap steps IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 31 -
Bootstrap example 1. Have some dataset x = (x1 ,
x2 , x3 , x4 , …, xn ) and function F = mean(x) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 32 If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife. -
Bootstrap example 1. Have some dataset x = (x1 ,
x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x4 , x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 33 If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife. -
Bootstrap example 1. Have some dataset x = (x1 ,
x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x4 , x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) 4. Repeat bootstrap iterations many times! xb = (x1 , x1 , x4 , …, xn ), Fb = mean(xb ) , xc = (x3 , x2 , x5 , …, xn ), Fc = mean(xc ) , etc. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 34 If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife. -
Bootstrap example 1. Have some dataset x = (x1 ,
x2 , x3 , x4 , …, xn ) and function F = mean(x) 2. Resample: xa = (x4 , x2 , x3 , x4 , …, xn ) 3. Evaluate Fa = mean(xa ) 4. Repeat bootstrap iterations many times! xb = (x1 , x1 , x4 , …, xn ), Fb = mean(xb ) , xc = (x3 , x2 , x5 , …, xn ), Fc = mean(xc ) , etc. 5. Now we have F from the original dataset, and Fa , Fb , Fc , …, Fn 6. Calculate σ2 = Variance (Fa , Fb , Fc , …, Fn ) and σ IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 35 If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife. -
Variations on the bootstrap Block bootstrap • Replicates correlation in
the data by sampling blocks or subsets of data IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 36 -
Variations on the bootstrap Block bootstrap • Replicates correlation in
the data by sampling blocks or subsets of data Paired bootstrap • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity in errors (variance isn’t the same for each point, e.g., different telescopes or different observing modes) IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 37 -
Variations on the bootstrap Block bootstrap • Replicates correlation in
the data by sampling blocks or subsets of data Paired bootstrap • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity in errors (variance isn’t the same for each point, e.g., different telescopes or different observing modes) Parametric bootstrap (esp. for simulations) • Get a functional approximation (like least-squares or MLE fit) of the original dataset, then take the random samples from the functional approximation IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 38 -
Variations on the bootstrap Block bootstrap • Replicates correlation in
the data by sampling blocks or subsets of data Paired bootstrap • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity in errors (variance isn’t the same for each point, e.g., different telescopes or different observing modes) Parametric bootstrap (esp. for simulations) • Get a functional approximation (like least-squares or MLE fit) of the original dataset, then take the random samples from the functional approximation Bayesian bootstrap • Non-uniform chance of sampling a point, can estimate the posterior distribution IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 39 -
Variations on the bootstrap Block bootstrap • Replicates correlation in
the data by sampling blocks or subsets of data Paired bootstrap • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity in errors (variance isn’t the same for each point, e.g., different telescopes or different observing modes) Parametric bootstrap (esp. for simulations) • Get a functional approximation (like least-squares or MLE fit) of the original dataset, then take the random samples from the functional approximation Bayesian bootstrap • Non-uniform chance of sampling a point, can estimate the posterior distribution Gaussian process regression bootstrap, wild bootstrap IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 40 -
Examples in the astronomy literature Bahramian et al. 2017: characteristics
of the X-ray binary 47 Tuc X-9 • Found a 28-minute periodic modulation in Chandra data, identified as the orbital period ØConfirmed the system as an ultracompact X-ray binary with a white dwarf companion • Used bootstrapping to get errors on the orbital period ØThink of it like simulating many more observations with the same properties IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 41 -
Examples in the astronomy literature Holwerda et al. 2009: dusty
disk in an occulting galaxy pair • Found spiral arms in an extended dusty disk, wanted to measure size and optical depth of the arms, estimate extent of dust and dust extinction ØComparing analysis techniques for this high-quality serendipitous observation, for use on likely lower-quality future observations • Bootstrapped extinction values in three filters to compare mean and standard deviation for their three analysis methods IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 42 -
Bootstrapping saved my PhD thesis! • Phase-resolved spectra weren’t strictly
independent in time for a given energy bin, but this correlation is non-physical (arises because of the analysis) • Wanted to fit the energy spectra, get errors on parameters that make sense • 198 segments of data that were being averaged together. Selected with replacement 5537 times, made a new average each time, fitting, got spectral parameters. Got robust errors! Took a week to run, though. IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 43 -
Tutorial time • In the Jupyter hub: tutorials/abigail_bootjack/boot-jack-workbook.ipynb IAA-SOSTAT 2021
☆ Abbie Stevens, MSU & UMich 44 Sources: • A.C. Cameron & P.K. Trivedi 2006, “Bootstrap Methods” (UC Davis lecture notes) • E. Feigelson & G.J. Babu 2012, “Statistical Challenges in Modern Astronomy” (book chapter) • S. Sawyer 2005, “Resampling Data: Using a Statistical Jackknife” (Washington University lecture notes) • S. Sinharay 2010, “An Overview of Statistics in Education” (book chapter) • The Wikipedia pages for “Bootstrapping (statistics)” and “Jackknife method (statistics)” are pretty good starting points