Dr. Abbie Stevens
November 30, 2021
120

# SOSTAT 2021: Bootstrap & Jackknife Statistical Techniques

Introduction lecture to bootstrap and jackknife statistical techniques for the 2nd Severo Ochoa School on Statistics, Data Mining, and Machine Learning in Granada, Spain: https://www.granadacongresos.com/sostat2021

Tutorial in Jupyter notebooks: https://github.com/abigailStev/bootjack-tutorial

More on Dr. Abbie Stevens: https://abigailstevens.com/

## Dr. Abbie Stevens

November 30, 2021

## Transcript

1. Bootstrap and Jackknife
Statistical Techniques
Dr. Abbie Stevens
Michigan State University & University of Michigan
[email protected]
@abigailStev
github.com/abigailstev

2. Outline
• Overview
• Jackknife
• Bootstrap
• Compare/contrast with examples
• Tutorial for trying them out and practicing
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 2

3. Brute-force vs shiny and new
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 3

4. Brute-force vs shiny and new
• Bootstrap and jackknife are not fancy, and not new
• Random re-sampling techniques for estimating error/accuracy (variance,
bias) of data (samples)
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 4

5. Brute-force vs shiny and new
• Bootstrap and jackknife are not fancy, and not new
• Random re-sampling techniques for estimating error/accuracy (variance,
bias) of data (samples)
• Important to select the right tool for the job!
• Commonly used in, e.g., biostatistics, industry data science, game science
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 5

6. Brute-force vs shiny and new
• Bootstrap and jackknife are not fancy, and not new
• Random re-sampling techniques for estimating error/accuracy (variance,
bias) of data (samples)
• Important to select the right tool for the job!
• Commonly used in, e.g., biostatistics, industry data science, game science
• Sometimes you get stuck, and this stuff usually works, very well
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 6

7. When to use them?
• Check the stability (variance) and
goodness of your choice of model and
model complexity
• Slight correlations between data points,
don’t know the extent, doesn’t affect the
underlying physics
• Can’t (or don’t want to) calculate
covariance matrix coefficients and derive
error
• Don’t know the statistical distribution of
the parameter space, esp. if no full
physical model to fit
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 7

8. When to use them?
• Check the stability (variance) and
goodness of your choice of model and
model complexity
• Slight correlations between data points,
don’t know the extent, doesn’t affect the
underlying physics
• Can’t (or don’t want to) calculate
covariance matrix coefficients and derive
error
• Don’t know the statistical distribution of
the parameter space, esp. if no full
physical model to fit
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 8
When not to use them?
• Very large data sets
• Complex operators that take a long
time to evaluate (unless you have a
lot of compute power at your
disposal!)
• Use caution with non-linear and/or
non-smooth or discrete operations,
like the median, or distributions
with long, asymmetric tails

9. Jackknife basics
• Name comes from a simple, useful little pocket knife
• Developed by Maurice Quenouille and later John Tukey in the 1950s
• M. Quenouille, “Notes on Bias in Estimation,” Biometrika, 1956
• J. Tukey, “Bias and Confidence in Not-quite Large Samples,” Annals of
Mathematical Statistics, 1958
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 9
Image credit: Tochanchai, Shutterstock via Collins Dictionary

10. Jackknife basics
• Name comes from a simple, useful little pocket knife
• Developed by Maurice Quenouille and later John Tukey in the 1950s
• M. Quenouille, “Notes on Bias in Estimation,” Biometrika, 1956
• J. Tukey, “Bias and Confidence in Not-quite Large Samples,” Annals of
Mathematical Statistics, 1958
• Using an “estimator” to resample observations to estimate, e.g., the
variance and bias
• Sample mean is often the “estimator” for the population mean
• Leaving out a single sample when re-sampling
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 10
Image credit: Tochanchai, Shutterstock via Collins Dictionary

11. Jackknife steps
compute on it.
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 11

12. Jackknife steps
compute on it.
2. Re-sample your data without replacement n-1 times.
• Without replacement and n-1 times are very important!
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 12

13. Jackknife steps
compute on it.
2. Re-sample your data without replacement n-1 times.
• Without replacement and n-1 times are very important!
3. Evaluate the function F on the resample.
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 13

14. Jackknife steps
compute on it.
2. Re-sample your data without replacement n-1 times.
• Without replacement and n-1 times are very important!
3. Evaluate the function F on the resample.
4. Repeat steps 2 and 3 n times, such that each data point has been left
out once in all of those jackknife iterations.
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 14

15. Jackknife steps
compute on it.
2. Re-sample your data without replacement n-1 times.
• Without replacement and n-1 times are very important!
3. Evaluate the function F on the resample.
4. Repeat steps 2 and 3 n times, such that each data point has been left
out once in all of those jackknife iterations.
5. Now you have n values of F from each jackknife iteration.
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 15

16. Jackknife steps
compute on it.
2. Re-sample your data without replacement n-1 times.
• Without replacement and n-1 times are very important!
3. Evaluate the function F on the resample.
4. Repeat steps 2 and 3 n times, such that each data point has been left
out once in all of those jackknife iterations.
5. Now you have n values of F from each jackknife iteration.
6. Calculate the variance *(n-1) and standard error on the distribution of
F resample values.
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 16

17. Jackknife example
1. Have some dataset x = (x1
, x2
, x3
, x4
, …, xn
) and function F = mean(x)
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 17

18. Jackknife example
1. Have some dataset x = (x1
, x2
, x3
, x4
, …, xn
) and function F = mean(x)
2. Resample: xa
= (x2
, x3
, x4
, …, xn
)
3. Evaluate Fa
= mean(xa
)
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 18

19. Jackknife example
1. Have some dataset x = (x1
, x2
, x3
, x4
, …, xn
) and function F = mean(x)
2. Resample: xa
= (x2
, x3
, x4
, …, xn
)
3. Evaluate Fa
= mean(xa
)
4. Repeat jackknife iterations!
xb
= (x1
, x3
, x4
, …, xn
), Fb
= mean(xb
)
xc
= (x1
, x2
, x4
, …, xn
), Fc
= mean(xc
)
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 19

20. Jackknife example
1. Have some dataset x = (x1
, x2
, x3
, x4
, …, xn
) and function F = mean(x)
2. Resample: xa
= (x2
, x3
, x4
, …, xn
)
3. Evaluate Fa
= mean(xa
)
4. Repeat jackknife iterations!
xb
= (x1
, x3
, x4
, …, xn
), Fb
= mean(xb
)
xc
= (x1
, x2
, x4
, …, xn
), Fc
= mean(xc
)
5. Now we have F from the original dataset, and Fa
, Fb
, Fc
, …, Fn
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 20

21. Jackknife example
1. Have some dataset x = (x1
, x2
, x3
, x4
, …, xn
) and function F = mean(x)
2. Resample: xa
= (x2
, x3
, x4
, …, xn
)
3. Evaluate Fa
= mean(xa
)
4. Repeat jackknife iterations!
xb
= (x1
, x3
, x4
, …, xn
), Fb
= mean(xb
)
xc
= (x1
, x2
, x4
, …, xn
), Fc
= mean(xc
)
5. Now we have F from the original dataset, and Fa
, Fb
, Fc
, …, Fn
6. Calculate σ2 = (n-1) * Variance (Fa
, Fb
, Fc
, …, Fn
) and σ
(gives a very conservative overestimate of error)
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 21
Source: Efron & Stein 1981, “The Jackknife Estimate of Variance”

22. Variations on the jackknife
Half-sample method
• Sample n/2 points for each iteration, evaluate function, do this n times
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 22

23. Variations on the jackknife
Half-sample method
• Sample n/2 points for each iteration, evaluate function, do this n times
Delete-k jackknife
• Resample n-k points for k < n instead of n-1 points for each jackknife iteration,
evaluate, etc.
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 23

24. Bootstrap basics
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 24
Image credit: vintagewesternwear.com
• Name comes from the very useful but unglamorous tab used to pull on
• More generalized version of the jackknife
• Efron & Tibshirani 1993, “An Introduction to the Bootstrap”
• Efron 2003, “Second Thoughts on the Bootstrap”

25. Bootstrap basics
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 25
Image credit: vintagewesternwear.com
• Name comes from the very useful but unglamorous tab used to pull on
• More generalized version of the jackknife
• Efron & Tibshirani 1993, “An Introduction to the Bootstrap”
• Efron 2003, “Second Thoughts on the Bootstrap”
• Like Monte Carlo from the original data, with uniform
probability of selecting an individual point
• Resampling with replacement for each iteration, many times
• Every time you run will give slightly different results,
because of random element in resample selection

26. 1. Start with your dataset of length n and the function F that you want to
compute on it.
Bootstrap steps
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 26

27. 1. Start with your dataset of length n and the function F that you want to
compute on it.
2. Re-sample your data with replacement n times.
• With replacement and n times are very important!
Bootstrap steps
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 27

28. 1. Start with your dataset of length n and the function F that you want to
compute on it.
2. Re-sample your data with replacement n times.
• With replacement and n times are very important!
3. Evaluate the function F on the resample.
Bootstrap steps
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 28

29. 1. Start with your dataset of length n and the function F that you want to
compute on it.
2. Re-sample your data with replacement n times.
• With replacement and n times are very important!
3. Evaluate the function F on the resample.
4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient,
especially if n>100, but if you have compute power, n[log(n)]2 is robust.
Bootstrap steps
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 29

30. 1. Start with your dataset of length n and the function F that you want to
compute on it.
2. Re-sample your data with replacement n times.
• With replacement and n times are very important!
3. Evaluate the function F on the resample.
4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient,
especially if n>100, but if you have compute power, n[log(n)]2 is robust.
5. Now you have n[log(n)]2 values of F from each bootstrap iteration.
Bootstrap steps
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 30

31. 1. Start with your dataset of length n and the function F that you want to
compute on it.
2. Re-sample your data with replacement n times.
• With replacement and n times are very important!
3. Evaluate the function F on the resample.
4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient,
especially if n>100, but if you have compute power, n[log(n)]2 is robust.
5. Now you have n[log(n)]2 values of F from each bootstrap iteration.
6. Calculate the variance and standard error on the distribution of F
resample values.
Bootstrap steps
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 31

32. Bootstrap example
1. Have some dataset x = (x1
, x2
, x3
, x4
, …, xn
) and function F = mean(x)
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 32
If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.

33. Bootstrap example
1. Have some dataset x = (x1
, x2
, x3
, x4
, …, xn
) and function F = mean(x)
2. Resample: xa
= (x4
, x2
, x3
, x4
, …, xn
)
3. Evaluate Fa
= mean(xa
)
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 33
If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.

34. Bootstrap example
1. Have some dataset x = (x1
, x2
, x3
, x4
, …, xn
) and function F = mean(x)
2. Resample: xa
= (x4
, x2
, x3
, x4
, …, xn
)
3. Evaluate Fa
= mean(xa
)
4. Repeat bootstrap iterations many times!
xb
= (x1
, x1
, x4
, …, xn
), Fb
= mean(xb
) ,
xc
= (x3
, x2
, x5
, …, xn
), Fc
= mean(xc
) , etc.
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 34
If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.

35. Bootstrap example
1. Have some dataset x = (x1
, x2
, x3
, x4
, …, xn
) and function F = mean(x)
2. Resample: xa
= (x4
, x2
, x3
, x4
, …, xn
)
3. Evaluate Fa
= mean(xa
)
4. Repeat bootstrap iterations many times!
xb
= (x1
, x1
, x4
, …, xn
), Fb
= mean(xb
) ,
xc
= (x3
, x2
, x5
, …, xn
), Fc
= mean(xc
) , etc.
5. Now we have F from the original dataset, and Fa
, Fb
, Fc
, …, Fn
6. Calculate σ2 = Variance (Fa
, Fb
, Fc
, …, Fn
) and σ
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 35
If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.

36. Variations on the bootstrap
Block bootstrap
• Replicates correlation in the data by sampling blocks or subsets of data
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 36

37. Variations on the bootstrap
Block bootstrap
• Replicates correlation in the data by sampling blocks or subsets of data
Paired bootstrap
• Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
in errors (variance isn’t the same for each point, e.g., different telescopes or
different observing modes)
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 37

38. Variations on the bootstrap
Block bootstrap
• Replicates correlation in the data by sampling blocks or subsets of data
Paired bootstrap
• Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
in errors (variance isn’t the same for each point, e.g., different telescopes or
different observing modes)
Parametric bootstrap (esp. for simulations)
• Get a functional approximation (like least-squares or MLE fit) of the original
dataset, then take the random samples from the functional approximation
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 38

39. Variations on the bootstrap
Block bootstrap
• Replicates correlation in the data by sampling blocks or subsets of data
Paired bootstrap
• Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
in errors (variance isn’t the same for each point, e.g., different telescopes or
different observing modes)
Parametric bootstrap (esp. for simulations)
• Get a functional approximation (like least-squares or MLE fit) of the original
dataset, then take the random samples from the functional approximation
Bayesian bootstrap
• Non-uniform chance of sampling a point, can estimate the posterior distribution
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 39

40. Variations on the bootstrap
Block bootstrap
• Replicates correlation in the data by sampling blocks or subsets of data
Paired bootstrap
• Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
in errors (variance isn’t the same for each point, e.g., different telescopes or
different observing modes)
Parametric bootstrap (esp. for simulations)
• Get a functional approximation (like least-squares or MLE fit) of the original
dataset, then take the random samples from the functional approximation
Bayesian bootstrap
• Non-uniform chance of sampling a point, can estimate the posterior distribution
Gaussian process regression bootstrap, wild bootstrap
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 40

41. Examples in the astronomy literature
Bahramian et al. 2017: characteristics of the X-ray binary 47 Tuc X-9
• Found a 28-minute periodic modulation in Chandra data, identified as the orbital
period
ØConfirmed the system as an ultracompact X-ray binary with a white dwarf
companion
• Used bootstrapping to get errors on the orbital period
ØThink of it like simulating many more observations with the same properties
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 41

42. Examples in the astronomy literature
Holwerda et al. 2009: dusty disk in an occulting galaxy pair
• Found spiral arms in an extended dusty disk, wanted to measure size and optical
depth of the arms, estimate extent of dust and dust extinction
ØComparing analysis techniques for this high-quality serendipitous
observation, for use on likely lower-quality future observations
• Bootstrapped extinction values in three filters to compare mean and standard
deviation for their three analysis methods
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 42

43. Bootstrapping saved my PhD thesis!
• Phase-resolved spectra weren’t strictly independent in time for a given
energy bin, but this correlation is non-physical (arises because of the
analysis)
• Wanted to fit the energy spectra, get errors on parameters that make
sense
• 198 segments of data that were being
averaged together. Selected with
replacement 5537 times, made a new
average each time, fitting, got spectral
parameters. Got robust errors! Took a
week to run, though.
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 43

44. Tutorial time
• In the Jupyter hub: tutorials/abigail_bootjack/boot-jack-workbook.ipynb
IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 44
Sources:
• A.C. Cameron & P.K. Trivedi 2006, “Bootstrap Methods” (UC Davis lecture notes)
• E. Feigelson & G.J. Babu 2012, “Statistical Challenges in Modern Astronomy” (book
chapter)
• S. Sawyer 2005, “Resampling Data: Using a Statistical Jackknife” (Washington
University lecture notes)
• S. Sinharay 2010, “An Overview of Statistics in Education” (book chapter)