SOSTAT 2021: Bootstrap & Jackknife Statistical Techniques

Introduction lecture to bootstrap and jackknife statistical techniques for the 2nd Severo Ochoa School on Statistics, Data Mining, and Machine Learning in Granada, Spain: https://www.granadacongresos.com/sostat2021

Tutorial in Jupyter notebooks: https://github.com/abigailStev/bootjack-tutorial

More on Dr. Abbie Stevens: https://abigailstevens.com/

Transcript

1. Bootstrap and Jackknife
   Statistical Techniques
   Dr. Abbie Stevens
   Michigan State University & University of Michigan
   alstev@msu.edu
   @abigailStev
   github.com/abigailstev
   

   View Slide

2. Outline
   • Overview
   • Jackknife
   • Bootstrap
   • Compare/contrast with examples
   • Tutorial for trying them out and practicing
   IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 2
   

   View Slide

3. Brute-force vs shiny and new
   IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 3
   

   View Slide

4. Brute-force vs shiny and new
   • Bootstrap and jackknife are not fancy, and not new
   • Random re-sampling techniques for estimating error/accuracy (variance,
   bias) of data (samples)
   IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 4
   

   View Slide

5. Brute-force vs shiny and new
   • Bootstrap and jackknife are not fancy, and not new
   • Random re-sampling techniques for estimating error/accuracy (variance,
   bias) of data (samples)
   • Building and adding to your toolbox for problem-solving
   • Important to select the right tool for the job!
   • Commonly used in, e.g., biostatistics, industry data science, game science
   IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 5
   

   View Slide

6. Brute-force vs shiny and new
   • Bootstrap and jackknife are not fancy, and not new
   • Random re-sampling techniques for estimating error/accuracy (variance,
   bias) of data (samples)
   • Building and adding to your toolbox for problem-solving
   • Important to select the right tool for the job!
   • Commonly used in, e.g., biostatistics, industry data science, game science
   • Sometimes you get stuck, and this stuff usually works, very well
   IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 6
   

   View Slide

7. When to use them?
   • Check the stability (variance) and
   goodness of your choice of model and
   model complexity
   • Slight correlations between data points,
   don’t know the extent, doesn’t affect the
   underlying physics
   • Can’t (or don’t want to) calculate
   covariance matrix coefficients and derive
   error
   • Don’t know the statistical distribution of
   the parameter space, esp. if no full
   physical model to fit
   IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 7
   

   View Slide

8. When to use them?
   • Check the stability (variance) and
   goodness of your choice of model and
   model complexity
   • Slight correlations between data points,
   don’t know the extent, doesn’t affect the
   underlying physics
   • Can’t (or don’t want to) calculate
   covariance matrix coefficients and derive
   error
   • Don’t know the statistical distribution of
   the parameter space, esp. if no full
   physical model to fit
   IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 8
   When not to use them?
   • Very large data sets
   • Complex operators that take a long
   time to evaluate (unless you have a
   lot of compute power at your
   disposal!)
   • Use caution with non-linear and/or
   non-smooth or discrete operations,
   like the median, or distributions
   with long, asymmetric tails
   

   View Slide

9. Jackknife basics
   • Name comes from a simple, useful little pocket knife
   • Developed by Maurice Quenouille and later John Tukey in the 1950s
   • M. Quenouille, “Notes on Bias in Estimation,” Biometrika, 1956
   • J. Tukey, “Bias and Confidence in Not-quite Large Samples,” Annals of
   Mathematical Statistics, 1958
   IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 9
   Image credit: Tochanchai, Shutterstock via Collins Dictionary
   

   View Slide

10. Jackknife basics
    • Name comes from a simple, useful little pocket knife
    • Developed by Maurice Quenouille and later John Tukey in the 1950s
    • M. Quenouille, “Notes on Bias in Estimation,” Biometrika, 1956
    • J. Tukey, “Bias and Confidence in Not-quite Large Samples,” Annals of
    Mathematical Statistics, 1958
    • Using an “estimator” to resample observations to estimate, e.g., the
    variance and bias
    • Sample mean is often the “estimator” for the population mean
    • Leaving out a single sample when re-sampling
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 10
    Image credit: Tochanchai, Shutterstock via Collins Dictionary
    

    View Slide

11. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 11
    

    View Slide

12. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data without replacement n-1 times.
    • Without replacement and n-1 times are very important!
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 12
    

    View Slide

13. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data without replacement n-1 times.
    • Without replacement and n-1 times are very important!
    3. Evaluate the function F on the resample.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 13
    

    View Slide

14. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data without replacement n-1 times.
    • Without replacement and n-1 times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n times, such that each data point has been left
    out once in all of those jackknife iterations.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 14
    

    View Slide

15. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data without replacement n-1 times.
    • Without replacement and n-1 times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n times, such that each data point has been left
    out once in all of those jackknife iterations.
    5. Now you have n values of F from each jackknife iteration.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 15
    

    View Slide

16. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data without replacement n-1 times.
    • Without replacement and n-1 times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n times, such that each data point has been left
    out once in all of those jackknife iterations.
    5. Now you have n values of F from each jackknife iteration.
    6. Calculate the variance *(n-1) and standard error on the distribution of
    F resample values.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 16
    

    View Slide

17. Jackknife example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 17
    

    View Slide

18. Jackknife example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 18
    

    View Slide

19. Jackknife example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    4. Repeat jackknife iterations!
    xb
    = (x1
    , x3
    , x4
    , …, xn
    ), Fb
    = mean(xb
    )
    xc
    = (x1
    , x2
    , x4
    , …, xn
    ), Fc
    = mean(xc
    )
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 19
    

    View Slide

20. Jackknife example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    4. Repeat jackknife iterations!
    xb
    = (x1
    , x3
    , x4
    , …, xn
    ), Fb
    = mean(xb
    )
    xc
    = (x1
    , x2
    , x4
    , …, xn
    ), Fc
    = mean(xc
    )
    5. Now we have F from the original dataset, and Fa
    , Fb
    , Fc
    , …, Fn
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 20
    

    View Slide

21. Jackknife example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    4. Repeat jackknife iterations!
    xb
    = (x1
    , x3
    , x4
    , …, xn
    ), Fb
    = mean(xb
    )
    xc
    = (x1
    , x2
    , x4
    , …, xn
    ), Fc
    = mean(xc
    )
    5. Now we have F from the original dataset, and Fa
    , Fb
    , Fc
    , …, Fn
    6. Calculate σ2 = (n-1) * Variance (Fa
    , Fb
    , Fc
    , …, Fn
    ) and σ
    (gives a very conservative overestimate of error)
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 21
    Source: Efron & Stein 1981, “The Jackknife Estimate of Variance”
    

    View Slide

22. Variations on the jackknife
    Half-sample method
    • Sample n/2 points for each iteration, evaluate function, do this n times
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 22
    

    View Slide

23. Variations on the jackknife
    Half-sample method
    • Sample n/2 points for each iteration, evaluate function, do this n times
    Delete-k jackknife
    • Resample n-k points for k < n instead of n-1 points for each jackknife iteration,
    evaluate, etc.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 23
    

    View Slide

24. Bootstrap basics
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 24
    Image credit: vintagewesternwear.com
    • Name comes from the very useful but unglamorous tab used to pull on
    boots (see also: “pull up by your bootstraps”)
    • More generalized version of the jackknife
    • Efron & Tibshirani 1993, “An Introduction to the Bootstrap”
    • Efron 2003, “Second Thoughts on the Bootstrap”
    

    View Slide

25. Bootstrap basics
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 25
    Image credit: vintagewesternwear.com
    • Name comes from the very useful but unglamorous tab used to pull on
    boots (see also: “pull up by your bootstraps”)
    • More generalized version of the jackknife
    • Efron & Tibshirani 1993, “An Introduction to the Bootstrap”
    • Efron 2003, “Second Thoughts on the Bootstrap”
    • Like Monte Carlo from the original data, with uniform
    probability of selecting an individual point
    • Resampling with replacement for each iteration, many times
    • Every time you run will give slightly different results,
    because of random element in resample selection
    

    View Slide

26. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 26
    

    View Slide

27. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data with replacement n times.
    • With replacement and n times are very important!
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 27
    

    View Slide

28. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data with replacement n times.
    • With replacement and n times are very important!
    3. Evaluate the function F on the resample.
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 28
    

    View Slide

29. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data with replacement n times.
    • With replacement and n times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient,
    especially if n>100, but if you have compute power, n[log(n)]2 is robust.
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 29
    

    View Slide

30. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data with replacement n times.
    • With replacement and n times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient,
    especially if n>100, but if you have compute power, n[log(n)]2 is robust.
    5. Now you have n[log(n)]2 values of F from each bootstrap iteration.
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 30
    

    View Slide

31. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data with replacement n times.
    • With replacement and n times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient,
    especially if n>100, but if you have compute power, n[log(n)]2 is robust.
    5. Now you have n[log(n)]2 values of F from each bootstrap iteration.
    6. Calculate the variance and standard error on the distribution of F
    resample values.
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 31
    

    View Slide

32. Bootstrap example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 32
    If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.
    

    View Slide

33. Bootstrap example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x4
    , x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 33
    If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.
    

    View Slide

34. Bootstrap example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x4
    , x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    4. Repeat bootstrap iterations many times!
    xb
    = (x1
    , x1
    , x4
    , …, xn
    ), Fb
    = mean(xb
    ) ,
    xc
    = (x3
    , x2
    , x5
    , …, xn
    ), Fc
    = mean(xc
    ) , etc.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 34
    If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.
    

    View Slide

35. Bootstrap example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x4
    , x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    4. Repeat bootstrap iterations many times!
    xb
    = (x1
    , x1
    , x4
    , …, xn
    ), Fb
    = mean(xb
    ) ,
    xc
    = (x3
    , x2
    , x5
    , …, xn
    ), Fc
    = mean(xc
    ) , etc.
    5. Now we have F from the original dataset, and Fa
    , Fb
    , Fc
    , …, Fn
    6. Calculate σ2 = Variance (Fa
    , Fb
    , Fc
    , …, Fn
    ) and σ
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 35
    If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.
    

    View Slide

36. Variations on the bootstrap
    Block bootstrap
    • Replicates correlation in the data by sampling blocks or subsets of data
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 36
    

    View Slide

37. Variations on the bootstrap
    Block bootstrap
    • Replicates correlation in the data by sampling blocks or subsets of data
    Paired bootstrap
    • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
    in errors (variance isn’t the same for each point, e.g., different telescopes or
    different observing modes)
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 37
    

    View Slide

38. Variations on the bootstrap
    Block bootstrap
    • Replicates correlation in the data by sampling blocks or subsets of data
    Paired bootstrap
    • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
    in errors (variance isn’t the same for each point, e.g., different telescopes or
    different observing modes)
    Parametric bootstrap (esp. for simulations)
    • Get a functional approximation (like least-squares or MLE fit) of the original
    dataset, then take the random samples from the functional approximation
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 38
    

    View Slide

39. Variations on the bootstrap
    Block bootstrap
    • Replicates correlation in the data by sampling blocks or subsets of data
    Paired bootstrap
    • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
    in errors (variance isn’t the same for each point, e.g., different telescopes or
    different observing modes)
    Parametric bootstrap (esp. for simulations)
    • Get a functional approximation (like least-squares or MLE fit) of the original
    dataset, then take the random samples from the functional approximation
    Bayesian bootstrap
    • Non-uniform chance of sampling a point, can estimate the posterior distribution
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 39
    

    View Slide

40. Variations on the bootstrap
    Block bootstrap
    • Replicates correlation in the data by sampling blocks or subsets of data
    Paired bootstrap
    • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
    in errors (variance isn’t the same for each point, e.g., different telescopes or
    different observing modes)
    Parametric bootstrap (esp. for simulations)
    • Get a functional approximation (like least-squares or MLE fit) of the original
    dataset, then take the random samples from the functional approximation
    Bayesian bootstrap
    • Non-uniform chance of sampling a point, can estimate the posterior distribution
    Gaussian process regression bootstrap, wild bootstrap
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 40
    

    View Slide

41. Examples in the astronomy literature
    Bahramian et al. 2017: characteristics of the X-ray binary 47 Tuc X-9
    • Found a 28-minute periodic modulation in Chandra data, identified as the orbital
    period
    ØConfirmed the system as an ultracompact X-ray binary with a white dwarf
    companion
    • Used bootstrapping to get errors on the orbital period
    ØThink of it like simulating many more observations with the same properties
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 41
    

    View Slide

42. Examples in the astronomy literature
    Holwerda et al. 2009: dusty disk in an occulting galaxy pair
    • Found spiral arms in an extended dusty disk, wanted to measure size and optical
    depth of the arms, estimate extent of dust and dust extinction
    ØComparing analysis techniques for this high-quality serendipitous
    observation, for use on likely lower-quality future observations
    • Bootstrapped extinction values in three filters to compare mean and standard
    deviation for their three analysis methods
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 42
    

    View Slide

43. Bootstrapping saved my PhD thesis!
    • Phase-resolved spectra weren’t strictly independent in time for a given
    energy bin, but this correlation is non-physical (arises because of the
    analysis)
    • Wanted to fit the energy spectra, get errors on parameters that make
    sense
    • 198 segments of data that were being
    averaged together. Selected with
    replacement 5537 times, made a new
    average each time, fitting, got spectral
    parameters. Got robust errors! Took a
    week to run, though.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 43
    

    View Slide

44. Tutorial time
    • In the Jupyter hub: tutorials/abigail_bootjack/boot-jack-workbook.ipynb
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 44
    Sources:
    • A.C. Cameron & P.K. Trivedi 2006, “Bootstrap Methods” (UC Davis lecture notes)
    • E. Feigelson & G.J. Babu 2012, “Statistical Challenges in Modern Astronomy” (book
    chapter)
    • S. Sawyer 2005, “Resampling Data: Using a Statistical Jackknife” (Washington
    University lecture notes)
    • S. Sinharay 2010, “An Overview of Statistics in Education” (book chapter)
    • The Wikipedia pages for “Bootstrapping (statistics)” and “Jackknife method
    (statistics)” are pretty good starting points
    

    View Slide