Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SOSTAT 2021: Bootstrap & Jackknife Statistical Techniques

SOSTAT 2021: Bootstrap & Jackknife Statistical Techniques

Introduction lecture to bootstrap and jackknife statistical techniques for the 2nd Severo Ochoa School on Statistics, Data Mining, and Machine Learning in Granada, Spain: https://www.granadacongresos.com/sostat2021

Tutorial in Jupyter notebooks: https://github.com/abigailStev/bootjack-tutorial

More on Dr. Abbie Stevens: https://abigailstevens.com/

Dr. Abbie Stevens

November 30, 2021
Tweet

More Decks by Dr. Abbie Stevens

Other Decks in Science

Transcript

  1. Bootstrap and Jackknife
    Statistical Techniques
    Dr. Abbie Stevens
    Michigan State University & University of Michigan
    [email protected]
    @abigailStev
    github.com/abigailstev

    View Slide

  2. Outline
    • Overview
    • Jackknife
    • Bootstrap
    • Compare/contrast with examples
    • Tutorial for trying them out and practicing
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 2

    View Slide

  3. Brute-force vs shiny and new
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 3

    View Slide

  4. Brute-force vs shiny and new
    • Bootstrap and jackknife are not fancy, and not new
    • Random re-sampling techniques for estimating error/accuracy (variance,
    bias) of data (samples)
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 4

    View Slide

  5. Brute-force vs shiny and new
    • Bootstrap and jackknife are not fancy, and not new
    • Random re-sampling techniques for estimating error/accuracy (variance,
    bias) of data (samples)
    • Building and adding to your toolbox for problem-solving
    • Important to select the right tool for the job!
    • Commonly used in, e.g., biostatistics, industry data science, game science
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 5

    View Slide

  6. Brute-force vs shiny and new
    • Bootstrap and jackknife are not fancy, and not new
    • Random re-sampling techniques for estimating error/accuracy (variance,
    bias) of data (samples)
    • Building and adding to your toolbox for problem-solving
    • Important to select the right tool for the job!
    • Commonly used in, e.g., biostatistics, industry data science, game science
    • Sometimes you get stuck, and this stuff usually works, very well
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 6

    View Slide

  7. When to use them?
    • Check the stability (variance) and
    goodness of your choice of model and
    model complexity
    • Slight correlations between data points,
    don’t know the extent, doesn’t affect the
    underlying physics
    • Can’t (or don’t want to) calculate
    covariance matrix coefficients and derive
    error
    • Don’t know the statistical distribution of
    the parameter space, esp. if no full
    physical model to fit
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 7

    View Slide

  8. When to use them?
    • Check the stability (variance) and
    goodness of your choice of model and
    model complexity
    • Slight correlations between data points,
    don’t know the extent, doesn’t affect the
    underlying physics
    • Can’t (or don’t want to) calculate
    covariance matrix coefficients and derive
    error
    • Don’t know the statistical distribution of
    the parameter space, esp. if no full
    physical model to fit
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 8
    When not to use them?
    • Very large data sets
    • Complex operators that take a long
    time to evaluate (unless you have a
    lot of compute power at your
    disposal!)
    • Use caution with non-linear and/or
    non-smooth or discrete operations,
    like the median, or distributions
    with long, asymmetric tails

    View Slide

  9. Jackknife basics
    • Name comes from a simple, useful little pocket knife
    • Developed by Maurice Quenouille and later John Tukey in the 1950s
    • M. Quenouille, “Notes on Bias in Estimation,” Biometrika, 1956
    • J. Tukey, “Bias and Confidence in Not-quite Large Samples,” Annals of
    Mathematical Statistics, 1958
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 9
    Image credit: Tochanchai, Shutterstock via Collins Dictionary

    View Slide

  10. Jackknife basics
    • Name comes from a simple, useful little pocket knife
    • Developed by Maurice Quenouille and later John Tukey in the 1950s
    • M. Quenouille, “Notes on Bias in Estimation,” Biometrika, 1956
    • J. Tukey, “Bias and Confidence in Not-quite Large Samples,” Annals of
    Mathematical Statistics, 1958
    • Using an “estimator” to resample observations to estimate, e.g., the
    variance and bias
    • Sample mean is often the “estimator” for the population mean
    • Leaving out a single sample when re-sampling
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 10
    Image credit: Tochanchai, Shutterstock via Collins Dictionary

    View Slide

  11. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 11

    View Slide

  12. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data without replacement n-1 times.
    • Without replacement and n-1 times are very important!
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 12

    View Slide

  13. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data without replacement n-1 times.
    • Without replacement and n-1 times are very important!
    3. Evaluate the function F on the resample.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 13

    View Slide

  14. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data without replacement n-1 times.
    • Without replacement and n-1 times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n times, such that each data point has been left
    out once in all of those jackknife iterations.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 14

    View Slide

  15. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data without replacement n-1 times.
    • Without replacement and n-1 times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n times, such that each data point has been left
    out once in all of those jackknife iterations.
    5. Now you have n values of F from each jackknife iteration.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 15

    View Slide

  16. Jackknife steps
    1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data without replacement n-1 times.
    • Without replacement and n-1 times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n times, such that each data point has been left
    out once in all of those jackknife iterations.
    5. Now you have n values of F from each jackknife iteration.
    6. Calculate the variance *(n-1) and standard error on the distribution of
    F resample values.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 16

    View Slide

  17. Jackknife example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 17

    View Slide

  18. Jackknife example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 18

    View Slide

  19. Jackknife example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    4. Repeat jackknife iterations!
    xb
    = (x1
    , x3
    , x4
    , …, xn
    ), Fb
    = mean(xb
    )
    xc
    = (x1
    , x2
    , x4
    , …, xn
    ), Fc
    = mean(xc
    )
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 19

    View Slide

  20. Jackknife example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    4. Repeat jackknife iterations!
    xb
    = (x1
    , x3
    , x4
    , …, xn
    ), Fb
    = mean(xb
    )
    xc
    = (x1
    , x2
    , x4
    , …, xn
    ), Fc
    = mean(xc
    )
    5. Now we have F from the original dataset, and Fa
    , Fb
    , Fc
    , …, Fn
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 20

    View Slide

  21. Jackknife example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    4. Repeat jackknife iterations!
    xb
    = (x1
    , x3
    , x4
    , …, xn
    ), Fb
    = mean(xb
    )
    xc
    = (x1
    , x2
    , x4
    , …, xn
    ), Fc
    = mean(xc
    )
    5. Now we have F from the original dataset, and Fa
    , Fb
    , Fc
    , …, Fn
    6. Calculate σ2 = (n-1) * Variance (Fa
    , Fb
    , Fc
    , …, Fn
    ) and σ
    (gives a very conservative overestimate of error)
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 21
    Source: Efron & Stein 1981, “The Jackknife Estimate of Variance”

    View Slide

  22. Variations on the jackknife
    Half-sample method
    • Sample n/2 points for each iteration, evaluate function, do this n times
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 22

    View Slide

  23. Variations on the jackknife
    Half-sample method
    • Sample n/2 points for each iteration, evaluate function, do this n times
    Delete-k jackknife
    • Resample n-k points for k < n instead of n-1 points for each jackknife iteration,
    evaluate, etc.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 23

    View Slide

  24. Bootstrap basics
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 24
    Image credit: vintagewesternwear.com
    • Name comes from the very useful but unglamorous tab used to pull on
    boots (see also: “pull up by your bootstraps”)
    • More generalized version of the jackknife
    • Efron & Tibshirani 1993, “An Introduction to the Bootstrap”
    • Efron 2003, “Second Thoughts on the Bootstrap”

    View Slide

  25. Bootstrap basics
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 25
    Image credit: vintagewesternwear.com
    • Name comes from the very useful but unglamorous tab used to pull on
    boots (see also: “pull up by your bootstraps”)
    • More generalized version of the jackknife
    • Efron & Tibshirani 1993, “An Introduction to the Bootstrap”
    • Efron 2003, “Second Thoughts on the Bootstrap”
    • Like Monte Carlo from the original data, with uniform
    probability of selecting an individual point
    • Resampling with replacement for each iteration, many times
    • Every time you run will give slightly different results,
    because of random element in resample selection

    View Slide

  26. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 26

    View Slide

  27. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data with replacement n times.
    • With replacement and n times are very important!
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 27

    View Slide

  28. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data with replacement n times.
    • With replacement and n times are very important!
    3. Evaluate the function F on the resample.
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 28

    View Slide

  29. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data with replacement n times.
    • With replacement and n times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient,
    especially if n>100, but if you have compute power, n[log(n)]2 is robust.
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 29

    View Slide

  30. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data with replacement n times.
    • With replacement and n times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient,
    especially if n>100, but if you have compute power, n[log(n)]2 is robust.
    5. Now you have n[log(n)]2 values of F from each bootstrap iteration.
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 30

    View Slide

  31. 1. Start with your dataset of length n and the function F that you want to
    compute on it.
    2. Re-sample your data with replacement n times.
    • With replacement and n times are very important!
    3. Evaluate the function F on the resample.
    4. Repeat steps 2 and 3 n[log(n)]2 times. Sometimes n times is sufficient,
    especially if n>100, but if you have compute power, n[log(n)]2 is robust.
    5. Now you have n[log(n)]2 values of F from each bootstrap iteration.
    6. Calculate the variance and standard error on the distribution of F
    resample values.
    Bootstrap steps
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 31

    View Slide

  32. Bootstrap example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 32
    If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.

    View Slide

  33. Bootstrap example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x4
    , x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 33
    If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.

    View Slide

  34. Bootstrap example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x4
    , x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    4. Repeat bootstrap iterations many times!
    xb
    = (x1
    , x1
    , x4
    , …, xn
    ), Fb
    = mean(xb
    ) ,
    xc
    = (x3
    , x2
    , x5
    , …, xn
    ), Fc
    = mean(xc
    ) , etc.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 34
    If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.

    View Slide

  35. Bootstrap example
    1. Have some dataset x = (x1
    , x2
    , x3
    , x4
    , …, xn
    ) and function F = mean(x)
    2. Resample: xa
    = (x4
    , x2
    , x3
    , x4
    , …, xn
    )
    3. Evaluate Fa
    = mean(xa
    )
    4. Repeat bootstrap iterations many times!
    xb
    = (x1
    , x1
    , x4
    , …, xn
    ), Fb
    = mean(xb
    ) ,
    xc
    = (x3
    , x2
    , x5
    , …, xn
    ), Fc
    = mean(xc
    ) , etc.
    5. Now we have F from the original dataset, and Fa
    , Fb
    , Fc
    , …, Fn
    6. Calculate σ2 = Variance (Fa
    , Fb
    , Fc
    , …, Fn
    ) and σ
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 35
    If this looks very similar to the jackknife example, that’s because bootstrap is a more generalized version of jackknife.

    View Slide

  36. Variations on the bootstrap
    Block bootstrap
    • Replicates correlation in the data by sampling blocks or subsets of data
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 36

    View Slide

  37. Variations on the bootstrap
    Block bootstrap
    • Replicates correlation in the data by sampling blocks or subsets of data
    Paired bootstrap
    • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
    in errors (variance isn’t the same for each point, e.g., different telescopes or
    different observing modes)
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 37

    View Slide

  38. Variations on the bootstrap
    Block bootstrap
    • Replicates correlation in the data by sampling blocks or subsets of data
    Paired bootstrap
    • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
    in errors (variance isn’t the same for each point, e.g., different telescopes or
    different observing modes)
    Parametric bootstrap (esp. for simulations)
    • Get a functional approximation (like least-squares or MLE fit) of the original
    dataset, then take the random samples from the functional approximation
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 38

    View Slide

  39. Variations on the bootstrap
    Block bootstrap
    • Replicates correlation in the data by sampling blocks or subsets of data
    Paired bootstrap
    • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
    in errors (variance isn’t the same for each point, e.g., different telescopes or
    different observing modes)
    Parametric bootstrap (esp. for simulations)
    • Get a functional approximation (like least-squares or MLE fit) of the original
    dataset, then take the random samples from the functional approximation
    Bayesian bootstrap
    • Non-uniform chance of sampling a point, can estimate the posterior distribution
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 39

    View Slide

  40. Variations on the bootstrap
    Block bootstrap
    • Replicates correlation in the data by sampling blocks or subsets of data
    Paired bootstrap
    • Selecting pairs of multivariate data points -- good for cases of heteroskedasticity
    in errors (variance isn’t the same for each point, e.g., different telescopes or
    different observing modes)
    Parametric bootstrap (esp. for simulations)
    • Get a functional approximation (like least-squares or MLE fit) of the original
    dataset, then take the random samples from the functional approximation
    Bayesian bootstrap
    • Non-uniform chance of sampling a point, can estimate the posterior distribution
    Gaussian process regression bootstrap, wild bootstrap
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 40

    View Slide

  41. Examples in the astronomy literature
    Bahramian et al. 2017: characteristics of the X-ray binary 47 Tuc X-9
    • Found a 28-minute periodic modulation in Chandra data, identified as the orbital
    period
    ØConfirmed the system as an ultracompact X-ray binary with a white dwarf
    companion
    • Used bootstrapping to get errors on the orbital period
    ØThink of it like simulating many more observations with the same properties
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 41

    View Slide

  42. Examples in the astronomy literature
    Holwerda et al. 2009: dusty disk in an occulting galaxy pair
    • Found spiral arms in an extended dusty disk, wanted to measure size and optical
    depth of the arms, estimate extent of dust and dust extinction
    ØComparing analysis techniques for this high-quality serendipitous
    observation, for use on likely lower-quality future observations
    • Bootstrapped extinction values in three filters to compare mean and standard
    deviation for their three analysis methods
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 42

    View Slide

  43. Bootstrapping saved my PhD thesis!
    • Phase-resolved spectra weren’t strictly independent in time for a given
    energy bin, but this correlation is non-physical (arises because of the
    analysis)
    • Wanted to fit the energy spectra, get errors on parameters that make
    sense
    • 198 segments of data that were being
    averaged together. Selected with
    replacement 5537 times, made a new
    average each time, fitting, got spectral
    parameters. Got robust errors! Took a
    week to run, though.
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 43

    View Slide

  44. Tutorial time
    • In the Jupyter hub: tutorials/abigail_bootjack/boot-jack-workbook.ipynb
    IAA-SOSTAT 2021 ☆ Abbie Stevens, MSU & UMich 44
    Sources:
    • A.C. Cameron & P.K. Trivedi 2006, “Bootstrap Methods” (UC Davis lecture notes)
    • E. Feigelson & G.J. Babu 2012, “Statistical Challenges in Modern Astronomy” (book
    chapter)
    • S. Sawyer 2005, “Resampling Data: Using a Statistical Jackknife” (Washington
    University lecture notes)
    • S. Sinharay 2010, “An Overview of Statistics in Education” (book chapter)
    • The Wikipedia pages for “Bootstrapping (statistics)” and “Jackknife method
    (statistics)” are pretty good starting points

    View Slide