GLMs by Nelder and Wedderburn, discussed by Ysé Wanono

presentation made by Ysé Wanono at the Reading Classics Seminar, TSI, Université Paris-Dauphine, Dec. 2, 2013

Transcript

1. 1/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions Generalized Linear Models by J.A. Nelder and R.W.M Wedderburn Yse Wanono 2/12/2013 Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
2. 2/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions A Simple Regression Model Assumption Z is determined by X via a function f Z = f (X) + where is a random error term The simplest way to represent a relationship between two continuous variables is by mean of a linear function. Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
3. 3/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The General Linear Model The General Linear Model can be written as E(z) = β0 + β1x1 + β2x2 + · · · + βkxk where x is an explanatory variable z the response variable The parameters of the model can be estimated by the method of least squares Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
4. 4/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Generalized Linear Model The GeneraliZED Linear Model can be written as g(E(z)) = β0 + β1x1 + β2x2 + · · · + b + βkxk where g is a link function Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
5. 5/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions General Idea of the article A class of the Generalized Linear Models The parameters of the models can be estimated by the method of the maximum likelihood A generalization of the analysis of variance using the log-likelihood Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
6. 6/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions Table of contents 1 Introduction 2 Section 1: Definition of the Models The Random component The Systematic Effects The Generalized Linear Model 3 Section 2: Fitting process and Generalization of the analysis of variance The Maximum Likelihood Equations Sufficient Statistics The Analysis of Deviance Generalization of Analysis of Variance 4 Section 3: Example for Special Distributions The Normal Distribution The Poisson Distribution Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
7. 7/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Random component The Systematic Effects The Generalized Linear Model The Random Component Z our observations The density function of Z is an exponential family π(z, θ, φ) = exp[α(φ)zθ − g(θ) + h(z) + β(φ, z)] where φ is a nuisance parameter, fixed α(φ) > 0 is the scale factor Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
8. 8/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Random component The Systematic Effects The Generalized Linear Model The Random Component Expression of the moments We use the results: E[δL/δθ] = 0 (1) E[δ2L/δθ2] = −E[δL/δθ]2 (2) Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
9. 9/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Random component The Systematic Effects The Generalized Linear Model The Random Component Expression of the moments So L = α(φ) {z.θ − g(θ) + h(z)} (1) δL/δθ = α(φ) {z − g (θ)} = 0 µ = E(z) = g (θ) (2) E[δ2L/δθ2] = −α(φ)(g (θ)) −E[δL/δθ]2 = −α(φ)2 E[(z − g (θ))2] = −α(φ)2 E[(z − µ)2] ⇒ −α(φ)2 V(z) = −α(φ)(g (θ)) ⇒ V = α(φ)V(z) = g (θ) = δµ/δθ Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
10. 10/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Random component The Systematic Effects The Generalized Linear Model The Random Component Expression of the moments To conclude, for a one-parameter exponential family, α(φ) = 1, we have that: (1)δL/δθ = z − µ (2) E[δ2L/δθ2] = −V(z) Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
11. 11/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Random component The Systematic Effects The Generalized Linear Model The linear Model for Systematic Effects The systematic components are Y = β1x1 + β2x2 + · · · + βmxm where xi are independent variates, supposed to be known βi are parameters They may have fixed (known) values or be unknown and require estimation Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
12. 12/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Random component The Systematic Effects The Generalized Linear Model The Generalized Linear Model So, the GLM consists of 3 components 1 A dependent variable z whose distribution with parameter θ is a member of an exponential family 2 A set of independent variables x1, · · · , xm and predicted such as Y = β1 x1 + β2 x2 + · · · + βm xm 3 A linking function θ = f (Y ) connecting the parameter θ of the distribution of z with the Y’s of the linear model Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
13. 13/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Maximum Likelihood Equations Sufficient Statistics The Analysis of Deviance Generalization of Analysis of Variance The Maximum Likelihood Equations Estimation of the βi In the first section, we saw that: (1)δL/δθ = α(φ) {z − µ} (2) V = δµ/δθ and Y = βi xi ⇒ δY δβi = xi δL/δβi = (δL δθ )( δθ δβi ) = α(φ) {z − µ} δθ δY xi = α(φ) {z − µ} δθ δµ δµ δY xi = α(φ)(z−µ) V δµ δY xi ⇒ So the equations for the Likelihood are: α(φ) (z − µ)xi V δµ δY = 0 Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
14. 14/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Maximum Likelihood Equations Sufficient Statistics The Analysis of Deviance Generalization of Analysis of Variance The Maximum Likelihood Equations Estimation of the βi The solutions of the maximum likelihood equation is equivalent to an iterative weighted least-squares procedure with a weight function w = (dµ/dY )2/V and a modified dependent variable y = Y + (z − µ)/(dµ/dY ) µ, Y and V are based on the current estimates Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
15. 15/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Maximum Likelihood Equations Sufficient Statistics The Analysis of Deviance Generalization of Analysis of Variance The Maximum Likelihood Equations The Newton-Raphson Algorithm 1 Take as first approximation µ = z and calculate Y from it 2 Calculate w as before and set y=Y 3 Then obtain the first approximation to the β s by regression Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
16. 16/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Maximum Likelihood Equations Sufficient Statistics The Analysis of Deviance Generalization of Analysis of Variance Sufficient Statistics Particular case, when θ the parameter of the distribution of the random element and Y the predicted value of the linear model coincide δL/βi = α(φ) {z − µ} δθ δY xi = α(φ) {z − µ} xi = 0 (z − ˜ µ)xik = 0 ⇒ zkxik = ˜ µkxik where zkxik is a set of sufficient statistics Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
17. 17/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Maximum Likelihood Equations Sufficient Statistics The Analysis of Deviance Generalization of Analysis of Variance The Analysis of Deviance Definition A linear model is said to be ordered if the fitting of the β’s is to be done in the same sequence as their declaration in the model Assumption The model under consideration is ordered and will be fitted sequentially a term at a time Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
18. 18/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Maximum Likelihood Equations Sufficient Statistics The Analysis of Deviance Generalization of Analysis of Variance The Analysis of Deviance The model-fitting process with an ordered model thus consists of proceeding a suitable distance from the minimal model towards the complete model The Deviance is: D∗ = 2[L(˜ µ, z) − L(z, z)] where ˜ µ is the maximum likelihood estimate Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
19. 19/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Maximum Likelihood Equations Sufficient Statistics The Analysis of Deviance Generalization of Analysis of Variance The Analysis of Deviance For the four Distributions in our example, the Deviance is: Normal (z−˜ µ)2 σ2 Poisson 2 { zln(z/˜ µ) − (z − ˜ µ)} Binomial 2 zln(z/˜ µ) − (n − z)ln((n−z) (n−˜ µ) Gamma 2p {− ln(z/˜ µ) + (z − ˜ µ)/˜ µ} Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
20. 20/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Maximum Likelihood Equations Sufficient Statistics The Analysis of Deviance Generalization of Analysis of Variance The Analysis of Deviance r the degrees of freedom is given by the rank of the X matrix For a sample of n independent observations, the deviance for the model has residual degrees of freedom (n-r) When (residual degrees of freedom × scale factor) is approximately equal to the deviance of the current model Then the model is well fitted Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
21. 21/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Maximum Likelihood Equations Sufficient Statistics The Analysis of Deviance Generalization of Analysis of Variance Generalization of Analysis of Variance The first differences of the deviance for a sequential model are approximately a χ2 Distribution Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
22. 22/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Normal Distribution The Poisson Distribution The Normal Distribution Fisher’s tuberculin test data 16 measurements of tuberculin response The variances of the observations are proportional to their expectation ⇒ Conclusion : The estimates obtained by this method agree with the authors’ estimates to about four decimal places Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
23. 23/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Normal Distribution The Poisson Distribution The Poisson Distribution Maxwell’s Contingency table This table gives the number of boys having disturbed dreams Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
24. 24/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Normal Distribution The Poisson Distribution The Poisson Distribution Maxwell’s Contingency table The Authors’ results: 18.38 is a χ2 variate with a 1 degree of freedom 14.08 is a χ2 variate with 11 degrees of freedom Conclusion: The data are adequately described by a linear×linear interaction ⇒ The dream rating tends to decrease with age Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur
25. 25/25 Introduction Section 1: Definition of the Models Section 2: Fitting process and Generalization of the analysis of variance Section 3: Example for Special Distributions The Normal Distribution The Poisson Distribution Thank you for your time and your attention! References Generalized Linear Models, P McCullagh, JA Nelder, 1989 Introduction au Modele Lineaire Generalise, Eric Wajnberg, 2011 http : //www.unice.fr/coquillard/UE7/courshttp://www.math.univ- toulouse.fr/ besse/Wikistat/pdf/st-m-modlin-mlg.pdf http : //maths.cnam.fr/IMG/pdf /PresentationMODGEN022007.pdf Erweiterungen des Linearen Modells, Marcus Hudec, Universitat Wien Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur