2.8k

# GLMs by Nelder and Wedderburn, discussed by Ysé Wanono

presentation made by Ysé Wanono at the Reading Classics Seminar, TSI, Université Paris-Dauphine, Dec. 2, 2013 ## Xi'an

December 05, 2013

## Transcript

1. 1/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
Generalized Linear Models by J.A. Nelder and
R.W.M Wedderburn
Yse Wanono
2/12/2013
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

2. 2/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
A Simple Regression Model
Assumption Z is determined by X via a function f
Z = f (X) +
where is a random error term
The simplest way to represent a relationship between two
continuous variables is by mean of a linear function.
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

3. 3/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The General Linear Model
The General Linear Model can be written as
E(z) = β0 + β1x1 + β2x2 + · · · + βkxk
where x is an explanatory variable
z the response variable
The parameters of the model can be estimated by the method
of least squares
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

4. 4/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Generalized Linear Model
The GeneraliZED Linear Model can be written as
g(E(z)) = β0 + β1x1 + β2x2 + · · · + b + βkxk
where g is a link function
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

5. 5/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
General Idea of the article
A class of the Generalized Linear Models
The parameters of the models can be estimated by the
method of the maximum likelihood
A generalization of the analysis of variance using the
log-likelihood
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

6. 6/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
1 Introduction
2 Section 1: Deﬁnition of the Models
The Random component
The Systematic Eﬀects
The Generalized Linear Model
3 Section 2: Fitting process and Generalization of the analysis of
variance
The Maximum Likelihood Equations
Suﬃcient Statistics
The Analysis of Deviance
Generalization of Analysis of Variance
4 Section 3: Example for Special Distributions
The Normal Distribution
The Poisson Distribution
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

7. 7/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Random component
The Systematic Eﬀects
The Generalized Linear Model
The Random Component
Z our observations
The density function of Z is an exponential family
π(z, θ, φ) = exp[α(φ)zθ − g(θ) + h(z) + β(φ, z)]
where φ is a nuisance parameter, ﬁxed
α(φ) > 0 is the scale factor
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

8. 8/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Random component
The Systematic Eﬀects
The Generalized Linear Model
The Random Component
Expression of the moments
We use the results:
E[δL/δθ] = 0 (1)
E[δ2L/δθ2] = −E[δL/δθ]2 (2)
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

9. 9/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Random component
The Systematic Eﬀects
The Generalized Linear Model
The Random Component
Expression of the moments
So L = α(φ) {z.θ − g(θ) + h(z)}
(1) δL/δθ = α(φ) {z − g (θ)} = 0
µ = E(z) = g (θ)
(2) E[δ2L/δθ2] = −α(φ)(g (θ))
−E[δL/δθ]2 = −α(φ)2
E[(z − g (θ))2] = −α(φ)2
E[(z − µ)2]
⇒ −α(φ)2
V(z) = −α(φ)(g (θ))
⇒ V = α(φ)V(z) = g (θ) = δµ/δθ
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

10. 10/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Random component
The Systematic Eﬀects
The Generalized Linear Model
The Random Component
Expression of the moments
To conclude,
for a one-parameter exponential family, α(φ) = 1, we have
that:
(1)δL/δθ = z − µ
(2) E[δ2L/δθ2] = −V(z)
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

11. 11/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Random component
The Systematic Eﬀects
The Generalized Linear Model
The linear Model for Systematic Eﬀects
The systematic components are
Y = β1x1 + β2x2 + · · · + βmxm
where xi are independent variates, supposed to be known
βi are parameters
They may have ﬁxed (known) values or be unknown and
require estimation
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

12. 12/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Random component
The Systematic Eﬀects
The Generalized Linear Model
The Generalized Linear Model
So, the GLM consists of 3 components
1 A dependent variable z whose distribution with parameter θ
is a member of an exponential family
2 A set of independent variables x1, · · · , xm
and predicted
such as
Y = β1
x1
+ β2
x2
+ · · · + βm
xm
3 A linking function θ = f (Y ) connecting the parameter θ of
the distribution of z with the Y’s of the linear model
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

13. 13/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Maximum Likelihood Equations
Suﬃcient Statistics
The Analysis of Deviance
Generalization of Analysis of Variance
The Maximum Likelihood Equations
Estimation of the βi
In the ﬁrst section, we saw that:
(1)δL/δθ = α(φ) {z − µ}
(2) V = δµ/δθ
and Y = βi xi ⇒ δY
δβi
= xi
δL/δβi = (δL
δθ
)( δθ
δβi
) = α(φ) {z − µ} δθ
δY
xi
= α(φ) {z − µ} δθ
δµ
δµ
δY
xi = α(φ)(z−µ)
V
δµ
δY
xi
⇒ So the equations for the Likelihood are:
α(φ)
(z − µ)xi
V
δµ
δY
= 0
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

14. 14/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Maximum Likelihood Equations
Suﬃcient Statistics
The Analysis of Deviance
Generalization of Analysis of Variance
The Maximum Likelihood Equations
Estimation of the βi
The solutions of the maximum likelihood equation is
equivalent to an iterative weighted least-squares
procedure
with a weight function w = (dµ/dY )2/V
and a modiﬁed dependent variable y = Y + (z − µ)/(dµ/dY )
µ, Y and V are based on the current estimates
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

15. 15/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Maximum Likelihood Equations
Suﬃcient Statistics
The Analysis of Deviance
Generalization of Analysis of Variance
The Maximum Likelihood Equations
The Newton-Raphson Algorithm
1 Take as ﬁrst approximation µ = z and calculate Y from it
2 Calculate w as before and set y=Y
3 Then obtain the ﬁrst approximation to the β s by regression
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

16. 16/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Maximum Likelihood Equations
Suﬃcient Statistics
The Analysis of Deviance
Generalization of Analysis of Variance
Suﬃcient Statistics
Particular case, when θ the parameter of the distribution of
the random element and Y the predicted value of the linear
model coincide
δL/βi = α(φ) {z − µ} δθ
δY
xi = α(φ) {z − µ} xi = 0
(z − ˜
µ)xik = 0
⇒ zkxik = ˜
µkxik
where zkxik is a set of suﬃcient statistics
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

17. 17/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Maximum Likelihood Equations
Suﬃcient Statistics
The Analysis of Deviance
Generalization of Analysis of Variance
The Analysis of Deviance
Deﬁnition
A linear model is said to be ordered if the ﬁtting of the β’s is to
be done in the same sequence as their declaration in the model
Assumption
The model under consideration is ordered and will be ﬁtted
sequentially a term at a time
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

18. 18/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Maximum Likelihood Equations
Suﬃcient Statistics
The Analysis of Deviance
Generalization of Analysis of Variance
The Analysis of Deviance
The model-ﬁtting process with an ordered model thus consists
of proceeding a suitable distance from the minimal model
towards the complete model
The Deviance is:
D∗ = 2[L(˜
µ, z) − L(z, z)]
where ˜
µ is the maximum likelihood estimate
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

19. 19/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Maximum Likelihood Equations
Suﬃcient Statistics
The Analysis of Deviance
Generalization of Analysis of Variance
The Analysis of Deviance
For the four Distributions in our example, the Deviance is:
Normal (z−˜
µ)2
σ2
Poisson 2 { zln(z/˜
µ) − (z − ˜
µ)}
Binomial 2 zln(z/˜
µ) − (n − z)ln((n−z)
(n−˜
µ)
Gamma 2p {− ln(z/˜
µ) + (z − ˜
µ)/˜
µ}
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

20. 20/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Maximum Likelihood Equations
Suﬃcient Statistics
The Analysis of Deviance
Generalization of Analysis of Variance
The Analysis of Deviance
r the degrees of freedom is given by the rank of the X
matrix For a sample of n independent observations, the
deviance for the model has residual degrees of freedom
(n-r)
When (residual degrees of freedom × scale factor) is
approximately equal to the deviance of the current model
Then the model is well ﬁtted
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

21. 21/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Maximum Likelihood Equations
Suﬃcient Statistics
The Analysis of Deviance
Generalization of Analysis of Variance
Generalization of Analysis of Variance
The ﬁrst diﬀerences of the deviance for a sequential model are
approximately a χ2 Distribution
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

22. 22/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Normal Distribution
The Poisson Distribution
The Normal Distribution
Fisher’s tuberculin test data
16 measurements of tuberculin response
The variances of the observations are proportional to their
expectation
⇒ Conclusion : The estimates obtained by this method
agree with the authors’ estimates to about four decimal places
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

23. 23/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Normal Distribution
The Poisson Distribution
The Poisson Distribution
Maxwell’s Contingency table
This table gives the number of boys having disturbed dreams
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

24. 24/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Normal Distribution
The Poisson Distribution
The Poisson Distribution
Maxwell’s Contingency table
The Authors’ results: 18.38 is a χ2 variate with a 1 degree
of freedom
14.08 is a χ2 variate with 11 degrees of freedom
Conclusion: The data are adequately described by a
linear×linear interaction
⇒ The dream rating tends to decrease with age
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

25. 25/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
The Normal Distribution
The Poisson Distribution