GLMs by Nelder and Wedderburn, discussed by Ysé Wanono

1/25
Introduction
Section 1: Deﬁnition of the Models
Section 2: Fitting process and Generalization of the analysis of variance
Section 3: Example for Special Distributions
Generalized Linear Models by J.A. Nelder and
R.W.M Wedderburn
Yse Wanono
2/12/2013
Yse Wanono Generalized Linear Models by J.A. Nelder and R.W.M Wedderbur

View Slide

2/25
Introduction
A Simple Regression Model
Assumption Z is determined by X via a function f
Z = f (X) +
where is a random error term
The simplest way to represent a relationship between two
continuous variables is by mean of a linear function.

View Slide

3/25
Introduction
The General Linear Model
The General Linear Model can be written as
E(z) = β0 + β1x1 + β2x2 + · · · + βkxk
where x is an explanatory variable
z the response variable
The parameters of the model can be estimated by the method
of least squares

View Slide

4/25
Introduction
The Generalized Linear Model
The GeneraliZED Linear Model can be written as
g(E(z)) = β0 + β1x1 + β2x2 + · · · + b + βkxk
where g is a link function

View Slide

5/25
Introduction
General Idea of the article
A class of the Generalized Linear Models
The parameters of the models can be estimated by the
method of the maximum likelihood
A generalization of the analysis of variance using the
log-likelihood

View Slide

6/25
Introduction
Table of contents
1 Introduction
2 Section 1: Definition of the Models
The Random component
The Systematic Effects
3 Section 2: Fitting process and Generalization of the analysis of
variance
The Maximum Likelihood Equations
Sufficient Statistics
The Analysis of Deviance
Generalization of Analysis of Variance
4 Section 3: Example for Special Distributions
The Normal Distribution
The Poisson Distribution

View Slide

7/25
Introduction
The Random Component
Z our observations
The density function of Z is an exponential family
π(z, θ, φ) = exp[α(φ)zθ − g(θ) + h(z) + β(φ, z)]
where φ is a nuisance parameter, ﬁxed
α(φ) > 0 is the scale factor

View Slide

8/25
Introduction
Expression of the moments
We use the results:
E[δL/δθ] = 0 (1)
E[δ2L/δθ2] = −E[δL/δθ]2 (2)

View Slide

9/25
Introduction
So L = α(φ) {z.θ − g(θ) + h(z)}
(1) δL/δθ = α(φ) {z − g (θ)} = 0
µ = E(z) = g (θ)
(2) E[δ2L/δθ2] = −α(φ)(g (θ))
−E[δL/δθ]2 = −α(φ)2
E[(z − g (θ))2] = −α(φ)2
E[(z − µ)2]
⇒ −α(φ)2
V(z) = −α(φ)(g (θ))
⇒ V = α(φ)V(z) = g (θ) = δµ/δθ

View Slide

10/25
Introduction
To conclude,
for a one-parameter exponential family, α(φ) = 1, we have
that:
(1)δL/δθ = z − µ
(2) E[δ2L/δθ2] = −V(z)

View Slide

11/25
Introduction
The linear Model for Systematic Eﬀects
The systematic components are
Y = β1x1 + β2x2 + · · · + βmxm
where xi are independent variates, supposed to be known
βi are parameters
They may have ﬁxed (known) values or be unknown and
require estimation

View Slide

12/25
Introduction
So, the GLM consists of 3 components
1 A dependent variable z whose distribution with parameter θ
is a member of an exponential family
2 A set of independent variables x1, · · · , xm
and predicted
such as
Y = β1
x1
+ β2
x2
+ · · · + βm
xm
3 A linking function θ = f (Y ) connecting the parameter θ of
the distribution of z with the Y’s of the linear model

View Slide

13/25
Introduction
Estimation of the βi
In the ﬁrst section, we saw that:
(1)δL/δθ = α(φ) {z − µ}
(2) V = δµ/δθ
and Y = βi xi ⇒ δY
δβi
= xi
δL/δβi = (δL
δθ
)( δθ
δβi
) = α(φ) {z − µ} δθ
δY
xi
= α(φ) {z − µ} δθ
δµ
δµ
δY
xi = α(φ)(z−µ)
V
δµ
δY
xi
⇒ So the equations for the Likelihood are:
α(φ)
(z − µ)xi
V
δµ
δY
= 0

View Slide

14/25
Introduction
Estimation of the βi
The solutions of the maximum likelihood equation is
equivalent to an iterative weighted least-squares
procedure
with a weight function w = (dµ/dY )2/V
and a modiﬁed dependent variable y = Y + (z − µ)/(dµ/dY )
µ, Y and V are based on the current estimates

View Slide

15/25
Introduction
The Newton-Raphson Algorithm
1 Take as ﬁrst approximation µ = z and calculate Y from it
2 Calculate w as before and set y=Y
3 Then obtain the ﬁrst approximation to the β s by regression

View Slide

16/25
Introduction
Particular case, when θ the parameter of the distribution of
the random element and Y the predicted value of the linear
model coincide
δL/βi = α(φ) {z − µ} δθ
δY
xi = α(φ) {z − µ} xi = 0
(z − ˜
µ)xik = 0
⇒ zkxik = ˜
µkxik
where zkxik is a set of suﬃcient statistics

View Slide

17/25
Introduction
Definition
A linear model is said to be ordered if the fitting of the β’s is to
be done in the same sequence as their declaration in the model
Assumption
The model under consideration is ordered and will be fitted
sequentially a term at a time

View Slide

18/25
Introduction
The model-ﬁtting process with an ordered model thus consists
of proceeding a suitable distance from the minimal model
towards the complete model
The Deviance is:
D∗ = 2[L(˜
µ, z) − L(z, z)]
where ˜
µ is the maximum likelihood estimate

View Slide

19/25
Introduction
For the four Distributions in our example, the Deviance is:
Normal (z−˜
µ)2
σ2
Poisson 2 { zln(z/˜
µ) − (z − ˜
µ)}
Binomial 2 zln(z/˜
µ) − (n − z)ln((n−z)
(n−˜
µ)
Gamma 2p {− ln(z/˜
µ) + (z − ˜
µ)/˜
µ}

View Slide

20/25
Introduction
r the degrees of freedom is given by the rank of the X
matrix For a sample of n independent observations, the
deviance for the model has residual degrees of freedom
(n-r)
When (residual degrees of freedom × scale factor) is
approximately equal to the deviance of the current model
Then the model is well ﬁtted

View Slide

21/25
Introduction
The ﬁrst diﬀerences of the deviance for a sequential model are
approximately a χ2 Distribution

View Slide

22/25
Introduction
Fisher’s tuberculin test data
16 measurements of tuberculin response
The variances of the observations are proportional to their
expectation
⇒ Conclusion : The estimates obtained by this method
agree with the authors’ estimates to about four decimal places

View Slide

23/25
Introduction
Maxwell’s Contingency table
This table gives the number of boys having disturbed dreams

View Slide

24/25
Introduction
Maxwell’s Contingency table
The Authors’ results: 18.38 is a χ2 variate with a 1 degree
of freedom
14.08 is a χ2 variate with 11 degrees of freedom
Conclusion: The data are adequately described by a
linear×linear interaction
⇒ The dream rating tends to decrease with age

View Slide

25/25
Introduction
Thank you for your time and your attention!
References
Generalized Linear Models, P McCullagh, JA Nelder, 1989
Introduction au Modele Lineaire Generalise, Eric Wajnberg,
2011
http :
//www.unice.fr/coquillard/UE7/courshttp://www.math.univ-
toulouse.fr/ besse/Wikistat/pdf/st-m-modlin-mlg.pdf
http :
//maths.cnam.fr/IMG/pdf /PresentationMODGEN022007.pdf
Erweiterungen des Linearen Modells, Marcus Hudec, Universitat
Wien

View Slide

GLMs by Nelder and Wedderburn, discussed by Ysé Wanono

GLMs by Nelder and Wedderburn, discussed by Ysé Wanono

Xi'an

More Decks by Xi'an

Other Decks in Education

Featured

Transcript