STAT 6110: Linear Models
Problem Set 3
- Consider a one-way ANOVA model
Yij = µ + αi + ij , ij ∼ N(0, σ2
), i = 1, 2, 3, j = 1, . . . , ni
, n1 = 3, n2 = 1, n3 = 2.
(a) Write this model in matrix notation.
(b) Show that µ + α1, α1 − α2 and α1 + α2 − 2α3 are estimable.
(c) Identify all of the estimable parameters.
(d) Construct the F test for the hypothesis
H0 : α1 + α2 − 2α3 = 0 and α1 − α2 = 0.
(e) Construct a 95% confidence ‘interval’ (!!) for α1 − α2.
sol) (a)
Y = Xβ + ,
where
Y = (Y11, Y12, Y13, Y21, Y31, Y32)
0
, X =
1 1 0 0
1 1 0 0
1 1 0 0
1 0 1 0
1 0 0 1
1 0 0 1
, β = (µ, α1, α2, α3)
0
,
= (11, 12, 13, 21, 31, 32)
0 ∼ N6(0, σ2
I).
(b) Recall that Xβ is always estimable (we can write Xβ = P
0Xβ for P = I). Note
that Xβ = ((µ + α1)j
0
3
, µ + α2,(µ + α3)j
0
2
)
0
.
Estimable functions are linear combinations of the rows of Xβ. All given functions
are linear combinations of the rows of Xβ, and hence are estimable.
(c) All linear combinations of the rows of Xβ.
(d) Observe that the hypothesis can be written as Λ0β = with Λ0 =
0 1 1 −2
0 1 −1 0
.
Thus we can choose P
0 =
1 0 0 1 −2 0
1 0 0 −1 0 0
such that Λ0 = P
0X. (Note
that P is not unique.) Using M = X(X0X)
−X0
(try to find this; you only need
the last three columns of X), the test statistic is given by
F =
Y
0MP(P
0MP)
−P
0MY/2
Y
0
(I − M)Y/3
∼ F(2, 3, γ),
where γ = β
0X0
(MP(P
0MP)
−P
0M)(Xβ)/(2σ
2
).
1
(e) Let βˆ be a least squares estimator of β such that Xβˆ = MY (you can find ‘a’ βˆ
using this expression). A 95% confidence interval for α1 − α2 is
(
d :
(λ
0βˆ − d)
0
(λ
0
(X0X)
−λ)
−(λ
0βˆ − d)
MSE ≤ c0.05)
(
d :
(λ
0βˆ − d)
2
MSE · λ
0
(X0X)
−λ
≤ c0.05)
n
d : λ
0βˆ −
p
c0.05 · MSE · λ
0
(X0X)
−λ ≤ d ≤ λ
0βˆ +
p
c0.05 · MSE · λ
0
(X0X)
−λ
o
,
where c0.05 = F(0.95, 1, 3) is the upper 95% percentile of a central F distribution
with degrees of freedom (1, 3) and λ
0 = (0, 1, −1, 0). (This shows that we can also
use a t-distribution.)
Since ρ
0 = (1, 0, 0, −1, 0, 0) satisfies λ
0 = ρ
0X, the ratio above can also be expressed
as
(ρ
0MY − d)
2
/(MSE · ρ
0M ρ).
- Consider a regression model Y = Xβ + , ∼ N(0, σ2
I) and suppose that we want to
predict the value of a future observation, say y0, that will be independent of Y and be
distributed N(x
0
0β, σ2
). Assume that X is of full-column rank.
(a) Find the distribution of
y0 − x
0
0βˆ
p
MSE · [1 + x
0
0
(X0X)
−1x0]
.
(βˆ is a least squares estimator of β.)
(b) Find a 95% prediction interval for y0.
(Note: A preprediction interval is similar to a confidence interval except that,
rather than finding parameter values that are consistent with the data and the
model, one finds new observations y0 that are consistent with the data and the
model as determined by an α level test.)
sol) (a) Since y0 and Y are independent, y0 − x
0
0βˆ ∼ N(0, σ2
(1 + x
0
0
(X0X)
−1x0)). Hence,
y0 − x
0
0βˆ
[σ
2
(1 + x
0
0
(X0X)
−1x0)]1/2
∼ N(0, 1).
Note also that
SSE
σ
2
(Y − Xβˆ)
0
(Y − Xβˆ)
σ
2
∼ χ
2
n−k
.
2
and
Cov(β, ˆ (I − M)Y ) =(X
0X)
−1X
0Cov(Y )(I − M)
=σ
2
(X
0X)
−1
[(I − M)X]
0
=0.
Hence, βˆ is independent of MSE, and thus the numerator is independent of the
denominator. Therefore,
y0 − x
0
0βˆ
[MSE(1 + x
0
0
(X0X)
−1x0)]1/2
∼ t(n − k).
(b) Using the above t-distribution,
(x
0
0βˆ−t0.025,n−k{MSE[1+x
0
0
(X
0X)
−1x0]}
1/2
, x0
0βˆ+t0.025,n−k{MSE[1+x
0
0
(X
0X)
−1x0]}
1/2
)
- Consider the model
Yi = β0 + β1xi1 + β2xi2 + β11x
2
i1 + β22x
2
i2 + β12xi1xi2 + i
,
where the predictor variables take on the following values.
i 1 2 3 4 5 6 7
xi1 1 1 −1 −1 0 0 0
xi2 1 −1 1 −1 0 0 0
Show that β0, β1, β2, β11 + β22, β12 are estimable and find algebraic forms for the estimates of these parameters. Find the MSE and the standard errors of the estimates.
sol) Observe that we can write the model as Y = Xβ + for
X =
1 1 1 1 1 1
1 1 −1 1 1 −1
1 −1 1 1 1 −1
1 −1 −1 1 1 1
1 0 0 0 0 0
1 0 0 0 0 0
1 0 0 0 0 0
and β = (β0, β1, β2, β11, β22, β12)
0
Let’s write the ith row of X as α
0
i
. Then,
3
ˆ β0 = (1, 0, 0, 0, 0, 0)0β = λ
0
0β
Since λ0 = α5 ∈ C(X0
), β0 is estimable.
ˆ β1 = (0, 1, 0, 0, 0, 0)0β = λ
0
1β
Since λ1 =
1
4
(α1 + α2 − α3 − α4) ∈ C(X0
), β1 is estimable.
ˆ β2 = (0, 0, 1, 0, 0, 0)0β = λ
0
2β
Since λ2 =
1
4
(α1 − α2 + α3 − α4) ∈ C(X0
), β2 is estimable.
ˆ β11 + β22 = (0, 0, 0, 1, 1, 0)0β = λ
0
3β
Since λ3 =
1
4
(α1 + α2 + α3 + α4) − α5 ∈ C(X0
), β11 + β22 is estimable.
ˆ β12 = (0, 0, 0, 0, 0, 1)0β = λ
0
4β
Since λ4 =
1
4
(α1 − α2 − α3 + α4) ∈ C(X0
), β12 is estimable.
These give ρ
0
0 = (0, 0, 0, 0, 1, 0, 0), ρ
0
1 = (1/4, 1/4, −1/4, −1/4, 0, 0, 0),
ρ
0
2 = (1/4, −1/4, 1/4, −1/4, 0, 0, 0), ρ
0
3 = (1/4, 1/4, 1/4, 1/4, −1, 0, 0),
ρ
0
4 = (1/4, −1/4, −1/4, 1/4, 0, 0, 0), for which we have λi = ρ
0
iX, i = 0, . . . , 4.
Let A be the matrix containing the orthonomal basis of X, then
A0 =
0 0 0 0 √
1
3
√
1
3
√
1
3
1
2
1
2 −
1
2 −
1
2
0 0 0
1
2 −
1
2
1
2 −
1
2
0 0 0
1
2
1
2
1
2
1
2
0 0 0
1
2 −
1
2 −
1
2
1
2
0 0 0
. Then the orthogonal projection operator
onto C(X) is M = AA0 =
1 0 0 0 0 0 0
0 1 0 0 0 0 0
0 0 1 0 0 0 0
0 0 0 1 0 0 0
0 0 0 0 1
3
1
3
1
3
0 0 0 0 1
3
1
3
1
3
0 0 0 0 1
3
1
3
1
3
. (You may have obtained M as
X(X0X)
−X0
, but using this matrix A is certainly convenient.)
Then,
ˆ βˆ
0 = λ
0
0βˆ = ρ
0
0MY =
1
3
(Y5 + Y6 + Y7).
ˆ βˆ
1 = λ
0
1βˆ = ρ
0
1MY =
1
4
(Y1 + Y2 − Y3 − Y4).
ˆ βˆ
2 = λ
0
2βˆ = ρ
0
2MY =
1
4
(Y1 − Y2 + Y3 − Y4).
ˆ βˆ
11 + βˆ
22 = λ
0
3βˆ = ρ
0
3MY =
1
4
(Y1 + Y2 + Y3 + Y4) −
1
3
(Y5 + Y6 + Y7).
ˆ βˆ
12 = λ
0
4βˆ = ρ
0
4MY =
1
4
(Y1 − Y2 − Y3 + Y4).
Observe
MSE =
Y
0
(I − M)Y
7 − 4
1
3
(Y5, Y6, Y7)
−2/3 1/3 1/3
1/3 −2/3 1/3
1/3 1/3 −2/3
(Y5, Y6, Y7)
0
,
and
4
ˆ V ar(β0) = MSE·ρ
0
0M ρ0 = MSE ·
1
3
, std(β0) =
√
MSE
√
3
,
ˆ V ar(β1) = MSE ·
1
4
, std(β1) =
√
MSE
2
,
ˆ V ar(β2) = MSE ·
1
4
, std(β2) =
√
MSE
2
,
ˆ V ar(β11 + β22) = MSE · (
1
4 +
1
3
) = MSE ·
7
12 , std(β1) =
√
MSE
2
,
ˆ V ar(β12) = MSE ·
1
4
, std(β12) =
√
MSE
2
.
- Show that β
0X0MMP Xβ = 0 if and only if Λ0β = 0.
sol)
β
0X
0MMP Xβ = 0 ⇔ kMMP Xβk
2 = 0
⇔ MMP Xβ = 0
⇔ Xβ ∈ C(MMP )
⊥
⇔ Xβ ∈ C(MP)
⊥
⇔ P
0MXβ = 0
⇔ P
0Xβ = 0
⇔ Λ
0β = 0 - Example 2.1.4
- Example 2.1.5
- Example 3.2.0
- Example 3.2.2
- Example 3.3.4
- Example 3.3.9
5
[Additional Exercises without Solutions] - Consider a linear model Y = Xβ + , ∼ Nn(0, σ2
I).
(a) For λ
0β estimable, find the distribution of
λ
0βˆ − λ
0β
p
MSE · λ
0
(X0X)
−λ
,
where λ
0βˆ is the least squares estimator (BLUE) of λ
0β.
(b) Find the form for a confidence interval for λ
0β. - Consider a linear model
Yi = β0 + β1xi1 + . . . + βkxik + i
, i = 1, 2, . . . , n. (1)
(a) Show that the noncentered model (1) can be written in the following centered
form:
Yi = α + β1(xi1 − x¯1) + . . . + βk(xik − x¯k) + i
, (2)
where ¯xi =
1
n
Pn
i=1 xij , j = 1, . . . , k. What is α?
(b) Show that the centered model (2) can be written in the following matrix form:
Y = [jn, Xc]
α
β
∗
- , (3)
where Y = (Y1, . . . , Yn)
0
, β∗ = (β1, . . . , βk)
0
, = (1, . . . , n)
0
Xc =
I −
1
n
Jn
X, X = [X1, . . . , Xk], Xj = (x1j
, . . . , xnj )
0
.
(c) Show that j
0
nXc is a zero vector.
(d) Assume that both X and Xc are of full column rank. Show that the estimators
for α and β
∗
in (3) are ˆα = Y¯ =
1
n
Pn
i=1 Yi and βˆ∗ = (X0
cXc)
−1X0
cY .
(e) Show that βˆ∗
is the same as the last three elements of βˆ = (X0
fXf )
−1X0
fY , where
Xf = [jn, X]. What is the relationship between ˆα and the first element of βˆ?
Note: Assume that both Xc and Xf are of full-column rank.
- Consider the linear model
Y = Xβ +
6
where Y = (Y1, Y2, Y3)
0
, β = (β1, β2)
0
,
X =
1 2
3 6
2 4
and ∼ N3(0, σ2
I). Construct the F test for the hypothesis
H0 : β1 + 2β2 = 1. - Exercise 2.11.3
- Exercise 2.11.4 (Recall that the normal equation is X0Xβ = X0Y .)
- Exercise 2.11.6
- Exercise 3.4
- Exercise 3.6
- Exercise 3.9.3