# On the performance of principal component Liu-type estimator under the mean square error criterion.

1. Introduction

Consider the multiple linear regression model

y = X[beta] + [epsilon], (1)

where y is an n x 1 vector of observation, X is an n x p known matrix of rank p, [beta] is a p x 1 vector of unknown parameters, and [epsilon] is an n x 1 vector of disturbances with expectation E([epsilon]) = 0 and variance-covariance matrix Cov([epsilon]) = [[sigma].sup.2][I.sub.n].

According to the Gauss-Markov theorem, the classical ordinary least squares estimator (OLSE) is obtained as follows:

[??] = [(X'X).sup.-1]X' y. (2)

The OLSE has been regarded as the best estimator for a long time. However, when multicollinearity is present and the matrix X'X is ill-conditioned, the OLSE is no longer a good estimator. To improve OLSE, many ways have been proposed. One way is to consider biased estimator, such as, principal component regression estimator [1], ridge estimator [2], Liu estimator [3], Liu-type estimator [4], two-parameter ridge estimator [5], r-k class estimator [6], r-d class estimator [7], and modified r-k class estimator [8].

An alternative method to overcome the multicollinearity is to consider the restrictions. Xu and Yang [9] introduced a stochastic restricted Liu estimator; Li and Yang [10] introduced a stochastic restricted ridge estimator.

To overcome multicollinearity, Hoerl and Kennard [2] solve the following problem:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII], (3)

where k is a Lagrangian multiplier and c is a constant, and obtain the ridge estimator (RE):

[??](k) = [(X'X + kI).sup.-1]X'y, k > 0. (4)

Liu [3] introduced the Liu estimator (LE):

[??](d) = [(X'X + I).sup.-1] (X'y + d[??]), 0 < d < 1, (5)

where [??] is OLSE. This estimator can be obtained by solving the following problem:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (6)

This estimator can also be obtained by the following ways. Suppose that [beta] satisfied d[??] = [beta] + e. Then, we use the mixed method [11]; we can obtain the Liu estimator.

Recently, Huang et al. [4] introduced a Liu-type estimator which includes the OLSE, RE, and LE, defined as follows:

[??](k, d) = [(X'X + kI).sup.-1] (X'y + d[??]), k > 0, 0 < d < 1, (7)

where [??] is OLSE. This estimator can be obtained by solving the following problem:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (8)

Let us consider the following transformation for the model (1):

y = XTT'[beta] + [epsilon] = Z[alpha] + [epsilon], (9)

where

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (10)

[[LAMBDA].sub.r] and [[LAMBDA].sub.p - r] are diagonal matrices such that that the main diagonal elements of the r x r matrix [[LAMBDA].sub.r] are the r largest eigenvalues of X'X, while [[LAMBDA].sub.p - r] are the remaining p - r eigenvalues. The p x p matrix T = ([T.sub.r], [T.sub.p - r]) is orthogonal with [T.sub.r] = ([t.sub.1], [t.sub.2],..., [t.sub.r]) consisting of its first r columns and [T.sub.p - r] = ([t.sub.r + 1], [t.sub.r + 2],..., [t.sub.p]) consisting of the remaining p-r columns of the matrix T. The PCRE for [beta] can be written as

[??]r = [T.sub.r][([T'.sub.r]X'X[T.sub.r]).sup.-1][T'.sub.r]X'y. (11)

Baye and Parker [6] introduced the application of ridge approach to improve the PCR estimator, namely, r-k class estimator as

[[??].sub.r](k) = [T.sub.r][([T'.sub.r]X'X[T.sub.r] + k[I.sub.r]).sup.-1][T'.sub.r]X'y, k > 0. (12)

Alternatively, Kaciranlar and Sakallioglu [7] introduced the r-d class estimator which is the combination of the LE and the PCRE, which is defined as follows:

[[??].sub.r](d) = [T.sub.r][([T'.sub.r]X'X[T.sub.r] + [I.sub.r]).sup.-1]([T'.sub.r]X'y + d[T'.sub.r][[??].sub.r]), 0 < d < 1. (13)

Wu [12] proposed the principal component Liu-type estimator (PCTTE), which is defined as

[[??].sub.r](k, d) = [T.sub.r][([T'.sub.r]X'X[T.sub.r] + k[I.sub.r]).sup.-1]([T'.sub.r]X'y + d[T'.sub.r][[??].sub.r]), k > 0, 0 < d < 1. (14)

In this paper, firstly we use a new method to propose the principal component Liu-type estimator. Then, we show that, under certain conditions, the PCTTE is superior to the related estimator in the mean square error criterion. Finally, we give a numerical example to illustrate the theoretical results.

2. The Principal Component Liu-Type Estimator

Using the symbols in (9) and (10), (1) can be written as follows:

y = X[T.sub.r][T'.sub.r][beta] + X[T.sub.p - r][T'.sub.p - r][beta] + [epsilon] = [Z.sub.r][[alpha].sub.r] + [Z.sub.p - r][[alpha].sub.p - r] + [epsilon]. (15)

The PCRE can be obtained by omitted [Z.sub.p - r][[alpha].sub.p - r], and the model (15) reduced to:

y = [Z.sub.r][[alpha].sub.r] + [epsilon]. (16)

Then, solve the following problem:

(y - [Z.sub.r][[alpha].sub.r])' (y - [Z.sub.r][[alpha].sub.r]), (17)

we obtain

[[??].sub.r] = [([Z'.sub.r][Z.sub.r]).sup.-1][Z'.sub.r]y. (18)

Then, transforming [[??].sub.r] to the original parameter space, we can get the PCRE of parameter [beta].

Now, we give a method to obtain the principal component Liu-type estimator. Let c be a constant and k a Lagrangian multiplier, minimizing

(y - [Z.sub.r][[alpha].sub.r])' (y - [Z.sub.r][[alpha].sub.r]) + 1/k [(k[[alpha].sub.r] - d[[??].sub.r])(k[[alpha].sub.r] - d[[??].sub.r])' - c], (19)

where [[??].sub.r] = [([Z'.sub.r][Z.sub.r]).sup.-1][Z'.sub.r]y. Then we get

[[??].sub.r](k, d) = [([Z'.sub.r][Z.sub.r] + k[I.sub.r]).sup.-1]([Z'.sub.r]y + d[[??].sub.r]). (20)

After transforming back to original parameters pace, we obtain

[[??].sub.r](k, d) = [T.sub.r][([T'.sub.r]X'X[T.sub.r] + k[I.sub.r]).sup.-1] x ([T'.sub.r]X'y + d[T'.sub.r][[??].sub.r]), k > 0, 0 < d < 1. (21)

This estimator can also be got by minimizing the function

(k[[alpha].sub.r] - d[[??].sub.r])(k[[alpha].sub.r] - d[[??].sub.r])' + k[(y - [Z.sub.r][[alpha].sub.r])' (y - [Z.sub.r][[alpha].sub.r]) - c]. (22)

It is easy to see that the new estimator [[??].sub.r](k, d) has the following properties:

(1) [[??].sub.r](1, 1) = [[??].sub.r] = [T.sub.r][([T'.sub.r]X'X[T.sub.r]).sup.-1][T'.sub.r]X'y is the PCRE;

(2) [[??].sub.p](1, 1) = [[??].sub.OLS] = [(X'X).sup.-1]X'y is the OLSE;

(3) [[??].sub.r](1, d) = [[??].sub.r](d) = [[??].sub.r](d) = [T.sub.r][([T'.sub.r]X'X[T.sub.r] + [I.sub.r]).sup.-1] ([T'.sub.r]X'y + d[T'.sub.r][[??].sub.r]) is the r-d class estimator;

(4) [[??].sub.p](1, d) = [??](d) = [(X'X + I).sup.-1] (X'y + d[[??].sub.OLS]) is the LE;

(5) [[??].sub.r](k, 0) = [[??].sub.r](k) = [T.sub.r][([T'.sub.r]X'X[T.sub.r] + k[I.sub.r]).sup.-1][T'.sub.r]X'y is the r-k class estimator;

(6) [[??].sub.p](k, 0) = [??](k) = [(X'X + kI).sup.-1]X'y is the RE;

(7) [[??].sub.p](k, d) = [??](k, d) = [(X'X + kI).sup.-1](X'y + kd[[??].sub.OLS]) is the LTE.

3. Superiority of the Principal Component Liu-Type Estimator over Some Estimators in the Mean Square Error Criterion

The mean square error (MSE) of an estimator [??] is defined as

M([??]) = E[([??] - [beta])' ([??] - [beta])] = tr V([??]) + [Bias([??])]' [Bias([??])], (23)

where V([??]) is the dispersion matrix and Bias([??]) is the bias vector. For two given estimators [[??].sub.1] and [[??].sub.2],the estimator [[??].sub.2] is said to be superior to [[??].sub.1] in the MSE criterion, if and only if

[DELTA]([[??].sub.1], [[??].sub.2]) = M([[??].sub.1]) - M([[??].sub.2]) [greater than or equal to]0. (24)

3.1. [[??].sub.r](k, d) versus [[??].sub.r]. Firstly, we compute that V([[??].sub.r](k, d)) = [[sigma].sup.2][T.sub.r][S.sub.r][(k).sup.-1][S.sub.r](d)[[LAMBDA].sup.-1.sub.r][S.sub.r](d) [S.sub.r][(k).sup.-1][T'.sub.r], E([[??].sub.r](k, d)) = [T.sub.r][S.sub.r][(k).sup.-1][S.sub.r](d)[T'.sub.r][beta], (25)

where [S.sub.r](k) = [[LAMBDA].sub.r] + k[I.sub.r]. Then, the mean square error (MSE) of [[??].sub.r](k, d) is given as follows:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (26)

Let k = din (26); we obtain the MSE of [[??].sub.r] as follows:

M([[??].sub.r]) = [r.summation over (i = 1)] [[sigma].sup.2]/[[lambda].sub.i] + [p.summation over (i = r + 1)][[alpha].sup.2.sub.i]. (27)

Now we consider the following difference:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (28)

If [[sigma].sup.2] - [[alpha].sup.2.sub.i][[lambda].sub.i] > 0, then when 0 < d < k, M([[??].sub.r](k, d)) - M([[??].sub.r]) < 0. If [[sigma].sup.2] - [[alpha].sup.2.sub.i][[lambda].sub.i] < 0, then when 0 < d < k < min{(([[alpha].sup.2.sub.i][[lambda].sub.i] + [[sigma].sup.2])d + 2[[sigma].sup.2][[lambda].sub.i])/([[alpha].sup.2.sub.i][[lambda].sub.i] - [[sigma].sup.2])} and max{(([[sigma].sup.2.sub.i][[lambda].sub.i] + [[sigma].sup.2])d + 2[[sigma].sup.2][[lambda].sub.i])/([[alpha].sup.2.sub.i][[lambda].sub.i] - [[sigma].sup.2])} < k < d < 1, M([[??].sub.r](k, d)) - M([[??].sub.r]) < 0. So we have the following theorem.

Theorem 1. The estimator [[??].sub.r](k, d) is superior to the estimator for [[??].sub.r] under the mean square error criterion for:

(a) 0<d< k if [[sigma].sup.2] - [[alpha].sup.2.sub.i][[lambda].sub.i] > 0 for all i = 1,..., r,

(b) 0 < d < k < min{(([[alpha].sup.2.sub.i][[lambda].sub.i] + [[sigma].sup.2])d + 2[[sigma].sup.2][[lambda].sub.i])/([[alpha].sup.2.sub.i][[lambda].sub.i] - [[sigma].sup.2])} and max{(([[alpha].sup.2.sub.i][[lambda].sub.i] + [[sigma].sup.2])d + 2[[sigma].sup.2][[lambda].sub.i])/([[alpha].sup.2.sub.i][[lambda].sub.i] - [[sigma].sup.2])} < k < d < 1, i = 1,..., r if [[sigma].sup.2] - [[alpha].sup.2.sub.i][[lambda].sub.i] < 0 for all i = 1,..., r.

3.2. [[??].sub.r](k, d) versus [[??].sub.r](k). From the definition of the [[??].sub.r](k), we know that let d = 0 in [[??].sub.r](k, d), and we obtain the [[??].sub.r](k).

Theorem 2. Let k[[alpha].sup.2.sub.i] - [[sigma].sup.2] > 0 for all i = 1,..., r. Then, there exists a strictly positive d such that [[??].sub.r](k, d) is superior to [[??].sub.r](k) in the mean square error criterion for 0 < d < ([[summation].sup.r.sub.i = 1]((k[[alpha].sup.2.sub.i] - [[sigma].sup.2])/[([[lambda].sub.i] + k).sup.2]))/([[summation].sup.r.sub.i = 1](([[sigma].sup.2] + [[alpha].sup.2.sub.i][[lambda].sub.i])/([[lambda].sub.i][([[lambda].sub.i] + k).sup.2]))) < 1.

Proof. We know that M([[??].sub.r](k)) = M[[??].sub.r](k, 0), so that by continuity it is sufficient to show that M([[??].sub.r](k, d)) decreasing in the neighborhood of =0.

Performing the calculus for fixed k, we can see that

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (29)

So when [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]; that is to say, M([[??].sub.r](k, d)) < M([[??].sub.r](k, 0)) = M([[??].sub.r](k)). The proof of Theorem 2 is completed.

3.3. [[??].sub.r](k, d) versus [[??].sub.r](d). From the definition of the [[??].sub.r](d), we know that let k = 1 in [[??].sub.r](k, d), and we obtain that [[??].sub.r](d),

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (30)

Now, we discuss the following difference:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (31)

Then by (31), if we want [DELTA] < 0, then

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (32)

when k < 1, it is easy to know that [d.sub.1] is always less than 1, and [d.sub.1] is to be bigger than 0, if

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (33)

Then, we get

k > (2[[lambda].sub.i] + 1)[[sigma].sup.2] - [[alpha].sup.2.sub.i][[lambda].sub.i]/[[alpha].sup.2.sub.i]([[lambda].sub.i] + 2) - [[sigma].sup.2] (34)

for [[alpha].sup.2.sub.i]([[lambda].sub.i] + 2) - [[sigma].sup.2] > 0 with all i = 1,..., r.

Thus, we obtain the following theorem.

Theorem 3. If [[alpha].sup.2.sub.i]([[lambda].sub.i] + 2) -

[[sigma].sup.2] > 0 and k > {max(((2[[lambda].sub.i] + 1)[[sigma].sup.2] - [[alpha].sup.2.sub.i][[lambda].sub.i])/([[alpha].sup.2.sub.i]([[lambda].sub.i] + 2) - [[sigma].sup.2])), 0} for all i = 1,..., r, then the [[??].sub.r](k, d) is better than the [[??].sub.r](d) in the mean square error sense for k < l and

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (35)

3.4. [[??].sub.r](k, d) versus [??]. In this subsection, we will give the comparison of the [[??].sub.r](k, d) and [??] under the mean square error criterion.

Let k = d and r = p in (26); we obtain the MSE of [[??].sub.r] as follows:

M([??]) = [p.summation over (i = 1)] [[sigma].sup.2]/[[lambda].sub.i]. (36)

In order to compare the [[??].sub.r](k, d) and [??], now we consider the following difference:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (37)

when [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII], then

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (38)

Since the lower bound of d is less than 1, the lower bound of d may be less than 0. If the lower bound of of d is less than 0, then we can choose any d in [0, l]. Thus, we can get the following theorem.

Theorem 4.

(1) If [[alpha].sup.2.sub.i][[lambda].sub.i] - [[sigma].sup.2] < 0 for all i = 1, 2,..., p, then [[??].sub.r](k, d) is superior to [??] in the mean square error sense for any k > 0 and 0 < d < 1.

(2) If [[alpha].sup.2.sub.i][[lambda].sub.i] - [[sigma].sup.2] < 0 for some i = 1,2,..., r, then [[??].sub.r](k, d) is superior to [??] in the mean square error sense for [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] when [[summation].sup.p.sub.i = r + 1](([[alpha].sup.2.sub.i][[lambda].sub.i] - [[sigma].sup.2])/[[lambda].sub.i]) < 0.

3.5. [[??].sub.r](k, d) versus [??](k, d). In this subsection, we will compare [[??].sub.r](k, d) with [??](k, d) in the mean square error sense.

Define [v.sub.i] = ([[lambda].sub.i] + d)/([[lambda].sub.i] + k), then the mean square error of [[??].sub.r](k, d) can be written as

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (39)

Firstly, we discuss the difference between M([[??].sub.r](k, d)) and M([[??].sub.r - 1] (k, d)):

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (40)

So [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII].

If [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII].

Then we have the following theorem.

Theorem 5.

(1) If [[sigma].sup.2] < [[alpha].sup.2.sub.i][[lambda].sub.i] + 2k[[alpha].sup.2.sub.i] for some i [member of] [N.sub.p], then there exists a nonnegative [d.sub.3] such that M([??](k, d)) > M([[??].sub.r](k, d)) for 0 < [d.sub.3] < d < 1, where [N.sub.p] = r + 1,..., p.

(2) If [[sigma].sup.2] < [[alpha].sup.2.sub.i][[lambda].sub.i] + 2k[[alpha].sup.2.sub.i] for all i [member of] [N.sub.p], then there exists a nonnegative [d.sub.3] such that M([??](k, d)) < M([[??].sub.r](k, d)) for 0 < d < [d.sub.4] < 1, where [N.sub.p] = r + 1,..., p, where

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (41)

4. Numerical Example

To illustrate our theoretical results, firstly we use a numerical example to investigate the estimators studied in the dataset originally due to Gruber [13], and later considered by Akdeniz and Erol [14]. We assemble the data as follows:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. (42)

Firstly, we can compute the OLSE of [beta] as follows:

[??] = (0.6921, 0.6258, -0.11H54, 0.2866, 0.0256)'. (43)

For the OLSE, PCRE, r-k class estimator, r-d class estimator, Liu-type estimator (LTE), and new estimator (PCTTE), their estimated mean square error (MSE) values are obtained by replacing all unknown model parameters by their, respectively, least squares estimators in corresponding expressions.

Firstly, we see the comparison of the OLSE and the PCTTE. When d = 0.6 is fixed, then if the values of k is big then the new estimator has smaller MSE values than OLSE; that is to say, the new estimator is better than the OLSE. So we see that the new estimator improved the OLSE.

From Figure 3, we see that when d, fixed, if 0 < d < k, then the new estimator is better than the PCRE. In Theorem 1, we see that if 0 < d < k, the new estimator is better. For the numerical example, when 0 < d < k, the new estimator is more efficient than the PCRE.

By Figures 1, 2, 3, 4, 5, 6, 7, and 8, we see that when 0 < d < k k, then new estimator is better than the LTE. So in practice, we can choose bigger k and smaller d.

5. Conclusion

In this paper, we use a new method to propose the principal component Liu-type estimator. Then, we discuss the superiority of the new estimator with the OLSE, PCRE, r-k class estimator, r-d class estimator, and Liu-type estimator in the sense of mean square error. We also give a method to choose the biasing parameters. Finally, we give a numerical example to illustrate the performance the various estimators.

http://dx.doi.org/10.1155/2013/858794

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

The author is grateful to three anonymous referees for their valuable comments which improved the quality of the paper. This work was supported by the Scientific Research Foundation of Chongqing University of Arts and Sciences (Grants nos. Y2013SC43, R2013SC12), Program for Innovation Team Building at Institutions of Higher Education in Chongqing (Grant no. KJTD201321), and the National Natural Science Foundation of China (nos. 11201505, 71271227).

References

[1] W. F. Massy, "Principal components regression in exploratory statistical reseach," Journal of the American Statistical Association, vol. 60, pp. 234-256, 1965.

[2] A. E. Hoerl and R. W. Kennard, "Ridge regression: biased estimation for nonorthogonal problems," Technometrics, vol. 12, no. 1, pp. 55-67, 1970.

[3] K. J. Liu, "A new class of biased estimate in linear regression," Communications in Statistics, vol. 22, no. 2, pp. 393-402, 1993.

[4] W. H. Huang, J. J. Qi, and N. T. Huang, "Liu-type estimation for a linear regression model with linear restrictions," Journal of Systems Science and Mathematical Sciences, vol. 29, no. 7, pp. 937-946, 2009.

[5] S. Toker and S. Kaciranlar, "On the performance of two parameter ridge estimator under the mean square error criterion," Applied Mathematics and Computation, vol. 219, no. 9, pp. 4718-4728, 2013.

[6] M. R. Baye and D. F. Parker, "Combining ridge and principal component regression: a money demand illustration," Communications in Statistics, vol. 13, no. 2, pp. 197-205, 1984.

[7] S. Kaciranlar and S. Sakallioglu, "Combining the Liu estimator and the principal component regression estimator," Communications in Statistics, vol. 30, no. 12, pp. 2699-2705, 2001.

[8] M. I. Alheety and B. M. G. Kibria, "Modified Liu-type estimator based on (r-k) class estimator, "Communications in Statistics, vol. 42, no. 2, pp. 304-319, 2013.

[9] J. Xu and H. Yang, "On the restricted almost unbiased estimators in linear regression," Journal of Applied Statistics, vol. 38, no. 3, pp. 605-617, 2011.

[10] Y. Li and H. Yang, "Two kinds of restricted modified estimators in linear regression model," Journal of Applied Statistics, vol. 38, no. 7, pp. 1447-1454, 2011.

[11] J. Durbin, "A note on regression when there is extraneous information that one of coecients," Journal of the American Statistical Association, vol. 48, pp. 799-808, 1953.

[12] J. B. Wu, Research on the properties of parameter estimation in linear regression model [Ph.D. papers], Chongqing University, Chongqing, China, 2013.

[13] M. H. J. Gruber, Improving Efficiency by Shrinkage: The James-Stein and Ridge Regression Estimators, vol. 156, Marcel Dekker, New York, NY, USA, 1998.

[14] F. Akdeniz and H. Erol, "Mean squared error matrix comparisons of some biased estimators in linear regression," Communications in Statistics, vol. 32, no. 12, pp. 2389-2413, 2003.

Jibo Wu (1, 2)

(1) School of Mathematics and Finances, Chongqing University of Arts and Sciences, Chongqing 402160, China

(2) Department of Mathematics and KLDAIP, Chongqing University of Arts and Sciences, Chongqing 402160, China

Correspondence should be addressed to Jibo Wu; linfen52@126.com

Received 10 October 2013; Accepted 12 November 2013

Academic Editor: Renat Zhdanov
COPYRIGHT 2013 Hindawi Limited
No portion of this article can be reproduced without the express written permission from the copyright holder.