In statistics, principal component regression (PCR) is a regression analysis technique that is based on principal component analysis (PCA). PCR is a form of reduced rank regression. More specifically, PCR is used for estimating the unknown regression coefficients in a standard linear regression model.
In PCR, instead of regressing the dependent variable on the explanatory variables directly, the principal components of the explanatory variables are used as regressors. One typically uses only a subset of all the principal components for regression, making PCR a kind of regularized procedure and also a type of shrinkage estimator.
Often the principal components with higher variances (the ones based on eigenvectors corresponding to the higher eigenvalues of the sample variance-covariance matrix of the explanatory variables) are selected as regressors. However, for the purpose of predicting the outcome, the principal components with low variances may also be important, in some cases even more important.
One major use of PCR lies in overcoming the multicollinearity problem which arises when two or more of the explanatory variables are close to being collinear.Dodge, Y. (2003) The Oxford Dictionary of Statistical Terms, OUP. PCR can aptly deal with such situations by excluding some of the low-variance principal components in the regression step. In addition, by usually regressing on only a subset of all the principal components, PCR can result in dimension reduction through substantially lowering the effective number of parameters characterizing the underlying model. This can be particularly useful in settings with high-dimensional covariates. Also, through appropriate selection of the principal components to be used for regression, PCR can lead to efficient prediction of the outcome based on the assumed model.
The principle
The PCR method may be broadly divided into three major steps:
1. Perform PCA on the observed data matrix for the explanatory variables to obtain the principal components, and then (usually) select a subset, based on some appropriate criteria, of the principal components so obtained for further use.
2. Now regress the observed vector of outcomes on the selected principal components as covariates, using ordinary least squares regression (linear regression) to get a vector of estimated regression coefficients (with dimension equal to the number of selected principal components).
3. Now transform this vector back to the scale of the actual covariates, using the selected PCA loadings (the eigenvectors corresponding to the selected principal components) to get the final PCR estimator (with dimension equal to the total number of covariates) for estimating the regression coefficients characterizing the original model.
Details of the method
Data representation: Let denote the vector of observed outcomes and denote the corresponding data matrix of observed covariates where, and denote the size of the observed sample and the number of covariates respectively, with . Each of the rows of denotes one set of observations for the dimensional covariate and the respective entry of denotes the corresponding observed outcome.
Data pre-processing: Assume that and each of the columns of have already been centered so that all of them have zero empirical means. This centering step is crucial (at least for the columns of ) since PCR involves the use of PCA on and PCA is sensitive to centering of the data.
Underlying model: Following centering, the standard Gauss–Markov linear regression model for on can be represented as: where denotes the unknown parameter vector of regression coefficients and denotes the vector of random errors with and for some unknown variance parameter
Objective: The primary goal is to obtain an efficient estimator for the parameter , based on the data. One frequently used approach for this is ordinary least squares regression which, assuming is full column rank, gives the unbiased estimator: of . PCR is another technique that may be used for the same purpose of estimating .
PCA step: PCR starts by performing a PCA on the centered data matrix . For this, let denote the singular value decomposition of where, with denoting the non-negative singular values of , while the columns of and are both orthonormal sets of vectors denoting the left and right singular vectors of respectively.
The principal components: gives a spectral decomposition of where with denoting the non-negative eigenvalues (also known as the principal values) of , while the columns of denote the corresponding orthonormal set of eigenvectors. Then, and respectively denote the principal component and the principal component direction (or PCA loading) corresponding to the largest principal value for each .
Derived covariates: For any , let denote the matrix with orthonormal columns consisting of the first columns of . Let denote the matrix having the first principal components as its columns. may be viewed as the data matrix obtained by using the transformed covariates instead of using the original covariates .
The PCR estimator: Let denote the vector of estimated regression coefficients obtained by ordinary least squares regression of the response vector on the data matrix . Then, for any , the final PCR estimator of based on using the first principal components is given by: .
Fundamental characteristics and applications of the PCR estimator
Two basic properties
The fitting process for obtaining the PCR estimator involves regressing the response vector on the derived data matrix which has orthogonal columns for any since the principal components are mutually orthogonal to each other. Thus in the regression step, performing a multiple linear regression jointly on the selected principal components as covariates is equivalent to carrying out independent simple linear regressions (or univariate regressions) separately on each of the selected principal components as a covariate.
When all the principal components are selected for regression so that , then the PCR estimator is equivalent to the ordinary least squares estimator. Thus, . This is easily seen from the fact that and also observing that is an orthogonal matrix.
Variance reduction
For any , the variance of is given by
In particular:
Hence for all we have:
Thus, for all we have:
where indicates that a square symmetric matrix is non-negative definite. Consequently, any given linear form of the PCR estimator has a lower variance compared to that of the same linear form of the ordinary least squares estimator.
Addressing multicollinearity
Under multicollinearity, two or more of the covariates are highly correlated, so that one can be linearly predicted from the others with a non-trivial degree of accuracy. Consequently, the columns of the data matrix that correspond to the observations for these covariates tend to become linearly dependent and therefore, tends to become rank deficient losing its full column rank structure. More quantitatively, one or more of the smaller eigenvalues of get(s) very close or become(s) exactly equal to under such situations. The variance expressions above indicate that these small eigenvalues have the maximum inflation effect on the variance of the least squares estimator, thereby destabilizing the estimator significantly when they are close to . This issue can be effectively addressed through using a PCR estimator obtained by excluding the principal components corresponding to these small eigenvalues.
Dimension reduction
PCR may also be used for performing dimension reduction. To see this, let denote any matrix having orthonormal columns, for any Suppose now that we want to approximate each of the covariate observations through the rank linear transformation for some .
Then, it can be shown that
is minimized at the matrix with the first principal component directions as columns, and the corresponding dimensional derived covariates. Thus the dimensional principal components provide the best linear approximation of rank to the observed data matrix .
The corresponding reconstruction error is given by:
Thus any potential dimension reduction may be achieved by choosing , the number of principal components to be used, through appropriate thresholding on the cumulative sum of the eigenvalues of . Since the smaller eigenvalues do not contribute significantly to the cumulative sum, the corresponding principal components may be continued to be dropped as long as the desired threshold limit is not exceeded. The same criteria may also be used for addressing the multicollinearity issue whereby the principal components corresponding to the smaller eigenvalues may be ignored as long as the threshold limit is maintained.
Regularization effect
Since the PCR estimator typically uses only a subset of all the principal components for regression, it can be viewed as some sort of a regularized procedure. More specifically, for any , the PCR estimator denotes the regularized solution to the following constrained minimization problem:
The constraint may be equivalently written as:
where:
Thus, when only a proper subset of all the principal components are selected for regression, the PCR estimator so obtained is based on a hard form of regularization that constrains the resulting solution to the column space of the selected principal component directions, and consequently restricts it to be orthogonal to the excluded directions.
Optimality of PCR among a class of regularized estimators
Given the constrained minimization problem as defined above, consider the following generalized version of it:
where, denotes any full column rank matrix of order with .
Let denote the corresponding solution. Thus
Then the optimal choice of the restriction matrix for which the corresponding estimator achieves the minimum prediction error is given by:
where
Quite clearly, the resulting optimal estimator is then simply given by the PCR estimator based on the first principal components.
Efficiency
Since the ordinary least squares estimator is unbiased for , we have
where, MSE denotes the mean squared error. Now, if for some , we additionally have: , then the corresponding is also unbiased for and therefore
We have already seen that
which then implies:
for that particular . Thus in that case, the corresponding would be a more efficient estimator of compared to , based on using the mean squared error as the performance criteria. In addition, any given linear form of the corresponding would also have a lower mean squared error compared to that of the same linear form of .
Now suppose that for a given . Then the corresponding is biased for . However, since
it is still possible that , especially if is such that the excluded principal components correspond to the smaller eigenvalues, thereby resulting in lower bias.
In order to ensure efficient estimation and prediction performance of PCR as an estimator of , Park (1981) proposes the following guideline for selecting the principal components to be used for regression: Drop the principal component if and only if Practical implementation of this guideline of course requires estimates for the unknown model parameters and . In general, they may be estimated using the unrestricted least squares estimates obtained from the original full model. Park (1981) however provides a slightly modified set of estimates that may be better suited for this purpose.