In linear algebra and functional analysis, a projection is a linear transformation from a vector space to itself (an endomorphism) such that . That is, whenever is applied twice to any vector, it gives the same result as if it were applied once (i.e. is idempotent). It leaves its image unchanged.Meyer, pp 386+387 This definition of "projection" formalizes and generalizes the idea of graphical projection. One can also consider the effect of a projection on a geometrical object by examining the effect of the projection on points in the object.
Definitions A projection on a vector space is a linear operator such that .
When has an inner product and is complete, i.e. when is a Hilbert space, the concept of orthogonality can be used. A projection on a Hilbert space is called an orthogonal projection if it satisfies for all . A projection on a Hilbert space that is not orthogonal is called an oblique projection.
Projection matrix A square matrix is called a projection matrix if it is equal to its square, i.e. if . A square matrix is called an orthogonal projection matrix if for a real matrix, and respectively for a complex matrix, where denotes the transpose of and denotes the adjoint or Hermitian transpose of . A projection matrix that is not an orthogonal projection matrix is called an oblique projection matrix. The eigenvalues of a projection matrix must be 0 or 1.
Examples
Orthogonal projection For example, the function which maps the point in three-dimensional space to the point is an orthogonal projection onto the xy-plane. This function is represented by the matrix
The action of this matrix on an arbitrary vector is
To see that is indeed a projection, i.e., , we compute
Observing that shows that the projection is an orthogonal projection.
Oblique projection A simple example of a non-orthogonal (oblique) projection is
Via matrix multiplication, one sees that
showing that is indeed a projection.
The projection is orthogonal if and only if because only then
Properties and classification
Idempotence By definition, a projection is idempotent (i.e. ).
Open map Every projection is an open map onto its image, meaning that it maps each open set in the domain to an open set in the subspace topology of the image. That is, for any vector and any ball (with positive radius) centered on , there exists a ball (with positive radius) centered on that is wholly contained in the image .
Complementarity of image and kernel Let be a finite-dimensional vector space and be a projection on . Suppose the subspaces and are the image and kernel of respectively. Then has the following properties:
is the identity operator on : We have a direct sum . Every vector may be decomposed uniquely as with and , and where
The image and kernel of a projection are complementary, as are and . The operator is also a projection as the image and kernel of become the kernel and image of and vice versa. We say is a projection along onto (kernel/image) and is a projection along onto .
Spectrum In infinite-dimensional vector spaces, the spectrum of a projection is contained in as
Only 0 or 1 can be an eigenvalue of a projection. This implies that an orthogonal projection is always a positive semi-definite matrix. In general, the corresponding eigenspaces are (respectively) the kernel and range of the projection. Decomposition of a vector space into direct sums is not unique. Therefore, given a subspace , there may be many projections whose range (or kernel) is .
If a projection is nontrivial it has minimal polynomial , which factors into distinct linear factors, and thus is diagonalizable.
Product of projections The product of projections is not in general a projection, even if they are orthogonal. If two projections commute then their product is a projection, but the converse is false: the product of two non-commuting projections may or may not be a projection.
If two orthogonal projections commute then their product is an orthogonal projection. If the product of two orthogonal projections is an orthogonal projection, then the two orthogonal projections commute (more generally: two self-adjoint endomorphisms commute if and only if their product is self-adjoint).
Orthogonal projections
When the vector space has an inner product and is complete (is a Hilbert space) the concept of orthogonality can be used. An orthogonal projection is a projection for which the range and the kernel are orthogonal subspaces. Thus, for every and in , . Equivalently:
A projection is orthogonal if and only if it is self-adjoint. Using the self-adjoint and idempotent properties of , for any and in we have , , and
where is the inner product associated with . Therefore, and are orthogonal projections.Meyer, p. 433 The other direction, namely that if is orthogonal then it is self-adjoint, follows from the implication from to
for every and in ; thus .
The existence of an orthogonal projection onto a closed subspace follows from the Hilbert projection theorem.
Properties and special cases An orthogonal projection is a bounded operator. This is because for every in the vector space we have, by the Cauchy–Schwarz inequality:
Thus .
For finite-dimensional complex or real vector spaces, the standard inner product can be substituted for .
Formulas A simple case occurs when the orthogonal projection is onto a line. If is a unit vector on the line, then the projection is given by the outer product
(If is complex-valued, the transpose in the above equation is replaced by a Hermitian transpose). This operator leaves u invariant, and it annihilates all vectors orthogonal to , proving that it is indeed the orthogonal projection onto the line containing u.Meyer, p. 431 A simple way to see this is to consider an arbitrary vector as the sum of a component on the line (i.e. the projected vector we seek) and another perpendicular to it, . Applying projection, we get
by the properties of the dot product of parallel and perpendicular vectors.
This formula can be generalized to orthogonal projections on a subspace of arbitrary dimension. Let be an orthonormal basis of the subspace , with the assumption that the integer , and let denote the matrix whose columns are , i.e., . Then the projection is given by:Meyer, equation (5.13.4)
which can be rewritten as
The matrix is the partial isometry that vanishes on the orthogonal complement of , and is the isometry that embeds into the underlying vector space. The range of is therefore the final space of . It is also clear that is the identity operator on .
The orthonormality condition can also be dropped. If is a (not necessarily orthonormal) basis with , and is the matrix with these vectors as columns, then the projection is:Meyer, equation (5.13.3)
The matrix still embeds into the underlying vector space but is no longer an isometry in general. The matrix is a "normalizing factor" that recovers the norm. For example, the rank-1 operator is not a projection if After dividing by we obtain the projection onto the subspace spanned by .
In the general case, we can have an arbitrary positive definite matrix defining an inner product , and the projection is given by . Then
When the range space of the projection is generated by a frame (i.e. the number of generators is greater than its dimension), the formula for the projection takes the form: . Here stands for the Moore–Penrose pseudoinverse. This is just one of many ways to construct the projection operator.
If is a non-singular matrix and (i.e., is the null space matrix of ),See also Linear least squares (mathematics) § Properties of the least-squares estimators. the following holds:
If the orthogonal condition is enhanced to with non-singular, the following holds:
All these formulas also hold for complex inner product spaces, provided that the conjugate transpose is used instead of the transpose. Further details on sums of projectors can be found in Banerjee and Roy (2014). Also see Banerjee (2004) for application of sums of projectors in basic spherical trigonometry.
Oblique projections The term oblique projections is sometimes used to refer to non-orthogonal projections. These projections are also used to represent spatial figures in two-dimensional drawings (see oblique projection), though not as frequently as orthogonal projections. Whereas calculating the fitted value of an ordinary least squares regression requires an orthogonal projection, calculating the fitted value of an instrumental variables regression requires an oblique projection.
A projection is defined by its kernel and the basis vectors used to characterize its range (which is a complement of the kernel). When these basis vectors are orthogonal to the kernel, then the projection is an orthogonal projection. When these basis vectors are not orthogonal to the kernel, the projection is an oblique projection, or just a projection.
A matrix representation formula for a nonzero projection operator Let be a linear operator such that and assume that is not the zero operator. Let the vectors form a basis for the range of , and assemble these vectors in the matrix . Then , otherwise and is the zero operator. The range and the kernel are complementary spaces, so the kernel has dimension . It follows that the orthogonal complement of the kernel has dimension . Let form a basis for the orthogonal complement of the kernel of the projection, and assemble these vectors in the matrix . Then the projection (with the condition ) is given by
This expression generalizes the formula for orthogonal projections given above.Meyer, equation (7.10.39) A standard proof of this expression is the following. For any vector in the vector space , we can decompose , where vector is in the image of , and vector So , and then is in the kernel of , which is the null space of In other words, the vector is in the column space of so for some dimension vector and the vector satisfies by the construction of . Put these conditions together, and we find a vector so that . Since matrices and are of full rank by their construction, the -matrix is invertible. So the equation gives the vector In this way, for any vector and hence .
In the case that is an orthogonal projection, we can take , and it follows that . By using this formula, one can easily check that . In general, if the vector space is over complex number field, one then uses the Hermitian transpose and has the formula . Recall that one can express the Moore–Penrose inverse of the matrix by since has full column rank, so .
Singular values is also an oblique projection. The singular values of and can be computed by an orthonormal basis of . Let be an orthonormal basis of and let be the orthogonal complement of . Denote the singular values of the matrix by the positive values . With this, the singular values for are:
and the singular values for are
This implies that the largest singular values of and are equal, and thus that the matrix norm of the oblique projections are the same. However, the condition number satisfies the relation , and is therefore not necessarily equal.
Finding projection with an inner product
Let be a vector space (in this case a plane) spanned by orthogonal vectors . Let be a vector. One can define a projection of onto as
where repeated indices are summed over (Einstein sum notation). The vector can be written as an orthogonal sum such that . is sometimes denoted as . There is a theorem in linear algebra that states that this is the smallest distance (the orthogonal distance) from to and is commonly used in areas such as machine learning.
Canonical forms
Any projection on a vector space of dimension over a field is a diagonalizable matrix, since its minimal polynomial divides , which splits into distinct linear factors. Thus there exists a basis in which has the form