In functional analysis, a discipline within mathematics, a reproducing kernel Hilbert space (RKHS) is a Hilbert space of functions in which point evaluation is a continuous linear functional. Specifically, a Hilbert space of functions from a set (to or ) is an RKHS if the point-evaluation functional , , is continuous for every . Equivalently, is an RKHS if for every there exists a function such that, for all ,The function is then called the reproducing kernel, and it reproduces the value of at via the inner product.

An immediate consequence of this property is that convergence in norm implies uniform convergence on any subset of on which is bounded. However, the converse does not necessarily hold. Often the set carries a topology, and depends continuously on , in which case: convergence in norm implies uniform convergence on compact subsets of .

It is not entirely straightforward to construct natural examples of a Hilbert space which are not an RKHS in a non-trivial fashion.Alpay, D., and T. M. Mills. "A family of Hilbert spaces which are not reproducing kernel Hilbert spaces." J. Anal. Appl. 1.2 (2003): 107–111. Some examples, however, have been found.Z. Pasternak-Winiarski, "On weights which admit reproducing kernel of Bergman type", International Journal of Mathematics and Mathematical Sciences, vol. 15, Issue 1, 1992.T. Ł. Żynda, "On weights which admit reproducing kernel of Szegő type", Journal of Contemporary Mathematical Analysis (Armenian Academy of Sciences), 55, 2020.

While, formally, L2 spaces are defined as Hilbert spaces of equivalence classes of functions, this definition can trivially be extended to a Hilbert space of functions by choosing a (total) function as a representative for each equivalence class. However, no choice of representatives can make this space an RKHS ( would need to be the non-existent Dirac delta function). However, there are RKHSs in which the norm is an L2-norm, such as the space of band-limited functions (see the example below).

An RKHS is associated with a kernel that reproduces every function in the space in the sense that for every in the set on which the functions are defined, "evaluation at " can be performed by taking an inner product with a function determined by the kernel. Such a reproducing kernel exists if and only if every evaluation functional is continuous.

The reproducing kernel was first introduced in the 1907 work of Stanisław ZarembaZaremba, S. L’équation biharmonique et une classe remarquable de fonctions fondamentales harmoniques. Krakauer Anzeiger, 147-196 (1907) concerning boundary value problems for harmonic and biharmonic functions. James Mercer simultaneously examined functions which satisfy the reproducing property in the theory of integral equations. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of Gábor Szegő, Stefan Bergman, and Salomon Bochner. The subject was eventually systematically developed in the early 1950s by Nachman Aronszajn and Stefan Bergman.Okutmustur

These spaces have wide applications, including complex analysis, harmonic analysis, and quantum mechanics. Reproducing kernel Hilbert spaces are particularly important in the field of statistical learning theory because of the celebrated representer theorem which states that every function in an RKHS that minimises an empirical risk functional can be written as a linear combination of the kernel function evaluated at the training points. This is a practically useful result as it effectively simplifies the empirical risk minimization problem from an infinite dimensional to a finite dimensional optimization problem.

For ease of understanding, we provide the framework for real-valued Hilbert spaces. The theory can be easily extended to spaces of complex-valued functions and hence include the many important examples of reproducing kernel Hilbert spaces that are spaces of analytic functions.Paulson

Definition Let be an arbitrary set and a Hilbert space of real-valued functions on , equipped with pointwise addition and pointwise scalar multiplication. The evaluation functional over the Hilbert space of functions is a linear functional that evaluates each function at a point ,

We say that H is a reproducing kernel Hilbert space if, for all in , is continuous at every in or, equivalently, if is a bounded operator on , i.e. there exists some such that

Although is assumed for all , it might still be the case that .

While property () is the weakest condition that ensures both the existence of an inner product and the evaluation of every function in at every point in the domain, it does not lend itself to easy application in practice. A more intuitive definition of the RKHS can be obtained by observing that this property guarantees that the evaluation functional can be represented by taking the inner product of with a function in . This function is the so-called reproducing kernel for the Hilbert space from which the RKHS takes its name. More formally, the Riesz representation theorem implies that for all in there exists a unique element of with the reproducing property,

Since is itself a function defined on with values in the field (or in the case of complex Hilbert spaces) and as is in we have that

where is the element in associated to .

This allows us to define the reproducing kernel of as a function (or in the complex case) by

From this definition it is easy to see that (or in the complex case) is both symmetric (resp. conjugate symmetric) and positive definite, i.e.

for every Durrett The Moore–Aronszajn theorem (see below) is a sort of converse to this: if a function satisfies these conditions then there is a Hilbert space of functions on for which it is a reproducing kernel.

Examples The simplest example of a reproducing kernel Hilbert space is the space where is a set and is the counting measure on . For , the reproducing kernel is the indicator function of the one point set .

Nontrivial reproducing kernel Hilbert spaces often involve analytic functions, as we now illustrate by example. Consider the Hilbert space of bandlimited continuous functions . Fix some cutoff frequency and define the Hilbert space

where is the set of square integrable functions, and is the Fourier transform of . As the inner product, we use

Since this is a closed subspace of , it is a Hilbert space. Moreover, the elements of are smooth functions on that tend to zero at infinity, essentially by the Riemann-Lebesgue lemma. In fact, the elements of are the restrictions to of entire holomorphic functions, by the Paley–Wiener theorem.

From the Fourier inversion theorem, we have

It then follows by the Cauchy–Schwarz inequality and Plancherel's theorem that, for all ,

This inequality shows that the evaluation functional is bounded, proving that is indeed a RKHS.

The kernel function in this case is given by

The Fourier transform of defined above is given by

which is a consequence of the time-shifting property of the Fourier transform. Consequently, using Plancherel's theorem, we have

Thus we obtain the reproducing property of the kernel.

in this case is the "bandlimited version" of the Dirac delta function, and that converges to in the weak sense as the cutoff frequency tends to infinity.

Moore–Aronszajn theorem We have seen how a reproducing kernel Hilbert space defines a reproducing kernel function that is both symmetric and positive definite. The Moore–Aronszajn theorem goes in the other direction; it states that every symmetric, positive definite kernel defines a unique reproducing kernel Hilbert space. The theorem first appeared in Aronszajn's Theory of Reproducing Kernels, although he attributes it to E. H. Moore.

Integral operators and Mercer's theorem We may characterize a symmetric positive definite kernel via the integral operator using Mercer's theorem and obtain an additional view of the RKHS. Let be a compact space equipped with a strictly positive finite Borel measure and a continuous, symmetric, and positive definite function. Define the integral operator as

where is the space of square integrable functions with respect to .

Mercer's theorem states that the spectral decomposition of the integral operator of yields a series representation of in terms of the eigenvalues and eigenfunctions of . This then implies that is a reproducing kernel so that the corresponding RKHS can be defined in terms of these eigenvalues and eigenfunctions. We provide the details below.

Under these assumptions is a compact, continuous, self-adjoint, and positive operator. The spectral theorem for self-adjoint operators implies that there is an at most countable decreasing sequence such that and , where the form an orthonormal basis of . By the positivity of for all One can also show that maps continuously into the space of continuous functions and therefore we may choose continuous functions as the eigenvectors, that is, for all Then by Mercer's theorem may be written in terms of the eigenvalues and continuous eigenfunctions as

for all such that

This above series representation is referred to as a Mercer kernel or Mercer representation of .

Furthermore, it can be shown that the RKHS of is given by

where the inner product of given by

This representation of the RKHS has application in probability and statistics, for example to the Karhunen–Loève representation for stochastic processes and kernel PCA.

Feature maps A feature map is a map , where is a Hilbert space which we will call the feature space. The first sections presented the connection between bounded/continuous evaluation functions, positive definite functions, and integral operators and in this section we provide another representation of the RKHS in terms of feature maps.

Every feature map defines a kernel via

Clearly is symmetric and positive definiteness follows from the properties of inner product in . Conversely, every positive definite function and corresponding reproducing kernel Hilbert space has infinitely many associated feature maps such that () holds.

For example, we can trivially take and for all . Then () is satisfied by the reproducing property. Another classical example of a feature map relates to the previous section regarding integral operators by taking and .

This connection between kernels and feature maps provides us with a new way to understand positive definite functions and hence reproducing kernels as inner products in . Moreover, every feature map can naturally define a RKHS by means of the definition of a positive definite function.

Lastly, feature maps allow us to construct function spaces that reveal another perspective on the RKHS. Consider the linear space

We can define a norm on by

It can be shown that is a RKHS with kernel defined by . This representation implies that the elements of the RKHS are inner products of elements in the feature space and can accordingly be seen as hyperplanes. This view of the RKHS is related to the kernel trick in machine learning.Rosasco

Properties

Useful properties of RKHSs:

Let be a sequence of sets and be a collection of corresponding positive definite functions on It then follows that

is a kernel on Let then the restriction of to is also a reproducing kernel. Consider a normalized kernel such that for all . Define a pseudo-metric on X as

By the Cauchy–Schwarz inequality,

This inequality allows us to view as a measure of similarity between inputs. If are similar then will be closer to 1 while if are dissimilar then will be closer to 0. The closure of the span of coincides with .Rosasco

Common examples

Bilinear kernels

The RKHS corresponding to this kernel is the dual space, consisting of functions satisfying .

Polynomial kernels

Radial basis function kernels These are another common class of kernels which satisfy . Some examples include:

Gaussian or squared exponential kernel:

Laplacian kernel: