Kernel-independent component analysis

In statistics, kernel-independent component analysis (kernel ICA) is an efficient algorithm for independent component analysis which estimates source components by optimizing a generalized variance contrast function, which is based on representations in a reproducing kernel Hilbert space.^[1][2] Those contrast functions use the notion of mutual information as a measure of statistical independence.

Main idea

Kernel ICA is based on the idea that correlations between two random variables can be represented in a reproducing kernel Hilbert space (RKHS), denoted by ${\displaystyle {\mathcal {F}}}$, associated with a feature map ${\displaystyle L_{x}:{\mathcal {F}}\mapsto \mathbb {R} }$ defined for a fixed ${\displaystyle x\in \mathbb {R} }$. The ${\displaystyle {\mathcal {F}}}$-correlation between two random variables ${\displaystyle X}$ and ${\displaystyle Y}$ is defined as

${\displaystyle \rho _{\mathcal {F}}(X,Y)=\max _{f,g\in {\mathcal {F}}}\operatorname {corr} (\langle L_{X},f\rangle ,\langle L_{Y},g\rangle )}$

where the functions ${\displaystyle f,g:\mathbb {R} \to \mathbb {R} }$ range over ${\displaystyle {\mathcal {F}}}$ and

${\displaystyle \operatorname {corr} (\langle L_{X},f\rangle ,\langle L_{Y},g\rangle ):={\frac {\operatorname {cov} (f(X),g(Y))}{\operatorname {var} (f(X))^{1/2}\operatorname {var} (g(Y))^{1/2}}}}$

for fixed ${\displaystyle f,g\in {\mathcal {F}}}$.^[1] Note that the reproducing property implies that ${\displaystyle f(x)=\langle L_{x},f\rangle }$ for fixed ${\displaystyle x\in \mathbb {R} }$ and ${\displaystyle f\in {\mathcal {F}}}$.^[3] It follows then that the ${\displaystyle {\mathcal {F}}}$-correlation between two independent random variables is zero.

This notion of ${\displaystyle {\mathcal {F}}}$-correlations is used for defining contrast functions that are optimized in the Kernel ICA algorithm. Specifically, if ${\displaystyle \mathbf {X}$ :=(x_{ij})\in \mathbb {R} ^{n\times m}} is a prewhitened data matrix, that is, the sample mean of each column is zero and the sample covariance of the rows is the ${\displaystyle m\times m}$ dimensional identity matrix, Kernel ICA estimates a ${\displaystyle m\times m}$ dimensional orthogonal matrix ${\displaystyle \mathbf {A} }$ so as to minimize finite-sample ${\displaystyle {\mathcal {F}}}$-correlations between the columns of ${\displaystyle \mathbf {S}$ :=\mathbf {X} \mathbf {A} ^{\prime }} .