2008年11月9日 星期日

Multivariate Normal Distribution

We want to characterize multivariate normal distribution.

Consider the random vector
\mathbf{X}=\begin{pmatrix} X_1,&\dots{}&,X_N \end{pmatrix}^\text{T}

There are several equivalent definitions of multivariate normal distribution. One simple definition is the explicite formula of its distribution function:
\frac{1}{\sqrt{(2\pi)^N|\Sigma|}}\exp{\left\{ -\frac{1}{2}(\mathbf{X}-\mathbf{\mu})^\text{T}\Sigma^{-1}(\mathbf{X}-\mathbf{\mu}) \right\}}

However, this definition relies on the invertability of the covariance matrix. If the covariance didn't have full rank, then the formula is not well-defined.

Other definitions get rid of the problem:
  1. All the linear combination of its entries is normal distributed
  2. X can be writen as the affined linear transformation of an independent standard normal vectors.
  3. Defined via the characteristic
Nevertheless, the distribution formula definition is the clearest one. Therefore, we want to extend the formula definition so that it is well-defined even if covariance matrix is rank-defected.
The idea is to approximate the distribution by the sequence of well-defined distributions. The theory behind it is weak convergence. For the one-dimensional case, when the variance of the sequence of normal random variables decreases to 0, the distribution tends to the Dirac delta function. That is
\lim_{n\rightarrow \infty}\mathcal{N}(0,\frac{1}{n})\xrightarrow{\mathcal{D}}\delta(x)

Similarly, for the multivariate case, we can approximate the rank-defected covariance matrx by full rank matrices. For the moment, assume the covariance matrix has full-rank. Since the covariance matrix is symmetirc, its eigen-decomposition always exist and it's unitary:
\Sigma = U\Lambda U^{\textrm{T}}
Suppose there are r positive eigenvalues in the rank-defected matrix, reorder the diagonal matrix such that the eigenvalues are in decreasing order. The decompostion can be partitioned as:
\Sigma= \begin{bmatrix} U_1 & U_2 \end{bmatrix} \begin{bmatrix} \Lambda_1 & 0 \\ 0 & \Lambda_2 \end{bmatrix} \begin{bmatrix} U_1^{\textrm{T}}\\ U_2^{\textrm{T}} \end{bmatrix}
Plug the above expression into the distribution formula:
\frac{1}{\sqrt{(2\pi)^N|\Lambda_1||\Lambda_2|}}\exp{\left\{ -\frac{1}{2}u^TU_1\Lambda_1^{-1}U_1^Tu-\frac{1}{2}u^TU_2\Lambda_2^{-1}U_2^Tu \right\}}
where u is x-\mu.
Now we can divide the distribution into two parts: the first part corresponds to the r positive eigenvalues:
\frac{1}{\sqrt{(2\pi)^r|\Lambda_1|}}\exp{\left\{ -\frac{1}{2}u^T\Sigma^+u\right\}}
And the second part:
\frac{1}{\sqrt{(2\pi)^{N-r}|\Lambda_2|}}\exp{\left\{ -\frac{1}{2}v^T\Lambda_2^{-1}v\right\}}
where v=U_2^Tu
As Lumbda2 tends to 0, the second part converges to multi-dimentional delta function:
\frac{1}{\sqrt{(2\pi)^{N-r}|\Lambda_2|}}\exp{\left\{ -\frac{1}{2}v^T\Lambda_2^{-1}v\right\}} \xrightarrow{\mathcal{D}}\delta(v_1)\delta(v_2)\dots\delta(v_{N-r})=\delta(\mathbf{v})
The final result is
f_{\mathbf{X}}(\mathbf{x})=f_{\mathbf{v}_1}(U_1^T\mathbf{(x-\mathbf{\mu})})\delta_{\mathbf{v}_2}(U_2^T\mathbf{(x-\mathbf{\mu})})
張貼者： 於
標籤：

沒有留言:

張貼留言

訂閱： 張貼留言 (Atom)