The covariance matrix of Z = CX is
The similarity of (3.2) and (2.10) is pretty clear. When C is a row vector cT = (c1, c2,…, cp), CX = cTX = c1X1 + … + cp Xp and
where μ and Σ are the mean vector and covariance matrix of X.
Let X1 and X2 denote two subvectors of X, i.e.,
where Σ11 = cov(X1) and Σ22 = cov(X2). The matrix Σ12 contains the covariance of each component in X1 and each component in X2. Based on the symmetry of Σ, we have
3.2 Density Function and Properties of Multivariate Normal Distribution
Normal distribution is the most commonly used distribution for continuous random variables. Many statistical models and inference methods are based on the univariate or multivariate normal distribution. One advantage of the normal distribution is its mathematical tractability. More importantly, the normal distribution turns out to be a good approximation to the “true” population distribution for many sample statistics and real-world data due to the central limit theorem, which says that the summation of a large number of independent observations from any population with the same mean and variance approximately follows a normal distribution.
Recall that a univariate random variable X with mean μ and variance σ2 is normally distributed, which is denoted by X ∼ N (μ, σ2), if it has the probability density function
The multivariate normal distribution is an extension of the univariate normal distribution. If a p-dimensional random vector X follows a multivariate normal distribution with mean vector μ and covariance matrix Σ, the probability density function of X has the form
We denote the p-dimensional normal distribution by Np(μ, Σ).
From (3.8), the density of a p-dimensional normal distribution depends on x through the term (x − μ)T Σ−1 (x − μ), which is the square of the distance from x to Σ standardized by the covariance matrix. Then it is clear that the set of x values yielding a constant height for the density form an ellipsoid. The set of points with the same height for the density is called a contour. The constant probability density contour of a p-dimensional normal distribution is:
which forms the surface of an ellipsoid centered at μ with standardized distance between x and μ equal to c. And the contour with larger distance c has a smaller height value for the density. It can be shown that the axes of the ellipsoid contours of constant density for the p-dimensional normal distribution are in the directions of the eigenvectors of Σ with lengths proportional to the square roots of the corresponding eigenvalues of Σ.
Example 3.1: Consider a bivariate (p = 2) normally distributed random vector X = (X1 X2)T. Suppose the mean vector is μ = (0 0)T and the covariance matrix is
So the variance of both variables is equal to one and the covariance matrix coincides with the correlation matrix. The inverse of the covariance matrix