General multivariate normal distribution

Dependencies:

  1. Standard multivariate normal distribution
  2. Independence of random variables (incomplete)
  3. Sum of independent normal random variables is normal
  4. /probability/rand-vars/variance/variance-of-affine-form-of-matrix
  5. Linearity of expectation for matrices
  6. /probability/rand-vars/var-matrix-is-psd
  7. /probability/rand-vars/moment-generating-function
  8. /probability/rand-vars/mgf-gives-distr
  9. /probability/normal-distr/mgf
  10. Identity matrix
  11. /linear-algebra/matrices/sqrt-of-symm-matrix
  12. Full-rank square matrix is invertible
  13. /linear-algebra/matrices/psd-full-rank

There is a class of probability distributions called multivariate normal distributions. Let $X = [X_1, X_2, \ldots, X_n]$ be a sequence of random variables. $X$ is said to follow a multivariate normal distribution (or more simply, $X$ is normal) iff \[ \forall a \in \mathbb{R}^n, \exists (\mu, \sigma), a^TX \sim \mathcal{N}(\mu, \sigma^2) \]

For a $k$-by-$n$ matrix $A$, $X$ is said to be $(\mu, A)$-normal iff $\exists Z \sim \mathcal{N}(0, 1)^k$ such that $X = \mu + AZ$.

We can prove that $X$ is normal iff $\exists \mu \in \mathbb{R}^n, \exists A \in \mathbb{R}^{k \times n}$ such that $X$ is $(\mu, A)$-normal. Therefore, $(\mu, A)$-normality can be used as an alternative definition of the class of multivariate normal distributions.

If $X$ is $(\mu, A)$-normal, then $\E(X) = \mu$ and $\Var(X) = AA^T$. We'll also prove that if $X$ is $(\mu, A)$-normal, then the probability distribution of $X$ is uniquely determined by $\mu$ and $AA^T$. Therefore, a multivariate normal distribution is uniquely defined by its mean $\mu$ and covariance matrix $\Sigma$. Denote this distribution by $\mathcal{N}(\mu, \Sigma)$. The standard normal distribution is therefore, $\mathcal{N}(0, I)$.

This finally gives us that $X \sim \mathcal{N}(\mu, \Sigma)$ iff \[ \exists A, (\Sigma = AA^T \textrm{ and } \exists Z \sim \mathcal{N}(0, I), X = \mu + AZ) \]

When $\Sigma$ is positive definite, $X \sim \mathcal{N}(\mu, \Sigma)$ has a density function given by \[ f_X(x) = \frac{1}{\sqrt{(2\pi)^k |\Sigma|}} \exp\left( -\frac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu) \right) \]

$\E(X)$ and $\Var(X)$

Suppose $X$ is $(\mu, A)$-normal. \[ \E(X) = \E(\mu + AZ) = \mu + A\E(Z) = \mu \] \[ \Var(X) = \Var(\mu + AZ) = A\Var(Z)A^T = AA^T \]

Proof that $(\mu, A)$-normal iff normal

Suppose $X$ is $(\mu, A)$-normal, where $A \in \mathbb{R}^{n \times k}$. So $X = \mu + AZ$, where $Z$ is standard normal.

Let $b = A^Ta$. Then $a^TX = a^T\mu + b^TZ$ Since $b^TZ$ is a linear combination of standard normal variables, $b^TZ \sim \mathcal{N}(0, b^Tb)$. So, $a^TX \sim \mathcal{N}(a^T\mu, a^TAA^Ta)$.

Since $\forall a \in \mathbb{R}^n, a^TX$ is normal, $X$ is normal.

The proof that normal implies $(\mu, A)$-normal can be found in theorem 1 in http://www2.stat.duke.edu/~st118/sta732/mvnormal.pdf. The proof uses these lemmas/definitions:

The theorem above also proves that if $X$ is $(\mu, A)$-normal, the moment generating function of $X$ depends only on $\mu$ and $AA^T$, and that $\E(X) = \mu$ and $\Var(X) = AA^T$.

Dependency for: None

Info:

Transitive dependencies:

  1. /linear-algebra/matrices/psd-full-rank
  2. /linear-algebra/matrices/sqrt-of-symm-matrix
  3. /probability/normal-distr/mgf
  4. /probability/rand-vars/mgf-gives-distr
  5. /probability/rand-vars/moment-generating-function
  6. /probability/rand-vars/var-matrix-is-psd
  7. /probability/rand-vars/variance/variance-of-affine-form-of-matrix
  8. /analysis/exponential-grows-superpolynomially
  9. /analysis/integration-by-parts
  10. /measure-theory/linearity-of-lebesgue-integral
  11. /measure-theory/lebesgue-integral
  12. /linear-algebra/matrices/gauss-jordan-algo
  13. /linear-algebra/vector-spaces/condition-for-subspace
  14. /sets-and-relations/equivalence-relation
  15. /sets-and-relations/de-morgan-laws
  16. /sets-and-relations/countable-set
  17. /analysis/topological-space
  18. Gaussian integral (incomplete)
  19. Group
  20. Ring
  21. Polynomial
  22. Field
  23. Vector Space
  24. Linear independence
  25. Span
  26. Integral Domain
  27. Comparing coefficients of a polynomial with disjoint variables
  28. Semiring
  29. Matrix
  30. Stacking
  31. System of linear equations
  32. Product of stacked matrices
  33. Matrix multiplication is associative
  34. Reduced Row Echelon Form (RREF)
  35. Elementary row operation
  36. Every elementary row operation has a unique inverse
  37. Row equivalence of matrices
  38. Matrices over a field form a vector space
  39. Row space
  40. Row equivalent matrices have the same row space
  41. RREF is unique
  42. Identity matrix
  43. Full-rank square matrix in RREF is the identity matrix
  44. Inverse of a matrix
  45. Inverse of product
  46. Elementary row operation is matrix pre-multiplication
  47. Row equivalence matrix
  48. Equations with row equivalent matrices have the same solution set
  49. Rank of a matrix
  50. Basis of a vector space
  51. Linearly independent set is not bigger than a span
  52. Homogeneous linear equations with more variables than equations
  53. Rank of a homogenous system of linear equations
  54. Full-rank square matrix is invertible
  55. σ-algebra
  56. σ-algebra is closed under countable intersections
  57. Measure
  58. Probability
  59. Conditional probability (incomplete)
  60. Independence of events
  61. Independence of composite events
  62. Generated σ-algebra
  63. Measurable function
  64. Borel algebra
  65. Generators of the real Borel algebra (incomplete)
  66. Random variable
  67. Expected value of a random variable
  68. Distribution of sum of random variables (incomplete)
  69. Linearity of expectation
  70. Covariance of 2 random variables
  71. Variance of a random variable
  72. Normal distribution
  73. Linearity of expectation for matrices
  74. Cross-covariance matrix
  75. Covariance matrix
  76. Independence of random variables (incomplete)
  77. Sum of independent normal random variables is normal
  78. Standard multivariate normal distribution