General multivariate normal distribution

Dependencies:

  1. Standard multivariate normal distribution
  2. Independence of random variables (incomplete)
  3. Sum of independent normal random variables is normal
  4. /probability/rand-vars/variance/variance-of-affine-form-of-matrix
  5. Linearity of expectation for matrices
  6. /probability/rand-vars/var-matrix-is-psd
  7. /probability/rand-vars/moment-generating-function
  8. /probability/rand-vars/mgf-gives-distr
  9. /probability/normal-distr/mgf
  10. Identity matrix
  11. /linear-algebra/matrices/sqrt-of-symm-matrix
  12. Full-rank square matrix is invertible
  13. /linear-algebra/matrices/psd-full-rank

There is a class of probability distributions called multivariate normal distributions. Let $X = [X_1, X_2, \ldots, X_n]$ be a sequence of random variables. $X$ is said to follow a multivariate normal distribution (or more simply, $X$ is normal) iff \[ \forall a \in \mathbb{R}^n, \exists (\mu, \sigma), a^TX \sim \mathcal{N}(\mu, \sigma^2) \]

For a $k$-by-$n$ matrix $A$, $X$ is said to be $(\mu, A)$-normal iff $\exists Z \sim \mathcal{N}(0, 1)^k$ such that $X = \mu + AZ$.

We can prove that $X$ is normal iff $\exists \mu \in \mathbb{R}^n, \exists A \in \mathbb{R}^{k \times n}$ such that $X$ is $(\mu, A)$-normal. Therefore, $(\mu, A)$-normality can be used as an alternative definition of the class of multivariate normal distributions.

If $X$ is $(\mu, A)$-normal, then $\E(X) = \mu$ and $\Var(X) = AA^T$. We'll also prove that if $X$ is $(\mu, A)$-normal, then the probability distribution of $X$ is uniquely determined by $\mu$ and $AA^T$. Therefore, a multivariate normal distribution is uniquely defined by its mean $\mu$ and covariance matrix $\Sigma$. Denote this distribution by $\mathcal{N}(\mu, \Sigma)$. The standard normal distribution is therefore, $\mathcal{N}(0, I)$.

This finally gives us that $X \sim \mathcal{N}(\mu, \Sigma)$ iff \[ \exists A, (\Sigma = AA^T \textrm{ and } \exists Z \sim \mathcal{N}(0, I), X = \mu + AZ) \]

When $\Sigma$ is positive definite, $X \sim \mathcal{N}(\mu, \Sigma)$ has a density function given by \[ f_X(x) = \frac{1}{\sqrt{(2\pi)^k |\Sigma|}} \exp\left( -\frac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu) \right) \]

$\E(X)$ and $\Var(X)$

Suppose $X$ is $(\mu, A)$-normal. \[ \E(X) = \E(\mu + AZ) = \mu + A\E(Z) = \mu \] \[ \Var(X) = \Var(\mu + AZ) = A\Var(Z)A^T = AA^T \]

Proof that $(\mu, A)$-normal iff normal

Suppose $X$ is $(\mu, A)$-normal, where $A \in \mathbb{R}^{n \times k}$. So $X = \mu + AZ$, where $Z$ is standard normal.

Let $b = A^Ta$. Then $a^TX = a^T\mu + b^TZ$ Since $b^TZ$ is a linear combination of standard normal variables, $b^TZ \sim \mathcal{N}(0, b^Tb)$. So, $a^TX \sim \mathcal{N}(a^T\mu, a^TAA^Ta)$.

Since $\forall a \in \mathbb{R}^n, a^TX$ is normal, $X$ is normal.

The proof that normal implies $(\mu, A)$-normal can be found in theorem 1 in http://www2.stat.duke.edu/~st118/sta732/mvnormal.pdf. The proof uses these lemmas/definitions:

The theorem above also proves that if $X$ is $(\mu, A)$-normal, the moment generating function of $X$ depends only on $\mu$ and $AA^T$, and that $\E(X) = \mu$ and $\Var(X) = AA^T$.

Dependency for: None

Info:

Transitive dependencies:

  1. /analysis/topological-space
  2. /sets-and-relations/countable-set
  3. /linear-algebra/vector-spaces/condition-for-subspace
  4. /linear-algebra/matrices/gauss-jordan-algo
  5. /sets-and-relations/equivalence-relation
  6. /linear-algebra/matrices/psd-full-rank
  7. /linear-algebra/matrices/sqrt-of-symm-matrix
  8. /probability/normal-distr/mgf
  9. /probability/rand-vars/mgf-gives-distr
  10. /probability/rand-vars/moment-generating-function
  11. /probability/rand-vars/var-matrix-is-psd
  12. /probability/rand-vars/variance/variance-of-affine-form-of-matrix
  13. /analysis/exponential-grows-superpolynomially
  14. /analysis/integration-by-parts
  15. /sets-and-relations/de-morgan-laws
  16. /measure-theory/linearity-of-lebesgue-integral
  17. /measure-theory/lebesgue-integral
  18. σ-algebra
  19. Generated σ-algebra
  20. Borel algebra
  21. Measurable function
  22. Generators of the real Borel algebra (incomplete)
  23. Measure
  24. σ-algebra is closed under countable intersections
  25. Group
  26. Ring
  27. Polynomial
  28. Integral Domain
  29. Comparing coefficients of a polynomial with disjoint variables
  30. Field
  31. Vector Space
  32. Linear independence
  33. Span
  34. Semiring
  35. Matrix
  36. Stacking
  37. System of linear equations
  38. Product of stacked matrices
  39. Matrix multiplication is associative
  40. Reduced Row Echelon Form (RREF)
  41. Matrices over a field form a vector space
  42. Row space
  43. Elementary row operation
  44. Every elementary row operation has a unique inverse
  45. Row equivalence of matrices
  46. Row equivalent matrices have the same row space
  47. RREF is unique
  48. Identity matrix
  49. Inverse of a matrix
  50. Inverse of product
  51. Elementary row operation is matrix pre-multiplication
  52. Row equivalence matrix
  53. Equations with row equivalent matrices have the same solution set
  54. Rank of a matrix
  55. Basis of a vector space
  56. Linearly independent set is not bigger than a span
  57. Homogeneous linear equations with more variables than equations
  58. Rank of a homogenous system of linear equations
  59. Full-rank square matrix in RREF is the identity matrix
  60. Full-rank square matrix is invertible
  61. Probability
  62. Conditional probability (incomplete)
  63. Independence of events
  64. Independence of composite events
  65. Random variable
  66. Expected value of a random variable
  67. Distribution of sum of random variables (incomplete)
  68. Independence of random variables (incomplete)
  69. Linearity of expectation
  70. Variance of a random variable
  71. Covariance of 2 random variables
  72. Linearity of expectation for matrices
  73. Cross-covariance matrix
  74. Covariance matrix
  75. Gaussian integral (incomplete)
  76. Normal distribution
  77. Sum of independent normal random variables is normal
  78. Standard multivariate normal distribution