Asymptotic Multinomial Distribution

The binomial distribution \(\operatorname{Bin}(n,p)\) is asymptotically normal \(\mathcal{N}(\mu=np, \sigma^{2}=np(1-p))\) provided \(p\) is not "too close" to zero or one. There are a variety of heuristics about when this holds (e.g., \(n\geq30\), etc.), but I'd like to discuss what happens with a multinomial distribution for large n.

A reasonable check (or consistency condition) is when there are \(k=2\) outcomes in our multinomial distribution, we recover the familiar Binomial result: we should have a univariate normal distribution. This suggests a k-nomial distribution is asymptotically a \((k-1)\)-variate normal distribution.

With this in mind, we would expect the asymptotic relation to be of the form \[ \operatorname{Multinom}(n, \vec{p}) \approx\mathcal{MVN}(\vec{\mu}=n\vec{p}, \Sigma = nC) \tag{1} \] where \(C\) is the covariance matrix for the multinomial distribution. To be fully explicit for my "future self", the components of the covariance matrix are \[ (C)_{ij} = p_{i}(\delta_{ij}-p_{j}). \tag{2} \] The problem with this conjecture is the multivariate normal is \(k\)-dimensional and degenerate (which could be seen by multiplying the covariance matrix by the vector with all k entries equal to 1).

Our consistency condition suggests discarding one component of the vector \(n\vec{p}\) and the corresponding row and column of the covariance matrix \(\Sigma = nC\). Conventions usually drop the last component of the mean vector as well as the last row and column of the covariance matrix.

Also note, the condition for the binomial distribution that "p is not too close to 1 or 0" also has a counter-part with the multinomial distribution. This is that the vector \(\vec{p}\) is not near the boundary of the n-simplex (i.e., no single component is "near 1 or 0").

There's nothing deep here. This is just a note to myself, since I find myself using this quite a bit when looking at state-space methods.