Bishop PRML - Ch2. Probability Distributions (2)

Multinomial Variables

• one hot encoding

  • $\mathbf{x} = (0,0,1,0,0,0)^\mathsf{T}$
  • $\sum_{k=1}^Kx_k = 1$
  • multinomial label을 표현하는 방법

• categorical distribution

  • $p(\mathbf{x}{\mid}\boldsymbol{\mu}) = \prod_{k=1}^K\mu_k^{x_k}$

    • $p(x_k=1) = \mu_k$
    • $\boldsymbol{\mu} = (\mu_1,\cdots,\mu_K)^\mathsf{T}$
  • $\sum_\mathbf{x}p(\mathbf{x}{\mid}\boldsymbol{\mu}) = \sum_{k=1}^K\mu_k = 1$
  • $\mathbb{E}[\mathbf{x}{\mid}]\boldsymbol{\mu}\mathbf{x} = \boldsymbol{\mu}$

  • likelihood

    • $p(\mathcal{D}{\mid}\boldsymbol{\mu}) = \prod_{n=1}^N\prod_{k=1}^K\mu_k^{x_{nk}} = \prod_{k=1}^K\mu_k^{\sum_nx_{nk}} = \prod_{k=1}^K\mu_k^{m_k}$

      • $m_k = \sum_nx_{nk}$ : sufficient statistic ($x_k = 1$인 관측값의 수)

  • MLE

    • Lagrange multiplier

      • $\mathcal{L} = \sum_{k=1}^Km_k\ln\mu_k + \lambda\left(\sum_{k=1}^K\mu_k-1\right)$

    • find extrema

      • $\mu_k = -m_k/\lambda$

    • substituting into constraint

      • $\lambda = -N$
    • $\mu_k^{ML} = {m_k\over{N}}$ (N개의 관측값 중 $x_k = 1$인 경우의 비율과 동일)

• multinomial distribution

  • $\operatorname{Mult}(m_1,m_2,\cdots,m_K{\mid}\boldsymbol{\mu},N) = {N\choose{m_1,\cdots,m_K}}\prod_{k=1}^K\mu_k^{m_k}$

    • ${N\choose{m_1,\cdots,m_K}} = {N!\over{m_1! \cdots m_K!}}$
    • $\sum_{k=1}^Km_k = N$

• The Dirichlet distribution

  • $\operatorname{Dir}(\boldsymbol{\mu}{\mid}\boldsymbol{\alpha}) = {\Gamma(\alpha_0)\over{\Gamma(\alpha_1)\cdots\Gamma(\alpha_K)}}\prod_{k=1}^K\mu_k^{\alpha_k-1}$

    • $\alpha_0 = \sum_{k=1}^K\alpha_k$
    • $0\leq\mu_k\leq 1$
    • $\sum_k\mu_k = 1$
    • 잘 보면, 이 Dirichlet distribution도 multinomial의 실수영역에서의 확장임

• posterior of multinomial likelihood & Dirichlet prior

  • $p(\boldsymbol{\mu}{\mid}\mathcal{D},\boldsymbol{\alpha})\propto{p(\mathcal{D}{\mid}\boldsymbol{\mu})p(\boldsymbol{\mu}{\mid}\boldsymbol{\alpha})}\propto{\prod_{k=1}^K\mu_k^{\alpha_k+m_k-1}}$

  • $p(\boldsymbol{\mu}{\mid}\mathcal{D},\boldsymbol{\alpha}) = \operatorname{Dir}(\boldsymbol{\mu}{\mid}\boldsymbol{\alpha}+\mathbf{m}) = {\Gamma(\alpha_0+N)\over{\Gamma(\alpha_1+m_1)\cdots\Gamma(\alpha_K+m_K)}}\prod_{k=1}^K\mu_k^{\alpha_k+m_k-1}$

    • $\mathbf{m} = (m_1,\cdots,m_K)^\mathsf{T}$
  • posterior가 Dirichlet임을 확인할 수 있고, Dirichlet prior가 multinomial likelihood의 conjugate prior임을 확인할 수 있음