變分推斷

變分推斷（英文：Variational inference）或者話變分貝葉斯方法（英文：Variational Bayesian methods）係指啲一系列嘅技術，攞嚟逼近啲喺貝葉斯推論同埋機械學習中出現到嘅難整積分嘅。「變分」係意指喺某個範圍內改變啲初始簡單嘅分佈，去逼近個實際分佈；「推斷」意指從啲外顯變數推斷返啲潛在變數。

背景

内文：潛在變數模型

變分推斷或者話變分貝葉斯方法，主要係攞嚟處理啲複雜嘅統計模型，啲既有外顯變數又有潛在變數同埋未知參數嘅，嚟幫啲變數之間啲關係做建模。

根據貝葉斯推論：

${\displaystyle P(\mathbf {z} \mid \mathbf {x} )={\frac {P(\mathbf {x} \mid \mathbf {z} )\cdot P(\mathbf {z} )}{P(\mathbf {x} )}}}$

其中${\displaystyle \textstyle \mathbf {x} }$係樣本，${\displaystyle \textstyle \mathbf {z} }$係樣本反映嘅潛在變數。${\displaystyle P(\mathbf {z} \mid \mathbf {x} )}$表示畀有樣本${\displaystyle \textstyle \mathbf {x} }$嗰陣啲${\displaystyle \textstyle \mathbf {z} }$嘅分佈。

對於生成模型嚟講，可以根據潛在空間嘅${\displaystyle P(\mathbf {z} )}$生成潛在變數${\displaystyle \textstyle \mathbf {z} }$，再根據觀察空間嘅${\displaystyle P(\mathbf {x} \mid \mathbf {z} )}$得到相應嘅新樣本${\displaystyle \textstyle \mathbf {x} }$。

反過嚟喺對啲${\displaystyle \textstyle \mathbf {x} }$建模出${\displaystyle P(\mathbf {x} )}$嗰陣，就需要用到${\displaystyle P(\mathbf {x} )}$嘅邊緣概似亦即係${\displaystyle \int P(\mathbf {z} )P(\mathbf {x} \mid \mathbf {z} )}{d\mathbf {z} }$；注意到${\displaystyle P(\mathbf {z} )P(\mathbf {x} \mid \mathbf {z} )}{d\mathbf {z} }$即係對${\displaystyle P(\mathbf {x} \mid \mathbf {z} )}$嘅期望形式，所以條式都可以寫成${\displaystyle \mathbb {E} _{\mathbf {z} \sim P(\mathbf {z} )}[P(\mathbf {x} \mid \mathbf {z} )]}$嘅形式。同時亦都要根據貝葉斯推論從${\displaystyle P(\mathbf {x} )}$抽出${\displaystyle P(\mathbf {z} \mid \mathbf {x} )}$亦即係潛在變數喺有嗰啲樣本嘅條件下個分佈情況。要對${\displaystyle P_{\theta }(\mathbf {x} )}$建模可以用對數最大概似估計，即最大化某個${\displaystyle f(\theta )}$。但因爲邊緣概似當中個積分冇解析解，所以難幫佢做數值積分或者求梯度，而變分推斷可以處理到種情況。

數學推導

最大化估計

對數最大概似可以嘸直接對${\displaystyle p_{\theta }(\mathbf {x} )}$建模，而係可以搵遞個簡單函數${\displaystyle g(\theta )}$係細過原本函數${\displaystyle f(\theta )}$嘅，作爲${\displaystyle f(\theta )}$下界（lower bound）之一，係噉有：

${\displaystyle f(\theta )\geq g(\theta )}$

希望係最大化個${\displaystyle g(\theta )}$概似理應畀到對應嘅最大化${\displaystyle f(\theta )}$概似，但由於${\displaystyle f(\theta )}$係好複雜嘅函數，所以單單靠一隻${\displaystyle g(\theta )}$係嘸夠組成個${\displaystyle f(\theta )}$啲下界，所以需要一堆壘${\displaystyle g\in {\mathcal {G}}}$ 喺個家族集合${\displaystyle {\mathcal {G}}}$裏頭又㔶齊${\displaystyle \theta }$個範圍嘅，再喺啲${\displaystyle g}$裏頭整返最大化${\displaystyle \max _{g\in {\mathcal {G}},\theta }g\left(\theta \right)}$嚟間接幫${\displaystyle f(\theta )}$做最大化${\displaystyle \max _{\theta }f\left(\theta \right)}$。

獲取啲下界函數

攞到啲下界函數${\displaystyle g(\theta )}$嘅方法可以係對個對數形式概率${\displaystyle p_{\theta }(\mathbf {x} )}$進行變形。對於潛在變數${\displaystyle \mathbf {z} }$同埋某個佢個分佈${\displaystyle q(\mathbf {z} )}$，顯在嘅對數概率${\displaystyle \log p_{\mathbf {\theta } }(\mathbf {x} )}$可以寫得成${\displaystyle \mathbb {E} _{\mathbf {z} \sim q(\mathbf {z} )}[\log p_{\mathbf {\theta } }(\mathbf {x} )]}$亦即係${\displaystyle \int q(\mathbf {z} )\log p_{\mathbf {\theta } }(\mathbf {x} )d\mathbf {z} }$嘅形式：

${\displaystyle \log p_{\mathbf {\theta } }(\mathbf {x} )=\int q(\mathbf {z} )\log p_{\mathbf {\theta } }(\mathbf {x} )d\mathbf {z} }$

由於${\displaystyle p_{\theta }(\mathbf {x} ,\mathbf {z} )=p_{\theta }(\mathbf {x} )p_{\theta }(\mathbf {z} \mid \mathbf {x} )}$，所以有：

${\displaystyle \log p_{\mathbf {\theta } }(\mathbf {x} )=\int q(\mathbf {z} )\log {\dfrac {p_{\theta }(\mathbf {x} ,\mathbf {z} )}{p_{\theta }(\mathbf {z} \mid \mathbf {x} )}}d\mathbf {z} }$

喺當中拆出${\textstyle q(\mathbf {z} )}$有：

${\displaystyle {\begin{aligned}\log p_{\mathbf {\theta } }(\mathbf {x} )&=\int q(\mathbf {z} )\log \left({\dfrac {p_{\theta }(\mathbf {x} ,\mathbf {z} )}{q(\mathbf {z} )}}\cdot {\dfrac {q(\mathbf {z} )}{p_{\theta }(\mathbf {z} \mid \mathbf {x} )}}\right)d\mathbf {z} \\&=\int q(\mathbf {z} )\log {\dfrac {p_{\theta }(\mathbf {x} ,\mathbf {z} )}{q(\mathbf {z} )}}d\mathbf {z} +\int q(\mathbf {z} )\log {\dfrac {q(\mathbf {z} )}{p_{\theta }(\mathbf {z} \mid \mathbf {x} )}}d\mathbf {z} \end{aligned}}}$

注意到左右兩䊆分別可以表示成期望同埋KL散度，似下式：

${\displaystyle \log p_{\mathbf {\theta } }(\mathbf {x} )={\mathbb {E} _{\mathbf {z} \sim q(\mathbf {z} )}[\log {\dfrac {p_{\theta }(\mathbf {x} ,\mathbf {z} )}{q(\mathbf {z} )}}]}+\mathbb {KL} [q(\mathbf {z} )\|p_{\theta }(\mathbf {z} \mid \mathbf {x} )]}$

個KL散度（即 ${\displaystyle \mathbb {KL} [q(\mathbf {z} )\|p_{\theta }(\mathbf {z} \mid \mathbf {x} )]}$ ）可以直觀啲噉理解爲：從${\displaystyle q(\mathbf {z} )}$嚟睇，${\displaystyle p_{\theta }(\mathbf {z} \mid \mathbf {x} )}$戥佢走差有幾多；KL散度等於零嗰陣，兩樣嘢基本可以睇作係喺所有埞方都相等。因爲KL散度係非負（大於等於零），所以有：

${\displaystyle \log p_{\mathbf {\theta } }(\mathbf {x} )\geq {\mathbb {E} _{\mathbf {z} \sim q(\mathbf {z} )}[\log {\dfrac {p_{\theta }(\mathbf {x} ,\mathbf {z} )}{q(\mathbf {z} )}}]}}$

即個${\displaystyle \mathbb {E} _{\mathbf {z} \sim q(\mathbf {z} )}[\log {\dfrac {p_{\theta }(\mathbf {x} ,\mathbf {z} )}{q(\mathbf {z} )}}]}$查實就係${\displaystyle \log p_{\mathbf {\theta } }(\mathbf {x} )}$嘅下界，亦即係證據下界（evidence lower bound，ELBO），記作${\displaystyle {\mathcal {L}}(\mathbf {\theta } ,q)}$。所以最好就係${\displaystyle q(\mathbf {z} )=p_{\theta }(\mathbf {z} \mid \mathbf {x} )}$，噉樣有KL散度爲零，個${\displaystyle {\mathcal {L}}(\mathbf {\theta } ,q)}$又得最大化。但因爲${\displaystyle p_{\theta }(\mathbf {z} \mid \mathbf {x} )}$好難攞到，好多時衹有好特殊嘅算法情況下先做得到令兩便相等，種情況係期望–最大化算法（expectation–maximization algorithm，EM Algorithm）。簡單嚟講EM–Algorithm係調校個${\displaystyle q(\mathbf {z} )}$嚟最細化個KL散度令到${\displaystyle {\mathcal {L}}(\mathbf {\theta } ,q)}$得最大化先（即「Expectation」），再喺個${\displaystyle q}$下搵返一個${\displaystyle \mathbf {\theta } }$最佳嘅令${\displaystyle \mathbb {E} _{\mathbf {z} \sim q(\mathbf {z} )}[\log p_{\theta }(\mathbf {x} ,\mathbf {z} )]}$最大，即又令到${\displaystyle {\mathcal {L}}(\mathbf {\theta } ,q)}$喺${\displaystyle \mathbf {\theta } }$方面得到最大化（即「Maximization」），再返去E動作繼續校細個${\displaystyle q}$……噉樣往復做落去最終去達到最佳。

睇埋

• 潛在變數模型
• 變分自編碼器