KL 散度,又叫相对熵,用于衡量两个分布之间的距离。设 $p(x)$, $q(x)$ 是关于随机变量 $x$ 的两个分布,则 $p$ 相对于 $q$ 的 KL 散度为:
$D_{K L}(p \| q)=E_{p(x)} \log \frac{p(x)}{q(x)}$
信息论中,熵 $H(P)$ 表示对来自 $P$ 的随机变量进行编码所需的最小字节数,而交叉熵 $H(P, Q)$ 则表示使用基于 $Q$ 的编码对来自 $P$ 的变量进行编码所需的字节数,它也可以作为损失函数,即交叉熵损失函数:
$H(p, q)=-\sum_{i=1}^{n} p\left(x_{i}\right) \log \left(q\left(x_{i}\right)\right)$
因此,KL散度可以认为是使用基于 $Q$ 的编码对来自 $P$ 的变量进行编码所需的“额外"字节数;显然,额外字节数必然非负,当且仅当 $P=Q$ 时,额外字节数为 $0$:
$\begin{aligned}D_{K L}(p \| q) & =\sum_{i=1}^{n} p\left(x_{i}\right) \log \left(p\left(x_{i}\right)\right)-\sum_{i=1}^{n} p\left(x_{i}\right) \log \left(q\left(x_{i}\right)\right) \\& =-H(p(x))+\left[-\sum_{i=1}^{n} p\left(x_{i}\right) \log \left(q\left(x_{i}\right)\right)\right]\end{aligned}$
CLASS torch.nn.KLDivLoss(size_average=None, reduce=None, reduction='mean', log_target=False)
$L\left(y_{\text {pred }}, y_{\text {true }}\right)=y_{\text {true }} \cdot \log \frac{y_{\text {true }}}{y_{\text {pred }}}=y_{\text {true }} \cdot\left(\log y_{\text {true }}-\log y_{\text {pred }}\right)$
To avoid underflow(下溢) issues when computing this quantity, this loss expects the argument input
in the log-space. The argument target
may also be provided in the log-space
To summarise, this function is roughly equivalent to computing
if not log_target: # default loss_pointwise = target * (target.log() - input) else: loss_pointwise = target.exp() * (target - input)
and then reducing this result depending on the argument reduction
as
if reduction == "mean": # default loss = loss_pointwise.mean() elif reduction == "batchmean": # mathematically correct loss = loss_pointwise.sum() / input.size(0) elif reduction == "sum": loss = loss_pointwise.sum() else: # reduction == "none" loss = loss_pointwise
reduction= “mean” doesn’t return the true KL divergence value, please use reduction= “batchmean” which aligns with the mathematical definition. In a future release, “mean” will be changed to be the same as “batchmean”.
使用:
kl_loss = nn.KLDivLoss(reduction="batchmean") # input should be a distribution in the log space input = F.log_softmax(torch.randn(3, 5, requires_grad=True)) # Sample a batch of distributions. Usually this would come from the dataset target = F.softmax(torch.rand(3, 5)) output = kl_loss(input, target) kl_loss = nn.KLDivLoss(reduction="batchmean", log_target=True) log_target = F.log_softmax(torch.rand(3, 5)) output = kl_loss(input, log_target)
标签:loss,right,log,散度,kl,reduction,计算,target,left From: https://www.cnblogs.com/BlairGrowing/p/17328494.html