Tips & Tricks

Table of Contents

Log-sum-exp

Why

Improves numerical stability.

How

\begin{equation*} \log \Big[ \sum_i \exp (x_i) \Big] = \alpha + \log \Big[\sum_i \exp (x_i - \alpha) \Big] \end{equation*} 1
  • $\alpha$ usually chosen to be $\underset{i}{\text{max }} x_i$
  • I can see that the introduction of $\alpha$ will make it more stable, by decreasing how much $x_i$ affects the sum, but at the same time you will get even smaller exponentials due to $x_i$. In my head that would imply even more instability, no? I guess the constant $\alpha$ somehow provides more stability than this additional instability..