Description

This paper is concerned with approximating entropy for Gaussian mixture model (GMM) probability distributions. It has two key contributions. The first is a Gaussian splitting algorithm that can permit entropy approximation to high accuracy. The second is a general method for computing tight upper and lower bounds to the entropy. Upper bounds are found according to the following insight: that any moment preserving (or moment matching) merging of Gaussian components provides an upper bound on the true entropy. An analogous approach may be applied to computing lower bounds: any moment preserving splitting of Gaussian components provides a lower bound on the true entropy. Furthermore, the closeness of these bounds is dependent on how well-separated the Gaussian components are from each other; this facilitates efficient merging/splitting heuristics to find tight bounds.

Abstract

For many practical probability density representations such as for the widely used Gaussian mixture densities, an analytic evaluation of the differential entropy is not possible and thus, approximate calculations are inevitable. For this purpose, the first contribution of this paper deals with a novel entropy approximation method for Gaussian mixture random vectors, which is based on a component-wise Taylor-series expansion of the logarithm of a Gaussian mixture and on a splitting method of Gaussian mixture components. The employed order of the Taylor-series expansion and the number of components used for splitting allows balancing between accuracy and computational demand. The second contribution is the determination of meaningful and efficient lower and upper bounds of the entropy, which can be also used for approximation purposes. In addition, a refinement method for the more important upper bound is proposed in order to approach the true entropy value.

Download

Full paper [pdf] (563 kb, 8 pages)