A Complete Characterization of Tensorizable $f$-divergences
2026-08-28 • Information Theory
Information Theory
AI summaryⓘ
The authors studied a group of mathematical tools called f-divergences, which measure how different two probability distributions are. While many such tools exist, only a few like the Kullback-Leibler divergence and the chi-squared divergence are commonly used because they have convenient properties when combining data from independent sources, known as tensorization. The authors improved an existing way to understand tensorization and showed that all such formulas can be described in a simple form controlled by one parameter. They also identified exactly which f-divergences have this tensorization property under their definition.
f-divergenceprobability distributionsKullback-Leibler divergencechi-squared divergencesquared Hellinger distancetensorizationproduct measuresmulti-affine forminformation theorystatistics
Authors
Rodrigo Cruz, Flavio P. Calmon, Qian Yu
Abstract
Csiszar's formulation of the $f$-divergence introduced a vast family of functionals for quantifying dissimilarity between probability distributions. However, many applications in statistics and information theory rely only on a few $f$-divergences, such as the Kullback-Leibler divergence, the $χ^2$-divergence, and the squared Hellinger distance. These divergences are especially useful because they admit simple compositional formulas under product measures, a property sometimes referred to as tensorization. In this work, we refine a formalism of tensorization previously introduced in the literature. Then, we show that any possible tensorization formula has a multi-affine form characterized by a single parameter, and identify all tensorizable $f$-divergences under our adopted notion of tensorization.