A Generalized Information Bottleneck Method: A Decision-Theoretic Perspective
2026-02-20 • Information Theory
Information Theory
AI summaryⓘ
The authors study a way to simplify data while keeping what’s important for making predictions, a method called the information bottleneck (IB). They explore a more general version of IB that uses a special type of information measure called H-mutual information, which connects to decision-making ideas. Using this connection, the authors develop a new algorithm that helps balance how much data is compressed versus how useful it remains for prediction. This work extends the classical IB method to a broader set of measures and practical interpretations.
Information BottleneckMutual InformationH-mutual InformationData CompressionStatistical Decision TheoryAlternating OptimizationSample InformationUtility MeasurementConcave Functions
Authors
Akira Kamatsuka, Takahiro Yoshida
Abstract
The information bottleneck (IB) method seeks a compressed representation of data that preserves information relevant to a target variable for prediction while discarding irrelevant information from the original data. In its classical formulation, the IB method employs mutual information to evaluate the compression between the original and compressed data and the utility of the representation for the target variable. In this study, we investigate a generalized IB problem, where the evaluation of utility is based on the $\mathcal{H}$-mutual information that satisfies the concave (\texttt{CV}) and averaging (\texttt{AVG}) conditions. This class of information measures admits a statistical decision-theoretic interpretation via its equivalence to the expected value of sample information. Based on this interpretation, we derive an alternating optimization algorithm to assess the tradeoff between compression and utility in the generalized IB problem.