Successive Refinement and Coarsening of the Information Bottleneck

Abstract

We study two central aspects of information processing in cognitive systems: one is the ability to incorporate fresh information to already learnt models; the other is the “trickling” of information through the many layers of a cognitive processing pipeline. We investigate the extent to which these specific structures of cognitive processing impact their informational optimal limits. To do so, we present mathematical characterisations and low-dimensional numerical examples, which explore formal properties of the Information Bottleneck method: namely, how it relates to successive refinement, and successive coarsening of information.