Structure and Information Parsimony: Emergence of Symmetries from Generalised Information Bottlenecks

The behaviour of embodied agents tends to unfold within a mesh of stringent constraints, each pulling in different directions. For instance, principled models of complexity-constrained behaviour suggest a fundamental tension between the enactement of the agent’s purposeful behaviour and the “informational resources” at its disposal. The main motivation of this thesis is the hypothesis that this tension induces agents to leverage the structure of the interaction with their environment, at a “granularity” adapted to their informational resources — as relying on such structure limits the “informational expenditure” necessary to enact purposeful behaviour. From the agent’s intrinsic perspective, this structure consists of regularities of the way its actions influence its sensory influx, i.e., of the sensorimotor contingencies (SMCs) that sensorimotor theories claim perception is based on — a concept that, despite the important progresses of the last decades, remains elusive in many respects. Here, I develop novel mathematical and computational tools, at the intersection of information theory, group theory and dynamical systems, that provide a theoretical framework to explore the above hypothesis — so as to unlock new avenues for progress in adaptive behaviour, structure discovery and sensorimotor perception. In particular, previous research suggests to formalise SMCs through the symmetries defined by the dynamics of the sensorimotor interface — i.e., well-chosen commutation relations involving the way the agent’s behaviour impacts its sensory influx. The above hypothesis thus becomes, at the formal level, that of a certain “duality” between a system’s symmetries and the informationally parsimonious descriptions of it that such symmetries make possible. The main focus of this thesis is to establish and investigate this duality, in different forms — in a group-theoretic but also closed-loop and stochastic setting — and at different granularities — where the symmetries’ granularity scales with the information parsimony that they make possible. More precisely, the results cover three different themes. First, I look at symmetries of stochastic channels through the information-theoretic lens (in the finite case). Group-theoretic invariances of a stochastic channel are explicitly characterised in terms of information parsimony through the “classic” Information Bottleneck (IB) framework. Channel equivariances, however, cannot be captured by the classic IB, but require the introduction of a novel framework, which we call the Divergence Information Bottleneck (DIB). Here, information parsimony is traded-off against the preservation of the divergence of the data distribution from a given exponential family. For a well-chosen exponential family, the corresponding DIB formalises the intuition of an optimal compression preserving the “information carried by a given channel”, and does characterise the channel’s equivariances. These information-theoretic reformulations of “exact” channel symmetries then yield principled definitions of soft channel invariances and equivariances, where the “softness” of the symmetry is parametrised by the granularity of the corresponding coarse-graining. Overall, this framework provides a novel building block for information theory-based symmetry discovery and symmetry-based coarse-graining. This formal progress could also be instrumental to understand the intrinsic structure of an agent’s sensorimotor interface — i.e., what is known as apparatus-related SMCs — through the transformations of the agent’s sensorimotor spaces that leave this interface unchanged. While the latter symmetries describe structure at an exclusively sensorimotor level, previous work suggests that perception also relies on commutation relations between sensorimotor and internal dynamics. These results resonate with SMC theory’s claim that the “attunement” of these internal dynamics makes them capture the invariants, but also the “structure of changes” induced by a given, ongoing agent-environment interaction—i.e., what is known as object-related SMCs. I propose to explicitate these concepts by extending the class-pose decomposition framework. In this line of work, the aim is to decompose a group action into one coordinate (the “class”) capturing the action’s invariants, i.e., its orbits, and a second coordinate (the “pose”) that is “strictly equivariant”, in that it equivariantly tracks the changes induced by the group action without capturing any invariant. This mathematical object is generalised in three directions: algebraic, dynamical and information-theoretic. The algebraic aspect starts from the observation that class-pose decomposition is only possible if all orbits are isomorphic—a highly non-generic assumption. To obtain a structure that can be built from any group action, we “reverse the arrows and break the bijectivity” in the commutation relations. This allows the transformations of the class-pose space to be richer than those of the original state-space. But we also require these commutation relations to be “maximally isomorphic”, yielding whatwe call a minimal joining of the orbits: i.e., intuitively, the “simplest” group action that simultaneously “simulates” the original group action on each orbit. The dynamical aspect consists in moving from the setting of group actions — ultimately a poor model of embodied agents’ own actions — to one that allows for a closed-loop “behaviour” made of non-invertible and stochastic actions: Markov Decision Processes (MDPs), here with fixed policy, no rewards and on standard Borel spaces. Classes become ergodic components of the MDP: i.e., intuitively, asymptotic attractors of the agent’s behaviour. Poses are then defined by a minimal joining of these ergodic components, which can be seen as a dynamical, measure-theoretic and policy-dependent counterpart of least common multiples for integers. Eventually, these generalisations of classes and poses are reformulated (in the finite case) in terms of information parsimony. This yields, in particular, an information-theoretic characterisation of a group action’s partition in orbits — an important step for formalising the links between symmetry and information. These formal structures provide new tools to the operationalisation of sensorimotor perception. However, rather than yielding a pure explicitation of existing theories, they suggest new conceptual directions: e.g., that the attunement of brain dynamics is induced by information parsimony constraints, and yields parsimonious fictions which emerge from sensorimotor history but are not reducible to ongoing sensorimotor dynamics. As most of the notions of structure above are defined for a continuous range of “granularities”, parametrised by the corresponding informational trade-offs, it is crucial to understand the relation between such structures at different granularities. As a first step in this direction, I study the relations between coarser or finer bottlenecks for the classic IB method. More precisely, I investigate a property known as successive refinement (SR), which asks whether a coarser bottleneck can be obtained as a coarse-graining of a finer bottleneck. This property is important to determine whether for the soft invariances defined above, coarser invariances always include finer ones. It is also relevant to incremental learning, as it is equivalent to whether one can, without incurring an additional informational cost, design a finer bottleneck by first designing a coarser one, and then adding new information from the source. SR is given (in the finite case and under mild assumptions) a geometric characterisation in terms of inclusion of convex hulls defined by bottleneck channels. This characterisation means, intuitively, that SR holds whenever the information captured by the coarser bottleneck is entirely “contained” in the information captured by the finer bottleneck. We then consider a soft notion of successive refinement, by quantifying the “lack” of it through a previously established notion of unique information. This allows to investigate, in synthetic numerical experiments, howthe “amount to which” the SR property is satisfied depends on the respective granularities of the coarser and finer bottlenecks. The experiments suggest that, generically, the “lack” of successive refinability is relatively mild, and SR is the “closest” to hold for trade-off parameters poised close to bifurcation values.