Object Empowerment: An Information-Theoretic Approach to Intrinsically Motivated Reinforcement Learning of Tool Use

Artificial intelligence aspires to build agents that act autonomously, adapt to novel situations, and discover meaningful strategies without external instruction. A crucial challenge in this regard is to enable agents with intrinsic drives that promote purposeful behaviour even in the absence of explicit goals or dense rewards. Within cognitive science and reinforcement learning (RL), such intrinsic motivations have been linked to the agent’s ability to influence its environment. Among these, empowerment stands out as a prominent form of intrinsic motivation. It is defined as the maximum mutual information between an agent’s actions and resulting states. It formalises this notion by quantifying how much control an agent possesses over its future. It provides a principled, information-theoretic foundation for autonomous exploration and self-organised behaviour. Building on this foundation, the present thesis advances empowerment from a general but unspecific model of control to a computational framework specifically designed for tool use. It argues that tool use, which is a hallmark of intelligent behaviour in both biological and artificial systems, can be understood as the process of maximising influence over taskrelevant objects through intermediary entities (tools). To capture this formally, the thesis introduces object empowerment, a novel formulation that conditions the empowerment channel on manipulable objects, thereby isolating the agent’s causal influence on specific environmental entities. The framework is then extended to learning tool–object interactions by integrating object empowerment into RL as an intrinsic reward regulariser. This allows agents to autonomously discover functional dependencies between tools and objects, even under sparsereward conditions. Subsequent chapters generalise the approach to environments with multiple tools and objects. This approach defines a multi-object empowerment model and a corresponding tool–object empowerment matrix that supports systematic tool comparison and selection. Finally, the thesis advances from tool selection to tool characterisation by introducing three empowerment-based measures: persistence, latency, and reliability. These measures quantify how long a tool remains effective, how quickly its effects manifest, and how robustly it performs under uncertainty, respectively. Empirical validation across custom grid-world and MiniHack environments demonstrates that agents trained with object empowerment regularisation converge faster, explore more efficiently, and exhibit interpretable tool-use behaviours compared to standard RL baselines. These experiments reveal that empowerment not only facilitates exploration but also provides a transparent, quantitative account of causal structure in tool-mediated interaction. Collectively, the thesis establishes object empowerment as a unifying principle for modelling and generating tool-use behaviour. By integrating information-theoretic control with RL, it bridges intrinsic motivation, causal reasoning, and autonomous skill acquisition. Thus, object empowerment offers more than an intrinsic drive: it constitutes a language for constructing agents that act not merely to explore, but to understand and shape their own possibilities for influence.