Leveraging Empowerment to Model Tool Use in Reinforcement Learning

Abstract

Intrinsic motivation plays a key role in learning how to use tools, a fundamental aspect of human cultural evolution and child development that remains largely unexplored within the context of Reinforcement Learning (RL). This paper introduces “object empowerment” as a novel concept within this realm, showing its role as information-theoretic intrinsic motivation that underpins tool discovery and usage. Using empowerment, we propose a new general framework to model the utilization of tools within RL. We explore how maximizing empowerment can expedite the RL of tasks involving tools, highlighting its capacity to solve the challenge posed by sparse reward signals. By employing object empowerment as an intrinsically motivated regulariser, we guide the RL agent in simple grid-worlds towards states beneficial for learning how to master tools for efficient task completion. We will show how object empowerment can be used to measure and compare the effectiveness of different tools in handling an object. Our findings indicate efficient strategies to learn tool use and insights into the integration and modeling of tool control in the context of RL.