Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning
Virtual Informal Systems Seminar (VISS) Centre for Intelligent Machines (CIM) and Groupe d'Etudes et de Recherche en Analyse des Decisions (GERAD)
Meeting ID: 910 7928 6959 聽 聽 聽 聽
Passcode: VISS
Speaker: 聽Kai Cui, PhD candidate, Technical University of Darmstadt
Abstract: The recent mean field game (MFG) formalism facilitates otherwise intractable computation of approximate Nash equilibria in many-agent settings. In this paper, we consider discrete-time finite MFGs subject to finite-horizon objectives. We show that all discrete-time finite MFGs with non-constant fixed point operators fail to be contractive as typically assumed in existing MFG literature, barring convergence via fixed point iteration. Instead, we incorporate entropy-regularization and Boltzmann policies into the fixed point iteration. As a result, we obtain provable convergence to approximate fixed points where existing methods fail, and reach the original goal of approximate Nash equilibria. All proposed methods are evaluated with respect to their exploitability, on both instructive examples with tractable exact solutions and high-dimensional problems where exact methods become intractable. In high-dimensional scenarios, we apply established deep reinforcement learning methods and empirically combine fictitious play with our approximations.
Bio: Kai Cui is a PhD candidate at the Bioinspired Communication Systems Lab under supervision of Professor Heinz Koeppl at Technical University of Darmstadt (Germany). Prior to joining BCS, he also received BSc and MSc degrees in Computer Science as well as Electrical Engineering and Information Technology at Technical University of Darmstadt. His current research interests are multi-agent systems, reinforcement learning, mean field games and UAV swarms.
