A robotic model of hippocampal reverse replay for reinforcement learning

WHELAN, Matthew T., JIMENEZ RODRIGUEZ, Alejandro, PRESCOTT, Tony J. and VASILAKI, Eleni (2022). A robotic model of hippocampal reverse replay for reinforcement learning. Bioinspiration & Biomimetics, 18 (1): 015007.

Preview

PDF Jimenez Rodriguez-RoboticModelHippocampal(VoR).pdf - Published Version Creative Commons Attribution. Download (1MB) | Preview

Abstract

Hippocampal reverse replay, a phenomenon in which recently active hippocampal cells reactivate in the reverse order, is thought to contribute to learning, particularly reinforcement learning (RL), in animals. Here, we present a novel computational model which exploits reverse replay to improve stability and performance on a homing task. The model takes inspiration from the hippocampal-striatal network, and learning occurs via a three-factor RL rule. To augment this model with hippocampal reverse replay, we derived a policy gradient learning rule that associates place-cell activity with responses in cells representing actions and a supervised learning rule of the same form, interpreting the replay activity as a ‘target’ frequency. We evaluated the model using a simulated robot spatial navigation task inspired by the Morris water maze. Results suggest that reverse replay can improve performance stability over multiple trials. Our model exploits reverse reply as an additional source for propagating information about desirable synaptic changes, reducing the requirements for long-time scales in eligibility traces combined with low learning rates. We conclude that reverse replay can positively contribute to RL, although less stable learning is possible in its absence. Analogously, we postulate that reverse replay may enhance RL in the mammalian hippocampal-striatal system rather than provide its core mechanism.

Item Type:	Article
Uncontrolled Keywords:	02 Physical Sciences; 06 Biological Sciences; 09 Engineering; Physiology; 31 Biological sciences; 40 Engineering; 51 Physical sciences
Identification Number:	https://doi.org/10.1088/1748-3190/ac9ffc
SWORD Depositor:	Symplectic Elements
Depositing User:	Symplectic Elements
Date Deposited:	05 Apr 2024 10:37
Last Modified:	05 Apr 2024 10:45
URI:	https://shura.shu.ac.uk/id/eprint/33531

Actions (login required)

View Item

Downloads

Publication Metrics