Short-term memory traces for action bias in human reinforcement learning.

Bogacz R
McClure SM
Li J
Cohen JD
Montague PR
Scientific Abstract

Recent experimental and theoretical work on reinforcement learning has shed light on the neural bases of learning from rewards and punishments. One fundamental problem in reinforcement learning is the credit assignment problem, or how to properly assign credit to actions that lead to reward or punishment following a delay. Temporal difference learning solves this problem, but its efficiency can be significantly improved by the addition of eligibility traces (ET). In essence, ETs function as decaying memories of previous choices that are used to scale synaptic weight changes. It has been shown in theoretical studies that ETs spanning a number of actions may improve the performance of reinforcement learning. However, it remains an open question whether including ETs that persist over sequences of actions allows reinforcement learning models to better fit empirical data regarding the behaviors of humans and other animals. Here, we report an experiment in which human subjects performed a sequential economic decision game in which the long-term optimal strategy differed from the strategy that leads to the greatest short-term return. We demonstrate that human subjects' performance in the task is significantly affected by the time between choices in a surprising and seemingly counterintuitive way. However, this behavior is naturally explained by a temporal difference learning model which includes ETs persisting across actions. Furthermore, we review recent findings that suggest that short-term synaptic plasticity in dopamine neurons may provide a realistic biophysical mechanism for producing ETs that persist on a timescale consistent with behavioral observations.

Citation

2007.Brain Res., 1153():111-21.

More Like This
Publication
Bogacz R

2017.J Math Psychol, 76(Pt B):198-211.

Unit Publication
Publication
Bogacz R
2020. eLife, 9:e53262
Unit Publication
Publication
Grogan J, Tsivos D, Smith L, Knight BE, Bogacz R, Whone AL, Coulthard EJ
2017. eLife;6:e26801
Unit Publication
Publication
Aron AR, Herz DM, Brown P, Forstmann BU, Zaghloul KA

2016.J. Neurosci., 36(45):11489-11495.

Unit Publication
Publication
Richards BA, Lillicrap TP, Beaudoin P, Bengio Y, Bogacz R, Christensen A, Clopath C, Costa RP, de Berker A, Ganguli S, Gillon CJ, Hafner D, Kepecs A, Kriegeskorte N, Latham P, Lindsay GW, Miller KD, Naud R, Pack CC, Poirazi P, Roelfsema P, Sacramento J, Saxe A, Scellier B, Schapiro AC, Senn W, Wayne G, Yamins D, Zenke F, Zylberberg J, Therien D, Kording KP
2019. Nat. Neurosci., 22:1761-1770.
Unit Publication