Uncertainty-guided learning with scaled prediction errors in the basal ganglia.

Möller M
Manohar S
Bogacz R

When feedback is less reliable, it should influence learning to a smaller extent. This paper proposes how this principle is achieved in nerve cell circuits responsible for reinforcement learning in a brain region known as the basal ganglia. The paper describes a computational model where the activity of nerve cells releasing the chemical messenger dopamine is scaled by the uncertainty, which explains the observed patterns of activity of these cells.

Scientific Abstract

To accurately predict rewards associated with states or actions, the variability of observations has to be taken into account. In particular, when the observations are noisy, the individual rewards should have less influence on tracking of average reward, and the estimate of the mean reward should be updated to a smaller extent after each observation. However, it is not known how the magnitude of the observation noise might be tracked and used to control prediction updates in the brain reward system. Here, we introduce a new model that uses simple, tractable learning rules that track the mean and standard deviation of reward, and leverages prediction errors scaled by uncertainty as the central feedback signal. We show that the new model has an advantage over conventional reinforcement learning models in a value tracking task, and approaches a theoretic limit of performance provided by the Kalman filter. Further, we propose a possible biological implementation of the model in the basal ganglia circuit. In the proposed network, dopaminergic neurons encode reward prediction errors scaled by standard deviation of rewards. We show that such scaling may arise if the striatal neurons learn the standard deviation of rewards and modulate the activity of dopaminergic neurons. The model is consistent with experimental findings concerning dopamine prediction error scaling relative to reward magnitude, and with many features of striatal plasticity. Our results span across the levels of implementation, algorithm, and computation, and might have important implications for understanding the dopaminergic prediction error signal and its relation to adaptive and effective learning.

Dopaminergic responses
Dopaminergic responses in the experiment by Tobler et al. and their simulations. A) Probability distributions of rewards associated with 3 stimuli used in the experiment. Each stimulus produced a reward with 50% probability, but the stimuli differed in the volume of predicted reward. B) Activity of dopaminergic neurons (Data), and simulations of Rescorla-Wagner (RW) and Scaled Prediction Error (SPE) models.
Citation

2022. PLoS Comput Biol, 18(5):e1009816.

DOI
10.1371/journal.pcbi.1009816
Publication
Mikhael JG, Bogacz R

2016.PLoS Comput. Biol., 12(9):e1005062.

Publication
Pavlides A, Hogan SJ, Bogacz R

2015.PLoS Comput. Biol., 11(12):e1004609.

Publication
Bogacz R
2020. eLife, 9:e53262
Publication
Lopes-Dos-Santos V, van de Ven GM, Morley A, Trouche S, Campo-Urriza N, Dupret D
2018. Neuron, 100(4):940–952.
Publication
West TO, Berthouze L, Halliday DM, Litvak V, Sharott A, Magill PJ, Farmer SF
2018. J. Neurophysiol., 119(5):1608-1628.