Learning the payoffs and costs of actions.

Möller M
Bogacz R

How do we decide what to do in any given situation? We need to work out the payoffs and costs of actions – what we will gain and what it might cost us. But what is the exact nature of the mechanism in the brain which weighs up these payoffs and costs? (Continue reading accessible summary.)

Video Abstract
Scientific Abstract

A set of sub-cortical nuclei called basal ganglia is critical for learning the values of actions. The basal ganglia include two pathways, which have been associated with approach and avoid behavior respectively and are differentially modulated by dopamine projections from the midbrain. Inspired by the influential opponent actor learning model, we demonstrate that, under certain circumstances, these pathways may represent learned estimates of the positive and negative consequences (payoffs and costs) of individual actions. In the model, the level of dopamine activity encodes the motivational state and controls to what extent payoffs and costs enter the overall evaluation of actions. We show that a set of previously proposed plasticity rules is suitable to extract payoffs and costs from a prediction error signal if they occur at different moments in time. For those plasticity rules, successful learning requires differential effects of positive and negative outcome prediction errors on the two pathways and a weak decay of synaptic weights over trials. We also confirm through simulations that the model reproduces drug-induced changes of willingness to work, as observed in classical experiments with the D2-antagonist haloperidol.

Basal ganglia as a scale: weighing payoffs against costs
Basal ganglia as a scale: weighing payoffs against costs
Citation
2019. PLoS Comput. Biol., 15(2):e1006285.
More Like This
Publication
Zénon A, Duclos Y, Carron R, Witjas T, Baunez C, Régis J, Azulay JP, Brown P, Eusebio A

2016.Brain, 139(Pt 6):1830-43.

Unit Publication
Publication
Kurzawa N, Summerfield C, Bogacz R

2017.Neural Comput, 29(2):368-393.

Unit Publication
Publication
Groschner LN, Chan Wah Hak L, Bogacz R, DasGupta S, Miesenböck G
2018. Cell, 173(4):894-905.
Unit Publication
Publication
Duszkiewicz AJ, McNamara CG, Takeuchi T, Genzel L
2019. Trends Neurosci., 42(2):102-114.
Unit Publication