Modeling the effects of motivation on choice and learning in the basal ganglia.

van Swieten MMH
Bogacz R

Animals and humans combine information about their environments and current energy reserves to make choices important for survival. We developed a mathematical model that describes how the brain can evaluate positive and negative consequences of actions to make the best decisions given current energy levels. We provide experimental evidence that supports our model.

Scientific Abstract

Decision making relies on adequately evaluating the consequences of actions on the basis of past experience and the current physiological state. A key role in this process is played by the basal ganglia, where neural activity and plasticity are modulated by dopaminergic input from the midbrain. Internal physiological factors, such as hunger, scale signals encoded by dopaminergic neurons and thus they alter the motivation for taking actions and learning. However, to our knowledge, no formal mathematical formulation exists for how a physiological state affects learning and action selection in the basal ganglia. We developed a framework for modelling the effect of motivation on choice and learning. The framework defines the motivation to obtain a particular resource as the difference between the desired and the current level of this resource, and proposes how the utility of reinforcements depends on the motivation. To account for dopaminergic activity previously recorded in different physiological states, the paper argues that the prediction error encoded in the dopaminergic activity needs to be redefined as the difference between utility and expected utility, which depends on both the objective reinforcement and the motivation. We also demonstrate a possible mechanism by which the evaluation and learning of utility of actions can be implemented in the basal ganglia network. The presented theory brings together models of learning in the basal ganglia with the incentive salience theory in a single simple framework, and it provides a mechanistic insight into how decision processes and learning in the basal ganglia are modulated by the motivation. Moreover, this theory is also consistent with data on neural underpinnings of overeating and obesity, and makes further experimental predictions.

A cartoon illustrating the usefulness of an action depends on the size of the reinforcementand the current physiological state.
The usefulness of an action (U) depends on the size of the reinforcement (r) and the current physiological state (S1). To maintain your physiological balance you should take actions that bring you closer to your desired state (S*). The motivation (m) for taking an action is expressed as the difference between your current state and your desired state. If the motivation is positive, an action is useful as this action brings you closer to the desired state. If the motivation is negative, the action is not useful, because taking this action will bring you further away from you desired state. The size of the reinforcement represents the change in state by taking the action. Changes in state resulting in an increase in desirability are indicated by a green arrow and decreases in desirability are indicated by a red arrow.
2020. PLoS Comput. Biol., 16(5):e1007465.