A Normative Account of Confirmation Bias During Reinforcement Learning.

Lefebvre G
Summerfield C
Bogacz R

Confirmation bias refers to our tendency to pay more attention to information consistent with our past experience or beliefs. Here, we used computer simulations to investigate whether and how confirmation bias can be useful in certain situations. We show that confirmation bias can enable more accurate decisions in the face of variability, because it leads to higher levels of confidence and helps ensure choices are less likely to be swayed by variability.

Scientific Abstract

Reinforcement learning involves updating estimates of the value of states and actions on the basis of experience. Previous work has shown that in humans, reinforcement learning exhibits a confirmatory bias: when the value of a chosen option is being updated, estimates are revised more radically following positive than negative reward prediction errors, but the converse is observed when updating the unchosen option value estimate. Here, we simulate performance on a multi-arm bandit task to examine the consequences of a confirmatory bias for reward harvesting. We report a paradoxical finding: that confirmatory biases allow the agent to maximize reward relative to an unbiased updating rule. This principle holds over a wide range of experimental settings and is most influential when decisions are corrupted by noise. We show that this occurs because on average, confirmatory biases lead to overestimating the value of more valuable bandits and underestimating the value of less valuable bandits, rendering decisions overall more robust in the face of noise. Our results show how apparently suboptimal learning rules can in fact be reward maximizing if decisions are made with finite computational precision.

Colored graph showing how reward changes with different levels of confirmation bias
Average reward for a simulated agent making noisy decisions (employing softmax rule) as a function of learning rate for confirmatory information (alpha^C) and disconfirmatory information (alpha^D).
Citation
2022. Neural Comput., 34(2):307-337.
Publication
Shah SA, Tan H, Brown P

2017.Int IEEE EMBS Conf Neural Eng, 2017():371-374.

Publication
van Swieten MMH, Bogacz R
2020. PLoS Comput. Biol., 16(5):e1007465.
Publication
Salvatori T, Song Y, Hong Y, Sha L, Frieder S, Xu Z, Bogacz R, Lukasiewicz T
2021. Adv. Neural Inf. Process Syst., 34:3874-3886.
Publication
Fischer P, Tan H, Pogosyan A, Brown P

2016.Eur. J. Neurosci., 44(5):2202-13.