Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models

Millidge B
Salvatori T
Song Y
Lukasiewicz T
Bogacz R

Many different computer models have been proposed for how the brain stores and retrieves memories. This paper shows that a lot of these models can be described in a common mathematical framework.

Scientific Abstract

A large number of neural network models of associative memory have been proposed in the literature. These include the classical Hopfield networks (HNs), sparse distributed memories (SDMs), and more recently the modern continuous Hopfield networks (MCHNs), which possess close links with self-attention in machine learning. In this paper, we propose a general framework for understanding the operation of such memory networks as a sequence of three operations: similarity, separation, and projection. We derive all these memory models as instances of our general framework with differing similarity and separation functions. We extend the mathematical framework of Krotov et al (2020) to express general associative memory models using neural network dynamics with local computation, and derive a general energy function that is a Lyapunov function of the dynamics. Finally, using our framework, we empirically investigate the capacity of using different similarity functions for these associative memory models, beyond the dot product similarity measure, and demonstrate empirically that Euclidean or Manhattan distance similarity metrics perform substantially better in practice on many tasks, enabling a more robust retrieval and higher memory capacity than existing models.

A diagram of the Universal Hopfield Network
Stages of retrieval of memories in the Universal Hopfield Network
2022. Proceedings of the 39th International Conference on Machine Learning, PMLR 162:15561-15583
Related Content
Duszkiewicz AJ, McNamara CG, Takeuchi T, Genzel L
2019. Trends Neurosci., 42(2):102-114.
Salvatori T, Song Y, Hong Y, Sha L, Frieder S, Xu Z, Bogacz R, Lukasiewicz T
2021. Adv. Neural Inf. Process Syst., 34:3874-3886.
Richards BA, Lillicrap TP, Beaudoin P, Bengio Y, Bogacz R, Christensen A, Clopath C, Costa RP, de Berker A, Ganguli S, Gillon CJ, Hafner D, Kepecs A, Kriegeskorte N, Latham P, Lindsay GW, Miller KD, Naud R, Pack CC, Poirazi P, Roelfsema P, Sacramento J, Saxe A, Scellier B, Schapiro AC, Senn W, Wayne G, Yamins D, Zenke F, Zylberberg J, Therien D, Kording KP
2019. Nat. Neurosci., 22:1761-1770.
Calder-Travis J, Bogacz R, Yeung N
2023, Journal of Mathematical Psychology, 117, 102815
Tang M, Salvatori T, Millidge B, Song Y, Lukasiewicz T, Bogacz R

2023. PLoS Comput Biol, 19(4)e1010719.