Abstract
Transformers have revolutionized machine learning models of language and vision, but their connection with neuroscience remains tenuous. Built from attention layers, they require a mass comparison of queries and keys that is difficult to perform using traditional neural circuits. Here, we show that neurons can implement attention-like computations using short-term, Hebbian synaptic potentiation. We call our mechanism the match-and-control principle and it proposes that when activity in an axon is synchronous, or matched, with the somatic activity of a neuron that it synapses onto, the synapse can be briefly strongly potentiated, allowing the axon to take over, or control, the activity of the downstream neuron for a short time. In our scheme, the keys and queries are represented as spike trains and comparisons between the two are performed in individual spines allowing for hundreds of key comparisons per query and roughly as many keys and queries as there are neurons in the network.
Funder
Brain and Behavior Research Foundation
Publisher
Public Library of Science (PLoS)
Subject
Computational Theory and Mathematics,Cellular and Molecular Neuroscience,Genetics,Molecular Biology,Ecology,Modeling and Simulation,Ecology, Evolution, Behavior and Systematics
Reference64 articles.
1. Attention Is All You Need;A Vaswani;Advances in Neural Information Processing Systems,2017
2. Saharia C, Chan W, Saxena S, Li L, Whang J, Denton E, et al. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. 2022; arxiv:2205.11487v1.
3. Ramesh A, Dhariwal P, Nichol A, Chu C, OpenAI MC. Hierarchical Text-Conditional Image Generation with CLIP Latents. 2022; arxiv:2204.06125v1.
4. A Survey of Transformers;T Lin;AI Open,2021
5. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding;J Devlin;NAACL HLT 2019—2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies—Proceedings of the Conference,2018