What Formal Languages Can Transformers Express? A Survey-Reference-Cited by-同舟云学术

What Formal Languages Can Transformers Express? A Survey

Published:2024 Issue: Volume:12 Page:543-561
ISSN:2307-387X
Container-title:Transactions of the Association for Computational Linguistics
language:en
Short-container-title:

Author:

Strobl Lena¹,Merrill William²,Weiss Gail³,Chiang David⁴,Angluin Dana⁵

Affiliation:

1. Umeå University, Sweden. lena.strobl@umu.se

2. New York University, USA. willm@nyu.edu

3. EPFL, Switzerland. gail.weiss@epfl.ch

4. University of Notre Dame, USA. dchiang@nd.edu

5. Yale University, USA. dana.angluin@yale.edu

Abstract

Abstract As transformers have gained prominence in natural language processing, some researchers have investigated theoretically what problems they can and cannot solve, by treating problems as formal languages. Exploring such questions can help clarify the power of transformers relative to other models of computation, their fundamental capabilities and limits, and the impact of architectural choices. Work in this subarea has made considerable progress in recent years. Here, we undertake a comprehensive survey of this work, documenting the diverse assumptions that underlie different results and providing a unified framework for harmonizing seemingly contradictory findings.

Publisher

MIT Press

Link

https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl_a_00663/2370911/tacl_a_00663.pdf

Reference89 articles.

1. A survey of neural networks and formal languages;Ackerman;arXiv preprint arXiv:2006.01338,2020

2. Physics of language models: Part 1, context-free grammar;Allen-Zhu;arXiv preprint arXiv:2305.13673,2023

3. The permanent requires large uniform threshold circuits;Allender;Chicago Journal of Theoretical Computer Science,1999

4. Masked hard-attention transformers and Boolean RASP recognize exactly the star-free languages;Angluin;arXiv preprint arXiv:2310.13897,2023

5. Computational Complexity