FACT: FFN-Attention Co-optimized Transformer Architecture with Eager Correlation Prediction-Reference-Cited by-同舟云学术

FACT: FFN-Attention Co-optimized Transformer Architecture with Eager Correlation Prediction

Published:2023-06-17 Issue: Volume: Page:
ISSN:
Container-title:Proceedings of the 50th Annual International Symposium on Computer Architecture
language:
Short-container-title:

Author:

Qin Yubin¹^ORCID,Wang Yang¹^ORCID,Deng Dazheng¹^ORCID,Zhao Zhiren¹^ORCID,Yang Xiaolong¹^ORCID,Liu Leibo¹^ORCID,Wei Shaojun¹^ORCID,Hu Yang¹^ORCID,Yin Shouyi¹^ORCID

Affiliation:

1. Tsinghua University, Beijing, China

Funder

National Natural Science Foundation of China

Beijing Municipal Science and Technology Commission

National Key Research and Development Program of China

Beijing National Research Center For Information Science And Technology

Beijing Innovation Center for Future Chip

Publisher

ACM

Reference73 articles.

1. Renzo Andri , Beatrice Bussolino , Antonio Cipolletta , Lukas Cavigelli , and Zhe Wang . 2022 . Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tiles . In 55th IEEE/ACM International Symposium on Microarchitecture, MICRO 2022 , Chicago, IL, USA, October 1--5 , 2022. IEEE, 582--598. Renzo Andri, Beatrice Bussolino, Antonio Cipolletta, Lukas Cavigelli, and Zhe Wang. 2022. Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tiles. In 55th IEEE/ACM International Symposium on Microarchitecture, MICRO 2022, Chicago, IL, USA, October 1--5, 2022. IEEE, 582--598.

2. Bahar Asgari , Ramyad Hadidi , Tushar Krishna , Hyesoon Kim , and Sudhakar Yalamanchili . 2020 . ALRESCHA: A Lightweight Reconfigurable Sparse-Computation Accelerator. In IEEE International Symposium on High Performance Computer Architecture, HPCA 2020 , San Diego, CA, USA, February 22--26 , 2020. IEEE, 249--260. Bahar Asgari, Ramyad Hadidi, Tushar Krishna, Hyesoon Kim, and Sudhakar Yalamanchili. 2020. ALRESCHA: A Lightweight Reconfigurable Sparse-Computation Accelerator. In IEEE International Symposium on High Performance Computer Architecture, HPCA 2020, San Diego, CA, USA, February 22--26, 2020. IEEE, 249--260.

3. Iz Beltagy , Matthew E. Peters , and Arman Cohan . 2020 . Longformer: The Long-Document Transformer. CoRR abs/2004.05150 (2020). arXiv:2004.05150 Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. CoRR abs/2004.05150 (2020). arXiv:2004.05150

4. Tom B. Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel M. Ziegler , Jeffrey Wu , Clemens Winter , Christopher Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , and Dario Amodei . 2020 . Language Models are Few-Shot Learners . In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 , NeurIPS 2020, December 6--12, 2020, virtual, Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6--12, 2020, virtual, Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).

5. Nicolas Carion , Francisco Massa , Gabriel Synnaeve , Nicolas Usunier , Alexander Kirillov , and Sergey Zagoruyko . 2020 . End-to-End Object Detection with Transformers. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23--28, 2020 , Proceedings, Part I (Lecture Notes in Computer Science , Vol. 12346), Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer, 213-- 229 . Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-End Object Detection with Transformers. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 12346), Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer, 213--229.