On Block g-Circulant Matrices with Discrete Cosine and Sine Transforms for Transformer-Based Translation Machine
-
Published:2024-05-29
Issue:11
Volume:12
Page:1697
-
ISSN:2227-7390
-
Container-title:Mathematics
-
language:en
-
Short-container-title:Mathematics
Author:
Asriani Euis1, Muchtadi-Alamsyah Intan23ORCID, Purwarianti Ayu34
Affiliation:
1. Doctoral Program of Mathematics, Faculty of Mathematics and Natural Sciences, Institut Teknologi Bandung, Bandung 40132, Indonesia 2. Algebra Research Group, Faculty of Mathematics and Natural Sciences, Institut Teknologi Bandung, Bandung 40132, Indonesia 3. University Center of Excellence Artificial Intelligence on Vision, Natural Language Processing and Big Data Analytics (U-CoE AI-VLB), Institut Teknologi Bandung, Bandung 40132, Indonesia 4. Informatics Research Group, School of Electrical Engineering and Informatics, Institut Teknologi Bandung, Bandung 40132, Indonesia
Abstract
Transformer has emerged as one of the modern neural networks that has been applied in numerous applications. However, transformers’ large and deep architecture makes them computationally and memory-intensive. In this paper, we propose the block g-circulant matrices to replace the dense weight matrices in the feedforward layers of the transformer and leverage the DCT-DST algorithm to multiply these matrices with the input vector. Our test using Portuguese-English datasets shows that the suggested method improves model memory efficiency compared to the dense transformer but at the cost of a slight drop in accuracy. We found that the model Dense-block 1-circulant DCT-DST of 128 dimensions achieved the highest model memory efficiency at 22.14%. We further show that the same model achieved a BLEU score of 26.47%.
Funder
Hibah PDD Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi 2023
Reference37 articles.
1. Mitsuda, K., Higashinaka, R., Sugiyama, H., Mizukami, M., Kinebuchi, T., Nakamura, R., Adachi, N., and Kawabata, H. (2022). Conversational AI for Natural Human-Centric Interaction: Proceedings of the 12th International Workshop on Spoken Dialogue System Technology, Singapore, IWSDS 2021, Springer Nature. 2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and dan Polosukhin, I. (2017). Attention is all you need. arXiv. 3. Ranganathan, J., and Abuka, G. (December, January 29). Text summarization using transformer model. Proceedings of the 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS), Milan, Italy. 4. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., and Schmid, C. (2021, January 10–17). Vivit: A video vision transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada. 5. Zeng, P., Zhang, H., Song, J., and Gao, L. (2022, January 23–29). S2 transformer for image captioning. Proceedings of the International Joint Conferences on Artificial Intelligence, Vienna, Austria.
|
|