Enhancing Privacy in Large Language Model with Homomorphic Encryption and Sparse Attention
-
Published:2023-12-11
Issue:24
Volume:13
Page:13146
-
ISSN:2076-3417
-
Container-title:Applied Sciences
-
language:en
-
Short-container-title:Applied Sciences
Author:
Zhang Lexin1, Li Changxiang1, Hu Qi1, Lang Jingjing1, Huang Sirui1, Hu Linyue1, Leng Jingwen1, Chen Qiuhan1, Lv Chunli1
Affiliation:
1. China Agricultural University, Beijing 100083, China
Abstract
In response to the challenges of personal privacy protection in the dialogue models of the information era, this study introduces an innovative privacy-preserving dialogue model framework. This framework seamlessly incorporates Fully Homomorphic Encryption (FHE) technology with dynamic sparse attention (DSA) mechanisms, aiming to enhance the response efficiency and accuracy of dialogue systems without compromising user privacy. Experimental comparative analyses have confirmed the advantages of the proposed framework in terms of precision, recall, accuracy, and latency, with values of 0.92, 0.91, 0.92, and 15 ms, respectively. In particular, the newly proposed DSA module, while ensuring data security, significantly improves performance by up to 100 times compared to traditional multi-head attention mechanisms.
Funder
National Natural Science Foundation of China
Subject
Fluid Flow and Transfer Processes,Computer Science Applications,Process Chemistry and Technology,General Engineering,Instrumentation,General Materials Science
Reference35 articles.
1. Zhang, Y., Wa, S., Liu, Y., Zhou, X., Sun, P., and Ma, Q. (2021). High-accuracy detection of maize leaf diseases CNN based on multi-pathway activation function module. Remote Sens., 13. 2. Lin, X., Wa, S., Zhang, Y., and Ma, Q. (2022). A dilated segmentation network with the morphological correction method in farming area image Series. Remote Sens., 14. 3. Zhang, Y., He, S., Wa, S., Zong, Z., Lin, J., Fan, D., Fu, J., and Lv, C. (2022). Symmetry GAN Detection Network: An Automatic One-Stage High-Accuracy Detection Network for Various Types of Lesions on CT Images. Symmetry, 14. 4. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA. 5. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
|
|