Numerical‐discrete‐scheme‐incorporated recurrent neural network for tasks in natural language processing

Author:

Liu Mei12,Luo Wendi12,Cai Zangtai2,Du Xiujuan2,Zhang Jiliang3,Li Shuai12ORCID

Affiliation:

1. School of Information Science and Engineering Lanzhou University Lanzhou China

2. The State Key Laboratory of Tibetan Intelligent Information Processing and Application Qinghai Normal University Xining China

3. Department of Electronic and Electrical Engineering The University of Sheffield Sheffield UK

Abstract

AbstractA variety of neural networks have been presented to deal with issues in deep learning in the last decades. Despite the prominent success achieved by the neural network, it still lacks theoretical guidance to design an efficient neural network model, and verifying the performance of a model needs excessive resources. Previous research studies have demonstrated that many existing models can be regarded as different numerical discretizations of differential equations. This connection sheds light on designing an effective recurrent neural network (RNN) by resorting to numerical analysis. Simple RNN is regarded as a discretisation of the forward Euler scheme. Considering the limited solution accuracy of the forward Euler methods, a Taylor‐type discrete scheme is presented with lower truncation error and a Taylor‐type RNN (T‐RNN) is designed with its guidance. Extensive experiments are conducted to evaluate its performance on statistical language models and emotion analysis tasks. The noticeable gains obtained by T‐RNN present its superiority and the feasibility of designing the neural network model using numerical methods.

Funder

National Natural Science Foundation of China

Natural Science Foundation of Gansu Province

Publisher

Institution of Engineering and Technology (IET)

Subject

Artificial Intelligence,Computer Networks and Communications,Computer Vision and Pattern Recognition,Human-Computer Interaction,Information Systems

Reference39 articles.

1. Cross‐domain sequence labelling using language modelling and parameter generating

2. Yang Z.L. et al.:Breaking the softmax bottleneck: a high‐rank RNN language model. arXiv preprint arXiv:1711.03953 (2017)

3. Gated Recurrent Fusion With Joint Training Framework for Robust End-to-End Speech Recognition

4. Advances in image processing using machine learning techniques;Dolecek G.J.;CAAI Trans. Intell. Technol.,2022

5. CNN‐RNN based method for license plate recognition

Cited by 6 articles. 订阅此论文施引文献 订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献

1. A novel hybrid BWO-BiLSTM-ATT framework for accurate offshore wind power prediction;Ocean Engineering;2024-11

2. An adaptive discretized RNN algorithm for posture collaboration motion control of constrained dual-arm robots;Frontiers in Neurorobotics;2024-05-22

3. Improving Optimizers by Runge-Kutta Method: A case study of SGD and Adam;2024 12th International Conference on Intelligent Control and Information Processing (ICICIP);2024-03-08

4. Data-driven Motion-force Control for Acceleration Minimization of Robots;2023 13th International Conference on Information Science and Technology (ICIST);2023-12-08

5. Natural Robot Guidance using Transformers;2023 IEEE 28th International Conference on Emerging Technologies and Factory Automation (ETFA);2023-09-12

同舟云学术

1.学者识别学者识别

2.学术分析学术分析

3.人才评估人才评估

"同舟云学术"是以全球学者为主线,采集、加工和组织学术论文而形成的新型学术文献查询和分析系统,可以对全球学者进行文献检索和人才价值评估。用户可以通过关注某些学科领域的顶尖人物而持续追踪该领域的学科进展和研究前沿。经过近期的数据扩容,当前同舟云学术共收录了国内外主流学术期刊6万余种,收集的期刊论文及会议论文总量共计约1.5亿篇,并以每天添加12000余篇中外论文的速度递增。我们也可以为用户提供个性化、定制化的学者数据。欢迎来电咨询!咨询电话:010-8811{复制后删除}0370

www.globalauthorid.com

TOP

Copyright © 2019-2024 北京同舟云网络信息技术有限公司
京公网安备11010802033243号  京ICP备18003416号-3