Affiliation:
1. College of Computer Science and Technology, Zhejiang University of Technology, HangZhou, China
2. School of Faculty of Education, University of Hong Kong, Hong Kong, China
Abstract
Diffusion models are a kind of math-based model that were first applied to image generation. Recently, they have drawn wide interest in natural language generation (NLG), a sub-field of natural language processing (NLP), due to their capability to generate varied and high-quality text outputs. In this article, we conduct a comprehensive survey on the application of diffusion models in text generation. We divide text generation into three parts (conditional, unconstrained, and multi-mode text generation, respectively) and provide a detailed introduction. In addition, considering that autoregressive-based pre-training models (PLMs) have recently dominated text generation, we conduct a detailed comparison between diffusion models and PLMs in multiple dimensions, highlighting their respective advantages and limitations. We believe that integrating PLMs into diffusion is a valuable research avenue. We also discuss current challenges faced by diffusion models in text generation and propose potential future research directions, such as improving sampling speed to address scalability issues and exploring multi-modal text generation. By providing a comprehensive analysis and outlook, this survey will serve as a valuable reference for researchers and practitioners interested in utilizing diffusion models for text generation tasks.
Funder
National Natural Science Foundation of China
Reference86 articles.
1. Structured denoising diffusion models in discrete state-spaces;Austin;Advances in Neural Information Processing Systems,2021
2. One transformer fits all distributions in multi-modal diffusion at scale;Bao,2023
3. Findings of the 2014 workshop on statistical machine translation;Bojar,2014
4. Findings of the 2016 conference on machine translation;Bojar,2016
5. Unleashing transformers: parallel token prediction with discrete absorbing diffusion for fast high-resolution image generation from vector-quantized codes;Bond-Taylor,2022
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献