Affiliation:
1. Research Division, Genie Enterprise, Donnersbergweg 1, 67059 Ludwigshafen, Germany
Abstract
Nowadays, individuals can be overwhelmed by a huge number of documents being present in daily life. Capturing the necessary details is often a challenge. Therefore, it is rather important to summarize documents to obtain the main information quickly. There currently exist automatic approaches to this task, but their quality is often not properly assessed. State-of-the-art metrics rely on human-generated summaries as a reference for the evaluation. If no reference is given, the assessment will be challenging. Therefore, in the absence of human-generated reference summaries, we investigated an alternative approach to how machine-generated summaries can be evaluated. For this, we focus on the original text or document to retrieve a metric that allows a direct evaluation of automatically generated summaries. This approach is particularly helpful in cases where it is difficult or costly to find reference summaries. In this paper, we present a novel metric called Summary Score without Reference—SUSWIR—which is based on four factors already known in the text summarization community: Semantic Similarity, Redundancy, Relevance, and Bias Avoidance Analysis, overcoming drawbacks of common metrics. Therefore, we aim to close a gap in the current evaluation environment for machine-generated text summaries. The novel metric is introduced theoretically and tested on five datasets from their respective domains. The conducted experiments yielded noteworthy outcomes, employing the utilization of SUSWIR.
Subject
Industrial and Manufacturing Engineering
Reference46 articles.
1. Literature Review on Extractive Text Summarization Approaches;Saziyabegum;Int. J. Comput. Appl.,2016
2. Others Automatic summarization;Nenkova;Found. Trends® Inf. Retr.,2011
3. The anatomy of a large-scale hypertextual web search engine;Brin;Comput. Netw. ISDN Syst.,1998
4. Torres-Moreno, J. (2014). Automatic Text Summarization, John Wiley & Sons.
5. Iskender, N., Polzehl, T., and Möller, S. (2021, January 19). Reliability of human evaluation for text summarization: Lessons learned and challenges ahead. Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), Kyiv, Ukraine.
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献