Applying machine learning methods to identify argumentative connections in scientific communication texts

Abstract

The paper presents the results of experiments to assess the machine learning methods applicability for solving the problem of identifying argumentative connections in scientific communication texts. Argumentative connection is understood as a relationship that connects the premise and the conclusion of a typical reasoning or an argument used by the author to persuade the readers. To assess the quality, the characteristics of accuracy, completeness and F-measure were used obtained when solving the problem of recognizing argumentative connections between adjacent text fragments of two types: sentences and clauses. The basis of the experiment was a Russian-language corpus of texts from the field of scientific communication with arguments marked up by linguistic experts. For markup, the ArgNetBank Studio tool was used, which allows creating collections of texts with detailed argumentation markup. Data sets for machine learning were built on the basis of labeled texts, in which the ratio of pairs of text fragments (sentences or clauses) connected and non-connected by argumentative relationships was 1 to 3. To improve the quality of model training, the sets were balanced in two ways. In the first case, a balance was achieved due to the fact that an equal number of pairs of both types were selected from each text; in the second, pairs were duplicated. Using the obtained data sets, experiments were carried out on linking text fragments using different types of machine learning methods. The range of changes in quality assessments when recognizing related fragments depending on their share in the training and test collections was experimentally determined. It has been established that, within the framework of the existing imbalance in real collections, the values of quality assessments can vary within 40–50%. The novelty of the work lies in the study of the range of possible discrepancies in quality assessments when applying different machine learning methods on balanced and unbalanced training and test collections in Russian-language material.