Information system for extraction of information from open web resources

Abstract

The purpose of the work is to develop a project of an information and reference system for finding answers to questions based on the highest degree of comparison using text content from open English- language web resources. Examples of such questions can be: “What is the best book ever?”, “What is the most popular IDE for Python”. The result of the functioning of the information and reference system is a ranked list of answers based on the frequency of appearance of each of the answer options. Also, a numerical characteristic of the probability of the preference of a particular answer over others is added to each element of the list. Based on this metric, the obtained results are ranked. This information and reference system works with questions to which there is no unequivocal answer, what differs it from classic information systems for finding answers to questions of the QA-system type. The latter have a hypothesis that there is only one true answer to the question, often such systems work with well-known facts. Examples of questions they answer can be, for example, the date of birth of a famous person, or the population of a certain country. Instead, the proposed information and reference system answers subjective questions, for example, “What is the best book in the fantasy genre?” or “What is the best programming language?”. The system is based on the popularity of one or another answer. Proper names based on the analysis of N-grams are also keywords for forming the answer to the question.