TextGraph - A lexicon based framework for concept extraction and visualization

Abstract

The shift in the news consumption from traditional newspapers to online news has led media analysts and researchers to apply powerful text mining techniques on the vast amount of news data. News has a profound influence on public as it informs about the events happening around them and may affect them. It keeps the people connected and allows them to engage in the decision making process. The words used in the news language are sometimes taken from the regional languages so as to express a new phenomenon, event or idea. In this paper, we have proposed a lexicon based framework named as TextGraph that automatically extracts the concepts from the Dawn news using the Term Frequency–Inverse Document Frequency (TF-IDF) weighting factor and visualizes them in a formal way. To achieve value-add insights, we have developed Pakistani English corpus and used it along with other existing dictionaries. Our proposed corpus incorporates the Pakistani English words used in the Dawn news stories, which are annotated and validated by a human expert. Experimental results show that our concept extraction method out performs and gives more specific concepts. Our research suggests that the proposed framework and corpus opens multiple directions for promising future research in this domain.