The efficient implementation of distributed indexing with Hadoop for digital investigations on Big Data

Abstract

Big Data brings new challenges to the field of e-Discovery or digital forensics and these challenges are mostly connected to the various methods for data processing. Considering that the most important factors are time and cost in determining success or failure of digital investigation, the development of a valid indexing method for efficient search should come first to more quickly and accurately find relevant evidence from Big Data. This paper, therefore, introduces a Distributed Text Processing System based on Hadoop called DTPS and explains about the distinctions between DTPS and other related researches to emphasize the necessity of it. In addition, this paper describes various experimental results in order to find the best implementation strategy in using Hadoop MapReduce for the distributed indexing and to analyze the worth for practical use of DTPS by comparative evaluation of its performance with similar tools. To be short, the ultimate purpose of this research is the development of useful search engine specially aimed at Big Data indexing as a major part for the future e-Discovery cloud service.