SourcererJBF: A Java Build Framework For Large-Scale Compilation

Abstract

Researchers and tool developers working on dynamic analysis, software testing, automated program repair, verification, and validation, need large compiled, compilable, and executable code corpora to test their ideas. The publicly available corpora are relatively small, and/or non-compilable, and/or non-executable. Developing a compiled code corpus is a laborious activity demanding significant manual effort and human intervention. To facilitate large-scale program analysis research, we develop SourcererJBF , a J ava B uild F ramework that can automatically build a large Java code corpus without project-specific instructions and human intervention. To generate a compiled code corpus, SourcererJBF creates an offline knowledge base by collecting external dependencies from the project directories and existing build scripts (if available). It constructs indices of those collected external dependencies that enable a fast search for resolving dependencies during the project compilation. As the output of the large-scale compilation, it produces JAigantic, a compilable Java corpus containing compiled projects, their bytecode, dependencies, normalized build script, and build command. We evaluated SourcererJBF’s effectiveness, correctness, performance, and scalability in a large collection of Java projects. Our experimental results demonstrate that SourcererJBF is significantly effective and scalable in building large Java code corpus. Besides, it substantiates reasonable performance and correctness similar to projects’ existing build systems.