Affiliation:
1. Southern University of Science and Technology, Shenzhen, China / Hong Kong University of Science and Technology, Hong Kong, China
2. Hong Kong University of Science and Technology, Hong Kong, China
3. Southern University of Science and Technology, Shenzhen, China
Abstract
As a powerful tool for developers, interactive debuggers help locate and fix errors in software. By using
debugging information included in binaries, debuggers can retrieve necessary program states about the
program. Unlike printf-style debugging, debuggers allow for more flexible inspection and modification of
program execution states. However, debuggers may incorrectly retrieve and interpret program execution,
causing confusion and hindering the debugging process.
Despite the wide usage of interactive debuggers, a scalable and comprehensive measurement of their functionality correctness does not exist yet. Existing works either fall short in scalability or focus more on the
“compiler-side” defects instead of debugger bugs. To facilitate a better assessment of debugger correctness,
we first propose and advocate a set of debugger testing criteria, covering both comprehensiveness (in terms
of debug information covered) and scalability (in terms of testing overhead). Moreover, we design comparative experiments to show that fulfilling these criteria is not only theoretically appealing, but also brings
major improvement to debugger testing. Furthermore, based on these criteria, we present DTD, a differential
testing (DT) framework for detecting bugs in interactive debuggers. DTD compares the behaviors of two
mainstream debuggers when processing an identical C executable — discrepancies indicate bugs in one of the
two debuggers.
DTD leverages a novel heuristic method to avoid the repetitive structures (e.g., loops) that exist in C
programs, which facilitates DTD to achieve full debug information coverage efficiently. Moreover, we have
also designed a Temporal Differential Filtering method to practically filter out the false positives caused by
the uninitialized variables in common C programs. With these carefully designed techniques, DTD fulfills
our proposed testing requirements and, therefore, achieves high scalability and testing comprehensiveness.
For the first time, it offers large-scale testing for C debuggers to detect debugger behavior discrepancies
when inspecting millions of program states. An empirical comparison shows that DTD finds 17× more error-triggering cases and detects 5× more bugs than the state-of-the-art debugger testing technique. We have used DTD to detect 13 bugs in the LLVM toolchain (Clang/LLDB) and 5 bugs in the GNU toolchain (GCC/GDB).
One of our fixes has already landed in the latest LLDB development branch.
Funder
National Natural Science Foundation of China
Shenzhen Science and Technology Program
Publisher
Association for Computing Machinery (ACM)