Affiliation:
1. Nanjing University, Nanjing, China
2. National University of Singapore, Singapore, Singapore
Abstract
Datalog is a popular and widely-used declarative logic programming language. Datalog engines apply many cross-rule optimizations; bugs in them can cause incorrect results. To detect such optimization bugs, we propose an automated testing approach called Incremental Rule Evaluation (IRE), which synergistically tackles the test oracle and test case generation problem. The core idea behind the test oracle is to compare the results of an optimized program and a program without cross-rule optimization; any difference indicates a bug in the Datalog engine. Our core insight is that, for an optimized, incrementally-generated Datalog program, we can evaluate all rules individually by constructing a reference program to disable the optimizations that are performed among multiple rules. Incrementally generating test cases not only allows us to apply the test oracle for every new rule generated—we also can ensure that every newly added rule generates a non-empty result with a given probability and eschew recomputing already-known facts. We implemented IRE as a tool named Deopt, and evaluated Deopt on four mature Datalog engines, namely Soufflé, CozoDB, μZ, and DDlog, and discovered a total of 30 bugs. Of these, 13 were logic bugs, while the remaining were crash and error bugs. Deopt can detect all bugs found by queryFuzz, a state-of-the-art approach. Out of the bugs identified by Deopt, queryFuzz might be unable to detect 5. Our incremental test case generation approach is efficient; for example, for test cases containing 60 rules, our incremental approach can produce 1.17× (for DDlog) to 31.02× (for Soufflé) as many valid test cases with non-empty results as the naive random method. We believe that the simplicity and the generality of the approach will lead to its wide adoption in practice.
Funder
National Natural Science Foundation of China
Fundamental Research Funds for the Central Universities
Publisher
Association for Computing Machinery (ACM)
Reference74 articles.
1. Data functions, datalog and negation
2. Serge Abiteboul, Richard Hull, and Victor Vianu. 1995. Foundations of databases. 8, Addison-Wesley Reading.
3. Magic Sets for disjunctive Datalog programs
4. Porting doop to Soufflé
5. Samuel Arch, Xiaowen Hu, David Zhao, Pavle Subotić, and Bernhard Scholz. 2022. Building a Join Optimizer for Soufflé. In Logic-Based Program Synthesis and Transformation, Alicia Villanueva (Ed.). Springer International Publishing, Cham. 83–102. isbn:978-3-031-16767-6