Regression Test List Sharding in a Distributed Test Environment

Abstract

One of the major issues during the regression test of the new version of Real Time Operating System (RTOS) is the time involved in test case execution. The main reason being a single embedded system device under test (DUT) is used to execute the test list containing several test cases. This traditional method of regression test also leads to wasted productivity of the other devices at hand that could be otherwise used during this regression test. Hence, in this paper, we propose a technique that aims at reducing the overall regression test cycle time of a newer version of a Real Time Operating System (RTOS) by employing a method known as “test-list sharding” in a distributed test environment. In the proposed work, multiple DUTs are connected to the test server via a communication network. The test server executes the test list containing several test cases and performs the test-list sharding, that is, distributing test cases to different DUTs and executing them in parallel. After the test is executed on the DUT, the test results are sent back to the test server which will summarize all the results. In the proposed work, the sharding is done by distributing the test cases without overloading or under loading any of the DUTs. Test list is sharded in such a way that the same tests are not sent to multiple DUTs. The main advantage of the proposed method is that the test sharding can be easily scalable to accommodate any number of devices that can be connected to the test server. Also, the test list sharding is done in a dynamic way so that the tests are distributed to an idle DUT that has completed a test execution and ready for another test to execute. The comparison study of executing a sample test list sequentially on a single DUT and distributed test system with multiple DUTs is performed. Results obtained showed the performance gain in terms of test cycle time reduction, scalability, equal load distribution and effective resource utilization.