Beyond Accuracy: An Empirical Study on Unit Testing in Open-source Deep Learning Projects-Reference-Cited by-同舟云学术

Beyond Accuracy: An Empirical Study on Unit Testing in Open-source Deep Learning Projects

Published:2023-12-20 Issue: Volume: Page:
ISSN:1049-331X
Container-title:ACM Transactions on Software Engineering and Methodology
language:en
Short-container-title:ACM Trans. Softw. Eng. Methodol.

Author:

Wang Han¹,Yu Sijia²,Chen Chunyang¹,Turhan Burak³,Zhu Xiaodong²

Affiliation:

1. Faculty of Information Technology, Monash University, Australia

2. Jilin University, Changchun, China

3. M3S Research Unit, Faculty of ITEE, University of Oulu, Finland

Abstract

Deep Learning (DL) models have rapidly advanced, focusing on achieving high performance through testing model accuracy and robustness. However, it is unclear whether DL projects, as software systems, are tested thoroughly or functionally correct when there is a need to treat and test them like other software systems. Therefore, we empirically study the unit tests in open-source DL projects, analyzing 9,129 projects from GitHub. We find that: 1) unit tested DL projects have positive correlation with the open-source project metrics and have a higher acceptance rate of pull requests, 2) 68% of the sampled DL projects are not unit tested at all, 3) the layer and utilities (utils) of DL models have the most unit tests. Based on these findings and previous research outcomes, we built a mapping taxonomy between unit tests and faults in DL projects. We discuss the implications of our findings for developers and researchers and highlight the need for unit testing in open-source DL projects to ensure their reliability and stability. The study contributes to this community by raising awareness of the importance of unit testing in DL projects and encouraging further research in this area.

Publisher

Association for Computing Machinery (ACM)

Subject

Software

Link

https://dl.acm.org/doi/pdf/10.1145/3638245

Reference81 articles.

1. 2015. pytest: helps you write better programs. https://docs.pytest.org/en/7.1.x/. Accessed: 2022-08-21. 2015. pytest: helps you write better programs. https://docs.pytest.org/en/7.1.x/. Accessed: 2022-08-21.

2. 2016. Unit tests for functions in a Jupyter notebook?https://stackoverflow.com/questions/40172281/unit-tests-for-functions-in-a-jupyter-notebook. Accessed: 2022-08-16. 2016. Unit tests for functions in a Jupyter notebook?https://stackoverflow.com/questions/40172281/unit-tests-for-functions-in-a-jupyter-notebook. Accessed: 2022-08-16.

3. 2017. How To Unit Test Machine Learning Code. https://www.kdnuggets.com/2017/11/unit-test-machine-learning-code.html. Accessed: 2022-03-09. 2017. How To Unit Test Machine Learning Code. https://www.kdnuggets.com/2017/11/unit-test-machine-learning-code.html. Accessed: 2022-03-09.

4. 2017. Tensorflow response is making no sense. https://www.reddit.com/r/MachineLearning/comments/6qyvvg/p_tensorflow_response_is_making_no_sense/. Accessed: 2022-08-09. 2017. Tensorflow response is making no sense. https://www.reddit.com/r/MachineLearning/comments/6qyvvg/p_tensorflow_response_is_making_no_sense/. Accessed: 2022-08-09.

5. 2021. driver-check.py. https://github.com/5gon12eder/msc-graphstudy/blob/master/test/driver-check.py. Accessed: 2022-01-09. 2021. driver-check.py. https://github.com/5gon12eder/msc-graphstudy/blob/master/test/driver-check.py. Accessed: 2022-01-09.