Mobile-UI-Repair: a deep learning based UI smell detection technique for mobile user interface

Abstract

The graphical user interface (GUI) in mobile applications plays a crucial role in connecting users with mobile applications. GUIs often receive many UI design smells, bugs, or feature enhancement requests. The design smells include text overlap, component occlusion, blur screens, null values, and missing images. It also provides for the behavior of mobile applications during their usage. Manual testing of mobile applications (app as short in the rest of the document) is essential to ensuring app quality, especially for identifying usability and accessibility that may be missed during automated testing. However, it is time-consuming and inefficient due to the need for testers to perform actions repeatedly and the possibility of missing some functionalities. Although several approaches have been proposed, they require significant performance improvement. In addition, the key challenges of these approaches are incorporating the design guidelines and rules necessary to follow during app development and combine the syntactical and semantic information available on the development forums. In this study, we proposed a UI bug identification and localization approach called Mobile-UI-Repair (M-UI-R). M-UI-R is capable of recognizing graphical user interfaces (GUIs) display issues and accurately identifying the specific location of the bug within the GUI. M-UI-R is trained and tested on the history data and also validated on real-time data. The evaluation shows that the average precision is 87.7% and the average recall is 86.5% achieved in the detection of UI display issues. M-UI-R also achieved an average precision of 71.5% and an average recall of 70.7% in the localization of UI design smell. Moreover, a survey involving eight developers demonstrates that the proposed approach provides valuable support for enhancing the user interface of mobile applications. This aids developers in their efforts to fix bugs.