Longitudinal Study on Retrospective Assessment of Perceived Usability: A New Method and Perspectives

Abstract

AbstractThis paper reports a longitudinal study on the retrospective evaluation of perceived usability. We propose a retest method based on SUS questionnaires to reduce the multiple-observation effects typically present in longitudinal research. We designed different forms of the SUS questionnaire to eliminate memory effects and distributed duplicate SUS copies to subjects. We first measured the alternative-form reliability of duplicate in Experiment I ($\textbf{r =0.814}$). We then found that the duplicates we designed significantly reduce memory effects compared to the standard SUS questionnaire. Experiment II involved tests on the retrospective evaluation of perceived usability at five measurement points after initial use. The retrospective evaluation of perceived usability changes over time. After 5 min (at point ${\mathrm{T}}_2$), we found that each subject’s retrospective evaluation changed significantly regardless of whether the task had higher or lower usability levels. Moreover, the `directions’of this change tended to differ. This indicates that time is an important factor in perceived usability evaluation. Our results altogether indicate that time affects the retrospective evaluation of perceived usability as well as the accuracy of perceived usability evaluation, which suggests that the process of perceived usability assessment requires strict time constraints.