Content Validation from Card Sorting

Abstract

During the content validation phase of the development of a hybrid heuristic/survey-based usability assessment technique, the authors of this paper conducted a card sort with Subject Matter Experts (SMEs) in Human Factors Psychology. The information provided would inform the initial usability dimensions (e.g., Help, Learnability, Graphic Design & Aesthetics) used in the resulting technique. While Hierarchical Cluster Analysis (HCA) is the standard approach to data analysis with card sorts, several factors (e.g., size of data set, application of system, nature of construct measured) led the research team to consider alternatives. As a result, data derived from this card sort were analyzed in two ways: 1) using Confirmatory Factor Analysis (CFA) and, 2) Using (HCA). The use of the two methods provided an opportunity for researchers to compare their quality and utility in informing dimensions used during the construct validation phase. While HCA was more detailed and provided a better visual description as to how individual items were related to others and broad dimensions, the results were far from parsimonious, making it difficult to determine the meaning of broad dimensions and exactly where those dimensions deviated from others. In contrast, CFA resulted in clear and distinct dimensions with individual items loading on no more than two dimensions (e.g., one primary and one secondary). With that said, the CFA resulted in the rejection of more items as many did not load on the extant dimensions. The general conclusions of this psychometric methodological comparison, for the validation of a Usability evaluation method, is that while HCA is an excellent method for understanding the intercorrelations and intricacies of a construct (i.e., Usability), it may not be the best suited approach for developing a usability method intended to provide systematic feedback to system developers. If the goal is the development of a usability assessment technique, researchers and practitioners alike can use CFA to analyze their card sort data and inform the construct validation phase of their measure.