Effectiveness Metrics for Ideation: Merging Genealogy Trees and Improving Novelty Metric

Abstract

Experiments on engineering design creativity are typically assessed using either process-based (e.g., protocol studies using a coding scheme) or outcome-based approaches (e.g., assessing the ideas generated using a set of metrics). The authors review existing metrics used in outcome-based creativity experiments and analyze their applicability and limitations. In particular, the focus is on a widely used set of metrics for engineering design creativity experiments developed by Shah et al. [17]. These metrics provide robust conceptual definitions for Quality, Novelty and Variety, and various authors use them as a basis for assessment, adapting the metrics to their implementations as needed. Some other authors have proposed modifications to Shah et al.’s metrics as well as improvements. These changes typically address specific implementation issues, and hence, their validity is limited to the experiments at hand. The authors of this paper believe that the detailed implementation of ideation metrics should not only work numerically, but should also be conceptually consistent. Upon discovering previously not discussed issues of these metrics, the authors present modifications to the Shah et al. Novelty and Variety metric implementations. These changes preserve the numerical and conceptual integrity of the original metric forms. Examples from current design engineering creativity experiments are included to better understand the proposed changes to the metrics.