Comparison and Extension of Novelty Metrics for Problem-Solving Tasks

Abstract

Shah’s metrics for measuring ideation effectiveness have been used extensively by the engineering design community to quantify the value of designed concepts. Shah measures novelty as the infrequency of an idea relative to a set of ideas. Vargas-Hernandez extended this novelty metric using partial genealogy trees to consider the frequency of ideas that share the same working principle. These genealogy trees capture differences between individual ideas organized by the following levels of abstraction: physical principle, working principle, and embodiment. Shah’s and Vargas-Hernandez’s metrics both require that all ideas be described at the lowest level (embodiment). This approach excludes ideas that are described at higher levels of abstraction. This paper proposes a new novelty metric that extends Vargas Hernandez’s metrics by including the higher levels of the genealogy trees, allowing abstract ideas to be properly evaluated. This paper compares the newly proposed novelty metric to Shah’s and Vargas Hernandez’s metrics using data from a previous study. The study required participants to perform problem-solving tasks in which they submitted a textual list of ideas for how to solve general day-to-day problems. The proposed novelty metric addresses limitations of the previous metrics when applied to the abstract ideas in the data set and meets established metric requirements. The proposed metric also broadens Shah’s metric in a similar manner as Vargas Hernandez but extends it to capture the entire genealogy tree rather than a subset of the tree.