An Empirical Comparison of Methods for Eliciting and Modeling Expert Knowledge

Abstract

The goal of this project in Human-Centered Computing was to apply a variety of methods of Cognitive Task Analysis (CTA) and Cognitive Field Research (CFR) to support a complete process going all the way from knowledge elicitation to leverage point identification and then to system prototyping, and also use this as an opportunity to empirically compare and evaluate the methods. The research relied upon the participation of expert, journeyman, and apprentice weather forecasters at the Naval Training Meteorology and Oceanography Facility at Pensacola Naval Air Station. Methods included Protocol Analysis, a number of types of structured interviews, workspace and work patterns analysis, the Critical Decision Method, the Knowledge Audit, Concept Mapping, and the Cognitive Modeling Procedure. The methods were compared in terms of (1) their yield of information that was useful in modeling expert knowledge, (2) their yield in terms of identification of leverage points (where the application of new technology might bring about positive change), and (3) their efficiency. Efficiency was gauged in terms of total effort (time to prepare to run a procedure, plus time to run the procedure, plus time to analyze the data) relative to the yield (number of leverage points identified, number of propositions suitable for use in a model of domain knowledge). CTA/CFR methods supported the identification of dozens of leverage points and also yielded behaviorally- validated models of the reasoning of expert forecasters. Knowledge modeling using Concept-Mapping resulted in over a thousand propositions covering domain knowledge. The Critical Decision Method yielded a number of richly-populated case studies with associated Decision Requirements Tables. Results speak to the relative efficiency of various methods of CTA/CFR, and also the strengths of each of the methods. In addition to extending our empirical base on the comparison of knowledge elicitation methods, a deliverable from the project was a knowledge model that illustrates human-centered computing in that it integrates training support and performance aiding.