A methodology for certification of modeling and simulation applications

Abstract

Certification of modeling and simulation (M&S) applications poses significant technical challenges for M&S program managers, engineers, and practitioners. Certification is becoming increasingly more important as M&S applications are used more and more for military training, complex system design evaluation, M&S-based acquisition, problem solving, and critical decision making. Certification, a very complex process, involves the measurement and evaluation of hundreds of qualitative and quantitative elements, mandates subject matter expert evaluation, and requires the integration of different evaluations. Planning and managing such measurements and evaluations requires a unifying methodology and should not be performed in an ad hoc manner. This paper presents such a methodology. The methodology consists of the following body of methods, rules, and postulates: (a) employment of subject matter experts, (b) construction of a hierarchy of indicators, (c) relative criticality weighting of indicators using the analytic hierarchy process, (d) using a rule-based expert knowledge base with an object-oriented specification language, (e) assignment of crisp, fuzzy, and nominal scores for the indicators, (f) aggregation of indicator scores, (g) graphical representation of the indicator scores and weights, (h) hypertext certification report, and (i) interpretation of the results. The methodology can be used for certification of any kind of M&S application either throughout the M&S development life cycle or after the development is completed.