A hierarchical tree-based decision-making approach for assessing the relative trustworthiness of risk assessment models

Abstract

Risk assessment provides information to support decision-making. Then, the confidence that can be put in its outcomes is fundamental, and this depends on the accuracy, representativeness and completeness of the models used in the risk assessment. A quantitative measure is needed to assess the credibility and trustworthiness of the outcomes obtained from such models, for decision-making purposes. This article proposes a four-level, top-down, hierarchical tree to identify the main attributes and criteria that affect the level of trustworthiness of models used in risk assessment. The level of trustworthiness (Level 1) is broken down into two attributes (Level 2), three sub-attributes and one “leaf” attribute (Level 3), and seven basic “leaf” sub-attributes (Level 4). Based on this hierarchical decomposition, a bottom-up, quantitative approach is employed for the assessment of model trustworthiness, using tangible information and data available for the basic “leaf” sub-attributes (Level 4). Analytical hierarchical process is adopted for evaluating and aggregating the sub-attributes, and Dempster–Shafer theory is adopted to consider the uncertainty and the inconsistency in the experts’ judgments. The approach is applied to a case study concerning the modeling of the residual heat removal system of a nuclear power plant, to compute its failure probability. The relative trustworthiness of two mathematical models of different complexity is evaluated: a fault tree and a multi-states physics-based model. The trustworthiness of the multi-states physics-based model is found to outweigh that of the fault tree model, which can be explained by the fact that multi-states physics-based model takes into account the components failure dependency relations and degradation effects. The feasibility and reasonableness of the approach are, thus, demonstrated, paving the way for its potential applicability to inform decision-making on safety-critical systems.