Seeing is believing: A graphical reference framework for multi-criteria evaluation

Abstract

This work proposes a method for assessment of evaluation processes, applicable to systems in which both measurable and policy-based performance criteria coexist. It intends to provide a mapping procedure, based solely on measurable criteria, aiming at visual inspection of the outcome, which allows the introduction of two types of indicators at different phases of the process. The definition of performance-based zones is suggested: two of them clearly correspond to high/low performance, in which the overall evaluation result is essentially established on the basis of measurable indicators, i.e. on the basis of objective criteria, and an in-between area, more prone to a less deterministic outcome. The method of evaluation is applied in three successive steps: (i) build the representation and identify the performance zones, (ii) introduce additional policy criteria to complete a first draft of the outcome, and, finally (iii) assess relative positioning, correct if necessary, and finalize. Performance maps are presented and inspected for a set of Research and Development institutions, to illustrate examples of application of the methodology proposed.