An Evaluation Framework for Reputation Management Systems*

Abstract

Reputation management (RM) is employed in distributed and peer-to-peer networks to help users compute a measure of trust in other users based on initial belief, observed behavior, and run-time feedback. These trust values influence how, or with whom, a user will interact. Existing literature on RM focuses primarily on algorithm development, not comparative analysis. To remedy this, the authors propose an evaluation framework based on the trace-simulator paradigm. Trace file generation emulates a variety of network configurations, and particular attention is given to modeling malicious user behavior. Simulation is trace-based and incremental trust calculation techniques are developed to allow experimentation with networks of substantial size. The described framework is available as open source so that researchers can evaluate the effectiveness of other reputation management techniques and/or extend functionality. This chapter reports on the authors’ framework’s design decisions. Their goal being to build a general-purpose simulator, the authors have the opportunity to characterize the breadth of existing RM systems. Further, they demonstrate their tool using two reputation algorithms (EigenTrust and a modified TNA-SL) under varied network conditions. The authors’ analysis permits them to make claims about the algorithms’ comparative merits. They conclude that such systems, assuming their distribution is secure, are highly effective at managing trust, even against adversarial collectives.