Use of Scripted Role-Play in Evaluation of Multiple-User Multiple-Service Mobile Social and Pervasive Systems

Abstract

Mobile and social computing is rapidly evolving towards a deeper integration with the physical world due to the proliferation of smart connected objects. It is widely acknowledged that involving end users in the design, development and evaluation of applications that function within the resulting complex socio-technical systems is crucial. However, reliable methods for managing evaluation of medium fidelity prototypes, whose utility is often dependent on rich data sets and/or the presence of multiple users simultaneously engaging in multiple activities, have not yet emerged. The authors report on the use of scripted role-play as an experimental approach applied in a mixed-methods evaluation of early prototypes of a suite of professional networking applications targeting a conference attendance scenario. Their evaluation was significantly constrained by the limited availability of a small cohort of end users for a relatively short period of time, which pose a challenge to define interactions that would ensure these users could experience and understand the novel application features. The authors observed that participatory role-play facilitated deeper user engagement with, exploration of, and discussion about, the mobile social applications than would have been possible with traditional usability approaches given the small user cohort and the time-constrained conditions.