Abstract
Purpose
– The purpose of this paper is to describe a pilot to test an approach to measuring inter-rater reliability of the Outcomes Star suite of tools. The intention, in publishing this account, is to show transparency in on-going development of the tool, and to invite further co-operative development.
Design/methodology/approach
– In total, 24 workers, trained to use the first edition Family Star, scored a tested case study. Scoring was analysed using two metrics on the ten-point scale and the underlying five-point Journey of Change. The case study approach and metrics were evaluated for validity and accessibility.
Findings
– This initial evaluation suggests this edition of the Family Star has good inter-rater reliability for the five-point Journey of Change, reaching the accepted threshold of 0.8 for the inter-rater reliability coefficient when three outlying workers are excluded. The reliability for the full ten point scale was moderate.
Research limitations/implications
– The sample size of 24 raters is small, though sufficient for an initial test of the approach, which will now be applied to larger samples, using other versions of the Outcomes Star.
Practical implications
– The findings indicate that it is important that service providers test worker understanding of the scales to ensure consistency of use. The second edition of the Family Star incorporates more precise definitions of the ten-point scales to help improve the reliability.
Originality/value
– The case study method and metrics provide an accessible measure of reliability, both for Star development and to enable managers to assess the reliability of an organisation's client data for internal and external purposes.
Subject
Sociology and Political Science,Geography, Planning and Development
Reference14 articles.
1. Brennan, C.W.
and
Daly, B.J.
(2014), “Methodological challenges of validating a clinical decision-making tool in the practice environment”, West J Nurs Res, available at: http://wjn.sagepub.com/content/early/2014/06/18/0193945914539738 (accessed 18 June 2014).
2. Burns, S.
and
MacKeith, J.
(2013), Family Star Organisation Guide, 2nd ed., Triangle Consulting Social Enterprise, London.
3. Burry-Stock, J.A.
,
Shaw, D.G.
,
Laurie, C.
and
Chissom, G.S.
(1996), “Rater agreement indexes for performance assessment”, Educational and Psychological Measurement, Vol. 56 No. 2, pp. 251-62.
4. Dickens, G.
,
Weleminsky, J.
,
Onifade, Y.
and
Sugarman, P.
(2012), “Recovery star: validating user recovery”, The Psychiatrist, Vol. 36, pp. 45-50.
5. Fleming, J.A.
,
McCracken, J.
and
Carran, D.
(2004), “A comparison of two methods of determining interrater reliability”, Assessment for Effective Intervention, Vol. 29, pp. 39-50.
Cited by
4 articles.
订阅此论文施引文献
订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献