The Reliability of a Promotional Job Knowledge Examination Scored by Numberof Items Right and by Four Confidence Weighting Procedures and its Corresponding Concurrent Validity Estimates Relative to Performance Criterion Ratings

Abstract

For a sample of 153 male police officers, five sets of scores on a multiple-choice job knowledge examination were derived from the use of one conventional scoring procedure representing the number of items correct and from four confidence weighting methods. These five sets of scores were correlated with ratings of supervisors on each of eight job performance dimensions and with standings on each of eight factor scales of a standardized personality inventory. Evidence concerning the realization of higher test reliabilities from use of confidence weighting procedures in relation to those obtained from the conventional scoring method was equivocal. Increments in validity coefficients obtained from the confidence weighting scoring methods were slight. None of the five sets of scores registered statistically significant correlations with any one of eight personality scales. It would appear that in the context of the particular public service job studied the employment of confidence weighting techniques in achievement test scoring would yield at best only small gains in either reliability or concurrent validity. The use of the fourth confidence weighting procedure that statistically controlled for variance in a job knowledge examination demonstrated not one statistically significant correlation with any one of the personality factor scales but afforded statistically significant validity coefficients with five of the eight criterion measures-a circumstance which suggests the possible presence of a unique construct deserving of further systematic research in a variety of employment settings.