A Method for Evaluating Computer Programs for Electrocardiographic Interpretation

Abstract

A simple method for testing reproducibility in ECG computer program performance results from using two digital representations of the same analog ECG tracing. Each digital representation is separated from the other by one millisecond in time. When the digital representations are processed by the Mayo Clinic program (1968), the diagnostic statements are identically reproduced in only 60% of 33 tracings. When the method is applied to version D of the PHS program and to the newly released IBM program of 1973, identical reproducibility is 43.3% and 76.0%, respectively, of 217 tracings. After analog filtering these figures are improved to 49.8% and 79.7%, respectively. These results show that reproducibility is most affected by a program's algorithms for pattern recognition, measurement, consistency checking, and noise handling. Reproducibility is less affected by attenuation of high frequency noise at the analog level. The relationship of reproducibility to program error rate in previous studies is discussed. Hence poor performance on this test obviates the need for a more time-consuming clinical evaluation. The need for human overview and quality checking is re-emphasized.