BACKGROUND
Clinical Natural Language Processing (NLP) researchers need access to directly comparable evaluation results for applications such as text de-identification across a range of corpus types and the means to easily test new systems or corpora within the same framework. Current systems, reported metrics, and personally identifiable information (PII) categories evaluated are not easily comparable.
OBJECTIVE
This study presents an open-source and extensible end-to-end evaluation framework for comparing clinical NLP system performance across corpora even when the annotation categories do not align.
METHODS
As use case for this new evaluation framework, we use 6 off-the-shelf text de-identification systems across three standard clinical text corpora for the task (two of which are publicly available) and one private corpus (all in English) for a total of four corpora, with annotation categories that are not directly analogous. The framework is built on shell scripts that can be extended to include new systems, corpora, and performance metrics. We present this open tool, multiple means for aligning PII categories during evaluation, and our initial timing and performance metric findings.
RESULTS
From this case study, we found an order of magnitude difference in processing speed between systems and that no single system uniformly outperformed the others across corpora and PII categories. Instead, a rich tapestry of performance trade-offs for PII categories and groups of categories appeared.
CONCLUSIONS
NLP systems in general, and de-identification systems (and new domain corpora) in our use case, tend to be evaluated in stand-alone research articles that only include a limited set of comparator systems. We hold that a single evaluation pipeline across multiple systems and corpora allows for more nuanced comparisons. The open pipeline we present should reduce barriers to evaluation and system advancement.