Validate your white matter tractography algorithms with a reappraised ISMRM 2015 Tractography Challenge scoring system

Abstract

Abstract Since 2015, research groups seek to produce the nec-plus-ultra tractography algorithms using the ISMRM 2015 Tractography Challenge as evaluation. In particular, since 2017, machine learning has made its entrance into the tractography world. The ISMRM 2015 Tractography Challenge is the most used phantom during tractography validation, although it contains limitations. We offer, here, a new Tractometer scoring system for this phantom, where segmentation of the bundles is now based on manually-defined regions of interest rather than on bundle recognition. Bundles are now more reliably segmented, offering more stable metrics with higher precision for future users. New code is available online. Scores of the initial 96 submissions to the challenge are updated. Overall, conclusions from the 2015 challenge are confirmed with the new scoring, but individual tractograms scores have changed, and the data is much improved at the bundle- and streamline-level. This work also led to the production of a ground truth tractogram with less noisy streamlines and an example of processed data, all available on the Tractometer website. This enhanced Tractometer scoring system and new data should continue to help researchers develop and evaluate the next generation of tractography techniques.