Time-lapse Image Super-resolution Neural Network with Reliable Confidence Quantification for Optical Microscopy

Abstract

AbstractSingle image super-resolution (SISR) neural networks for optical microscopy have shown great capability to directly transform a low-resolution (LR) image into its super-resolution (SR) counterpart, enabling low-cost long-term live-cell SR imaging. However, when processing time-lapse data, current SISR models failed to exploit the important temporal dependencies between neighbor frames, often resulting in temporally inconsistent outputs. Besides, SISR models are subject to inference uncertainty that is hard to accurately quantify, therefore it is difficult to determine to what extend can we trust the inferred SR images. Here, we first build a large-scale, high-quality fluorescence microscopy dataset for the time-lapse image super-resolution (TISR) task, and conducted a comprehensive evaluation on two essential components of TISR neural networks, i.e., propagation and alignment. Second, we devised a deformable phase-space alignment (DPA) based TISR neural network (DPA-TISR), which adaptively enhances the cross-frame alignment in the phase domain and outperforms existing state-of-the-art SISR and TISR models. Third, we combined the Bayesian training scheme and Monte Carlo dropout with DPA-TISR, developing Bayesian DPA-TISR, and designed an expected calibration error (ECE)minimization framework to obtain a well-calibrated confidence map along with each output SR image, which reliably implicates potential inference errors. We demonstrate the unique characteristics of Bayesian DPA-TISR underlie the ultralong-term live-cell SR imaging capability with high spatial fidelity, superb temporal consistency, and accurate confidence quantification on a wide variety of bioprocesses.