Exact Tile-Based Segmentation Inference for Images Larger than GPU Memory

Abstract

We address the problem of performing exact (tiling-error free) out-of-core semantic segmentation inference of arbitrarily large images using fully convolutional neural networks (FCN). FCN models have the property that once a model is trained, it can be applied on arbitrarily sized images, although it is still constrained by the available GPU memory. This work is motivated by overcoming the GPU memory size constraint without numerically impacting the fnal result. Our approach is to select a tile size that will ft into GPU memory with a halo border of half the network receptive feld. Next, stride across the image by that tile size without the halo. The input tile halos will overlap, while the output tiles join exactly at the seams. Such an approach enables inference to be performed on whole slide microscopy images, such as those generated by a slide scanner. The novelty of this work is in documenting the formulas for determining tile size and stride and then validating them on U-Net and FC-DenseNet architectures. In addition, we quantify the errors due to tiling confgurations which do not satisfy the constraints, and we explore the use of architecture effective receptive felds to estimate the tiling parameters.