Show me your neighbour and I tell what you are: fisheye transformation for deep learning-based single-cell phenotyping

Abstract

AbstractRecently we have concluded that image-based features derived from the microenvironment have an enormous impact on successfully determining the class of an object1. Here we demonstrate that deep learning-based phenotypic analysis of cells with a properly chosen microenvironment-size provides results comparable to our earlier neighbourhood-based methods that utilise hand-crafted image features. We hypothesised that treating cells with equal weight, regardless of their position within the cellular microenvironment, is suboptimal, and direct neighbours have a larger impact on the phenotype of the cell-of-interest than cells in its larger proximity. Hence we present a novel approach that (1) considers the fully featured view of the cell-of-interest, (2) includes the neighbourhood and (3) gives lesser weight to cells that are far from the cell. To achieve this, we present a transformation similar to those characteristic for fisheye cameras. Such a transformation satisfies all the above defined criteria, with a fast rate of transform for any images. Using the proposed transformation with proper settings we could significantly increase the accuracy of single-cell phenotyping, both in case of cell culture and tissue-based microscopy images. The range of potential applications of the proposed method goes beyond microscopy, as we present improved results on the iWildCam 2020 dataset containing images of wild animals.