A VR-based volumetric medical image segmentation and visualization system with natural human interaction

Abstract

AbstractVolume rendering produces informative two-dimensional (2D) images from a 3-dimensional (3D) volume. It highlights the region of interest and facilitates a good comprehension of the entire data set. However, volume rendering faces a few challenges. First, a high-dimensional transfer function is usually required to differentiate the target from its neighboring objects with subtle variance. Unfortunately, designing such a transfer function is a strenuously trial-and-error process. Second, manipulating/visualizing a 3D volume with a traditional 2D input/output device suffers dimensional limitations. To address all the challenges, we design NUI-VR$$^2$$ 2 , a natural user interface-enabled volume rendering system in the virtual reality space. NUI-VR$$^2$$ 2 marries volume rendering and interactive image segmentation. It transforms the original volume into a probability map with image segmentation. A simple linear transfer function will highlight the target well in the probability map. More importantly, we set the entire image segmentation and volume rendering pipeline in an immersive virtual reality environment with a natural user interface. NUI-VR$$^2$$ 2 eliminates the dimensional limitations in manipulating and perceiving 3D volumes and dramatically improves the user experience.