Exploiting Augmented Reality and Computer Vision for Healthcare Education: The Case of Pharmaceutical Substances Visualization and Information Retrieval

Abstract

Augmented Reality (AR) is already used as the primary visualization and user interaction tool in several scientific and business areas. At the same time new AR technologies and frameworks considerably facilitate both the development of innovative applications and also their wide adoption in different domains of everyday life. In the area of healthcare AR solutions make use of mobile or wearable devices and glasses to support, among others, education and healthcare professionals training. The aim of this paper is to present a prototype mHealth app for education, which uses AR and computer vision technologies for pharmaceutical substances recognition on drug packaging. The conceptual design of the system includes three main components which are responsible for a) Text recognition, b) Drug identification and c) AR operations for interactivity. The prototype application is available in Android or iOS platforms and has been evaluated in real-world scenarios. Camera and screen of the mobile phones fulfill the text recognition and AR operations, which eliminates the need for special equipment, while PubChem and 3D Model databases provide assets required for the drug identification and AR visualizations. The results highlight the value of AR for educational purposes, especially when combined with advanced image recognition technologies to build interactive AR encyclopedias.