Virtual cleaning of works of art using deep convolutional neural networks

Abstract

AbstractA varnish layer that is applied to a painting, generally to protect it, yellows over time, deteriorating the original look of the painting. This prompts conservators to undertake a cleaning process to remove the old varnish and apply a new one. Providing the conservators with the likely appearance of the painting before the cleaning process starts can be helpful to them, which can be done through virtual cleaning. Virtual cleaning is simply the simulation of the cleaning process. Previous works in this area required the method to have access to black and white paint regions, or physically removing the varnish first at a few spots. Through looking at the problem of virtual cleaning differently, we try to address those shortcomings. To do so, we propose using a convolutional neural network (CNN) to tackle the problem of virtual cleaning. The CNN is trained on artificially yellowed images of people, urban and rural areas, and color charts, as well as their original versions. The network is then applied to various paintings with similar scene content. The results of the method are first compared to the only physical model in the virtual cleaning field. We compare the outputs from the proposed method and the physical model by visualization as well as a quantitative measure that calculates the spectral similarity between the outputs and the reference images. These results show that the proposed method outperforms the physical model. The CNN is also applied to images of the Mona Lisa and The Virgin and Child with Saint Anne, both painted by Leonardo da Vinci. Results show both a qualitative and quantitative improvement in the color quality of the resulting image compared to their reference images. The CNN developed here is also compared to a CNN that has been developed for the purpose of image colorization in the literature to demonstrate the effectiveness of the CNN devised here, showing that the CNN architecture herein leads to a better result. The novelty of the work proposed herein lies in two premises. First, the accuracy of the method, which is demonstrated through comparison with the only physical approach derived until now. Second is the generalizability of the method which is shown through blindly applying the method to two famous works of art for which no information but an RGB image of the uncleaned artwork is known.