Virtual cleaning of works of art using a deep generative network: spectral reflectance estimation

Abstract

AbstractGenerally applied to a painting for protection purposes, a varnish layer becomes yellow over time, making the painting undergo an appearance change. Upon this change, the conservators start a process that entails removing the old layer of varnish and applying a new one. As widely discussed in the literature, helping the conservators through supplying them with the probable outcome of the varnish removal can be of great value to them, aiding in the decision making process regarding varnish removal. This help can be realized through virtual cleaning, which in simple terms, refers to simulation of the cleaning process outcome. There have been different approaches devised to tackle the problem of virtual cleaning, each of which tries to develop a method that virtually cleans the artwork in a more accurate manner. Although successful in some senses, the majority of them do not possess a high level of accuracy. Prior approaches suffer from a range of shortcomings such as a reliance on identifying locations of specific colors on the painting, the need to access a large set of training data, or their lack of applicability to a wide range of paintings. In this work, we develop a Deep Generative Network to virtually clean the artwork. Using this method, only a small area of the painting needs to be physically cleaned prior to virtual cleaning. Using the cleaned and uncleaned versions of this small area, the entire unvarnished painting can be estimated. It should be noted that this estimation is performed in the spectral reflectance domain and herein it is applied to hyperspectral imagery of the work. The model is first applied to a Macbeth ColorChecker target (as a proof of concept) and then to real data of a small impressionist panel by Georges Seurat (known as ‘Haymakers at Montfermeil’ or just ‘Haymakers’). The Macbeth ColorChecker is simulated in both varnished and unvarnished forms, but in the case of the ‘Haymakers’, we have real hyperspectral imagery belonging to both states. The results of applying the Deep Generative Network show that the proposed method has done a better job virtually cleaning the artwork compared to a physics-based method in the literature. The results are presented through visualization in the sRGB color space and also by computing Euclidean distance and spectral angle (calculated in the spectral reflectance domain) between the virtually cleaned artwork and the physically cleaned one. The ultimate goal of our virtual cleaning algorithm is to enable pigment mapping and identification after virtual cleaning of the artwork in a more accurate manner, even before the process of physical cleaning.