Assessing the Potential of Multi-Temporal Conditional Generative Adversarial Networks in SAR-to-Optical Image Translation for Early-Stage Crop Monitoring

Abstract

The incomplete construction of optical image time series caused by cloud contamination is one of the major limitations facing the application of optical satellite images in crop monitoring. Thus, the construction of a complete optical image time series via image reconstruction of cloud-contaminated regions is essential for thematic mapping in croplands. This study investigates the potential of multi-temporal conditional generative adversarial networks (MTcGANs) that use a single synthetic aperture radar (SAR) image acquired on a prediction date and a pair of SAR and optical images acquired on a reference date in the context of early-stage crop monitoring. MTcGAN has an advantage over conventional SAR-to-optical image translation methods as it allows input data of various compositions. As the prediction performance of MTcGAN depends on the input data composition, the variations in the prediction performance should be assessed for different input data combination cases. Such an assessment was performed through experiments using Sentinel-1 and -2 images acquired in the US Corn Belt. MTcGAN outperformed existing SAR-to-optical image translation methods, including Pix2Pix and supervised CycleGAN (S-CycleGAN), in cases representing various input compositions. In particular, MTcGAN was substantially superior when there was little change in crop vitality between the reference and prediction dates. For the SWIR1 band, the root mean square error of MTcGAN (0.021) for corn was significantly improved by 54.4% and 50.0% compared to Pix2Pix (0.046) and S-CycleGAN (0.042), respectively. Even when there were large changes in crop vitality, the prediction accuracy of MTcGAN was more than twice that of Pix2Pix and S-CycleGAN. Without considering the temporal intervals between input image acquisition dates, MTcGAN was found to be beneficial when crops were visually distinct in both SAR and optical images. These experimental results demonstrate the potential of MTcGAN in SAR-to-optical image translation for crop monitoring during the early growth stage and can serve as a guideline for selecting appropriate input images for MTcGAN.