Optical implementation and robustness validation for multi-scale masked autoencoder

Abstract

Masked Autoencoders (MAEs), the state-of-the-art self-supervised neural network architecture in miscellaneous vision tasks, show surprisingly effective potential in reconstructing images distorted by random masking. This paper first introduces an optical implementation of MAEs, employing digital micromirror devices in the optical path to capture partially blocked images. MAEs with multi-scale patches are deployed in the reconstruction procedure. By using an optical-specialized version of the reconstruction network, the system can reconstruct original scenes of high quality. Simulations and experimental measurements showed a significant performance, achieving 24.41 dB average peak-signal-to-noise on Davis2017 datasets and 29.92 dB (masked areas) on authentic captured images under 70% of pixels being blocked. This paves the way for the application of low-bandwidth sampling of high-throughput high-resolution images.