Recent advances in in-sensor computational vision sensors: from mechanisms to applications

Abstract

Abstract The number of vision sensors continues to increase with the rapid development of intelligent systems. The effective transmitting and processing of the sensing data become difficult due to the sensing, computing and memory units being physically separated. In-sensor computing architecture inspired by biological visual systems with efficient information processing has attracted increasing attention for overcoming these performance limitations. Bipolar cells in the retina can generate ON/OFF information processing channels to amplify marginal information. The synaptic structure is plastic and can enhance the output information that is repeated many times. In recent years, numerous new material and device strategies to implement in-sensor computing by mimicking the functions of bipolar cells and synapses have been reported: ON/OFF optical responses have been realized on two-dimensional materials by band-modulation and tunneling; synaptic responses, such as short-term plasticity and long-term plasticity, have been realized by phase transition and carrier regulating. In this review, we will summarize the biological vision processes, analyse the physical mechanisms behind the in-sensor computational vision sensors (ICVSs), and then overview the emerging physical artificial neural networks implemented with ICVSs. After that, we will discuss ICVS design based on biological mechanisms beyond ON/OFF bipolar-cell-response and synaptic response.