Computational power is the most important cornerstone and engine of the intelligent era, while vision is the most important way for both humans and machines to perceive the world. Electronic computing platforms have high energy consumption and limited computing speed, whereas optical neural networks offer significant advantages, including fast computing speed, high parallelism, and low power consumption, making them the most promising next-generation parallel computing solution. In this work, we propose a new integrated in-sensor computing solution: the Spectral Convolutional Neural Network (SCNN) chip for matter meta-imaging. By massively integrating spectral filters on the surface of the image sensor, large-scale parallel computations of vector inner products can be achieved in the spectral dimension. The image sensor integrated with spectral filters is considered as both the input layer and the first convolutional layer. Combining it with subsequent small-size electronic computing convolutional layers results in an optoelectronic hybrid neural network. The SCNN breaks through the current challenge where most optical neural networks cannot be practically applied to real-world applications, making it possible to tackle complex visual computing tasks in the real world. As an application, we used the same spectral chip to achieve two completely different real-world complex tasks—pathological diagnosis and face authentication—both achieving over 96% accuracy, demonstrating the immense potential of the SCNN for real-world applications on edge devices.

This work has published on Nature Communications(https://doi.org/10.1038/s41467-024-55558-3) at 2^nd Jan. 2025 with the title of “Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light”. The first authors of this paper is Ph.D candidate Shijie Rao and Prof. Kaiyu Cui. The corresponding author of this paper is Prof. Kaiyu Cui and Prof. Yidong Huang.