On-chip Pixel Reconstruction using Simple CNN for Sparsely Read CMOS Image Sensor

CMOS image sensors have gained popularity due to low power consumption, high speed and their ability to scale to smaller sizes. These factors can further be improved by either improving the circuitry of the imager or by enhancing the processing capabilities of the entire imaging system through on-chip processing of the pixel information. One such improvement is to read only a small fraction of the pixel array information and reconstruct the original information from it. The on-chip prediction of the unread pixels to reconstruct the acquired image is still an open problem. Here we propose a solution to the problem which relies on on-chip implementation of simple convolutional neural network (CNN) for pixel prediction and image reconstruction. The system strives to selectively read only a minimal number of pixels (10% to 15%), while skipping the rest. This attributes to a considerable saving on power and checks the issue of latency at the ADC. This selective process in reading of only few pixels generates a sparse image signal. The proposed network is trained to reconstruct the estimation of the actual image from the sparse set. The hardware accelerator that incorporates the CNN model also considers hardware limitations and constraints such as power consumption and latency. The PSNR achieved for the SiDCNet variant with SqueezeNet achieves for test images is \sim30dB for 85% of pixels skipped, and the quantized model does not lose much in PSNR.