Infrared Images with Super-resolution Based on Deep Convolutional Neural Network
-
摘要: 由于器件及工艺等技术限制,红外图像分辨率相对可见光图像较低,存在细节纹理特征模糊等不足。对此,本文提出一种基于深度卷积神经网络(convolutional neural network,CNN)的红外图像超分辨率重建方法。该方法改进残差模块,降低激活函数对信息流影响的同时加深网络,充分利用低分辨率红外图像的原始信息。结合高效通道注意力机制和通道-空间注意力模块,使重建过程中有选择性地捕获更多特征信息,有利于对红外图像高频细节更准确地进行重建。实验结果表明,本文方法重建红外图像峰值信噪比(peak signal to noise ratio,PSNR)优于传统的Bicubic插值法以及基于CNN的SRResNet、EDSR、RCAN模型。当尺度因子为×2和×4时,重建图像的平均PSNR值比传统Bicubic插值法分别提高了4.57 dB和3.37 dB。Abstract: Owing to technical limitations regarding the device and process, the resolution of infrared images is relatively low compared to that of visible images, and deficiencies occur such as blurred textural features. In this study, we proposed a super-resolution reconstruction method based on a deep convolutional neural network (CNN) for infrared images. The method improves the residual module, reduces the influence of the activation function on the information flow while deepening the network, and makes full use of the original information of low-resolution infrared images. Combined with an efficient channel attention mechanism and channel-space attention module, the reconstruction process selectively captures more feature information and facilitates a more accurate reconstruction of the high-frequency details of infrared images. The experimental results show that the peak signal-to-noise ratio (PSNR) of the infrared images reconstructed using this method outperforms those of the traditional Bicubic interpolation method, as well as the CNN-based SRResNet, EDSR, and RCAN models. When the scale factor is ×2 and ×4, the average PSNR values of the reconstructed images improved by 4.57 and 3.37 dB, respectively, compared with the traditional Bicubic interpolation method.
-
Key words:
- infrared images /
- super-resolution /
- convolutional neural network /
- attention mechanism
-
表 1 ERAB(包括IRB和ECA)和CSA的研究
Table 1. Investigations of ERAB (including IRB and ECA) and CSA
Base M1 M2 M3 M4 IRB √ √ √ √ CSA √ √ ECA √ √ PSNR/dB 42.44 42.78 42.97 42.86 43.06 SSIM 0.9543 0.9549 0.9552 0.9550 0.9554 表 2 典型图像实验结果对比
Table 2. Comparison of typical images experiment results
Images Scale Bicubic
(PSNR/dB)/SSIMSRResNet
(PSNR/dB)/SSIMEDSR
(PSNR/dB)/SSIMRCAN
(PSNR/dB)/SSIMOurs
(PSNR/dB)/SSIMTEST_1 ×2 38.0631/0.9710 43.0650/0.9868 43.1183/0.9869 43.1627/0.9870 43.3488/0.9875 TEST_2 30.8528/0.7505 32.1850/0.7875 32.1907/0.7878 32.1908/0.7880 32.2152/0.7881 TEST_3 ×4 31.8568/0.9091 35.4462/0.9498 35.3748/0.9493 35.7971/0.9516 36.0539/0.9538 TEST_4 28.6532/0.8888 35.3544/0.9551 35.3770/0.9548 35.5273/0.9559 36.1061/0.9586 表 3 测试图像集超分辨率重建结果比较
Table 3. Comparison of super-resolution reconstruction results of test image set
Methods (PSNR/dB)/SSIM(×2) (PSNR/dB)/SSIM(×4) Bicubic 38.4937/0.9395 32.3882/0.8699 SRResNet 41.6272/0.9220 34.8284/0.8999 EDSR 41.6705/0.9221 35.4601/0.9035 RCAN 42.5441/0.9546 35.5871/0.9042 OURS 43.0616/0.9554 35.7599/0.9053 -
[1] Baker S, Kanade T. Limits on super resolution and how to break them[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2002, 24(9): 1167-1183. http://www.onacademic.com/detail/journal_1000035551419310_c5fb.html [2] Hou H, Andrews H. Cubic splines for image interpolation and digital filtering[J]. IEEE Transactions on acoustics, speech, and signal processing, 1978, 26(6): 508-517. doi: 10.1109/TASSP.1978.1163154 [3] Freeman W T, Pasztor E C, Carmichael O T. Learning low-level vision[J]. International Journal of Computer Vision, 2000, 40(1): 25-47. doi: 10.1023/A:1026501619075 [4] YANG J, Wright J, HUANG T S, et al. Image super-resolution via sparse representation[J]. IEEE transactions on image processing, 2010, 19(11): 2861-2873. doi: 10.1109/TIP.2010.2050625 [5] DONG C, Loy C C, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Trans Pattern Anal Mach Intell, 2016, 38(2): 295-307. doi: 10.1109/TPAMI.2015.2439281 [6] DONG C, Loy C C, TANG X. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the European conference on computer vision (ECCV), 2016: 391-407. [7] SHI W, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016: 1874-1883. [8] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778. [9] Ledig C, Theis L, Huszár F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4681-4690. [10] Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017: 136-144. [11] ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 286-301. [12] DAI T, CAI J, ZHANG Y, et al. Second-order attention network for single image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 11065-11074. [13] NIU B, WEN W, REN W, et al. Single image super-resolution via a holistic attention network[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2020: 191-207. [14] Choi Y, Kim N, Hwang S, et al. Thermal image enhancement using convolutional neural network[C]//2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2016: 223-230. [15] HE Z, TANG S, YANG J, et al. Cascaded deep networks with multiple receptive fields for infrared image super-resolution[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 29(8): 2310-2322. http://www.xueshufan.com/publication/2886209123 [16] ZOU Y, ZHANG L, LIU C, et al. Super-resolution reconstruction of infrared images based on a convolutional neural network with skip connections[J]. Optics and Lasers in Engineering, 2021, 146: 106717. doi: 10.1016/j.optlaseng.2021.106717 [17] YU J, FAN Y, YANG J, et al. Wide activation for efficient and accurate image super-resolution[J/OL]. arXiv preprint arXiv: 1808.08718, 2018. [18] WANG Q, WU B, ZHU P, et al. ECA-Net: Efficient channel attention for deep convolutional neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 11534-11542.