Single-frame Infrared Image Super-Resolution Reconstruction for Real Scenes
-
摘要: 现有的红外图像超分辨率重建方法主要依赖实验数据进行设计,但在面对真实环境中的复杂退化情况时,它们往往无法稳定地表现。针对这一挑战,本文提出了一种基于深度学习的新颖方法,专门针对真实场景下的红外图像超分辨率重建,构建了一个模拟真实场景下红外图像退化的模型,并提出了一个融合通道注意力与密集连接的网络结构。该结构旨在增强特征提取和图像重建能力,从而有效地提升真实场景下低分辨率红外图像的空间分辨率。通过一系列消融实验和与现有超分辨率方法的对比实验,本文方法展现了其在真实场景下红外图像处理中的有效性和优越性。实验结果显示,本文方法能够生成更锐利的边缘,并有效地消除噪声和模糊,从而显著提高图像的视觉质量。Abstract: Current infrared image super-resolution reconstruction methods, which are primarily designed based on experimental data, often fail in complex degradation scenarios encountered in real-world environments. To address this challenge, this paper presents a novel deep learning-based approach tailored for the super-resolution reconstruction of infrared images in real scenarios. The significant contributions of this research include the development of a model that simulates infrared image degradation in real-life settings and a network structure that integrates channel attention with dense connections. This structure enhances feature extraction and image reconstruction capabilities, effectively increasing the spatial resolution of low-resolution infrared images in realistic scenarios. The effectiveness and superiority of the proposed approach for processing infrared images in real-world contexts are demonstrated through a series of ablation studies and comparative experiments with existing super-resolution methods. The experimental results indicate that this method produces sharper edges and effectively eliminates noise and blur, thereby significantly improving the visual quality of the images.
-
Key words:
- infrared image /
- deep learning /
- super-resolution /
- real scene /
- degradation model
-
表 1 CADB模块中的密集连接结构参数
Table 1. Parameters of the densely connected structure in the CADB module
Layer type Kernel size Input channels Output channels Activation function Conv1 3×3 64 16 PReLU Conv2 3×3 80 16 PReLU Conv3 3×3 96 16 PReLU Conv4 3×3 112 16 PReLU Conv5 3×3 128 64 - 表 2 CADB模块中的通道注意力结构参数
Table 2. Parameters of the channel attention structure in the CADB module
Layer type Kernel size Input channels Output channels Activation function Conv1 3×3 64 16 GELU Conv2 3×3 16 64 - Pooling 1×1 64 64 - Conv3 1×1 64 4 ReLU Conv4 1×1 4 64 Sigmoid 表 3 重建模块参数
Table 3. Parameters of the reconstruction module
Layer type Kernel size Input channels Output channels Activation function Conv1 3×3 64 64 LReLU Conv2 3×3 64 32 LReLU Conv3 3×3 32 16 LReLU Conv4 3×3 16 1 - 表 4 不同超分倍数下本文方法与无退化模型变体的无参考图像质量评价指标比较
Table 4. Comparison of no-reference image quality assessment metrics between our method and the no degradation variant at different scaling scales
Scale Methods BRISQUE NIQE PI 2× Ours-ND 37.84 6.494 6.892 Ours 20.902 4.800 5.167 4× Ours-ND 46.208 6.931 7.692 Ours 28.480 5.628 5.384 表 5 不同超分倍数下本文方法与其他超分辨率方法在无参考图像质量评价指标上的比较
Table 5. Comparison of no-reference image quality assessment metrics between our method and other super-resolution methods at different scaling factors
Scale Methods BRISQUE NIQE PI 2× SRCNN 35.298 6.375 6.800 ESRGAN 26.559 5.139 6.206 SwinIR 34.998 5.515 6.381 Oz 39.161 6.483 6.954 Zou 40.697 6.116 6.750 Ours 20.902 4.800 5.167 4× SRCNN 53.581 6.758 7.321 ESRGAN 31.071 5.835 6.982 SwinIR 55.269 6.577 7.225 Oz 53.088 7.313 7.651 Zou 63.166 8.162 8.023 Ours 28.480 5.628 5.384 -
[1] WANG Z, CHEN J, Hoi S C H. Deep learning for image super-resolution: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(10): 3365-3387. [2] LI J, PEI Z, ZENG T. From beginner to master: A survey for deep learning-based single-image super-resolution[J]. arXiv preprint arXiv: 2109.14335, 2021. [3] DONG C, LOY C C, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(2): 295-307. [4] SHI W, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1874-1883. [5] LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of The IEEE Conference on Computer Vision And Pattern Recognition Workshops, 2017: 136-144. [6] WANG X, YU K, WU S, et al. Esrgan: Enhanced super-resolution generative adversarial networks[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 63-79. [7] SUN C, LV J, LI J, et al. A rapid and accurate infrared image super-resolution method based on zoom mechanism[J]. Infrared Physics & Technology, 2018, 88: 228-238. [8] Suryanarayana G, TU E, YANG J. Infrared super-resolution imaging using multi-scale saliency and deep wavelet residuals[J]. Infrared Physics & Technology, 2019, 97: 177-186. [9] YAO T, LUO Y, HU J, et al. Infrared image super-resolution via discriminative dictionary and deep residual network[J]. Infrared Physics & Technology, 2020, 107: 103314. [10] Oz N, Sochen N, Markovich O, et al. Rapid super resolution for infrared imagery[J]. Optics Express, 2020, 28(18): 27196-27209. doi: 10.1364/OE.389926 [11] ZOU Y, ZHANG L, LIU C, et al. Super-resolution reconstruction of infrared images based on a convolutional neural network with skip connections[J]. Optics and Lasers in Engineering, 2021, 146: 106717. doi: 10.1016/j.optlaseng.2021.106717 [12] 李方彪, 何昕, 魏仲慧, 等. 生成式对抗神经网络的多帧红外图像超分辨率重建[J]. 红外与激光工程, 2018, 47(2): 26-33. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ201802004.htmLI F, HE X, WEI Z, et al. Multiframe infrared image super-resolution reconstruction using generative adversarial networks[J]. Infrared and Laser Engineering, 2018, 47(2): 26-33. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ201802004.htm [13] 魏子康, 刘云清. 改进的RDN灰度图像超分辨率重建方法[J]. 红外与激光工程, 2020, 49(S1): 20200173. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ2020S1022.htmWEI Z, LIU Y. Gray image super-resolution reconstruction based on improved RDN method[J]. Infrared and Laser Engineering, 2020, 49(S1): 20200173. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ2020S1022.htm [14] 胡蕾, 王足根, 陈田, 等. 一种改进的SRGAN红外图像超分辨率重建算法[J]. 系统仿真学报, 2021, 33(9): 2109-2118. https://www.cnki.com.cn/Article/CJFDTOTAL-XTFZ202109013.htmHU L, WANG Z, CHEN T, et al. An improved SRGAN infrared image super-resolution reconstruction algorithm[J]. Journal of System Simulation, 2021, 33(9): 2109-2118. https://www.cnki.com.cn/Article/CJFDTOTAL-XTFZ202109013.htm [15] 邱德粉, 江俊君, 胡星宇, 等. 高分辨率可见光图像引导红外图像超分辨率的Transformer网络[J]. 中国图象图形学报, 2023, 28(1): 196-206. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202301012.htmQIU D, JIANG J, HU X, et al. Guided transformer for high-resolution visible image guided infrared image super-resolution[J]. Journal of Image and Graphics, 2023, 28(1): 196-206. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202301012.htm [16] ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 286-301. [17] TONG T, LI G, LIU X, et al. Image super-resolution using dense skip connections[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 4799-4807. [18] ZHANG K, Liang J, Van Gool L, et al. Designing a practical degradation model for deep blind image super-resolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 4791-4800. [19] WANG X, XIE L, DONG C, et al. Real-esrgan: Training real-world blind super-resolution with pure synthetic data[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 1905-1914. [20] ZHANG W, SHI G, LIU Y, et al. A closer look at blind super-resolution: Degradation models, baselines, and performance upper bounds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 527-536. [21] LIANG J, CAO J, SUN G, et al. Swinir: Image restoration using swin transformer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 1833-1844. [22] Huynh-Thu Q, Ghanbari M. Scope of validity of PSNR in image/video quality assessment[J]. Electronics Letters, 2008, 44(13): 800-801. doi: 10.1049/el:20080522 [23] Hanhart P, Korshunov P, Ebrahimi T. Benchmarking of quality metrics on ultra-high definition video sequences[C]//18th International Conference on Digital Signal Processing (DSP)of IEEE, 2013: 1-8. [24] Kundu D, Evans B L. Full-reference visual quality assessment for synthetic images: A subjective study[C]// IEEE International Conference on Image Processing (ICIP), 2015: 2374-2378. [25] Mittal A, Soundararajan R, Bovik A C. Making a "completely blind" image quality analyzer[J]. IEEE Signal Processing Letters, 2012, 20(3): 209-212. [26] Mittal A, Moorthy A K, Bovik A C. No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on Image Processing, 2012, 21(12): 4695-4708. [27] Blau Y, Mechrez R, Timofte R, et al. The 2018 PIRM challenge on perceptual image super-resolution[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 334-355.