面向真实场景的单帧红外图像超分辨率重建

师奕峰; 陈楠; 朱芳; 毛文彪; 李发明; 王添福; 张济清; 姚立斌

面向真实场景的单帧红外图像超分辨率重建

昆明物理研究所, 云南昆明 650223

详细信息

作者简介:
师奕峰（1998-），男，硕士研究生，主要从事图像处理方面的研究

通讯作者:
陈楠（1985-），男，博士，正高级工程师，博士生导师，主要从事混合信号集成电路设计方面的研究。E-mail：chennan_kip@163.com

张济清（1987-），男，博士，高级工程师，硕士生导师，主要从事混合信号集成电路设计方面的研究。E-mail：jiqingzhang@163.com

中图分类号: TP391
计量
- 文章访问数: 35
- HTML全文浏览量: 17
- PDF下载量: 17
- 被引次数: 0
出版历程
- 收稿日期: 2023-12-06
- 修回日期: 2024-01-19
- 刊出日期: 2024-04-20

Single-frame Infrared Image Super-Resolution Reconstruction for Real Scenes

Kunming Institute of Physics, Kunming 650223, China

摘要

摘要: 现有的红外图像超分辨率重建方法主要依赖实验数据进行设计，但在面对真实环境中的复杂退化情况时，它们往往无法稳定地表现。针对这一挑战，本文提出了一种基于深度学习的新颖方法，专门针对真实场景下的红外图像超分辨率重建，构建了一个模拟真实场景下红外图像退化的模型，并提出了一个融合通道注意力与密集连接的网络结构。该结构旨在增强特征提取和图像重建能力，从而有效地提升真实场景下低分辨率红外图像的空间分辨率。通过一系列消融实验和与现有超分辨率方法的对比实验，本文方法展现了其在真实场景下红外图像处理中的有效性和优越性。实验结果显示，本文方法能够生成更锐利的边缘，并有效地消除噪声和模糊，从而显著提高图像的视觉质量。
- 红外图像 /
- 深度学习 /
- 超分辨 /
- 真实场景 /
- 退化模型
Abstract: Current infrared image super-resolution reconstruction methods, which are primarily designed based on experimental data, often fail in complex degradation scenarios encountered in real-world environments. To address this challenge, this paper presents a novel deep learning-based approach tailored for the super-resolution reconstruction of infrared images in real scenarios. The significant contributions of this research include the development of a model that simulates infrared image degradation in real-life settings and a network structure that integrates channel attention with dense connections. This structure enhances feature extraction and image reconstruction capabilities, effectively increasing the spatial resolution of low-resolution infrared images in realistic scenarios. The effectiveness and superiority of the proposed approach for processing infrared images in real-world contexts are demonstrated through a series of ablation studies and comparative experiments with existing super-resolution methods. The experimental results indicate that this method produces sharper edges and effectively eliminates noise and blur, thereby significantly improving the visual quality of the images.
- infrared image /
- deep learning /
- super-resolution /
- real scene /
- degradation model

HTML全文

图 1 本文提出的红外图像退化模型

Figure 1. The proposed infrared image degradation model

下载: 全尺寸图片幻灯片

图 2 红外图像超分辨率重建网络结构

Figure 2. Structure of infrared image super-resolution reconstruction network

下载: 全尺寸图片幻灯片

图 3 训练流程示意图

Figure 3. Schematic diagram of the training process

下载: 全尺寸图片幻灯片

图 4 本文方法与无退化模型变体的2×超分结果对比

Figure 4. Comparison of 2× super-resolution results between our method and the no degradation variant

下载: 全尺寸图片幻灯片

图 5 不同方法在场景1下2×倍超分结果对比

Figure 5. Comparison of 2× super-resolution results under scene 1 using different methods

下载: 全尺寸图片幻灯片

图 6 不同方法在场景2下2×倍超分结果对比

Figure 6. Comparison of 2× super-resolution results under scene 2 using different methods

下载: 全尺寸图片幻灯片

图 7 不同方法在场景3下4×倍超分结果对比

Figure 7. Comparison of 4× super-resolution results under scene 3 using different methods

下载: 全尺寸图片幻灯片

图 8 不同方法在场景4下4×倍超分结果对比

Figure 8. Comparison of 4× super-resolution results under scene 4 using different methods

下载: 全尺寸图片幻灯片

表 1 CADB模块中的密集连接结构参数

Table 1. Parameters of the densely connected structure in the CADB module

Layer type	Kernel size	Input channels	Output channels	Activation function
Conv1	3×3	64	16	PReLU
Conv2	3×3	80	16	PReLU
Conv3	3×3	96	16	PReLU
Conv4	3×3	112	16	PReLU
Conv5	3×3	128	64	-

下载: 导出CSV

表 2 CADB模块中的通道注意力结构参数

Table 2. Parameters of the channel attention structure in the CADB module

Layer type	Kernel size	Input channels	Output channels	Activation function
Conv1	3×3	64	16	GELU
Conv2	3×3	16	64	-
Pooling	1×1	64	64	-
Conv3	1×1	64	4	ReLU
Conv4	1×1	4	64	Sigmoid

下载: 导出CSV

表 3 重建模块参数

Table 3. Parameters of the reconstruction module

Layer type	Kernel size	Input channels	Output channels	Activation function
Conv1	3×3	64	64	LReLU
Conv2	3×3	64	32	LReLU
Conv3	3×3	32	16	LReLU
Conv4	3×3	16	1	-

下载: 导出CSV

表 4 不同超分倍数下本文方法与无退化模型变体的无参考图像质量评价指标比较

Table 4. Comparison of no-reference image quality assessment metrics between our method and the no degradation variant at different scaling scales

Scale	Methods	BRISQUE	NIQE	PI
2×	Ours-ND	37.84	6.494	6.892
2×	Ours	20.902	4.800	5.167
4×	Ours-ND	46.208	6.931	7.692
4×	Ours	28.480	5.628	5.384

下载: 导出CSV

表 5 不同超分倍数下本文方法与其他超分辨率方法在无参考图像质量评价指标上的比较

Table 5. Comparison of no-reference image quality assessment metrics between our method and other super-resolution methods at different scaling factors

Scale	Methods	BRISQUE	NIQE	PI
2×	SRCNN	35.298	6.375	6.800
	ESRGAN	26.559	5.139	6.206
	SwinIR	34.998	5.515	6.381
	Oz	39.161	6.483	6.954
	Zou	40.697	6.116	6.750
	Ours	20.902	4.800	5.167
4×	SRCNN	53.581	6.758	7.321
	ESRGAN	31.071	5.835	6.982
	SwinIR	55.269	6.577	7.225
	Oz	53.088	7.313	7.651
	Zou	63.166	8.162	8.023
	Ours	28.480	5.628	5.384

下载: 导出CSV

参考文献(27)

[1]	WANG Z, CHEN J, Hoi S C H. Deep learning for image super-resolution: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(10): 3365-3387.
[2]	LI J, PEI Z, ZENG T. From beginner to master: A survey for deep learning-based single-image super-resolution[J]. arXiv preprint arXiv: 2109.14335, 2021.
[3]	DONG C, LOY C C, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(2): 295-307.
[4]	SHI W, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1874-1883.
[5]	LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of The IEEE Conference on Computer Vision And Pattern Recognition Workshops, 2017: 136-144.
[6]	WANG X, YU K, WU S, et al. Esrgan: Enhanced super-resolution generative adversarial networks[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 63-79.
[7]	SUN C, LV J, LI J, et al. A rapid and accurate infrared image super-resolution method based on zoom mechanism[J]. Infrared Physics & Technology, 2018, 88: 228-238.
[8]	Suryanarayana G, TU E, YANG J. Infrared super-resolution imaging using multi-scale saliency and deep wavelet residuals[J]. Infrared Physics & Technology, 2019, 97: 177-186.
[9]	YAO T, LUO Y, HU J, et al. Infrared image super-resolution via discriminative dictionary and deep residual network[J]. Infrared Physics & Technology, 2020, 107: 103314.
[10]	Oz N, Sochen N, Markovich O, et al. Rapid super resolution for infrared imagery[J]. Optics Express, 2020, 28(18): 27196-27209. doi: 10.1364/OE.389926
[11]	ZOU Y, ZHANG L, LIU C, et al. Super-resolution reconstruction of infrared images based on a convolutional neural network with skip connections[J]. Optics and Lasers in Engineering, 2021, 146: 106717. doi: 10.1016/j.optlaseng.2021.106717
[12]	李方彪, 何昕, 魏仲慧, 等. 生成式对抗神经网络的多帧红外图像超分辨率重建[J]. 红外与激光工程, 2018, 47(2): 26-33. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ201802004.htm LI F, HE X, WEI Z, et al. Multiframe infrared image super-resolution reconstruction using generative adversarial networks[J]. Infrared and Laser Engineering, 2018, 47(2): 26-33. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ201802004.htm
[13]	魏子康, 刘云清. 改进的RDN灰度图像超分辨率重建方法[J]. 红外与激光工程, 2020, 49(S1): 20200173. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ2020S1022.htm WEI Z, LIU Y. Gray image super-resolution reconstruction based on improved RDN method[J]. Infrared and Laser Engineering, 2020, 49(S1): 20200173. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ2020S1022.htm
[14]	胡蕾, 王足根, 陈田, 等. 一种改进的SRGAN红外图像超分辨率重建算法[J]. 系统仿真学报, 2021, 33(9): 2109-2118. https://www.cnki.com.cn/Article/CJFDTOTAL-XTFZ202109013.htm HU L, WANG Z, CHEN T, et al. An improved SRGAN infrared image super-resolution reconstruction algorithm[J]. Journal of System Simulation, 2021, 33(9): 2109-2118. https://www.cnki.com.cn/Article/CJFDTOTAL-XTFZ202109013.htm
[15]	邱德粉, 江俊君, 胡星宇, 等. 高分辨率可见光图像引导红外图像超分辨率的Transformer网络[J]. 中国图象图形学报, 2023, 28(1): 196-206. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202301012.htm QIU D, JIANG J, HU X, et al. Guided transformer for high-resolution visible image guided infrared image super-resolution[J]. Journal of Image and Graphics, 2023, 28(1): 196-206. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202301012.htm
[16]	ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 286-301.
[17]	TONG T, LI G, LIU X, et al. Image super-resolution using dense skip connections[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 4799-4807.
[18]	ZHANG K, Liang J, Van Gool L, et al. Designing a practical degradation model for deep blind image super-resolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 4791-4800.
[19]	WANG X, XIE L, DONG C, et al. Real-esrgan: Training real-world blind super-resolution with pure synthetic data[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 1905-1914.
[20]	ZHANG W, SHI G, LIU Y, et al. A closer look at blind super-resolution: Degradation models, baselines, and performance upper bounds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 527-536.
[21]	LIANG J, CAO J, SUN G, et al. Swinir: Image restoration using swin transformer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 1833-1844.
[22]	Huynh-Thu Q, Ghanbari M. Scope of validity of PSNR in image/video quality assessment[J]. Electronics Letters, 2008, 44(13): 800-801. doi: 10.1049/el:20080522
[23]	Hanhart P, Korshunov P, Ebrahimi T. Benchmarking of quality metrics on ultra-high definition video sequences[C]//18th International Conference on Digital Signal Processing (DSP)of IEEE, 2013: 1-8.
[24]	Kundu D, Evans B L. Full-reference visual quality assessment for synthetic images: A subjective study[C]// IEEE International Conference on Image Processing (ICIP), 2015: 2374-2378.
[25]	Mittal A, Soundararajan R, Bovik A C. Making a "completely blind" image quality analyzer[J]. IEEE Signal Processing Letters, 2012, 20(3): 209-212.
[26]	Mittal A, Moorthy A K, Bovik A C. No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on Image Processing, 2012, 21(12): 4695-4708.
[27]	Blau Y, Mechrez R, Timofte R, et al. The 2018 PIRM challenge on perceptual image super-resolution[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 334-355.