Citation: | QU Haicheng, HU Qianqian, ZHANG Xuecong. Infrared and Visible Image Fusion Combining Information Perception and Multiscale Features[J]. Infrared Technology , 2023, 45(7): 685-695. |
Existing image fusion algorithms based on deep learning are unable to satisfy the demands of computational efficiency and fusion effect. Most have also adopted a fusion strategy based on a single-scale model, which cannot effectively extract the contextual information in images. This study proposes an end-to-end image fusion network based on information perception and multiscale features. The network consists of an encoder, a fusion strategy, and decoder. Specifically, the multiscale features of the infrared and visible images were extracted by the encoder, and a feature complementary enhancement module was designed to fuse different modal multiscale features. Finally, the lightweight decoder was designed to combine the low-level details and high-level semantic information. In addition, the information entropy of the source image was used to construct an information-sensing loss function to train the fusion network and generate the fused image with rich information. The proposed fusion framework was evaluated on the TNO and MSRS datasets. The results show that compared with existing fusion methods, the proposed network was superior to other methods in terms of both subjective visual description and objective index evaluation, with higher computational efficiency.
[1] |
白玉, 侯志强, 刘晓义, 等. 基于可见光图像和红外图像决策级融合的目标检测算法[J]. 空军工程大学学报: 自然科学版, 2020, 21(6): 53-59. https://www.cnki.com.cn/Article/CJFDTOTAL-KJGC202006009.htm
BAI Yu, HOU Zhiqiang, LIU Xiaoyi, et al. An object detection algorithm based on decision-level fusion of visible light image and infrared image[J]. Journal of Air Force Engineering University: Natural Science Edition, 2020, 21(6): 53-59. https://www.cnki.com.cn/Article/CJFDTOTAL-KJGC202006009.htm
|
[2] |
CAO Yanpeng, GUAN Dayan, HUANG Weilin, et al. Pedestrian detection with unsupervised multispectral feature learning using deep neural networks[J]. Information Fusion, 2019, 46: 206-217. DOI: 10.1016/j.inffus.2018.06.005
|
[3] |
段辉军, 王志刚, 王彦. 基于改进YOLO网络的双通道显著性目标识别算法[J]. 激光与红外, 2020, 50(11): 1370-1378. DOI: 10.3969/j.issn.1001-5078.2020.11.014
DUAN Huijun, WANG Zhigang, WANG Yan. Two-channel saliency object recognition algorithm based on improved YOLO network[J]. Laser & Infrared, 2020, 50(11): 1370-1378. DOI: 10.3969/j.issn.1001-5078.2020.11.014
|
[4] |
TANG Linfeng, YUAN Jiteng, MA Jiayi. Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network[J]. Information Fusion, 2022, 82: 28-42. DOI: 10.1016/j.inffus.2021.12.004
|
[5] |
CHEN Jun, LI Xuejiao, LUO Linbo, et al. Infrared and visible image fusion based on target-enhanced multiscale transform decomposition[J]. Information Sciences, 2020, 508: 64-78. DOI: 10.1016/j.ins.2019.08.066
|
[6] |
LIU Xingbin, MEI Wenbo, DU Huiqian. Structure tensor and nonsubsampled shearlet transform based algorithm for CT and MRI image fusion[J]. Neurocomputing, 2017, 235: 131-139. DOI: 10.1016/j.neucom.2017.01.006
|
[7] |
LIU Yipeng, JIN Jing, WANG Qiang, et al. Region level based multi-focus image fusion using quaternion wavelet and normalized cut[J]. Signal Processing, 2014, 97: 9-30. DOI: 10.1016/j.sigpro.2013.10.010
|
[8] |
ZHANG Qiong, Maldague X. An adaptive fusion approach for infrared and visible images based on NSCT and compressed sensing[J]. Infrared Physics & Technology, 2016, 74: 11-20.
|
[9] |
LI Hui, WU Xiaojun J, Kittler J. MDLatLRR: A novel decomposition method for infrared and visible image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4733-4746. DOI: 10.1109/TIP.2020.2975984
|
[10] |
LIU Yu, CHEN Xun, Ward R K, et al. Image fusion with convolutional sparse representation[J]. IEEE signal Processing Letters, 2016, 23(12): 1882-1886. DOI: 10.1109/LSP.2016.2618776
|
[11] |
FU Zhizhong, WANG Xue, XU Jin, et al. Infrared and visible images fusion based on RPCA and NSCT[J]. Infrared Physics & Technology, 2016, 77: 114-123.
|
[12] |
MOU Jiao, GAO Wei, SONG Zongxi. Image fusion based on non-negative matrix factorization and infrared feature extraction[C]//6th International Congress on Image and Signal Processing (CISP). IEEE, 2013, 2: 1046-1050.
|
[13] |
MA Jiayi, CHEN Chen, LI Chang, et al. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31: 100-109. DOI: 10.1016/j.inffus.2016.02.001
|
[14] |
LIU Yu, LIU Shuping, WANG Zengfu. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015, 24: 147-164. DOI: 10.1016/j.inffus.2014.09.004
|
[15] |
LI Hui, WU Xiaojun. DenseFuse: A fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623.
|
[16] |
MA Jiayi, YU Wei, LIANG Pengwei, et al. FusionGAN: A generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26. DOI: 10.1016/j.inffus.2018.09.004
|
[17] |
武圆圆, 王志社, 王君尧, 等. 红外与可见光图像注意力生成对抗融合方法研究[J]. 红外技术, 2022, 44(2): 170-178. http://hwjs.nvir.cn/article/id/7f2ae6e4-af9c-4929-a689-cb053b4dda85
WU Yuanyuan, WANG Zhishi, WANG Junyao, et al. Infrared and visible image fusion using attention-based generative adversarial networks[J]. Infrared Technology, 2022, 44(2): 170-178. http://hwjs.nvir.cn/article/id/7f2ae6e4-af9c-4929-a689-cb053b4dda85
|
[18] |
HOU Ruichao, ZHOU Dongming, NIE Rencan, et al. VIF-Net: an unsupervised framework for infrared and visible image fusion[J]. IEEE Transactions on Computational Imaging, 2020, 6: 640-651. DOI: 10.1109/TCI.2020.2965304
|
[19] |
TANG Linfeng, YUAN Jiteng, ZHANG Hao, et al. PIAFusion: A progressive infrared and visible image fusion network based on illumination aware[J]. Information Fusion, 2022, 83: 79-92.
|
[20] |
LI Hui, WU Xiaojun, Durrani T. NestFuse: An infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656.
|
[21] |
LI Hui, WU Xiaojun, Kittler J. RFN-Nest: An end-to-end residual fusion network for infrared and visible images[J]. Information Fusion, 2021, 73: 72-86.
|
[22] |
Toet Alexander. TNO Image Fusion Dataset[EB/OL]. [2022-08-20]. https://doi.org/10.6084/m9.figshare.1008029.v2.
|
[23] |
WANG Qilong, WU Banggu, ZHU Pengfei, et al. ECA-Net: efficient channel attention for deep convolutional neural networks[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 11531-11539.
|
[24] |
Kumar B S. Image fusion based on pixel significance using cross bilateral filter[J]. Signal Image Video Process, 2015, 9(5): 1193-1204.
|
[25] |
MA Jinlei, ZHOU Zhiqiang, WANG Bo, et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics & Technology, 2017, 82: 8-17.
|
1. |
白皓,白廷柱. 基于深度残差神经网络的红外图像超分辨率重构算法. 红外技术. 2024(02): 176-182 .
![]() | |
2. |
上官小雨,于跃波. 视觉传达的多帧平面图像纹理细节增强算法. 吉林大学学报(信息科学版). 2023(02): 359-366 .
![]() | |
3. |
赵利芬,常晓通. SAR图像增强的建筑基坑底面下沉风险监测仿真. 计算机仿真. 2023(09): 291-295 .
![]() | |
4. |
胡德敏,闵天悦. 改进型轻量级GAN的红外图像超分辨率方法. 小型微型计算机系统. 2022(08): 1711-1717 .
![]() | |
5. |
仲崇丽,刘华. 改进卷积神经网络的热红外成像人脸识别. 激光杂志. 2022(12): 117-121 .
![]() | |
6. |
马龙祥,杨浩,宋婷婷. 基于数据上采样的医学图像分割研究. 电子设计工程. 2021(05): 7-11 .
![]() | |
7. |
左岑,杨秀杰,张捷,王璇. 基于轻量级金字塔密集残差网络的红外图像超分辨增强. 红外技术. 2021(03): 251-257 .
![]() | |
8. |
李萍,刘以安,徐安林. 基于多尺度耦合的密集残差网络红外图像增强. 电子测量与仪器学报. 2021(07): 148-155 .
![]() | |
9. |
史永祥,蒋斌,黄雍晫,杨桂生,李庆武,张志良. 基于深度学习的红外图像超分辨率重建. 应用科技. 2020(04): 8-13 .
![]() |