Infrared and Visible Image Fusion Based on Transform Domain VGGNet19

LI Yongping; YANG Yanchun; DANG Jianwu; WANG Yangping

Volume 44 Issue 12

Dec. 2022

Turn off MathJax

Article Contents

Article Navigation > Infrared Technology > 2022 > 44(12): 1293-1300

LI Yongping, YANG Yanchun, DANG Jianwu, WANG Yangping. Infrared and Visible Image Fusion Based on Transform Domain VGGNet19[J]. Infrared Technology , 2022, 44(12): 1293-1300.

Citation:

LI Yongping, YANG Yanchun, DANG Jianwu, WANG Yangping. Infrared and Visible Image Fusion Based on Transform Domain VGGNet19[J]. Infrared Technology , 2022, 44(12): 1293-1300.

LI Yongping, YANG Yanchun, DANG Jianwu, WANG Yangping. Infrared and Visible Image Fusion Based on Transform Domain VGGNet19[J]. Infrared Technology , 2022, 44(12): 1293-1300.

Citation:

LI Yongping, YANG Yanchun, DANG Jianwu, WANG Yangping. Infrared and Visible Image Fusion Based on Transform Domain VGGNet19[J]. Infrared Technology , 2022, 44(12): 1293-1300.

PDF( 1034 KB)

Infrared and Visible Image Fusion Based on Transform Domain VGGNet19

School of Electronic and Information Engineering, Lanzhou Jiaotong University, Lanzhou 730070, China

Received Date: 2022-01-15
Rev Recd Date: 2022-02-28
Publish Date: 2022-12-20

Abstract

Abstract

To address the problems of loss of detailed information and blurred edges in the fusion of infrared and visible images, an infrared and visible image fusion method through the VGGNet19 network in the transform domain is proposed. Firstly, in order to extract more accurate basic and detailed data from the source images during the decomposition process, the source images are decomposed using a multi-scale guided filter with edge-preserving smoothing function into a base layer and multiple detailed layers. Then, the Laplacian energy with the characteristics of retaining the main energy information is used to fuse the basic layer to obtain the basic fusion map. Subsequently, to prevent the fusion result from losing some detailed edge information, the VGGNet19 network is used to extract the features of the detail layers, L1 regularization, upsampling and final weighted average, thus the fused detail. Finally, the final fusion is obtained by adding two fusion graphs. The experimental results show that the method proposed can better extract the edge and detailed information in the source images, and achieve better results in terms of both subjective and objective evaluation indicators.
- image fusion,
- guided filter,
- VGGNet19,
- Laplacian energy,
- infrared and visible image

FullText(HTML)

References(23)

References

[1]	MA Jiayi, MA Yong, LI Chang. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. doi: 10.1016/j.inffus.2018.02.004
[2]	叶坤涛, 李文, 舒蕾蕾, 等. 结合改进显著性检测与NSST的红外与可见光图像融合方法[J]. 红外技术, 2021, 43(12): 1212-1221. http://hwjs.nvir.cn/article/id/bfd9f932-e0bd-4669-b698-b02d42e31805 YE Kuntao, LI Wen, SHU Leilei, et al. Infrared and visible image fusion method based on improved saliency detection and non-subsampled Shearlet transform[J]. Infrared Technology, 2021, 43(12): 1212-1221. http://hwjs.nvir.cn/article/id/bfd9f932-e0bd-4669-b698-b02d42e31805
[3]	LI Shutao, KANG Xudong, FANG Leyuan, et al. Pixel-level image fusion: a survey of the state of the art[J]. Information Fusion, 2017, 33: 100-112. doi: 10.1016/j.inffus.2016.05.004
[4]	MA Cong, MIAO Zhenjiang, ZHANG Xiaoping, et al. A saliency prior context model for real-time object tracking[J]. IEEE Transactions on Multimedia, 2017, 19(11): 24152424.
[5]	HU Wenrui, YANG Yehui, ZHANG Wensheng, et al. Moving object detection using Tensor based low-rank and saliently fused-sparse decomposition[J]. IEEE Transactions on Image Processing, 2017, 26(2): 724-737. doi: 10.1109/TIP.2016.2627803
[6]	杨九章, 刘炜剑, 程阳. 基于对比度金字塔与双边滤波的非对称红外与可见光图像融合[J]. 红外技术, 2021, 43(9): 840-844. http://hwjs.nvir.cn/article/id/1c7de46d-f30d-48dc-8841-9e8bf3c91107 YANG Jiuzhang, LIU Weijian, CHENG Yang. Asymmetric infrared and visible image fusion based on contrast pyramid and bilateral filtering[J]. Infrared Technology, 2021, 43(9): 840-844. http://hwjs.nvir.cn/article/id/1c7de46d-f30d-48dc-8841-9e8bf3c91107
[7]	罗迪, 王从庆, 周勇军. 一种基于生成对抗网络与注意力机制的可见光和红外图像融合方法[J]. 红外技术, 2021, 43(6): 566-574. http://hwjs.nvir.cn/article/id/3403109e-d8d7-45ed-904f-eb4bc246275a LUO Di, WANG Congqing, ZHOU Yongjun. A visible and infrared image fusion method based on generative adversarial networks and attention mechanism[J]. Infrared Technology, 2021, 43(6): 566-574. http://hwjs.nvir.cn/article/id/3403109e-d8d7-45ed-904f-eb4bc246275a
[8]	AZARANG A, HAFEZ E, MANOOCHEHRI, et al. Convolutional autoencoder-based multispectral image fusion[J]. IEEE Access, 2019, 7: 35673-35683. doi: 10.1109/ACCESS.2019.2905511
[9]	HOU Ruichao, ZHOU Dongming, NIE Rencan, et al. VIF-net: an unsupervised framework for infrared and visible image fusion[J]. IEEE Transactions on Computational Imaging, 2020(6): 640-6521.
[10]	LIU Yu, CHEN Xun, HU Peng, et al. Multi-focus image fusion with a deep convolutional neural network[J]. Information Fusion, 2017, 36: 191-207. doi: 10.1016/j.inffus.2016.12.001
[11]	MA Jiayi, YU Wei, LIANG Pengwei, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26. doi: 10.1016/j.inffus.2018.09.004
[12]	唐丽丽, 刘刚, 肖刚. 基于双路级联对抗机制的红外与可见光图像融合方法[J]. 光子学报, 2021, 50(9): 0910004. https://www.cnki.com.cn/Article/CJFDTOTAL-GZXB202109035.htm TANG Lili, LIU Gang, XIAO Gang. Infrared and visible image fusion method based on dual-path cascade adversarial mechanism[J]. Acta Photonica Sinica, 2021, 50(9): 0910004. https://www.cnki.com.cn/Article/CJFDTOTAL-GZXB202109035.htm
[13]	ZHANG Yu, LIU Yu, SUN Peng, IFCNN: a general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118. doi: 10.1016/j.inffus.2019.07.011
[14]	郝永平, 曹昭睿, 白帆, 等. 基于兴趣区域掩码卷积神经网络的红外-可见光图像融合与目标识别算法研究[J]. 光子学报, 2021, 50(2): 0210002. https://www.cnki.com.cn/Article/CJFDTOTAL-GZXB202102010.htm HAO Yongping, CAO Zhaorui, BAI Fan, et al. Research on infrared visible image fusion and target recognition algorithm based on region of interest mask convolution neural network[J]. Acta Photonica Sinica, 2021, 50(2): 0210002. https://www.cnki.com.cn/Article/CJFDTOTAL-GZXB202102010.htm
[15]	刘佳, 李登峰. 马氏距离与引导滤波加权的红外与可见光图像融合[J]. 红外技术, 2021, 43(2): 162-169. http://hwjs.nvir.cn/article/id/56484763-c7b0-4273-a087-8d672e8aba9a LIU Jia, LI Dengfeng. Infrared and visible light image fusion based on Mahalanobis distance and guided filter weighting[J]. Infrared Technology, 2021, 43(2): 162-169. http://hwjs.nvir.cn/article/id/56484763-c7b0-4273-a087-8d672e8aba9a
[16]	LI Hui, WU Xiaojun, KITTLER J. Infrared and visible image fusion using a deep learning framework[C]// 24th International Conference on Pattern Recognition of IEEE, 2018: 8546006-1.
[17]	LIU Yu, CHEN Xun, WARD R K, et al. Image fusion with convolutional sparse representation[J]. IEEE Signal Processing Letters, 2016, 23(12): 1882-1886. https://ieeexplore.ieee.org/document/7593316/
[18]	LIU C H, QI Y, DING W R. Infrared and visible image fusion method based on saliency detection in sparse domain[J]. Infrared Physics & Technology, 2017, 83: 94-102. https://www.sciencedirect.com/science/article/pii/S1350449516307150
[19]	MA Jinlei, ZHOU Zhiqian, WANG Bo, et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics & Technology, 2017, 82: 8-17. https://www.sciencedirect.com/science/article/pii/S1350449516305928
[20]	MA Jiayi, ZHOU Yi. Infrared and visible image fusion via gradientlet filter[J]. Computer Vision and Image Understanding, 2020(197-198): 103016.
[21]	QU Xiaobo, YAN Jingwen, XIAO Hongzhi, et al. Image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled contourlet transform domain[J]. Acta Automatica Sinica, 2008, 34(12): 1508-1514. https://www.sciencedirect.com/science/article/pii/S1874102908601743
[22]	LIU Yu, CHEN Xun, CHENG Juan, et al. Infrared and visible image fusion with convolutional neural networks[J]. International Journal of Wavelets, Multiresolution and Information Processing, 2018, 16(3): 1850018. doi: 10.1142/S0219691318500182
[23]	HAGHIGHAT M, RAZIAN M A. Fast-FMI: non-reference image fusion metric[C]//International Conference on Application of Information and Communication Technologies(AICT), 2014: 1-3.