Advanced Search
中文
Volume 45 Issue 9
Sep.  2023
Turn off MathJax
Article Contents
Abstract
References
Related Articles
SHEN Yu, LIANG Li, WANG Hailong, YAN Yuan, LIU Guanghui, SONG Jing. Infrared and Visible Image Fusion Based on N-RGAN Model[J]. Infrared Technology , 2023, 45(9): 897-906.
Citation: SHEN Yu, LIANG Li, WANG Hailong, YAN Yuan, LIU Guanghui, SONG Jing. Infrared and Visible Image Fusion Based on N-RGAN Model[J]. Infrared Technology , 2023, 45(9): 897-906.
shu

Infrared and Visible Image Fusion Based on N-RGAN Model

  • School of Electronic and Information Engineering, Lanzhou Jiaotong University, Lanzhou 730070, China

More Information
  • Received Date: June 02, 2022
  • Revised Date: July 19, 2022
    Graphical Abstract
    • Abstract
  • At present, infrared and visible image fusion algorithms still have problems such as low applicability to complex scenes, large loss of detail and texture information in fusion images, and low contrast and sharpness of fusion images. In view of the above problems, this study proposes an N-RGAN model that combines a non-subsampled shearlet transform (NSST) and a residual network (ResNet). Infrared and visible images are decomposed into high- and low-frequency sub-bands using NSST. The high-frequency sub-bands are spliced and input into the generator improved by the residual module, and the source infrared image is taken as the decision standard to improve network fusion performance, fusion image detail description, and target-highlighting ability. The salient features of infrared and visible images are extracted, and the low-frequency sub-bands are fused by adaptive weighting to improve image contrast and sharpness. The fusion results of the high- and low-frequency sub-bands are obtained by the NSST inverse transformation. Based on a comparison of various fusion algorithms, the proposed method improves peak signal-to-noise ratio (PSNR), average gradient (AVG), image entropy (IE), spatial frequency (SF), edge strength (ES), and image clarity (IC), thereby improving infrared and visible light image fusion effects in complex scenes, alleviating information loss in image detail texture, and enhancing image contrast and resolution.
    • FullText(HTML)
    • References (28)
  • [1]
    HUI L, XIAO J W. Dense Fuse: a fusion approach to infrared and visible images[J]. IEEE Trans. Image Processing, 2019, 28(5): 2614-2623. DOI: 10.1109/TIP.2018.2887342
    [2]
    HE G Q, JI J Q, DONG D D, et al. Infrared and visible image fusion method by using hybrid representation learning[J]. IEEE Geosci. Remote Sensing Lett, 2019, 16(11): 1796-1800. DOI: 10.1109/LGRS.2019.2907721
    [3]
    沈英, 黄春红, 黄峰, 等. 红外与可见光图像融合技术的研究进展[J]. 红外与激光工程, 2021, 50(9): 152-169. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ202109014.htm

    SHEN Y, HUANG C H, HUANG F, et al. Research progress of infrared and visible image fusion technology[J]. Infrared and Laser Engineering, 2021, 50(9): 152-169. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ202109014.htm
    [4]
    廖宁, 陈怡然. 红外和可见光图像高效融合的人工智能技术研究[J]. 激光杂志, 2022, 43(3): 109-113. DOI: 10.14016/j.cnki.jgzz.2022.03.109

    LIAO N, CHEN Y R. Research on artificial intelligence technology for efficient fusion of infrared and visible images[J]. Laser Magazine, 2022, 43(3): 109-113. DOI: 10.14016/j.cnki.jgzz.2022.03.109
    [5]
    黄颖杰, 梅领亮, 王勇, 等. 基于红外与可见光图像融合的无人机探测研究[J]. 电脑知识与技术, 2022, 18(7): 1-8. https://www.cnki.com.cn/Article/CJFDTOTAL-DNZS202207001.htm

    HUANG Y J, MEI L L, WANG Y, et al. Research on UAV detection based on infrared and visible image fusion[J]. Computer Knowledge and Technology, 2022, 18(7): 1-8. https://www.cnki.com.cn/Article/CJFDTOTAL-DNZS202207001.htm
    [6]
    沈瑜, 陈小朋. 基于DLatLRR与VGG Net的红外与可见光图像融合[J]. 北京航空航天大学学报, 2021, 47(6): 1105-1114. https://www.cnki.com.cn/Article/CJFDTOTAL-BJHK202106003.htm

    SHEN Y, CHEN X P. Infrared and visible image fusion based on DLatLRR and VGG Net[J]. Journal of Beijing University of Aeronautics and Astronautics, 2021, 47(6): 1105-1114. https://www.cnki.com.cn/Article/CJFDTOTAL-BJHK202106003.htm
    [7]
    LIU C H, QI Y, DING W R. Infrared and visible image fusion method based on saliency detection in sparse domain[J]. Infrared Physics & Technology, 2017, 83: 94-102.
    [8]
    GUO L, YANG B. Fusion of infrared and visible images based on visual saliency[J]. Computer Science, 2015, 42(6): 211-235.
    [9]
    胡雪凯, 罗蓬, 李铁成, 等. 基于自适应加权的多尺度图像融合研究[J]. 红外技术, 2022, 44(4): 404-409. http://hwjs.nvir.cn/article/id/514624de-3b87-466d-9683-f374c5b565b8

    HU X K, LUO P, LI T C, et al. Multi-scale image fusion based on adaptive weighting[J]. Infrared Technology, 2022, 44(4): 404-409. http://hwjs.nvir.cn/article/id/514624de-3b87-466d-9683-f374c5b565b8
    [10]
    陈国洋, 吴小俊, 徐天阳. 基于深度学习的无监督红外图像与可见光图像融合算法[J]. 激光与光电子学进展, 2022, 59(4): 151-160. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202204014.htm

    CHEN G X, WU X J, XU T Y. Unsupervised infrared image and visible image fusion algorithm based on deep learning[J]. Progress in Laser and Optoelectronics, 2022, 59(4): 151-160. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202204014.htm
    [11]
    李春艳, 孙韬, 谢俊峰. EMF深度学习可见光/红外图像融合算法[J]. 国外电子测量技术, 2020, 39(10): 25-32. https://www.cnki.com.cn/Article/CJFDTOTAL-GWCL202010008.htm

    LI C Y, SUN T, XIE J F. EMF deep learning visible/infrared image fusion algorithm[J]. Foreign Electronic Measurement Technology, 2020, 39(10): 25-32. https://www.cnki.com.cn/Article/CJFDTOTAL-GWCL202010008.htm
    [12]
    杨雪, 郑婷婷, 戴阳. 基于孪生卷积神经网络的图像融合[J]. 计算机系统应用, 2020, 29(5): 196-201. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY202005027.htm

    YANG X, ZHEN T T, DAI Y. Image fusion based on twin convolutional neural network[J]. Computer System Application, 2020, 29(5): 196-201. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY202005027.htm
    [13]
    沈瑜, 陈小朋, 杨倩. 多方向Laplacian能量和与tetrolet变换的图像融合[J]. 中国图象图形学报, 2020, 25(4): 721-731. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202004008.htm

    SHEN Y, CHEN X P, YANG Q. Image fusion based on multi-direction Laplacian energy and Tetrolet transform[J]. Journal of Image and Graphics, 2020, 25(4): 721-731. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB202004008.htm
    [14]
    郝帅, 安倍逸, 付周兴, 等. 基于小波变换和各向异性扩散的红外和可见光图像融合算法[J]. 西安科技大学学报, 2022, 42(1): 184-190. https://www.cnki.com.cn/Article/CJFDTOTAL-XKXB202201024.htm

    HAO S, AN B Y, FU Z X, et al. Infrared and visible image fusion algorithm based on wavelet transform and anisotropic diffusion[J]. Journal of Xi 'an University of Science and Technology, 2022, 42(1): 184-190. https://www.cnki.com.cn/Article/CJFDTOTAL-XKXB202201024.htm
    [15]
    姜寒雪, 郭立强. 一种基于NSCT和对比度拉伸的红外与可见光图像融合算法[J]. 淮阴师范学院学报: 自然科学版, 2022, 21(1): 17-23. https://www.cnki.com.cn/Article/CJFDTOTAL-HYSK202201003.htm

    JIANG H X, GUO L Q. An algorithm for infrared and visible image fusion based on NSCT and contrast stretching[J]. Journal of Huaiyin Normal University: Natural Science Edition, 2022, 21(1): 17-23. https://www.cnki.com.cn/Article/CJFDTOTAL-HYSK202201003.htm
    [16]
    李威, 李忠民. NSST域红外和可见光图像感知融合[J]. 激光与光电子学进展, 2021, 58(20): 202-210. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202120021.htm

    LI W, LI Z M. Perception fusion of infrared and visible images in NSST domain[J]. Progress in Laser and Optoelectronics, 2021, 58(20): 202-210. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202120021.htm
    [17]
    沈瑜, 陈小朋, 苑玉彬, 等. 基于显著矩阵与神经网络的红外与可见光图像融合[J]. 激光与光电子学进展, 2020, 57(20): 76-86. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202020009.htm

    SHEN Y, CHEN X P, YUAN Y B, et al. Infrared and visible image fusion based on saliency matrix and neural network[J]. Progress in Laser and Optoelectronics, 2020, 57(20): 76-86. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202020009.htm
    [18]
    刘栋, 聂仁灿, 周冬明, 等. 结合NSST与GA参数优化PCNN图像融合[J]. 计算机工程与应用, 2018, 54(19): 158-163, 171. https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG201819025.htm

    LIU D, NIE R C, ZHOU D M, et al. Optimization of PCNN image fusion based on NSST and GA parameters[J]. Computer Engineering and Applications, 2018, 54(19): 158-163, 171. https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG201819025.htm
    [19]
    魏亚南, 曲怀敬, 王纪委, 等. 基于NSCT和卷积稀疏表示的红外与可见光图像融合[J]. 计算机与数字工程, 2022, 50(2): 276-283. https://www.cnki.com.cn/Article/CJFDTOTAL-JSSG202202011.htm

    WEI Y N, QU H J, WANG J W, et al. Infrared and visible image fusion based on NSCT and convolution sparse representation[J]. Computer and Digital Engineering, 2022, 50(2): 276-283. https://www.cnki.com.cn/Article/CJFDTOTAL-JSSG202202011.htm
    [20]
    陈卓, 方明, 柴旭, 等. 红外与可见光图像融合的U-GAN模型[J]. 西北工业大学学报, 2020, 38(4): 904-912. https://www.cnki.com.cn/Article/CJFDTOTAL-XBGD202004027.htm

    CHENG Z, FANG M, CHAI X, et al. U-GAN Model for infrared and visible image fusion[J]. Journal of Northwestern Polytechnical University, 2020, 38(4): 904-912. https://www.cnki.com.cn/Article/CJFDTOTAL-XBGD202004027.htm
    [21]
    YANG Z, ZENG S. TPFusion: Texture preserving fusion of infrared and visible images via dense networks[J]. Entropy, 2022, 24(2): 294.
    [22]
    JING L, HUO H T, CHANG L, et al. AttentionFGAN: infrared and visible image fusion using attention-based generative adversarial networks[J]. IEEE Transactions on Multimedia, 2020, 23: 1383-1396.
    [23]
    杨艳春, 高晓宇, 党建武, 等. 基于NSST与IFCNN的红外可见光图像融合算法[J]. 激光与光电子学进展, 2021, 58(20): 118-126. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202120012.htm

    YANG Y C, GAO X Y, DANG J W, et al. Infrared visible image Fusion algorithm based on NSST and IFCNN[J]. Progress in Laser and Optoelectronics, 2021, 58(20): 118-126. https://www.cnki.com.cn/Article/CJFDTOTAL-JGDJ202120012.htm
    [24]
    汪美琴, 袁伟伟, 张继业. 生成对抗网络GAN的研究综述[J]. 计算机工程与设计, 2021, 42(12): 3389-3395. https://www.cnki.com.cn/Article/CJFDTOTAL-SJSJ202112012.htm

    WANG M Q, YUAN W W, ZHANG J Y. A review of generative adversarial networks[J]. Computer Engineering and Design, 2021, 42(12): 3389-3395. https://www.cnki.com.cn/Article/CJFDTOTAL-SJSJ202112012.htm
    [25]
    MA J Y, XU H, JIANG J J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2020, 29: 4980-4995.
    [26]
    李国梁, 向文豪, 张顺利, 等. 基于残差网络和注意力机制的红外与可见光图像融合算法[J]. 无人系统技术, 2022, 5(2): 9-21. https://www.cnki.com.cn/Article/CJFDTOTAL-UMST202202002.htm

    LI G L, XIANG W H, ZHANG S L, et al. Infrared and visible image fusion algorithm based on residual network and attentional mechanism[J]. Unmanned System Technology, 2022, 5(2): 9-21. https://www.cnki.com.cn/Article/CJFDTOTAL-UMST202202002.htm
    [27]
    YU L, LIU S, WANG Z. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015, 24: 147-164.
    [28]
    YU L, XUN C, Ward R K, et al. Image fusion with convolutional sparse representation[J]. IEEE Signal Processing Letters, 2016, 23(12): 1882-1886.
    • Related Articles

  • Related Articles

    [1]LIAO Guangfeng, GUAN Zhiwei, CHEN Qiang. An Improved Dual Discriminator Generative Adversarial Network Algorithm for Infrared and Visible Image Fusion[J]. Infrared Technology , 2025, 47(3): 367-375.
    [2]YUAN Hongchun, ZHANG Bo, CHENG Xin. Underwater Image Enhancement Algorithm Combining Transformer and Generative Adversarial Network[J]. Infrared Technology , 2024, 46(9): 975-983.
    [3]XING Yanchao, NIU Zhenhua. Infrared and Visible Image Fusion Based on Global Energy Features and Improved PCNN[J]. Infrared Technology , 2024, 46(8): 902-911.
    [4]LI Li, YI Shi, LIU Xi, CHENG Xinghao, WANG Cheng. Infrared Image Deblurring Based on Dense Residual Generation Adversarial Network[J]. Infrared Technology , 2024, 46(6): 663-671.
    [5]CHONG Fating, DONG Zhangyu, YANG Xuezhi, ZENG Qingwang. SAR and Multispectral Image Fusion Based on Dual-channel Multi-scale Feature Extraction and Attention[J]. Infrared Technology , 2024, 46(1): 61-73.
    [6]DONG Yifei, WANG Xiaojie, WANG Renshu, XU Jun, SHU Shengwen, TAO Yiqing. Thermal Defect Detection of Composite Insulator Based on One-dimensional Residual Network[J]. Infrared Technology , 2023, 45(6): 663-670.
    [7]WANG Xiaona, PAN Qing, TIAN Nili. Multi-modality Image Fusion Algorithm Based on NSST-DWT-ICSAPCNN[J]. Infrared Technology , 2022, 44(5): 497-503.
    [8]WU Yuanyuan, WANG Zhishe, WANG Junyao, SHAO Wenyu, CHEN Yanlin. Infrared and Visible Image Fusion Using Attention- Based Generative Adversarial Networks[J]. Infrared Technology , 2022, 44(2): 170-178.
    [9]HUANG Mengtao, GAO Na, LIU Bao. Image Deblurring Method Based on a Dual-Discriminator Weighted Generative Adversarial Network[J]. Infrared Technology , 2022, 44(1): 41-46.
    [10]LUO Di, WANG Congqing, ZHOU Yongjun. A Visible and Infrared Image Fusion Method based on Generative Adversarial Networks and Attention Mechanism[J]. Infrared Technology , 2021, 43(6): 566-574.

Catalog

    Get Citation
    PDF
    XML
    Article views (244) PDF downloads (73) Cited by()
    Related

    Copyright © 《Infrared Technology》Editorial Office京ICP备000000000号-1

    Supported by: Beijing Renhe Information Technology Co., Ltd.  Baidu Analytics

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return