留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于潜在低秩表示的红外和可见光图像融合

孙彬 诸葛吴为 高云翔 王梓萱

孙彬, 诸葛吴为, 高云翔, 王梓萱. 基于潜在低秩表示的红外和可见光图像融合[J]. 红外技术, 2022, 44(8): 853-862.
引用本文: 孙彬, 诸葛吴为, 高云翔, 王梓萱. 基于潜在低秩表示的红外和可见光图像融合[J]. 红外技术, 2022, 44(8): 853-862.
SUN Bin, ZHUGE Wuwei, GAO Yunxiang, WANG Zixuan. Infrared and Visible Image Fusion Based on Latent Low-Rank Representation[J]. Infrared Technology , 2022, 44(8): 853-862.
Citation: SUN Bin, ZHUGE Wuwei, GAO Yunxiang, WANG Zixuan. Infrared and Visible Image Fusion Based on Latent Low-Rank Representation[J]. Infrared Technology , 2022, 44(8): 853-862.

基于潜在低秩表示的红外和可见光图像融合

基金项目: 

四川省科技计划资助 2020YFG0231

四川省中央引导地方科技发展专项 2020ZYD016

电子科技大学基于项目的研究生创新培养计划建设项目 XMZ20203-9

详细信息
    作者简介:

    孙彬(1984-),副教授,主要研究方向为信息融合、图像处理、导航定位。E-mail: sunbinhust@uestc.edu.cn

  • 中图分类号: TP391

Infrared and Visible Image Fusion Based on Latent Low-Rank Representation

  • 摘要: 红外和可见光图像融合广泛应用于目标跟踪、检测和识别等领域。为了保留细节的同时增强对比度,本文提出一种基于潜在低秩表示的红外和可见光图像融合方法。潜在低秩分解将源图像分解为基层和显著层,其中基层包含主要内容和结构信息,显著层包含能量相对集中的局部区域。进一步利用比例金字塔分解得到低频和高频的基层子带,并针对不同层的特点设计对应的融合规则。利用稀疏表示表达低频基层较分散的能量,设计L1范数最大和稀疏系数最大规则,加权平均融合策略保留不同的显著特征;绝对值最大增强高频基层的对比度信息;而显著层则利用局部方差度量局部显著性,加权平均方式突出对比度较强的目标区域。在TNO数据集上的定性和定量实验分析表明方法具有良好的融合性能。基于低秩分解的方法能够增强红外和可见光融合图像中目标对比度的同时保留了丰富的细节信息。
  • 图  1  LatLRR分解结果示例

    Figure  1.  Example of LatLRR decomposition results

    图  2  提出的算法框架图

    Figure  2.  The proposed algorithm framework

    图  3  各层分解示例

    Figure  3.  Example of each layer

    图  4  比率金字塔-稀疏表示融合算法示意

    Figure  4.  Schematic diagram of ratio pyramid-sparse representation algorithm

    图  5  不同融合规则融合结果

    Figure  5.  Base layer images of different methods

    图  6  使用的10对源图像

    Figure  6.  Ten pairs of source images

    图  7  9种方法在‘soldier behind smoke’上的结果

    Figure  7.  Results of the nine methods on 'soldier behind smoke'

    图  8  9种方法在“street”上的结果

    Figure  8.  Results of the nine methods on 'street'

    图  9  9种方法在“boat”上的结果

    Figure  9.  Results of the nine methods on 'boat'

    表  1  9种方法在‘soldier behind smoke’、‘street’和‘boat’上的客观指标

    Table  1.   Objective evaluation indexes of nine methods on'soldier behind smoke', 'street' and'boat'

    Source image Methods
    ADF CSR TE-MST FPDE GF LatLRR MSVD TIF OURS
    Soldier behind smoke EN 6.892 6.925 6.563 6.899 7.166 6.924 6.894 7.053 7.297
    AG 3.604 3.580 4.500 3.567 3.369 2.674 2.871 3.863 3.232
    CE 3.766 3.014 2.349 3.109 2.982 2.870 3.173 2.485 1.544
    SF 10.606 11.225 11.930 10.161 9.767 7.755 9.693 11.360 8.558
    SD 33.280 34.120 28.180 33.346 39.749 33.797 33.101 36.420 41.472
    EI 33.163 34.478 43.443 32.901 32.719 26.200 25.596 38.837 33.934
    Street EN 6.040 6.013 6.776 6.067 6.133 6.030 5.935 6.414 6.852
    AG 2.682 2.525 3.597 4.105 2.714 2.413 2.176 3.298 5.551
    CE 3.278 3.255 2.994 3.111 3.447 1.076 3.494 2.222 4.863
    SF 9.790 11.064 12.593 12.484 10.445 9.184 9.964 11.612 18.467
    SD 21.981 23.207 35.386 22.172 31.453 27.485 21.149 30.346 42.875
    EI 27.827 26.632 38.053 41.107 28.833 25.682 22.388 35.128 56.952
    Boat EN 5.005 4.965 6.436 4.984 5.362 5.076 4.892 5.226 6.373
    AG 1.728 1.456 2.169 1.681 1.530 1.406 1.215 1.969 2.348
    CE 3.999 3.955 1.396 3.367 4.186 1.571 3.573 3.540 1.780
    SF 6.041 5.764 7.233 5.176 5.429 4.756 3.897 6.303 8.537
    SD 11.082 11.250 33.517 10.800 15.426 12.892 10.456 13.169 27.118
    EI 17.479 15.085 22.265 17.052 15.792 14.562 12.183 20.446 23.994
    下载: 导出CSV

    表  2  9种方法在10对源图像上的客观指标平均值

    Table  2.   Average values of objective evaluation indexes of nine methods on ten pairs of source images

    Average Methods
    ADF CSR TE-MST FPDE GF LatLRR MSVD TIF Ours
    EN 6.083 6.068 6.724 6.078 6.310 6.198 6.019 6.340 6.862
    AG 2.743 2.388 3.464 2.840 2.336 2.233 2.074 3.001 3.529
    CE 2.306 2.266 1.558 1.911 2.218 1.354 1.998 1.847 1.393
    SF 7.283 7.011 9.218 7.272 6.725 6.158 6.602 8.144 9.768
    SD 21.866 22.238 32.147 21.771 30.854 24.791 21.416 26.060 35.556
    EI 26.600 23.911 34.097 27.485 23.694 22.300 19.541 30.484 34.776
    下载: 导出CSV

    表  3  9种方法在3种不同分辨率源图像上的运行时间以及平均值

    Table  3.   The running time and average value of nine methods on three different resolution source images (seconds per image pair) s

    Source image Resolution Methods
    ADF CSR TE-MST FPDE GF LatRR MSVD TIF Ours
    Boat street 505×510 0.421 52.381 0.006 0.952 0.101 97.959 0.295 0.031 99.247
    Soldier 632×496 0.486 68.214 0.012 1.119 0.232 126.849 0.374 0.028 127.150
    Behind smoke 768×576 0.699 79.660 0.015 1.682 0.146 215.724 0.490 0.088 217.110
    Average - 0.535 66.752 0.011 1.251 0.160 146.844 0.386 0.049 147.836
    下载: 导出CSV
  • [1] 沈英, 黄春红, 黄峰, 等. 红外与可见光融合技术的研究进展[J]. 红外与激光工程, 2021(9): 1-16. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ202109014.htm

    SHEN Y, HUANG C H, HUANG F, et al. Infrared and visible image fusion: review of key technologies [J]. Infrared and Laser Engineering, 2021(9): 1-16. https://www.cnki.com.cn/Article/CJFDTOTAL-HWYJ202109014.htm
    [2] 杨孙运, 奚峥皓, 王汉东, 等. 基于NSCT和最小化-局部平均梯度的图像融合[J]. 红外技术, 2021, 43(1): 13-20. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS202101003.htm

    YANG S Y, XI Z H, WANG H D, et al. Image fusion based on NSCT and minimum-local mean gradient [J]. Infrared Technology, 2021, 43(1): 13-20. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS202101003.htm
    [3] ZHANG X, YE P, XIAO G. VIFB: A Visible and Infrared Image Fusion Benchmark[C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops IEEE, 2020: 468-478.
    [4] CHEN J, WU K, CHENG Z, et al, A saliency-based multiscale approach for infrared and visible image fusion[J]. Signal Processing, 2021, 182(4): 107936.
    [5] LIU Y, CHEN X, Ward R K, et al. Image fusion with convolutional sparse representation[J]. IEEE signal processing letters, 2016, 23(12): 1882-1886. doi:  10.1109/LSP.2016.2618776
    [6] MA J Y, CHEN C, LI C, et al. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31: 100-109. doi:  10.1016/j.inffus.2016.02.001
    [7] ZHANG H, XU H, TIAN X, et al. Image fusion meets deep learning: A survey and perspective[J]. Information Fusion, 2021, 76(11): 323-336.
    [8] Bavirisetti D P, D Huli R. Two-scale image fusion of visible and infrared images using saliency detection[J]. Infrared Physics & Technology, 2016, 76: 52-64.
    [9] LI H, WU X J. Infrared and visible image fusion using latent low-rank representation[J/OL] [2018-04-24]. arXiv preprint. https://arxiv.org/abs/1804.08992.
    [10] LIU G, YAN S. Latent low-rank representation for subspace segmentation and feature extraction[C]//International Conference on Computer Vision, 2011: 1615-1622.
    [11] 刘琰煜, 周冬明, 聂仁灿, 等. 低秩表示和字典学习的红外与可见光图像融合算法[J]. 云南大学学报: 自然科学版, 2019, 41(4): 689-698. https://www.cnki.com.cn/Article/CJFDTOTAL-YNDZ201904007.htm

    LIU Y Y, ZHOU D M, NIE R C, et al. Infrared and visible image fusion scheme using low rank representation and dictionary learning[J]. Journal of Yunnan University: Natural Sciences Edition, 2019, 41(4): 689-698. https://www.cnki.com.cn/Article/CJFDTOTAL-YNDZ201904007.htm
    [12] 王凡, 王屹, 刘洋. 利用结构化和一致性约束的稀疏表示模型进行红外和可见光图像融合[J]. 信号处理, 2020, 36(4): 572-583. https://www.cnki.com.cn/Article/CJFDTOTAL-XXCN202004012.htm

    WANG F, WANG Y, LIU Y. Infrared and visible image fusion method based on sparse representation with structured and spatial consistency constraints[J]. Journal of Signal Processing, 2020, 36(4): 572-583. https://www.cnki.com.cn/Article/CJFDTOTAL-XXCN202004012.htm
    [13] LIU Y, LIU S, WANG Z. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015(24): 147-164.
    [14] LI S, KANG X, FANG L, et al. Pixel-level image fusion: a survey of the state of the art[J]. Information Fusion, 2017, 33: 100-112. doi:  10.1016/j.inffus.2016.05.004
    [15] Toet Alexander. The TNO Multiband image data collection[J]. Data in Brief, 2017, 15: 249-251. doi:  10.1016/j.dib.2017.09.038
    [16] Bavirisetti D P, Dhuli R. Fusion of infrared and visible sensor images based on anisotropic diffusion and Karhunen-Loeve transform[J]. IEEE Sensors Journal, 2015, 16(1): 203-209.
    [17] CHEN J, LI X J, LUO L B, et al. Infrared and visible image fusion based on target-enhanced multiscale transform decomposition[J]. Information Sciences, 2020, 508: 64-78. doi:  10.1016/j.ins.2019.08.066
    [18] Ba Virisetti D P. Multi-sensor image fusion based on fourth order partial differential equations[C]//20th International Conference on Information Fusion (Fusion) of IEEE, 2017: 1-9.
    [19] MA J Y, ZHOU Y. Infrared and visible image fusion via gradientlet filter[J]. Computer Vision and Image Understanding, 2020, 197: 103016.
    [20] Bavirisetti D P, D Huli R. Two-scale image fusion of visible and infrared images using saliency detection[J]. Infrared Physics & Technology, 2016, 76: 52-64.
    [21] Naidu V. Image fusion technique using multi-resolution singular value decomposition[J]. Defence Science Journal, 2011, 61(5): 479-484. doi:  10.14429/dsj.61.705
    [22] 刘智嘉, 贾鹏, 夏寅辉. 基于红外与可见光图像融合技术发展与性能评价[J]. 激光与红外, 2019, 49(5): 123-130. https://www.cnki.com.cn/Article/CJFDTOTAL-JGHW201905022.htm

    LIU Z J, JIA P, XIA Y H, et al. Development and performance evaluation of infrared and visual image fusion technology[J]. Laser & Infrared, 2019, 49(5): 123-130. https://www.cnki.com.cn/Article/CJFDTOTAL-JGHW201905022.htm
  • 加载中
图(9) / 表(3)
计量
  • 文章访问数:  196
  • HTML全文浏览量:  48
  • PDF下载量:  49
  • 被引次数: 0
出版历程
  • 收稿日期:  2021-08-20
  • 修回日期:  2021-09-25
  • 刊出日期:  2022-08-20

目录

    /

    返回文章
    返回