基于双通道多尺度特征提取和注意力的SAR与多光谱图像融合

种法亭, 董张玉, 杨学志, 曾庆旺

种法亭, 董张玉, 杨学志, 曾庆旺. 基于双通道多尺度特征提取和注意力的SAR与多光谱图像融合[J]. 红外技术, 2024, 46(1): 61-73.
引用本文: 种法亭, 董张玉, 杨学志, 曾庆旺. 基于双通道多尺度特征提取和注意力的SAR与多光谱图像融合[J]. 红外技术, 2024, 46(1): 61-73.
CHONG Fating, DONG Zhangyu, YANG Xuezhi, ZENG Qingwang. SAR and Multispectral Image Fusion Based on Dual-channel Multi-scale Feature Extraction and Attention[J]. Infrared Technology , 2024, 46(1): 61-73.
Citation: CHONG Fating, DONG Zhangyu, YANG Xuezhi, ZENG Qingwang. SAR and Multispectral Image Fusion Based on Dual-channel Multi-scale Feature Extraction and Attention[J]. Infrared Technology , 2024, 46(1): 61-73.

基于双通道多尺度特征提取和注意力的SAR与多光谱图像融合

基金项目: 

安徽省重点研究与开发计划项目 202004a07020030

安徽省自然科学基金项目 2108085MF233

中央高校基本科研业务费专项 JZ2021HGTB0111

详细信息
    作者简介:

    种法亭(1997-),男,硕士研究生。主要研究方向:遥感图像融合。Email:fatingchong@163.com

    通讯作者:

    董张玉(1986-),男,副教授,硕士生导师。主要研究方向:遥感图像处理、数字信号处理。Email:dzyhfut@hfut.edu.cn

  • 中图分类号: TP391

SAR and Multispectral Image Fusion Based on Dual-channel Multi-scale Feature Extraction and Attention

  • 摘要: 图像融合的根本任务是提取图像特征,由于合成孔径雷达(Synthetic Aperture Radar,SAR)图像和多光谱(Multi Spectral,MS)图像存在通道差异,针对现有算法难以充分提取和利用SAR图像的高频细节信息和多光谱图像的低频光谱信息,融合图像存在细节丢失和光谱失真问题。本文提出了一种基于双通道多尺度特征提取和混合注意力的图像融合算法。首先采用双通道网络提取SAR和多光谱图像的多尺度高频细节特征和低频光谱特征,并连续使用不同空洞率的扩张卷积扩大感受野。然后将提取的特征映射到混合注意力模块中进行特征增强,再将这些增强特征与上采样的多光谱图像叠加。同时构建了基于光谱角度距离的损失函数,可以进一步缓解细节丢失和光谱失真。最后通过解码网络重建图像,得到高分辨率的融合图像。实验结果表明,本文算法达到了领先水平,并且融合图像在细节和光谱上保持了较好的平衡。
    Abstract: The fundamental task of image fusion is to extract image features. Because of the channel differences between synthetic aperture radar (SAR) images and multispectral (MS) images, existing algorithms have difficulty in fully extracting and utilizing the high-frequency detail information of SAR images and low-frequency spectral information of multispectral images, and the fused images have problems with detail loss and spectral distortion. In this study, an image fusion algorithm based on dual-channel multiscale feature extraction and hybrid attention is proposed. First, a dual-channel network is used to extract multi-scale high-frequency detail features and low-frequency spectral features of SAR and multispectral images, and successively expand the perceptual field using dilated convolution with different void rates. The extracted features are then mapped to the hybrid attention module for feature enhancement, and these enhanced features are superimposed on the upsampled multispectral images. A loss function based on the spectral angular distance was also constructed, which could further alleviate the problems of detail loss and spectral distortion. Finally, the image is reconstructed using a decoding network to obtain a high-resolution fused image. The experimental results show that the proposed algorithm achieves the best performance and that the fused image maintains a good balance of details and spectra.
  • 图  1   本文算法网络框架

    Figure  1.   Algorithmic network framework

    图  2   3种不同尺度的MFEB

    Figure  2.   Three different scales of MFEB

    图  3   细节特征与光谱特征叠加流程图

    Figure  3.   Flow chart of superposition of detailed features and spectral features

    图  4   不同分布方式的扩张卷积感受野对比。(a) 3×3的标准卷积感受野;(b) 3×3的扩张卷积感受野(空洞率=2);(c) 锯齿状分布扩张卷积感受野

    Figure  4.   Comparison of dilated convolution receptive field with different distribution modes (a)The receptive field of 3×3 standard convolutions; (b) The receptive field of 3×3 dilated convolution (dilation ratio=2); (c) Serrated distribution dilated convolution receptive field

    图  5   MAB结构

    Figure  5.   The structure of MAB

    图  6   通道注意力模块

    Figure  6.   Channel attention block

    图  7   空间注意力模块

    Figure  7.   Spatial attention block

    图  8   第1组实验数据

    Figure  8.   The first set experiment data

    图  9   第2组实验数据

    Figure  9.   The second set experiment data

    图  10   第1组对比实验融合结果。(a) PCA; (b) NSCT_SM_PCNN; (c) RSIFNN; (d) DRPNN; (e) PanNet; (f)双分支; (g) U2Fusion; (h) HANet; (i) DMRN-Net; (j)本文算法

    Figure  10.   Fusion results of the first set of comparative experiments. (a) PCA; (b) NSCT_SM_PCNN; (c) RSIFNN; (d) DRPNN; (e) PanNet; (f) Double branch; (g) U2Fusion; (h) HANet; (i) DMRN-Net; (j) Proposed algorithm

    图  11   第2组对比实验融合结果。(a) PCA; (b) NSCT_SM_PCNN; (c) RSIFNN; (d) DRPNN; (e) PanNet; (f) 双分支; (g) U2Fusion; (h) HANet; (i) DMRN-Net; (j)本文算法

    Figure  11.   Fusion results of the second set of comparative experiments. (a) PCA; (b) NSCT_SM_PCNN; (c) RSIFNN; (d) DRPNN; (e) PanNet; (f) Double branch; (g) U2Fusion; (h) HANet; (i) DMRN-Net (j) Proposed algorithm

    图  12   第1组消融实验结果。(a1)MFEB+MAB;(b1)MFEB+MAB+DC;(c1)MFEB+MAB+DC+loss;(a2)、(b2)和(c2)分别为它们的区域放大图

    Figure  12.   The first set of ablation experiment results. (a1) MFEB+MAB; (b1) MFEB+MAB+DC; (c1)MFEB+MAB+DC+loss; (a2), (b2) and (c2) are their regional enlarged views respectively

    图  13   第2组消融实验结果。(a1)MFEB+MAB;(b1)MFEB+MAB+DC;(c1)MFEB+MAB+DC+loss;(a2)、(b2)和(c2)分别为它们的区域放大图

    Figure  13.   The second set of ablation experiment results. (a1) MFEB+MAB; (b1) MFEB+MAB+DC; (c1)MFEB+MAB+DC+loss; (a2), (b2) and (c2) are their regional enlarged views respectively

    表  1   第1组不同算法实验数据

    Table  1   The first set of experimental data of different algorithms

    Algorithms CC↑ PSNR↑ SAM↓ sCC↑ UIQI↑ Time/s↓
    PCA 0.4373 17.2903 8.3840 0.5512 0.6587 0.0643
    NSCT_SM_PCNN 0.9318 28.4791 4.0874 0.7985 0.8091 86.7521
    RSIFNN 0.8734 20.5217 8.0587 0.7785 0.8127 0.8032
    DRPNN 0.9421 28.3581 3.8354 0.7821 0.8548 1.5214
    PanNet 0.9721 30.5549 2.1245 0.7743 0.8611 0.7749
    双分支 0.9784 29.8364 1.9743 0.8019 0.8546 0.8544
    U2Fusion 0.9816 30.8673 1.7374 0.8251 0.8852 0.7965
    HANet 0.9894 31.1665 1.6885 0.8378 0.9045 0.8247
    DMRN-Net 0.9931 31.7482 1.6401 0.8414 0.9136 0.8472
    Proposed 0.9953 33.2601 1.5946 0.8468 0.9244 0.7826
    下载: 导出CSV

    表  2   第2组不同算法实验数据

    Table  2   The second set of experimental data of different algorithms

    Algorithms CC↑ PSNR↑ SAM↓ sCC↑ UIQI↑ Time/s↓
    PCA 0.3679 16.8396 7.8233 0.4315 0.6218 0.0658
    NSCT_SM_PCNN 0.9142 27.7532 4.3214 0.7783 0.8324 80.6325
    RSIFNN 0.8612 18.3357 7.8521 0.7652 0.8052 0.7935
    DRPNN 0.9217 26.2145 3.7412 0.7624 0.8375 1.3254
    PanNet 0.9654 30.1842 2.2156 0.7839 0.8501 0.7683
    双分支 0.9751 29.2644 2.0546 0.8121 0.8478 0.8774
    U2Fusion 0.9807 29.8485 1.7681 0.8335 0.8813 0.8157
    HANet 0.9887 30.8674 1.6997 0.8364 0.9082 0.8344
    DMRN-Net 0.9916 31.3149 1.6648 0.8374 0.9123 0.8548
    Proposed 0.9947 32.8233 1.6073 0.8387 0.9221 0.7978
    下载: 导出CSV

    表  3   第1组无参考客观实验数据

    Table  3   No reference objective experimental data for the first set

    Algorithms Dλ DS QNR↑
    PCA 0.1587 0.1697 0.6985
    NSCT_SM_PCNN 0.1042 0.1127 0.7948
    RSIFNN 0.0932 0.1088 0.8081
    DRPNN 0.0898 0.0927 0.8258
    PanNet 0.0723 0.0764 0.8568
    双分支 0.0714 0.0743 0.8596
    U2Fusion 0.0756 0.0785 0.8518
    HANet 0.0693 0.0681 0.8673
    DMRN-Net 0.0583 0.0635 0.8819
    Proposed 0.0574 0.0617 0.8844
    下载: 导出CSV

    表  4   第2组无参考客观实验数据

    Table  4   No reference objective experimental data for the second set

    Algorithms Dλ DS QNR↑
    PCA 0.1658 0.1754 0.6879
    NSCT_SM_PCNN 0.1325 0.1388 0.7471
    RSIFNN 0.0974 0.1243 0.7904
    DRPNN 0.0951 0.0934 0.8204
    PanNet 0.0758 0.0803 0.8400
    双分支 0.0743 0.0785 0.8530
    U2Fusion 0.0769 0.0797 0.8495
    HANet 0.0708 0.0757 0.8589
    DMRN-Net 0.0621 0.0686 0.8736
    Proposed 0.0594 0.0621 0.8822
    下载: 导出CSV

    表  5   第1组消融实验数据

    Table  5   The first set of ablation experiment data

    Methods CC↑ PSNR↑ SAM↓ sCC↑ UIQI↑
    a 0.9913 32.4855 1.6343 0.8394 0.9167
    b 0.9934 32.8586 1.6257 0.8434 0.9175
    c 0.9953 33.2601 1.5946 0.8468 0.9244
    下载: 导出CSV

    表  6   第2组消融实验数据

    Table  6   The second set of ablation experiment data

    Methods CC↑ PSNR↑ SAM↓ sCC↑ UIQI↑
    a 0.9906 31.8541 1.6371 0.8342 0.9134
    b 0.9927 32.4387 1.6288 0.8366 0.9161
    c 0.9947 32.8233 1.6073 0.8387 0.9221
    下载: 导出CSV
  • [1]

    TU T M, HUANG P S, HUNG C L, et al. A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery[J]. IEEE Geoscience and Remote Sensing Letters, 2004, 1(4): 309-312. DOI: 10.1109/LGRS.2004.834804

    [2]

    Pal S K, Majumdar T J, Bhattacharya A K. ERS-2 SAR and IRS-1C LISS Ⅲ data fusion: A PCA approach to improve remote sensing based geological interpretation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2007, 61(5): 281-297. DOI: 10.1016/j.isprsjprs.2006.10.001

    [3]

    TU T M, LEE Y C, CHANG C P, et al. Adjustable intensity-hue-saturation and Brovey transform fusion technique for IKONOS/QuickBird imagery[J]. Optical Engineering, 2005, 44(11): 116201. DOI: 10.1117/1.2124871

    [4]

    Burt P J, Adelson E H. The Laplacian Pyramid as a Compact Image Code[M]. Readings in Computer Vision. Morgan Kaufmann, 1987: 671-679.

    [5]

    Ranchin T, Wald L. The wavelet transform for the analysis of remotely sensed images[J]. International Journal of Remote Sensing, 1993, 14(3): 615-619. DOI: 10.1080/01431169308904362

    [6]

    GUO K, Labate D, Lim W Q. Edge analysis and identification using the continuous shearlet transform[J]. Applied and Computational Harmonic Analysis, 2009, 27(1): 24-46. DOI: 10.1016/j.acha.2008.10.004

    [7]

    DA Cunha A L, ZHOU J, DO M N. The nonsubsampled contourlet transform: theory, design, and applications[J]. IEEE Transactions on Image Processing, 2006, 15(10): 3089-3101. DOI: 10.1109/TIP.2006.877507

    [8]

    Masi G, Cozzolino D, Verdoliva L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7): 594. DOI: 10.3390/rs8070594

    [9]

    WEI Y, YUAN Q, SHEN H, et al. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(10): 1795-1799. DOI: 10.1109/LGRS.2017.2736020

    [10]

    YANG J, FU X, HU Y, et al. PanNet: A deep network architecture for pan-sharpening[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 5449-5457.

    [11] 吴佼华, 杨学志, 方帅, 等. 基于双分支卷积神经网络的SAR与多光谱图像融合实验[J]. 地理与地理信息科学, 2021, 37(2): 22-30. DOI: 10.3969/j.issn.1672-0504.2021.02.004

    WU J H, YANG X Z, FANG S, et al. SAR and multispectral image fusion experiment based on dual branch convolutional neural network [J]. Geography and Geo-information Science, 2021, 37(2): 22-30. DOI: 10.3969/j.issn.1672-0504.2021.02.004

    [12]

    XU H, MA J, JIANG J, et al. U2Fusion: A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 502-518.

    [13]

    LIU Q, HAN L, TAN R, et al. Hybrid attention based residual network for pansharpening[J]. Remote Sensing, 2021, 13(10): 1962. DOI: 10.3390/rs13101962

    [14] 董张玉, 许道礼, 张晋, 等. 基于双分支多尺度残差融合嵌套的SAR和多光谱图像融合架构与实验[J]. 地理与地理信息科学, 2023, 39(1): 23-30.

    DONG Z Y, XU D L, ZHANG J, et al. Architecture and experiments of SAR and multispectral image fusion based on double-branch multiscale residual-fusion nesting[J]. Geography and Geo-information Science, 2023, 39(1): 23-30.

    [15]

    MIN A, GUO Z, LI H, et al. JMnet: Joint metric neural network for hyperspectral unmixing[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-12.

    [16] 郭彭浩. 基于卷积神经网络和贝叶斯理论的遥感图像Pansharpening算法研究[D]. 南京: 南京信息工程大学, 2021.

    GUO P H. Research on Pansharpening Algorithm of Remote Sensing Image Based on Convolution Neural Network and Bayesian Theory[D]. Nanjing: Nanjing University of Information Engineering, 2021.

    [17] 申兴成, 杨学志, 董张玉, 等. 结合扩张卷积的残差网络SAR图像去噪[J]. 测绘科学, 2021, 46(12): 106-114.

    SHEN X C, YANG X Z, DONG Z Y, et al. Residual network combined with dilated convolution for SAR image denoising[J]. Science of Surveying and Mapping, 2021, 46(12): 106-114.

    [18] 黄玲琳, 李强, 路锦正, 等. 基于多尺度和注意力模型的红外与可见光图像融合[J]. 红外技术, 2023, 45(2): 143-149. http://hwjs.nvir.cn/article/id/10e9d4ea-fb05-43a5-817a-bcad09f693b8

    HUANG L L, LI Q, LU J Z, et al. Infrared and visible image fusion based on multi-scale and attention model[J]. Infrared Technology, 2023, 45(2): 143-149. http://hwjs.nvir.cn/article/id/10e9d4ea-fb05-43a5-817a-bcad09f693b8

    [19]

    HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.

    [20]

    WANG Q, WU B, ZHU P, et al. Supplementary material for 'ECA-Net: Efficient channel attention for deep convolutional neural networks[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 13-19.

    [21]

    LIU Y, SHAO Z, Hoffmann N. Global attention mechanism: retain information to enhance channel-spatial interactions[J/OL]. arXiv preprint arXiv, 2021, https://arxiv.org/abs/2112.05561.

    [22]

    DONG C, LOY C C, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(2): 295-307.

    [23]

    LI P, LEE S H, HSU H Y, et al. Nonlinear fusion of multispectral citrus fruit image data with information contents[J]. Sensors, 2017, 17(1): 142.

    [24]

    ZHOU J, Civco D L, Silander J A. A wavelet transform method to merge Landsat TM and SPOT panchromatic data[J]. International Journal of Remote Sensing, 1998, 19(4): 743-757.

    [25]

    WANG Z, Bovik A C. A universal image quality index[J]. IEEE Signal Processing Letters, 2002, 9(3): 81-84.

    [26]

    CHENG J, LIU H, LIU T, et al. Remote sensing image fusion via wavelet transform and sparse representation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 104: 158-173.

    [27]

    WANG X L, CHEN C X. Image fusion for synthetic aperture radar and multispectral images based on sub-band-modulated non-subsampled contourlet transform and pulse coupled neural network methods[J]. The Imaging Science Journal, 2016, 64(2): 87-93.

    [28]

    SHAO Z, CAI J. Remote sensing image fusion with deep convolutional neural network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(5): 1656-1669.

  • 期刊类型引用(4)

    1. 蔡志鹏,黄文登,杨创华,娄本浊,何军锋. 不同梯度掺杂透射式GaAs光电阴极的平均时间衰减常数及瞬态响应理论研究. 真空科学与技术学报. 2025(05): 398-407 . 百度学术
    2. 吕行,富容国,常本康,郭欣,王芝. 透射式GaAs光电阴极性能提高以及结构优化. 物理学报. 2024(03): 250-256 . 百度学术
    3. 廖宇翔,王子辰,唐林,冯俞铭,赵小艳,刘頔威,张开春. 基于Geant4-CST联合仿真的条纹变像管光电子产生与控制. 光学学报. 2024(03): 308-315 . 百度学术
    4. 严仕凯,司曙光,王兴超,任玲,金睦淳,江涛,王亮,吴凯,金真,黄国瑞,涂蕴超. 阴极制备工艺对高温光电倍增管性能的影响. 光学学报. 2023(23): 28-35 . 百度学术

    其他类型引用(4)

图(13)  /  表(6)
计量
  • 文章访问数:  115
  • HTML全文浏览量:  91
  • PDF下载量:  35
  • 被引次数: 8
出版历程
  • 收稿日期:  2023-03-07
  • 修回日期:  2023-05-17
  • 刊出日期:  2024-01-19

目录

    /

    返回文章
    返回