SAR and Multispectral Image Fusion Based on Dual-channel Multi-scale Feature Extraction and Attention
-
摘要: 图像融合的根本任务是提取图像特征,由于合成孔径雷达(Synthetic Aperture Radar,SAR)图像和多光谱(Multi Spectral,MS)图像存在通道差异,针对现有算法难以充分提取和利用SAR图像的高频细节信息和多光谱图像的低频光谱信息,融合图像存在细节丢失和光谱失真问题。本文提出了一种基于双通道多尺度特征提取和混合注意力的图像融合算法。首先采用双通道网络提取SAR和多光谱图像的多尺度高频细节特征和低频光谱特征,并连续使用不同空洞率的扩张卷积扩大感受野。然后将提取的特征映射到混合注意力模块中进行特征增强,再将这些增强特征与上采样的多光谱图像叠加。同时构建了基于光谱角度距离的损失函数,可以进一步缓解细节丢失和光谱失真。最后通过解码网络重建图像,得到高分辨率的融合图像。实验结果表明,本文算法达到了领先水平,并且融合图像在细节和光谱上保持了较好的平衡。Abstract: The fundamental task of image fusion is to extract image features. Because of the channel differences between synthetic aperture radar (SAR) images and multispectral (MS) images, existing algorithms have difficulty in fully extracting and utilizing the high-frequency detail information of SAR images and low-frequency spectral information of multispectral images, and the fused images have problems with detail loss and spectral distortion. In this study, an image fusion algorithm based on dual-channel multiscale feature extraction and hybrid attention is proposed. First, a dual-channel network is used to extract multi-scale high-frequency detail features and low-frequency spectral features of SAR and multispectral images, and successively expand the perceptual field using dilated convolution with different void rates. The extracted features are then mapped to the hybrid attention module for feature enhancement, and these enhanced features are superimposed on the upsampled multispectral images. A loss function based on the spectral angular distance was also constructed, which could further alleviate the problems of detail loss and spectral distortion. Finally, the image is reconstructed using a decoding network to obtain a high-resolution fused image. The experimental results show that the proposed algorithm achieves the best performance and that the fused image maintains a good balance of details and spectra.
-
-
图 4 不同分布方式的扩张卷积感受野对比。(a) 3×3的标准卷积感受野;(b) 3×3的扩张卷积感受野(空洞率=2);(c) 锯齿状分布扩张卷积感受野
Figure 4. Comparison of dilated convolution receptive field with different distribution modes (a)The receptive field of 3×3 standard convolutions; (b) The receptive field of 3×3 dilated convolution (dilation ratio=2); (c) Serrated distribution dilated convolution receptive field
图 10 第1组对比实验融合结果。(a) PCA; (b) NSCT_SM_PCNN; (c) RSIFNN; (d) DRPNN; (e) PanNet; (f)双分支; (g) U2Fusion; (h) HANet; (i) DMRN-Net; (j)本文算法
Figure 10. Fusion results of the first set of comparative experiments. (a) PCA; (b) NSCT_SM_PCNN; (c) RSIFNN; (d) DRPNN; (e) PanNet; (f) Double branch; (g) U2Fusion; (h) HANet; (i) DMRN-Net; (j) Proposed algorithm
图 11 第2组对比实验融合结果。(a) PCA; (b) NSCT_SM_PCNN; (c) RSIFNN; (d) DRPNN; (e) PanNet; (f) 双分支; (g) U2Fusion; (h) HANet; (i) DMRN-Net; (j)本文算法
Figure 11. Fusion results of the second set of comparative experiments. (a) PCA; (b) NSCT_SM_PCNN; (c) RSIFNN; (d) DRPNN; (e) PanNet; (f) Double branch; (g) U2Fusion; (h) HANet; (i) DMRN-Net (j) Proposed algorithm
表 1 第1组不同算法实验数据
Table 1 The first set of experimental data of different algorithms
Algorithms CC↑ PSNR↑ SAM↓ sCC↑ UIQI↑ Time/s↓ PCA 0.4373 17.2903 8.3840 0.5512 0.6587 0.0643 NSCT_SM_PCNN 0.9318 28.4791 4.0874 0.7985 0.8091 86.7521 RSIFNN 0.8734 20.5217 8.0587 0.7785 0.8127 0.8032 DRPNN 0.9421 28.3581 3.8354 0.7821 0.8548 1.5214 PanNet 0.9721 30.5549 2.1245 0.7743 0.8611 0.7749 双分支 0.9784 29.8364 1.9743 0.8019 0.8546 0.8544 U2Fusion 0.9816 30.8673 1.7374 0.8251 0.8852 0.7965 HANet 0.9894 31.1665 1.6885 0.8378 0.9045 0.8247 DMRN-Net 0.9931 31.7482 1.6401 0.8414 0.9136 0.8472 Proposed 0.9953 33.2601 1.5946 0.8468 0.9244 0.7826 表 2 第2组不同算法实验数据
Table 2 The second set of experimental data of different algorithms
Algorithms CC↑ PSNR↑ SAM↓ sCC↑ UIQI↑ Time/s↓ PCA 0.3679 16.8396 7.8233 0.4315 0.6218 0.0658 NSCT_SM_PCNN 0.9142 27.7532 4.3214 0.7783 0.8324 80.6325 RSIFNN 0.8612 18.3357 7.8521 0.7652 0.8052 0.7935 DRPNN 0.9217 26.2145 3.7412 0.7624 0.8375 1.3254 PanNet 0.9654 30.1842 2.2156 0.7839 0.8501 0.7683 双分支 0.9751 29.2644 2.0546 0.8121 0.8478 0.8774 U2Fusion 0.9807 29.8485 1.7681 0.8335 0.8813 0.8157 HANet 0.9887 30.8674 1.6997 0.8364 0.9082 0.8344 DMRN-Net 0.9916 31.3149 1.6648 0.8374 0.9123 0.8548 Proposed 0.9947 32.8233 1.6073 0.8387 0.9221 0.7978 表 3 第1组无参考客观实验数据
Table 3 No reference objective experimental data for the first set
Algorithms Dλ↓ DS↓ QNR↑ PCA 0.1587 0.1697 0.6985 NSCT_SM_PCNN 0.1042 0.1127 0.7948 RSIFNN 0.0932 0.1088 0.8081 DRPNN 0.0898 0.0927 0.8258 PanNet 0.0723 0.0764 0.8568 双分支 0.0714 0.0743 0.8596 U2Fusion 0.0756 0.0785 0.8518 HANet 0.0693 0.0681 0.8673 DMRN-Net 0.0583 0.0635 0.8819 Proposed 0.0574 0.0617 0.8844 表 4 第2组无参考客观实验数据
Table 4 No reference objective experimental data for the second set
Algorithms Dλ↓ DS↓ QNR↑ PCA 0.1658 0.1754 0.6879 NSCT_SM_PCNN 0.1325 0.1388 0.7471 RSIFNN 0.0974 0.1243 0.7904 DRPNN 0.0951 0.0934 0.8204 PanNet 0.0758 0.0803 0.8400 双分支 0.0743 0.0785 0.8530 U2Fusion 0.0769 0.0797 0.8495 HANet 0.0708 0.0757 0.8589 DMRN-Net 0.0621 0.0686 0.8736 Proposed 0.0594 0.0621 0.8822 表 5 第1组消融实验数据
Table 5 The first set of ablation experiment data
Methods CC↑ PSNR↑ SAM↓ sCC↑ UIQI↑ a 0.9913 32.4855 1.6343 0.8394 0.9167 b 0.9934 32.8586 1.6257 0.8434 0.9175 c 0.9953 33.2601 1.5946 0.8468 0.9244 表 6 第2组消融实验数据
Table 6 The second set of ablation experiment data
Methods CC↑ PSNR↑ SAM↓ sCC↑ UIQI↑ a 0.9906 31.8541 1.6371 0.8342 0.9134 b 0.9927 32.4387 1.6288 0.8366 0.9161 c 0.9947 32.8233 1.6073 0.8387 0.9221 -
[1] TU T M, HUANG P S, HUNG C L, et al. A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery[J]. IEEE Geoscience and Remote Sensing Letters, 2004, 1(4): 309-312. DOI: 10.1109/LGRS.2004.834804
[2] Pal S K, Majumdar T J, Bhattacharya A K. ERS-2 SAR and IRS-1C LISS Ⅲ data fusion: A PCA approach to improve remote sensing based geological interpretation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2007, 61(5): 281-297. DOI: 10.1016/j.isprsjprs.2006.10.001
[3] TU T M, LEE Y C, CHANG C P, et al. Adjustable intensity-hue-saturation and Brovey transform fusion technique for IKONOS/QuickBird imagery[J]. Optical Engineering, 2005, 44(11): 116201. DOI: 10.1117/1.2124871
[4] Burt P J, Adelson E H. The Laplacian Pyramid as a Compact Image Code[M]. Readings in Computer Vision. Morgan Kaufmann, 1987: 671-679.
[5] Ranchin T, Wald L. The wavelet transform for the analysis of remotely sensed images[J]. International Journal of Remote Sensing, 1993, 14(3): 615-619. DOI: 10.1080/01431169308904362
[6] GUO K, Labate D, Lim W Q. Edge analysis and identification using the continuous shearlet transform[J]. Applied and Computational Harmonic Analysis, 2009, 27(1): 24-46. DOI: 10.1016/j.acha.2008.10.004
[7] DA Cunha A L, ZHOU J, DO M N. The nonsubsampled contourlet transform: theory, design, and applications[J]. IEEE Transactions on Image Processing, 2006, 15(10): 3089-3101. DOI: 10.1109/TIP.2006.877507
[8] Masi G, Cozzolino D, Verdoliva L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7): 594. DOI: 10.3390/rs8070594
[9] WEI Y, YUAN Q, SHEN H, et al. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(10): 1795-1799. DOI: 10.1109/LGRS.2017.2736020
[10] YANG J, FU X, HU Y, et al. PanNet: A deep network architecture for pan-sharpening[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 5449-5457.
[11] 吴佼华, 杨学志, 方帅, 等. 基于双分支卷积神经网络的SAR与多光谱图像融合实验[J]. 地理与地理信息科学, 2021, 37(2): 22-30. DOI: 10.3969/j.issn.1672-0504.2021.02.004 WU J H, YANG X Z, FANG S, et al. SAR and multispectral image fusion experiment based on dual branch convolutional neural network [J]. Geography and Geo-information Science, 2021, 37(2): 22-30. DOI: 10.3969/j.issn.1672-0504.2021.02.004
[12] XU H, MA J, JIANG J, et al. U2Fusion: A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 502-518.
[13] LIU Q, HAN L, TAN R, et al. Hybrid attention based residual network for pansharpening[J]. Remote Sensing, 2021, 13(10): 1962. DOI: 10.3390/rs13101962
[14] 董张玉, 许道礼, 张晋, 等. 基于双分支多尺度残差融合嵌套的SAR和多光谱图像融合架构与实验[J]. 地理与地理信息科学, 2023, 39(1): 23-30. DONG Z Y, XU D L, ZHANG J, et al. Architecture and experiments of SAR and multispectral image fusion based on double-branch multiscale residual-fusion nesting[J]. Geography and Geo-information Science, 2023, 39(1): 23-30.
[15] MIN A, GUO Z, LI H, et al. JMnet: Joint metric neural network for hyperspectral unmixing[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-12.
[16] 郭彭浩. 基于卷积神经网络和贝叶斯理论的遥感图像Pansharpening算法研究[D]. 南京: 南京信息工程大学, 2021. GUO P H. Research on Pansharpening Algorithm of Remote Sensing Image Based on Convolution Neural Network and Bayesian Theory[D]. Nanjing: Nanjing University of Information Engineering, 2021.
[17] 申兴成, 杨学志, 董张玉, 等. 结合扩张卷积的残差网络SAR图像去噪[J]. 测绘科学, 2021, 46(12): 106-114. SHEN X C, YANG X Z, DONG Z Y, et al. Residual network combined with dilated convolution for SAR image denoising[J]. Science of Surveying and Mapping, 2021, 46(12): 106-114.
[18] 黄玲琳, 李强, 路锦正, 等. 基于多尺度和注意力模型的红外与可见光图像融合[J]. 红外技术, 2023, 45(2): 143-149. http://hwjs.nvir.cn/article/id/10e9d4ea-fb05-43a5-817a-bcad09f693b8 HUANG L L, LI Q, LU J Z, et al. Infrared and visible image fusion based on multi-scale and attention model[J]. Infrared Technology, 2023, 45(2): 143-149. http://hwjs.nvir.cn/article/id/10e9d4ea-fb05-43a5-817a-bcad09f693b8
[19] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[20] WANG Q, WU B, ZHU P, et al. Supplementary material for 'ECA-Net: Efficient channel attention for deep convolutional neural networks[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 13-19.
[21] LIU Y, SHAO Z, Hoffmann N. Global attention mechanism: retain information to enhance channel-spatial interactions[J/OL]. arXiv preprint arXiv, 2021, https://arxiv.org/abs/2112.05561.
[22] DONG C, LOY C C, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(2): 295-307.
[23] LI P, LEE S H, HSU H Y, et al. Nonlinear fusion of multispectral citrus fruit image data with information contents[J]. Sensors, 2017, 17(1): 142.
[24] ZHOU J, Civco D L, Silander J A. A wavelet transform method to merge Landsat TM and SPOT panchromatic data[J]. International Journal of Remote Sensing, 1998, 19(4): 743-757.
[25] WANG Z, Bovik A C. A universal image quality index[J]. IEEE Signal Processing Letters, 2002, 9(3): 81-84.
[26] CHENG J, LIU H, LIU T, et al. Remote sensing image fusion via wavelet transform and sparse representation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 104: 158-173.
[27] WANG X L, CHEN C X. Image fusion for synthetic aperture radar and multispectral images based on sub-band-modulated non-subsampled contourlet transform and pulse coupled neural network methods[J]. The Imaging Science Journal, 2016, 64(2): 87-93.
[28] SHAO Z, CAI J. Remote sensing image fusion with deep convolutional neural network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(5): 1656-1669.
-
期刊类型引用(4)
1. 蔡志鹏,黄文登,杨创华,娄本浊,何军锋. 不同梯度掺杂透射式GaAs光电阴极的平均时间衰减常数及瞬态响应理论研究. 真空科学与技术学报. 2025(05): 398-407 . 百度学术
2. 吕行,富容国,常本康,郭欣,王芝. 透射式GaAs光电阴极性能提高以及结构优化. 物理学报. 2024(03): 250-256 . 百度学术
3. 廖宇翔,王子辰,唐林,冯俞铭,赵小艳,刘頔威,张开春. 基于Geant4-CST联合仿真的条纹变像管光电子产生与控制. 光学学报. 2024(03): 308-315 . 百度学术
4. 严仕凯,司曙光,王兴超,任玲,金睦淳,江涛,王亮,吴凯,金真,黄国瑞,涂蕴超. 阴极制备工艺对高温光电倍增管性能的影响. 光学学报. 2023(23): 28-35 . 百度学术
其他类型引用(4)