Advanced Search
中文
Volume 46 Issue 1
Jan.  2024
Turn off MathJax
Article Contents
Abstract
References
Related Articles
CHONG Fating, DONG Zhangyu, YANG Xuezhi, ZENG Qingwang. SAR and Multispectral Image Fusion Based on Dual-channel Multi-scale Feature Extraction and Attention[J]. Infrared Technology , 2024, 46(1): 61-73.
Citation: CHONG Fating, DONG Zhangyu, YANG Xuezhi, ZENG Qingwang. SAR and Multispectral Image Fusion Based on Dual-channel Multi-scale Feature Extraction and Attention[J]. Infrared Technology , 2024, 46(1): 61-73.
shu

SAR and Multispectral Image Fusion Based on Dual-channel Multi-scale Feature Extraction and Attention

  • 1.

    College of Computer and Information, Hefei University of Technology, Hefei 230031, China

  • 2.

    Anhui Province Key Laboratory of Industry Safety and Emergency Technology, Hefei 230031, China

  • 3.

    Anhui Province Laboratory of Intelligent Interconnection System, Hefei 230031, China

More Information
  • Received Date: March 07, 2023
  • Revised Date: May 17, 2023
    Graphical Abstract
    • Abstract
  • The fundamental task of image fusion is to extract image features. Because of the channel differences between synthetic aperture radar (SAR) images and multispectral (MS) images, existing algorithms have difficulty in fully extracting and utilizing the high-frequency detail information of SAR images and low-frequency spectral information of multispectral images, and the fused images have problems with detail loss and spectral distortion. In this study, an image fusion algorithm based on dual-channel multiscale feature extraction and hybrid attention is proposed. First, a dual-channel network is used to extract multi-scale high-frequency detail features and low-frequency spectral features of SAR and multispectral images, and successively expand the perceptual field using dilated convolution with different void rates. The extracted features are then mapped to the hybrid attention module for feature enhancement, and these enhanced features are superimposed on the upsampled multispectral images. A loss function based on the spectral angular distance was also constructed, which could further alleviate the problems of detail loss and spectral distortion. Finally, the image is reconstructed using a decoding network to obtain a high-resolution fused image. The experimental results show that the proposed algorithm achieves the best performance and that the fused image maintains a good balance of details and spectra.
    • FullText(HTML)
    • References (28)
  • [1]
    TU T M, HUANG P S, HUNG C L, et al. A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery[J]. IEEE Geoscience and Remote Sensing Letters, 2004, 1(4): 309-312. DOI: 10.1109/LGRS.2004.834804
    [2]
    Pal S K, Majumdar T J, Bhattacharya A K. ERS-2 SAR and IRS-1C LISS Ⅲ data fusion: A PCA approach to improve remote sensing based geological interpretation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2007, 61(5): 281-297. DOI: 10.1016/j.isprsjprs.2006.10.001
    [3]
    TU T M, LEE Y C, CHANG C P, et al. Adjustable intensity-hue-saturation and Brovey transform fusion technique for IKONOS/QuickBird imagery[J]. Optical Engineering, 2005, 44(11): 116201. DOI: 10.1117/1.2124871
    [4]
    Burt P J, Adelson E H. The Laplacian Pyramid as a Compact Image Code[M]. Readings in Computer Vision. Morgan Kaufmann, 1987: 671-679.
    [5]
    Ranchin T, Wald L. The wavelet transform for the analysis of remotely sensed images[J]. International Journal of Remote Sensing, 1993, 14(3): 615-619. DOI: 10.1080/01431169308904362
    [6]
    GUO K, Labate D, Lim W Q. Edge analysis and identification using the continuous shearlet transform[J]. Applied and Computational Harmonic Analysis, 2009, 27(1): 24-46. DOI: 10.1016/j.acha.2008.10.004
    [7]
    DA Cunha A L, ZHOU J, DO M N. The nonsubsampled contourlet transform: theory, design, and applications[J]. IEEE Transactions on Image Processing, 2006, 15(10): 3089-3101. DOI: 10.1109/TIP.2006.877507
    [8]
    Masi G, Cozzolino D, Verdoliva L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7): 594. DOI: 10.3390/rs8070594
    [9]
    WEI Y, YUAN Q, SHEN H, et al. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(10): 1795-1799. DOI: 10.1109/LGRS.2017.2736020
    [10]
    YANG J, FU X, HU Y, et al. PanNet: A deep network architecture for pan-sharpening[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 5449-5457.
    [11]
    吴佼华, 杨学志, 方帅, 等. 基于双分支卷积神经网络的SAR与多光谱图像融合实验[J]. 地理与地理信息科学, 2021, 37(2): 22-30. DOI: 10.3969/j.issn.1672-0504.2021.02.004

    WU J H, YANG X Z, FANG S, et al. SAR and multispectral image fusion experiment based on dual branch convolutional neural network [J]. Geography and Geo-information Science, 2021, 37(2): 22-30. DOI: 10.3969/j.issn.1672-0504.2021.02.004
    [12]
    XU H, MA J, JIANG J, et al. U2Fusion: A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 502-518.
    [13]
    LIU Q, HAN L, TAN R, et al. Hybrid attention based residual network for pansharpening[J]. Remote Sensing, 2021, 13(10): 1962. DOI: 10.3390/rs13101962
    [14]
    董张玉, 许道礼, 张晋, 等. 基于双分支多尺度残差融合嵌套的SAR和多光谱图像融合架构与实验[J]. 地理与地理信息科学, 2023, 39(1): 23-30.

    DONG Z Y, XU D L, ZHANG J, et al. Architecture and experiments of SAR and multispectral image fusion based on double-branch multiscale residual-fusion nesting[J]. Geography and Geo-information Science, 2023, 39(1): 23-30.
    [15]
    MIN A, GUO Z, LI H, et al. JMnet: Joint metric neural network for hyperspectral unmixing[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-12.
    [16]
    郭彭浩. 基于卷积神经网络和贝叶斯理论的遥感图像Pansharpening算法研究[D]. 南京: 南京信息工程大学, 2021.

    GUO P H. Research on Pansharpening Algorithm of Remote Sensing Image Based on Convolution Neural Network and Bayesian Theory[D]. Nanjing: Nanjing University of Information Engineering, 2021.
    [17]
    申兴成, 杨学志, 董张玉, 等. 结合扩张卷积的残差网络SAR图像去噪[J]. 测绘科学, 2021, 46(12): 106-114.

    SHEN X C, YANG X Z, DONG Z Y, et al. Residual network combined with dilated convolution for SAR image denoising[J]. Science of Surveying and Mapping, 2021, 46(12): 106-114.
    [18]
    黄玲琳, 李强, 路锦正, 等. 基于多尺度和注意力模型的红外与可见光图像融合[J]. 红外技术, 2023, 45(2): 143-149. http://hwjs.nvir.cn/article/id/10e9d4ea-fb05-43a5-817a-bcad09f693b8

    HUANG L L, LI Q, LU J Z, et al. Infrared and visible image fusion based on multi-scale and attention model[J]. Infrared Technology, 2023, 45(2): 143-149. http://hwjs.nvir.cn/article/id/10e9d4ea-fb05-43a5-817a-bcad09f693b8
    [19]
    HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
    [20]
    WANG Q, WU B, ZHU P, et al. Supplementary material for 'ECA-Net: Efficient channel attention for deep convolutional neural networks[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 13-19.
    [21]
    LIU Y, SHAO Z, Hoffmann N. Global attention mechanism: retain information to enhance channel-spatial interactions[J/OL]. arXiv preprint arXiv, 2021, https://arxiv.org/abs/2112.05561.
    [22]
    DONG C, LOY C C, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(2): 295-307.
    [23]
    LI P, LEE S H, HSU H Y, et al. Nonlinear fusion of multispectral citrus fruit image data with information contents[J]. Sensors, 2017, 17(1): 142.
    [24]
    ZHOU J, Civco D L, Silander J A. A wavelet transform method to merge Landsat TM and SPOT panchromatic data[J]. International Journal of Remote Sensing, 1998, 19(4): 743-757.
    [25]
    WANG Z, Bovik A C. A universal image quality index[J]. IEEE Signal Processing Letters, 2002, 9(3): 81-84.
    [26]
    CHENG J, LIU H, LIU T, et al. Remote sensing image fusion via wavelet transform and sparse representation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 104: 158-173.
    [27]
    WANG X L, CHEN C X. Image fusion for synthetic aperture radar and multispectral images based on sub-band-modulated non-subsampled contourlet transform and pulse coupled neural network methods[J]. The Imaging Science Journal, 2016, 64(2): 87-93.
    [28]
    SHAO Z, CAI J. Remote sensing image fusion with deep convolutional neural network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(5): 1656-1669.
    • Related Articles

  • Related Articles

    [1]LIAO Guangfeng, GUAN Zhiwei, CHEN Qiang. An Improved Dual Discriminator Generative Adversarial Network Algorithm for Infrared and Visible Image Fusion[J]. Infrared Technology , 2025, 47(3): 367-375.
    [2]YUAN Hongchun, ZHANG Bo, CHENG Xin. Underwater Image Enhancement Algorithm Combining Transformer and Generative Adversarial Network[J]. Infrared Technology , 2024, 46(9): 975-983.
    [3]LI Li, YI Shi, LIU Xi, CHENG Xinghao, WANG Cheng. Infrared Image Deblurring Based on Dense Residual Generation Adversarial Network[J]. Infrared Technology , 2024, 46(6): 663-671.
    [4]DI Jing, REN Li, LIU Jizhao, GUO Wenqing, LIAN Jing. Infrared and Visible Image Fusion Based on Three-branch Adversarial Learning and Compensation Attention Mechanism[J]. Infrared Technology , 2024, 46(5): 510-521.
    [5]CHEN Xin. Infrared and Visible Image Fusion Using Double Attention Generative Adversarial Networks[J]. Infrared Technology , 2023, 45(6): 639-648.
    [6]WANG Tianyuan, LUO Xiaoqing, ZHANG Zhancheng. Infrared and Visible Image Fusion Based on Self-attention Learning[J]. Infrared Technology , 2023, 45(2): 171-177.
    [7]FU Tian, DENG Changzheng, HAN Xinyue, GONG Mengqing. Infrared and Visible Image Registration for Power Equipments Based on Deep Learning[J]. Infrared Technology , 2022, 44(9): 936-943.
    [8]LI Yunhong, LIU Yudong, SU Xueping, LUO Xuemin, YAO Lan. Review of Infrared and Visible Image Registration[J]. Infrared Technology , 2022, 44(7): 641-651.
    [9]HUANG Mengtao, GAO Na, LIU Bao. Image Deblurring Method Based on a Dual-Discriminator Weighted Generative Adversarial Network[J]. Infrared Technology , 2022, 44(1): 41-46.
    [10]LUO Di, WANG Congqing, ZHOU Yongjun. A Visible and Infrared Image Fusion Method based on Generative Adversarial Networks and Attention Mechanism[J]. Infrared Technology , 2021, 43(6): 566-574.

Catalog

    Get Citation
    PDF
    XML
    Article views (104) PDF downloads (29) Cited by()
    Related

    Copyright © 《Infrared Technology》Editorial Office京ICP备000000000号-1

    Supported by: Beijing Renhe Information Technology Co., Ltd.  Baidu Analytics

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return