| Citation: |
XU Guangxian, ZHOU Weijie, MA Fei. Fusion of Hyperspectral and Multispectral Images Using a CNN Joint Multi-Scale Transformer[J]. Infrared Technology , 2025, 47(1): 52-62.
|
Hyperspectral images contain rich spectral information, and multispectral images have exquisite geometric features. More comprehensive remote sensing images can be obtained by merging high-resolution multispectral and low-resolution hyperspectral images. However, most existing fusion networks are based on convolutional neural networks. For remote sensing images with complex structures, convolution operations dependent on the kernel size tend to lead to a lack of global context information in the feature fusion stage. To ensure the quality of image fusion, this study proposes a convolutional neural network (CNN) combined with a multi-scale transformer network to realize multispectral and hyperspectral image fusion, combining the feature extraction capability of the CNN and the global modeling advantage of the transformer. The network divides the fusion task into two stages: feature extraction and fusion. In the feature extraction stage, different modules are designed for feature extraction based on the CNN. In the fusion stage, a multi-scale transformer module is used to establish a long-distance correlation between local and global information, and the features are mapped into high-resolution hyperspectral images through multilayer convolution layers. Experimental results on the CAVE and Harvard datasets show that the proposed algorithm can improve the quality of fused images better than other classical algorithms.
| [1] |
YANG Q, XU Y, WU Z, et al. Hyperspectral And Multispectral Image Fusion Based On Deep Attention Network[C]//Proceedings of the 2019 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), 2019: 1-5.
|
| [2] |
童庆禧, 张兵, 张立福. 中国高光谱遥感的前沿进展[J]. 遥感学报, 2016, 20(5): 689-707.
TONG Qingxi, ZHANG Bing, ZHANG Lifu. Current progress of hyperspectral remote sensing in China[J]. Journal of Remote Sensing, 2016, 20(5): 689-707.
|
| [3] |
Akbari H, Kosugi Y, Kojima K, et al. Detection and analysis of the intestinal ischemia using visible and invisible hyperspectral imaging[J]. IEEE Transactions on Biomedical Engineering, 2010, 57(8): 2011-2017. DOI: 10.1109/TBME.2010.2049110
|
| [4] |
Zhihong P, Healey G, Prasad M, et al. Face recognition in hyperspectral images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(12): 1552-1560. DOI: 10.1109/TPAMI.2003.1251148
|
| [5] |
DIAN R, LI S, SUN B, et al. Recent advances and new guidelines on hyperspectral and multispectral image fusion[J]. Information Fusion, 2021, 69: 40-51. DOI: 10.1016/j.inffus.2020.11.001
|
| [6] |
Fasbender D, Radoux J, Bogaert P. Bayesian data fusion for adaptable image pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(6): 1847-1857. DOI: 10.1109/TGRS.2008.917131
|
| [7] |
CHEN Z, PU H, WANG B, et al. Fusion of hyperspectral and multispectral images: a novel framework based on generalization of pan-sharpening methods[J]. IEEE Geoscience and Remote Sensing Letters, 2014, 11(8): 1418-1422. DOI: 10.1109/LGRS.2013.2294476
|
| [8] |
SELVA M, AIAZZI B, BUTERA F, et al. Hyper-sharpening of hyperspectral data: A first approach[C]//Proceedings of the 2014 6th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), 2014: 24-27.
|
| [9] |
Zurita-Milla R, Clevers J G P W, Schaepman M E. Unmixing-based landsat TM and MERIS FR data fusion[J]. IEEE Geoscience and Remote Sensing Letters, 2008, 5(3): 453-457. DOI: 10.1109/LGRS.2008.919685
|
| [10] |
Yokoya N, Yairi T, Iwasaki A. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2012, 50(2): 528-537. DOI: 10.1109/TGRS.2011.2161320
|
| [11] |
Lanaras C, Baltsavias E, Schindler K. Hyperspectral super-resolution by coupled spectral unmixing[C]// Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), 2015: 3586-3594.
|
| [12] |
WEI Q, Dobigeon N, Tourneret J Y. Fast fusion of multi-band images based on solving a sylvester equation[J]. IEEE Transactions on Image Processing, 2015, 24(11): 4109-4121. DOI: 10.1109/TIP.2015.2458572
|
| [13] |
Akhtar N, Shafait F, Mian A. Bayesian sparse representation for hyperspectral image super resolution[C]// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015: 7-12.
|
| [14] |
孙佳敏, 宋慧慧. 基于DWT和生成对抗网络的高光谱多光谱图像融合[J]. 无线电工程, 2021, 51(12): 1434-1441.
SUN Jiamin, SONG Huihui. Hyperspectral and multispectral image fusion based on discrete wavelet transform and fenerative adversarial networks[J]. Radio Engineering, 2021, 51(12): 1434-1441.
|
| [15] |
DIAN R, FANG L, LI S. Hyperspectral image super-resolution via non-local sparse tensor factorization[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017: 21-26.
|
| [16] |
DIAN R, LI S, FANG L, et al. Nonlocal sparse tensor factorization for semiblind hyperspectral and multispectral image fusion[J]. IEEE Transactions on Cybernetics, 2020, 50(10): 4469-4480. DOI: 10.1109/TCYB.2019.2951572
|
| [17] |
LI S, DIAN R, FANG L, et al. Fusing hyperspectral and multispectral images via coupled sparse tensor factorization[J]. IEEE Transactions on Image Processing, 2018, 27(8): 4118-4130. DOI: 10.1109/TIP.2018.2836307
|
| [18] |
Kanatsoulis C I, Fu X, Sidiropoulos N D, et al. Hyperspectral super-resolution: a coupled tensor factorization approach[J]. IEEE Transactions on Signal Processing, 2018, 66(24): 6503-6517. DOI: 10.1109/TSP.2018.2876362
|
| [19] |
LI J, ZHENG K, YAO J, et al. Deep unsupervised blind hyperspectral and multispectral data fusion[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 1-5.
|
| [20] |
WANG X, WANG X, ZHAO K, et al. FSL-Unet: full-scale linked unet with spatial–spectral joint perceptual attention for hyperspectral and multispectral image fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-14.
|
| [21] |
DONG M, LI W, LIANG X, et al. MDCNN: multispectral pansharpening based on a multiscale dilated convolutional neural network[J]. Journal of Applied Remote Sensing, 2021, 15: 036516.
|
| [22] |
Benzenati T, Kessentini Y, Kallel A. Pansharpening approach via two-stream detail injection based on relativistic generative adversarial networks[J]. Expert Systems with Applications, 2022, 188: 115996. DOI: 10.1016/j.eswa.2021.115996
|
| [23] |
FU Y, XU T, WU X, et al. PPT Fusion: pyramid patch transformerfor a case study in image fusion[J]. arXiv preprint arXiv: 2107.13967, 2021.
|
| [24] |
GAO S H, CHENG M M, ZHAO K, et al. Res2Net: a new multi-scale backbone architecture[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(2): 652-662. DOI: 10.1109/TPAMI.2019.2938758
|
| [25] |
LIU Z, LIN Y, CAO Y, et al. Swin Transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021: 10-17.
|
| [26] |
Kingma D P, BA J. Adam: a method for stochastic optimization[J/OL]. arXiv: 1412.6980, 2014, https://doi.org/10.48550/arXiv.1412.6980.
|
| [27] |
ZHANG X, HUANG W, WANG Q, et al. SSR-NET: spatial–spectral reconstruction network for hyperspectral and multispectral image fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(7): 5953-5965. DOI: 10.1109/TGRS.2020.3018732
|
| [28] |
LIU X, LIU Q, WANG Y. Remote sensing image fusion based on two-stream fusion network[J]. Information Fusion, 2020, 55: 1-15. DOI: 10.1016/j.inffus.2019.07.010
|
| [29] |
XIE Q, ZHOU M, ZHAO Q, et al. MHF-Net: an interpretable deep network for multispectral and hyperspectral image fusion[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(3): 1457-1473. DOI: 10.1109/TPAMI.2020.3015691
|
| [30] |
HU J F, HUANG T Z, DENG L J, et al. Hyperspectral image super-resolution via deep spatiospectral attention convolutional neural networks [J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(12): 7251-7265. DOI: 10.1109/TNNLS.2021.3084682
|
| [31] |
HU J F, HUANG T Z, DENG L J, et al. Fusformer: a transformer-based fusion network for hyperspectral image super-resolution[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 1-5.
|
Supported by: Beijing Renhe Information Technology Co., Ltd. 
DownLoad: