Infrared and Visible Image Fusion Based on Information Bottleneck Siamese Autoencoder Network

MA Luyao; LUO Xiaoqing; ZHANG Zhancheng

Volume 46 Issue 3

Mar. 2024

Turn off MathJax

Article Contents

Abstract

References

Infrared Technology > 2024 > 46(3): 314-324.

MA Luyao, LUO Xiaoqing, ZHANG Zhancheng. Infrared and Visible Image Fusion Based on Information Bottleneck Siamese Autoencoder Network[J]. Infrared Technology , 2024, 46(3): 314-324.

Citation:

MA Luyao, LUO Xiaoqing, ZHANG Zhancheng. Infrared and Visible Image Fusion Based on Information Bottleneck Siamese Autoencoder Network[J]. Infrared Technology , 2024, 46(3): 314-324.

Citation:

MA Luyao, LUO Xiaoqing, ZHANG Zhancheng. Infrared and Visible Image Fusion Based on Information Bottleneck Siamese Autoencoder Network[J]. Infrared Technology , 2024, 46(3): 314-324.

PDF (3267 KB)

Infrared and Visible Image Fusion Based on Information Bottleneck Siamese Autoencoder Network

MA Luyao^{1, 2, 3},
LUO Xiaoqing^{1, 2, 3, ,},
ZHANG Zhancheng⁴

1.
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China
2.
Institute of Advanced Technology, Jiangnan University, Wuxi 214122, China
3.
Jiangsu Laboratory of Pattern Recognition and Computational Intelligence, Wuxi 214122, China
4.
School of Electronics and Information Engineering, Suzhou University of Science and Technology, Suzhou 215000, China

More Information

Received Date: November 23, 2022
Revised Date: December 29, 2022

Graphical Abstract

Abstract

Abstract

Infrared and visible image fusion methods have problems such as insufficient information extraction, feature decoupling, and low interpretability. In order to fully extract and fuse the effective information of the source image, this paper proposes an infrared and visible image fusion method based on information bottleneck siamese autoencoder network (DIBF: Double Information Bottleneck Fusion). This method realizes the disentanglement of complementary features and redundant features by constructing an information bottleneck module on the twin branch. The expression process of complementary information corresponds to the feature fitting process of the first half of the information bottleneck. The compression process of redundant features corresponds to the feature compression process in the second half of the information bottleneck. This method cleverly expresses information extraction and fusion in image fusion as an information bottleneck trade-off problem, and achieves fusion by finding the optimal expression of information. In the information bottleneck module, the network obtains the information weight map of the feature through training, and uses the mean feature to compress the redundant features according to the information weight map. This method promotes the expression of complementary information through the loss function, and the two parts of compression and expression are balanced and optimized simultaneously. In this process, redundant information and complementary information are also decoupled. In the fusion stage, the information weight map is applied in the fusion rules, which improves the information richness of the fused images. Through subjective and objective experiments on the standard TNO dataset, compared with traditional and recent fusion methods, the results show that the method in this paper can effectively fuse useful information in infrared and visible images, and achieved good results on both visual perception and quantitative indicators.
- information bottleneck,
- Siamese,
- disentangled representations,
- infrared and visible,
- image fusion

FullText(HTML)

References (29)

References

[1]	张冬冬, 王春平, 付强. 深度学习框架下的红外与可见光图像融合算法综述[J]. 激光与红外, 2022, 52(9): 1288-1298. DOI: 10.3969/j.issn.1001-5078.2022.09.004 ZHANG D D, WANG C P, FU Q. Overview of infrared and visible image fusion algorithms based on deep learning framework[J]. Laser & Infrared, 2022, 52(9): 1288-1298. DOI: 10.3969/j.issn.1001-5078.2022.09.004
[2]	MA J, MA Y, LI C. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. DOI: 10.1016/j.inffus.2018.02.004
[3]	陈永, 张娇娇, 王镇. 多尺度密集连接注意力的红外与可见光图像融合[J]. 光学精密工程, 2022, 30(18): 2253-2266. CHEN Y, ZHANG J J, WANG Z. Infrared and visible image fusion based on multi-scale dense attention connection network[J]. Optics and Precision Engineering, 2022, 30(18): 2253-2266.
[4]	孙彬, 诸葛吴为, 高云翔, 等. 基于潜在低秩表示的红外和可见光图像融合[J]. 红外技术, 2022, 44(8): 853-862. http://hwjs.nvir.cn/article/id/7fc3a60d-61bb-454f-ad00-e925eeb54576 SUN B, ZHUGE W W, GAO Y X, et al. Infrared and visible lmage fusion based on latent low-rank representation[J]. Infrared Technology, 2022, 44(8): 853-862. http://hwjs.nvir.cn/article/id/7fc3a60d-61bb-454f-ad00-e925eeb54576
[5]	杨孙运, 奚峥皓, 王汉东, 等. 基于NSCT和最小化-局部平均梯度的图像融合[J]. 红外技术, 2021, 43(1): 13-20. http://hwjs.nvir.cn/article/id/144252d1-978c-4c1e-85ad-e0b8d5e03bf6 YANG S Y, XI Z H, WANG H D, et al. Image fusion based on NSCT and minimum-local mean gradient [J]. Infrared Technology, 2021, 43(1): 13-20. http://hwjs.nvir.cn/article/id/144252d1-978c-4c1e-85ad-e0b8d5e03bf6
[6]	刘智嘉, 贾鹏, 夏寅辉. 基于红外与可见光图像融合技术发展与性能评价[J]. 激光与红外, 2019, 49(5): 123-130. https://www.cnki.com.cn/Article/CJFDTOTAL-JGHW201905022.htm LIU Z J, JIA P, XIA Y H, et al. Development and performance evaluation of infrared and visual image fusion technology[J]. Laser & Infrared, 2019, 49(5): 123-130. https://www.cnki.com.cn/Article/CJFDTOTAL-JGHW201905022.htm
[7]	Lee H Y, Tseng H Y, Mao Q, et al. Drit++: Diverse image-to-image translation via disentangled representations[J]. International Journal of Computer Vision, 2020, 128(10): 2402-2417.
[8]	马梁, 苟于涛, 雷涛, 等. 基于多尺度特征融合的遥感图像小目标检测[J]. 光电工程, 2022, 49(4): 49-65. https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC202204005.htm MA L, GOU Y T, LEI T, et al. Small object detection based on multi-scale feature fusion using remote sensing images[J]. Opto-Electronic Engineering, 2022, 49(4): 49-65. https://www.cnki.com.cn/Article/CJFDTOTAL-GDGC202204005.htm
[9]	雷大江, 杜加浩, 张莉萍, 等. 联合多流融合和多尺度学习的卷积神经网络遥感图像融合方法[J]. 电子与信息学报, 2022, 44(1): 237-244. https://www.cnki.com.cn/Article/CJFDTOTAL-DZYX202201025.htm LEI D J, DU J H, ZHANG L P, et al. Multi-stream architecture and multi-scale convolutional neural network for remote sensing image fusion[J]. Journal of Electronics & Information Technology, 2022, 44(1): 237-244. https://www.cnki.com.cn/Article/CJFDTOTAL-DZYX202201025.htm
[10]	李明, 刘帆, 李婧芝. 结合卷积注意模块与卷积自编码器的细节注入遥感图像融合[J]. 光子学报, 2022, 51(6): 406-418. https://www.cnki.com.cn/Article/CJFDTOTAL-GZXB202206038.htm LI M, LIU F, LI J Z. Combining convolutional attention module and convolutional autoencoder for detail injection remote sensing image fusion[J]. Acta Photonica Sinica, 2022, 51(6): 406-418. https://www.cnki.com.cn/Article/CJFDTOTAL-GZXB202206038.htm
[11]	刘博, 韩广良, 罗惠元. 基于多尺度细节的孪生卷积神经网络图像融合算法[J]. 液晶与显示, 2021, 36(9): 1283-1293. https://www.cnki.com.cn/Article/CJFDTOTAL-YJYS202109009.htm LIU B, HAN G L, LUO H Y. Image fusion algorithm based on multi-scale detail siamese convolutional neural network[J]. Chinese Journal of Liquid Crystals and Displays, 2021, 36(9): 1283-1293. https://www.cnki.com.cn/Article/CJFDTOTAL-YJYS202109009.htm
[12]	Krishna V A, Reddy A A, Nagajyothi D. Signature recognition using siamese neural networks[C]//IEEE International Conference on Mobile Networks and Wireless Communications (ICMNWC), 2021: 1-4.
[13]	LI H, WU X J. DenseFuse: A fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623.
[14]	LI H, WU X J, Durrani T. NestFuse: An infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656. DOI: 10.1109/TIM.2020.3005230
[15]	LU B, CHEN J C, Chellappa R. Unsupervised domain-specific deblurring via disentangled representations[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 10225-10234.
[16]	WANG G, HAN H, SHAN S, et al. Cross-domain face presentation attack detection via multi-domain disentangled representation learning[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 6678-6687.
[17]	文载道, 王佳蕊, 王小旭, 等. 解耦表征学习综述[J]. 自动化学报, 2022, 48(2): 351-374. https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO202202003.htm WEN Z D, WANG J R, WANG X X, et al. A review of disentangled representation learning[J]. Acta Automatica Sinica, 2022, 48(2): 351-374. https://www.cnki.com.cn/Article/CJFDTOTAL-MOTO202202003.htm
[18]	ZHAO Z, XU S, ZHANG C, et al. DIDFuse: Deep image decomposition for infrared and visible image fusion[J]. arXiv preprint arXiv: 2003.09210, 2020.
[19]	XU H, WANG X, MA J. DRF: Disentangled representation for visible and infrared image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-13.
[20]	XU H, GONG M, TIAN X, et al. CUFD: An encoder–decoder network for visible and infrared image fusion based on common and unique feature decomposition[J]. Computer Vision and Image Understanding, 2022, 218: 103407. DOI: 10.1016/j.cviu.2022.103407
[21]	Tishby N, Pereira F C, Bialek W. The information bottleneck method[J]. arXiv preprint physics/0004057, 2000.
[22]	Tishby N, Zaslavsky N. Deep learning and the information bottleneck principle[C]// IEEE Information Theory Workshop (ITW). IEEE, 2015: 1-5.
[23]	Shwartz-Ziv R, Tishby N. Opening the black box of deep neural networks via information[J]. arXiv preprint arXiv: 1703.00810, 2017.
[24]	Alemi A A, Fischer I, Dillon J V, et al. Deep variational information bottleneck[J]. arXiv preprint arXiv: 1612.00410, 2016.
[25]	Tishby N, Zaslavsky N. Deep learning and the information bottleneck principle[C]//IEEE Information Theory Workshop (ITW). IEEE, 2015: 1-5.
[26]	ZHANG Y, LIU Y, SUN P, et al. IFCNN: A general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118. DOI: 10.1016/j.inffus.2019.07.011
[27]	MA J, CHEN C, LI C, et al. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31: 100-109. DOI: 10.1016/j.inffus.2016.02.001
[28]	ZHANG H, MA J. SDNet: A versatile squeeze-and-decomposition network for real-time image fusion[J]. International Journal of Computer Vision, 2021, 129(10): 2761-2785. DOI: 10.1007/s11263-021-01501-8
[29]	LIU Y, LIU S, WANG Z. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015, 24: 147-164. DOI: 10.1016/j.inffus.2014.09.004

Cited By

Get Citation

PDF

XML

Article views (117) PDF downloads (30)

Infrared and Visible Image Fusion Based on Information Bottleneck Siamese Autoencoder Network

Abstract

References

Catalog

Related

Infrared and Visible Image Fusion Based on Information Bottleneck Siamese Autoencoder Network

Abstract

References

Catalog

Related

Export File

Citation

Format

Content