全局-局部注意力引导的红外图像恢复算法

刘晓朋, 张涛

刘晓朋, 张涛. 全局-局部注意力引导的红外图像恢复算法[J]. 红外技术, 2024, 46(7): 791-801.
引用本文: 刘晓朋, 张涛. 全局-局部注意力引导的红外图像恢复算法[J]. 红外技术, 2024, 46(7): 791-801.
LIU Xiaopeng, ZHANG Tao. Global-Local Attention-Guided Reconstruction Network for Infrared Image[J]. Infrared Technology , 2024, 46(7): 791-801.
Citation: LIU Xiaopeng, ZHANG Tao. Global-Local Attention-Guided Reconstruction Network for Infrared Image[J]. Infrared Technology , 2024, 46(7): 791-801.

全局-局部注意力引导的红外图像恢复算法

基金项目: 

船舶总体性能创新研究开放基金项目 14422102

详细信息
    作者简介:

    刘晓朋(1998-),男,陕西汉中人,硕士研究生,主要从事深度学习,图像处理。E-mail: 6201910027@stu.jiangnan.edu.cn

  • 中图分类号: TP394.1

Global-Local Attention-Guided Reconstruction Network for Infrared Image

  • 摘要:

    针对真实世界的红外图像恢复算法中存在的图像模糊、纹理失真、参数过大等问题,提出了一种用于真实红外图像的全局-局部注意力引导的超分辨率重建算法。首先,设计了一种跨尺度的全局-局部特征融合模块,利用多尺度卷积和Transformer并行融合不同尺度的信息,并通过可学习因子引导全局和局部信息的有效融合。其次,提出了一种新颖的退化算法,即域随机化退化算法,以适应真实红外场景图像的退化域。最后,设计了一种新的混合损失函数,利用权重学习和正则化惩罚来增强网络的恢复能力,同时加快收敛速度。在经典退化图像和真实场景红外图像上的测试结果表明,与现有方法相比,该算法恢复的图像纹理更逼真,边界伪影更少,同时参数总数最多可减少20%。

    Abstract:

    To solve the problems of image blur smoothing, texture distortion, and excessively large parameters in real-world infrared-image recovery algorithms, a global-local attention-guided super-resolution reconstruction algorithm for infrared images is proposed. First, a cross-scale global-local feature fusion module utilizes multi-scale convolution and a transformer to fuse information at different scales in parallel and to guide the effective fusion of global and local information by learnable factors. Second, a novel domain randomization degradation model accommodates the degradation domain of real-world infrared images. Finally, a new hybrid loss based on weight learning and regularization penalty enhances the recovery capability of the network while speeding up convergence. Test results on classical degraded images and real-world infrared images show that, compared with existing methods, the images recovered by the proposed algorithm have more realistic textures and fewer boundary artifacts. Moreover, the total number of parameters can be reduced by up to 20%.

  • 图  1   比例因子为2的退化模型的示意图

    Figure  1.   Schematic illustration of the proposed degradation model for a scale factor of 2

    图  2   用于图像恢复的GLAGSR的架构

    Figure  2.   The architecture of the proposed GLAGSR for image restoration

    图  3   全局-局部特征融合块

    Figure  3.   Global-local feature fusion block

    图  4   多尺度卷积块(a)和分组残差GFF块(b), β是残差比例参数

    Figure  4.   The multi-scale convolution block (a)and the grouped residual GFF block (GR-GFF Block) (b), and β is the residual scaling parameter

    图  5   GLAGSR不同设置下的消融研究. (a)不同的GR-GFF数量块; (b)不同的LR图像块; (c)不同的数量块

    Figure  5.   Ablation study on different settings of GLAGSR. (a)Different GR-GFF block numbers; (b) Different Patch sizes; (c) Different block numbers

    图  6   鉴别器设计的消融研究

    Figure  6.   Ablation of two discriminator designs

    图  7   对于Urban100上的图像SR(×2),PSNR结果与不同方法的参数总数相比较

    Figure  7.   PSNR results vs the total number of parameters for different methods for image SR (×2) on Urban100

    图  8   超分辨率(×4)方法在红外图像上的视觉比较

    Figure  8.   Visual comparison of super-resolution (×4) methods on real-world infrared images

    表  1   GFF模块数量设计的消融研究表

    Table  1   Ablation study on GFF block number design

    GFF block number 2 3 4
    PSNR 32.60 32.68 32.64
    SSIM 0.8999 0.9010 0.9011
    下载: 导出CSV

    表  2   权重因子的消融研究表

    Table  2   Ablation study on weight factor

    Weight Factor w0: w1=1 w0: w1=2 w0: w1=0.5
    PSNR 32.15 32.68 32.04
    SSIM 0.8887 0.9012 0.8786
    下载: 导出CSV

    表  3   损失函数的消融研究表

    Table  3   Ablation study of the proposed hybrid loss

    Index Loss1 Loss2 Loss3 Loss4 Loss5
    L1 × × ×
    Lp × × ×
    Lg × × ×
    Ld × ×
    PSNR 32.62 32.58 32.53 32.51 32.68
    SSIM 0.9000 0.8994 0.8985 08987 0.9011
    下载: 导出CSV

    表  4   基准数据集上双三次退化图像的超分辨率性能(PSNR/SSIM)与最新方法的定量比较

    Table  4   Quantitative comparison of super-resolution performance (average PSNR/SSIM) with the state-of-the-art methods for bicubic degradation images on benchmark datasets

    Method Scale Training dataset Set5[10] Set14[8] BSD100[7] Urban100[7]
    PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM
    SRCNN[6] ×2 DIV2K 36.66 0.9542 32.45 0.9067 31.36 0.8879 29.50 0.8946
    EDSR[9] ×2 DIV2K 38.11 0.9602 33.92 0.9195 32.32 0.9013 32.93 0.9773
    RDN[14] ×2 DIV2K 38.24 0.9614 34.01 0.9212 32.34 0.9017 33.39 0.9353
    RCAN[22] ×2 DIV2K 38.27 0.9614 34.12 0.9216 32.41 0.9027 33.34 0.9384
    SAN[28] ×2 DIV2K 38.31 0.9620 34.07 0.9213 32.42 0.9028 33.10 0.9370
    HAN[23] ×2 DIV2K 38.27 0.9614 34.16 0.9217 32.42 0.9027 33.35 0.9385
    NLSA[2] ×2 DIV2K 38.34 0.9618 34.08 0.9231 32.43 0.9027 33.42 0.9394
    GLAGSR (Ours) ×2 DIV2K 38.37 0.9616 34.17 0.9221 32.48 0.9029 33.49 0.9395
    SRCNN[6] ×3 DIV2K 36.66 0.9542 32.45 0.9067 31.36 0.8879 29.50 0.8946
    EDSR[9] ×3 DIV2K 34.76 0.9290 30.66 0.8481 29.32 0.8104 29.02 0.8685
    RDN[14] ×3 DIV2K 34.58 0.9280 30.53 0.8447 29.23 0.8079 28.46 0.8582
    GLAGSR (Ours) ×3 DIV2K 34.90 0.9314 30.80 0.8498 29.40 0.8130 29.55 0.8751
    SRCNN[6] ×4 DIV2K 30.84 0.8628 27.50 0.7513 26.90 0.7101 24.52 0.7221
    EDSR[9] ×4 DIV2K 32.46 0.8968 28.80 0.7876 27.71 0.7420 26.64 0.8033
    RDN[14] ×4 DIV2K 32.47 0.8990 28.81 0.7871 27.72 0.7419 26.61 0.8028
    RCAN[22] ×4 DIV2K 32.63 0.9002 28.87 0.7889 27.77 0.7436 26.82 0.8087
    SAN[28] ×4 DIV2K 32.64 0.9003 28.92 0.7888 27.78 0.7436 26.79 0.8068
    HAN[23] ×4 DIV2K 32.64 0.9002 28.90 0.7890 27.80 0.7442 26.85 0.8094
    NLSA[2] ×4 DIV2K 32.59 0.9000 28.87 0.7891 27.78 0.7444 26.96 0.8109
    GLAGSR (Ours) ×4 DIV2K 32.80 0.9029 29.03 0.7928 27.89 0.7461 27.02 0.8135
    下载: 导出CSV
  • [1] 胡德敏, 闵天悦. 改进型轻量级GAN的红外图像超分辨率方法[J]. 小型微型计算机系统, 2022, 43(8): 1711-1717. https://www.cnki.com.cn/Article/CJFDTOTAL-XXWX202208021.htm

    HU Demin, MIN Tianyue. Infrared image super-resolution via improved lightweight GAN[J]. Journal of Chinese Computer Systems, 2022, 43(8): 1711-1717. https://www.cnki.com.cn/Article/CJFDTOTAL-XXWX202208021.htm

    [2]

    MEI Y, FAN Y, ZHOU Y. Image super-resolution with non-local sparse attention[C]//Proc. CVPR, 2021: 3517-3526. DOI: 10.1109/CVPR46437.2021.00352.

    [3] 马敏慧, 王红茹, 王佳. 基于改进的MSRCR-CLAHE融合的水下图像增强算法[J]. 红外技术, 2023, 45(1): 23-32. http://hwjs.nvir.cn/cn/article/id/69e5b90e-9c0c-43c6-b4e2-dedede3eb414

    MA Minhui, WANG Hongru, WANG Jia. An underwater image enhancement algorithm based on improved MSRCR-CLAHE fusion[J]. Infrared Technology, 2023, 45(1): 23-32. http://hwjs.nvir.cn/cn/article/id/69e5b90e-9c0c-43c6-b4e2-dedede3eb414

    [4]

    ZHANG D, SHAO J, LI X, et al. Remote sensing image super-resolution via mixed high-order attention network[J]. IEEE Trans. Geosci. Remote Sens. , 2021, 59(6): 5183-5196. DOI: 10.1109/TGRS.2020.3009918

    [5] 刘云峰, 赵洪山, 杨晋彪, 等. 基于GNR先验的电力设备热成像超分辨率方法[J]. 红外技术, 2023, 45(1): 40-48. http://hwjs.nvir.cn/cn/article/id/3f88d6d0-ab5c-4cd7-999e-b50ffce93699

    LIU Yunfeng, ZHAO Hongshan, YANG Jinbiao, et al. Super resolution method for power equipment infrared imaging based on gradient norm-ratio prior[J]. Infrared Technology, 2023, 45(1): 40-48. http://hwjs.nvir.cn/cn/article/id/3f88d6d0-ab5c-4cd7-999e-b50ffce93699

    [6]

    DONG C, Loy C, HE K, et al. Learning a deep convolutional network for image super-resolution[C]//Proc. Computer Vision, ECCV, 2014: 184-199. DOI: 10.1007/978-3-319-10593-2_13.

    [7] 聂丰英, 侯利霞, 万里勇. 自适应双边滤波与方向梯度的红外图像增强[J]. 红外技术, 2022, 44(12): 1309-1315. http://hwjs.nvir.cn/cn/article/id/8fbb7847-4550-463c-a220-9e97cd402968

    NIE Fengying, HOU Lixia, WAN Liyong. Infrared image enhancement based on adaptive bilateral filtering and directional gradient[J]. Infrared Technology, 2022, 44(12): 1309-1315. http://hwjs.nvir.cn/cn/article/id/8fbb7847-4550-463c-a220-9e97cd402968

    [8]

    Timofte R, Rothe R, Van Gool L. Seven ways to improve example-based single image super resolution[C]//Proc. CVPR, 2016: 1865-1873. DOI: 10.1109/CVPR.2016.206

    [9]

    Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proc. CVPRW, 2017: 136-144. Doi: 10.1109/CVPRW.2017.151

    [10]

    Bevilacqua M, Roumy A, Guillemot C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]// Proc. BMVC, 2012: 135-141. DOI: 10.5244/C.26.135.

    [11]

    TANG Y, GONG W, CHEN X, et al. Deep inception-residual Laplacian pyramid networks for accurate single-image super-resolution[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 31(5): 1514-1528.

    [12]

    CAI J, ZENG H, YONG H, et al. Toward real-world single image super-resolution: a new benchmark and a new model[C]//Proc. ICCV, 2019: 3086-3095. DOI: 10.1109/ICCV.2019.00318.

    [13]

    SHI W, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proc. CVPR, 2016: 1874-1883. DOI: 10.1109/CVPR.2016.207.

    [14]

    ZHANG Y, TIAN Y, KONG Y, et al. Residual dense network for image super-resolution[C]//Proc. CVPR, 2018: 2472-2481. DOI: 10.1109/CVPR.2018.00262.

    [15]

    Esser P, Rombach R, Ommer B. Taming transformers for high-resolution image synthesis[C]//Proc. CVPR, 2021: 12873-12883. DOI: 10.1109/CVPR46437.2021.01268.

    [16]

    JIANG Y, CHANG S, WANG Z. Transgan: two transformers can make one strong GAN[J/OL]. Computer Vision and Pattern Recognition, 2021, 34: https://arxiv.org/abs/2102.07074.

    [17]

    ZHANG K, LIANG J, Van Gool L, et al. Designing a practical degradation model for deep blind image super-resolution[C]//Proc. ICCV, 2021: 4771-4780. DOI: 10.1109/ICCV48922.2021.00475.

    [18]

    WANG X, XIE L, DONG C, et al. Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data[C]//Proc. ICCVW, 2021: 1905-1914. DOI: 10.1109/ICCVW54120.2021.00217

    [19]

    Ledig C, Theis L, Huszar F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proc. CVPR, 2017: 4681-4690. DOI: 10.1109/CVPR.2017.19.

    [20]

    ZHOU Y, WU G, FU Y, et al. Cross-MPI: cross-scale stereo for image super-resolution using multiplane images[C]//Proc. CVPR, 2021: 14842-14851. DOI: 10.1109/CVPR46437.2021.01460.

    [21] 烟台艾睿光电科技有限公司. 艾睿光电红外开源数据库[EB/OL]. [2023-02-26]. http://openai.raytrontek.com/apply/Super_resolution.html/.

    IRay Technology Co., Ltd. IRay Optoelectronic Infrared Open Source Database[EB/OL]. [2023-02-26]. http://openai.raytrontek.com/apply/Super_resolution.html/.

    [22]

    ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proc. ECCV, 2018: 286-301. DOI: 10.1007/978-3-030-01234-2_18.

    [23]

    NIU B, WEN W, REN W, et al. Single image super-resolution via a holistic attention network[C]//Proc. ECCV, 2020, 12357: 191-207.

    [24]

    WANG X, YU K, WU S, et al. ESRGAN: Enhanced super-resolution generative adversarial networks[C]// Proc. ECCV, 2019: 63-79. DOI: 10.1007/978-3-030-11021-5_5.

    [25]

    LIU Z, LIN Y, CAO Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]//Proc. ICCV, 2021: 9992-10002. DOI: 10.1109/ICCV48922.2021.00986.

    [26]

    WANG Y, WANG L, WANG H, et al. Resolution-aware network for image super-resolution[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 29(5): 1259-1269.

    [27]

    Andreas Lugmayr, Martin Danelljan, Radu Timofte. Unsupervised learning for real-world super-resolution[J]. ICCV Workshop, 2019, 2(3): 3408-3416.

    [28]

    DAI T, CAI J, ZHANG Y, et al. Second-order attention network for single image super-resolution[C]//Proc. CVPR, 2019: 11057-11066. DOI: 10.1109/CVPR.2019.01132.

    [29]

    Agustsson E, Timofte R. Ntire 2017 challenge on single image super-resolution: Dataset and study[C]// Proc. CVPRW, 2017: 126-135. DOI: 10.1109/CVPRW.2017.150.

    [30]

    LIANG J, CAO J, SUN G, et al. SwinIR: Image restoration using swin transformer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 1833-1844. DOI: 10.1109/ICCVW54120.2021.00210.

    [31]

    Blau Y, Mechrez R, Timofte R, et al. The 2018 PIRM challenge on perceptual image super-resolution[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018, DOI: 10.1007/978-3-030-11021-5_21

    [32]

    MA C, YANG C Y, YANG X, et al. Learning a no-reference quality metric for single-image super-resolution[J]. Computer Vision and Image Understanding, 2017, 158: 1-16. DOI: 10.1016/j.cviu.2016.12.009

  • 期刊类型引用(3)

    1. 吴丹,张艳. 多特征匹配的高光谱图像配准方法. 现代电子技术. 2021(21): 56-59 . 百度学术
    2. 史国川,拓浩男. SIFT和SURF算法在偏振图像匹配中的应用研究. 舰船电子工程. 2020(10): 93-95+148 . 百度学术
    3. 刘丁瑜,易加维,张徐洲,张畅,刘飞,方慧,何勇. 单子叶植物叶片双向反射分布的测量与分析. 光谱学与光谱分析. 2019(07): 2100-2106 . 百度学术

    其他类型引用(5)

图(8)  /  表(4)
计量
  • 文章访问数:  74
  • HTML全文浏览量:  19
  • PDF下载量:  39
  • 被引次数: 8
出版历程
  • 收稿日期:  2023-02-25
  • 修回日期:  2023-03-30
  • 网络出版日期:  2024-07-24
  • 刊出日期:  2024-07-19

目录

    /

    返回文章
    返回
    x 关闭 永久关闭

    尊敬的专家、作者、读者:

    端午节期间因系统维护,《红外技术》网站(hwjs.nvir.cn)将于2024年6月7日20:00-6月10日关闭。关闭期间,您将暂时无法访问《红外技术》网站和登录投审稿系统,给您带来不便敬请谅解!

    预计6月11日正常恢复《红外技术》网站及投审稿系统的服务。您如有任何问题,可发送邮件至编辑部邮箱(irtek@china.com)与我们联系。

    感谢您对本刊的支持!

    《红外技术》编辑部

    2024年6月6日