Review of Polarization Image Fusion Based on Deep Learning
-
摘要: 偏振图像融合旨在通过光谱信息和偏振信息的结合改善图像整体质量，在图像增强、空间遥感、目标识别和军事国防等领域具有广泛应用。本文在回顾基于多尺度变换、稀疏表示和伪彩色等传统融合方法基础上，重点介绍基于深度学习的偏振图像融合方法研究现状。首先阐述基于卷积神经网络和生成对抗网络的偏振图像融合研究进展，然后给出在目标检测、语义分割、图像去雾和三维重建领域的相关应用，同时整理公开的高质量偏振图像数据集，最后对未来研究进行展望。Abstract: Polarization image fusion improves overall image quality by combining spectral and polarization information. It is used in different fields, such as image enhancement, spatial remote sensing, target identification and military defense. In this study, based on a review of traditional fusion methods using multi-scale transform, sparse representation, pseudo-coloration, etc. we focus on the current research status of polarization image fusion methods based on deep learning. First, the research progress of polarization image fusion based on convolutional neural networks and generative adversarial networks is presented. Next, related applications in target detection, semantic segmentation, image defogging, and three-dimensional reconstruction are described. Some publicly available high-quality polarization image datasets are collated. Finally, an outlook on future research is presented.
-
Keywords:
- polarization image /
- image fusion /
- deep learning
-
0. 引言
高光谱探测技术能够获取目标连续光谱信息，通过精确分析目标光谱特征能够识别目标的物化特性，进而区分不同的目标。这使得高光谱探测技术在物质检测领域具有巨大的潜力，近十余年涌现出大量利用高光谱探测对物质特性或杂质进行检测的实例。对于空基高光谱成像探测，以JPL研制的AVIRIS为开端，各国研制了多型机载成像光谱仪，目前在使用中的机载高光谱载荷有Hymap[1]、CASI-SASI[2]、APEX[3]、AVIRIS-NG[4]、AISA FENIX[5]以及我国的SITP AHSI[6]等，上述载荷的光谱范围通常为0.4~2.5 μm，光谱分辨率为5~15 nm，视场角28°~60°不等。由于这些载荷的总重量通常在150 kg以上，其中最轻的AISA FENIX重量也达到32 kg，功耗超过150 W，载荷必须搭载在具有三轴稳定平台的载重能力较大的飞机上。
随着无人机遥感技术的快速发展，受航线约束小，实验成本低廉，时间灵活的无人机平台成为机载高分辨率相机、激光雷达、多光谱、高光谱相机等载荷开展遥感模型及应用研究的重要平台。与之相应，高光谱载荷小型化需求也得到了快速提升，根据web of science对关键词UAV和hyperspectral的检索，每年相关文献的数量自2011年的6篇增长到2022年的175篇。其中，应用领域主要涉及卫星遥感数据的地表真实性检验和定标，农业遥感测量、生态环境监测、工业视觉等。为满足不断增长的市场需求，SPECIM[7-8]、Headwall[9-10]、Resonon[11-12]、HySpex[13-14]等公司均推出了适用于无人机平台的成像光谱仪，载荷指标如表 1所示。对比表 1中的9款无人机成像光谱仪，其光谱范围主要分为可见光-近红外谱段的0.4~1.0 μm和近红外-短波红外谱段的0.9~1.7 μm两种。除此之外，HySpex在SWIR-620上使用斯特林制冷型MCT探测器实现了0.97~2.5 μm谱段的高光谱成像。对比上述成像光谱仪的光谱分辨率，可见光谱段为1.9~10 nm，短波红外谱段为5.1~8 nm。可见光谱段的空间维像元数量多数为1000 pixel左右，少数可达1600 pixel，短波红外谱段为620~640 pixel。视场角以20°为主，最大可达38°。为适应无人机平台，不考虑前光学系统和数据采集终端，成像光谱仪的重量应小于4.5 kg，目前使用最普遍的FX10、Nano HyperSpec、Pika L、Pika XC2重量在0.64~2.51 kg之间。
表 1 常用无人机平台光谱仪Table 1. Commonly used UAV platform spectrometerManufacturers Project Spectral region/μm Spectral resolution /nm Pixel elements /pixel Field angle/° Mass/kg Specim FX10 0.4-1.0 5.5 b 1024 38c 1.26 d FX17 0.9-1.7 8 b 640 1.7 d Headwall Nano HyperSpec 0.4-1.0 6 b 640 17c 1.2 d Micro HyperSpec 0.4-1.0 5.8 b 1600 34c 3.9 d Resonon Pika L 0.4-1.0 3.3 b 900 13c 1.59 e Pika XC2 0.4-1.0 1.9 b 1600 23c 3.37 e Pika IR+ 0.9-1.7 5.6 b 640 21c 4.31 e HySpex Mjolnir V-1024 0.4-1.0 3 a 1024 20c 4 d Mjolnir S-620 0.97-2.5 5.1 a 620 4.5 d Note：a Spectral sampling interval. b Full width of spectral response function at half maximum. c FOV at common focal length.
d Lens and image acquisition system not included. e Including image acquisition system.尽管目前市场上已经有多个型号的无人机平台高光谱成像仪，但这些仪器在各领域的推广仍然面临各种问题。导致目前无人机应用主要采用多光谱成像仪，而高光谱成像仪主要用于科学研究[15]。根据F. Nex等的总结，目前无人机平台应用高光谱成像仪时仍存在一些问题，例如成本高、重量大、数据量大且不直观、探测效率低等；还需面临同时适应不同的应用需求，进行实时数据处理和分析，生产不同应用的数值产品等挑战[16]。
针对上述问题，本文将实时高光谱数据处理平台、紧凑型高通量分光系统、高速工业CMOS相机、像方光学系统、物方扫描机构相结合，设计研制了可同时满足工业视觉检测、植被监测、土地管理、地物光谱特性测量研究等多种需求的轻小型高光谱成像仪。成像仪光学系统和数据处理模块总重小于1.5 kg，结合扫描模块系统后总重小于3 kg，能够满足无人机。
1. 基本原理
1.1 系统总体结构设计
针对轻小型高光谱成像仪搭载与无人机平台的潜在应用需求，成像仪需要具备以下特点。首先仪器光谱分辨率需要与现有星载高光谱成像仪相当，参考目前在轨的GF-5 AHSI[17]、PRISMA[18]、HISUI[19]，设计光谱分辨率5 nm。光谱范围上，根据工业生产、农业及生态监测、土地管理、植被及相关地物光谱特性测量等应用方向的需求，设计为使用最广泛的0.4~1.0 μm[7, 9-11, 13]。空间分辨率上，针对工业生产需优于5 mm[20-22]，观测植被叶片需达到厘米量级[23-24]，土地管理中需分辨率优于0.5 m。视场角既关系到数据采集效率、光学系统的像质和复杂程度，又关系到数据采集中双向反射分布函数（Bidirectional Reflectance Distribution Function，BRDF）特性。综合考虑上述因素，同时参考表 1中无人机高光谱成像仪的视场角，设计载荷在推扫模式下的FOV（Field of View）为25°，摆扫模式下视场总FOV＞38°。数据处理层面，表 1中的高光谱载荷在进行无人机飞行任务中开展了数据预处理。例如Specim和Resonon仅提供了成像控制软件和配套的定标参数，后续的数据处理流程均在地面完成，其中Resonon的后处理软件提供了反射率反演和基于主成分的目标分类功能[25]。Headwall的航拍系统采用与Specim相同的思路，机上高光谱成像仪仅按预设的成像参数开展图像采集。但针对时效性更高的工业视觉应用，Headwall的最新型号MV.X将嵌入式计算平台与光谱仪相结合，实现了高光谱分类结果的实时生产。根据实际应用中对及时提取场景中的目标分布和光谱异常的需求，数据处理模块需要具备实时光谱异常提取和进行监督/非监督分类的功能和性能。
根据上述设计需求，轻小型高光谱成像仪的总体架构如图 1所示。前光学系统计划采用像方远心光路，分光系统采用Offner分光构型，从而最大程度降低大视场光谱成像的像差和畸变。探测器焦面及视频电路采用集成多路读出设计，配合自动开窗的读取方式以适配不同应用环境对帧频的要求。数据处理模块采用低功耗紧凑型板卡为硬件平台与视频电路、驱动电路和GNSS+IMU系统相连接。软件平台包含数据采集模块、预处理模块、数据反演模块、交互模块等，实现高光谱数据采集到信息提取的全链路处理。除此之外，为实现地物多角度光谱特征的采集和宽幅高光谱数据采集，系统配备扫描模块，模块通过数据处理模块驱动。
物方扫描机构采用传统的线型扫描方式。根据工作模式分为前向多角度测量、天底方向固定推扫、摆扫成像3种，如图 2所示。前向多角度测量模式通过扫描机构在沿轨方向调整多个观测天顶角，配合3~5条航带对地表形成上半球空间多个方位的BRDF光谱测量，如图 2(a)所示。天底方向固定推扫模式下，固定摆镜对地表进行线阵推扫光谱探测，如图 2(b)所示。摆扫成像模式通过扫描机构在穿轨方向进行40°视场角的线型扫描，如图 2(c)所示。
轻小型高光谱成像仪安装于无人机的稳定平台上，由于扫描过程中平台沿轨飞行，摆扫轨迹需要利用稳定平台进行扫描线校正，消除物方扫描的重叠和漏扫[26]。设计扫描模块及其与高光谱成像仪连接方式如图 3所示。
1.2 光学系统设计
根据1.1节的技术指标分析，光学系统FOV为25°，目前无人机实验中航高多设计为50~100 m[23-24]，幅宽对应为22~44 m。系统空间分辨率为1~2 cm。综合考虑成本、功耗及噪声特性，采用像元尺寸为5.5 μm的CMV4000ES探测器。为便于产品统型，采用复消色差的25 mm焦距像方远心镜头作为前光学系统。由于该探测器满阱电荷数较小，而主要目标植被的反射率差异较大。为确保信噪比，利用探测器光谱维的全部像元进行光谱探测，采用光谱维合并的方式提升探测的动态范围和信噪比。最终光学结构及其调制传递函数（modulation transfer function，MTF）分别如图 4、图 5所示，系统设计指标如表 2所示。
表 2 光学系统设计指标Table 2. Optical system design indexesParameters Indexes Spectrum range /μm 0.4~1.0 F number 2.4 Focal length /mm 25 FOV/° 25 Scanning angle /° -20~20 Spectral resolution /nm 5 Spectral sampling interval/nm 2.5 Slits size 11.3 mm×0.09 μm Diffraction order -1 MTF ≥0.45 1.3 数据采集与实时处理模块设计
根据轻小型高光谱成像仪的应用需求，成像仪的数据采集与实时处理模块需要同时具备以下功能：数据采集、预处理、数据反演以及相应的人机交互。模块总体架构如图 6所示。
![]()
图 6 数据采集与实时处理模块结构Figure 6. Structure diagram of data acquisition and real-time processing module1.3.1 数据采集子模块设计
为实现数据的实时处理，需要根据不同的应用需求针对每个功能中耗时较长的环节进行优化。首先，数据采集子模块充分利用探测器电路的多路读出功能，将探测模式由全谱段读取模式，改为全谱段读取模式和针对不同数值产品生产的多谱段选择读取模式两种。全谱段读取模式主要用于物质特性测量，数据采集时采用多路读出方式加速读取全部数据。多谱段选择读取模式用于工业视觉检测和大面积制图需要高速多光谱数据采集的场景。这种模式在视频电路层面根据预设的中心波长和波段宽度进行分区图像采集，通过在成像过程中筛选特定波长的方式减小数据量提升探测效率。在第三章的工业视觉检测实验中可见，此模式能实现超过2000行/s的超高速高光谱数据采集和处理。
1.3.2 数据预处理子模块设计
数据预处理子模块集成多种近地表遥感实验常用的算法以满足不同的遥感实验要求。为确保处理效率，辐射校正和大气校正算法都采用基于查找表的加速处理。对于辐射校正算法，根据谱段设置对定标数据进行预处理，使校正过程可以通过一次矩阵运算实现。对于大气校正，根据飞行航迹对应的实验条件确定部分大气校正参数，包括大气类型、气溶胶类型、照明-成像几何参数飞行高度对应的大气分层等。利用上述大气校正参数对水汽参数查找表、气溶胶参数查找表及大气校正光学参量查找表进行简化。最后在成像中实时计算大气光谱特征，利用查找表提取大气光学参量，求解地表反射率。
1.3.3 数据反演子模块设计
根据应用需求，目前数据反演子模块中集成了植被光谱指数算法、光谱异常提取算法和基于二维奇异谱分解的高光谱非监督分类算法。植被光谱指数算法用于对地表农作物及其他植被的分布和生长胁迫情况进行及时调查。其中集成目前生态研究和植被胁迫研究使用的各种植被光谱指数[27]，在获取植被冠层表观反射率后，根据指数的定义对每个像元进行光谱特征计算。光谱异常提取算法用于对工业检测中的产品质量进行在线检测，以及在其他应用场景中，进行弱小目标的实时提取。由于不同应用中场景的光谱特征有显著差异，光谱异常检测对不同应用场景所需的波段也不同。
实际应用中，需要首先利用全谱段采集模式获取的光谱数据进行波段选择。确定显著性波段后再利用光谱异常检测算法对由显著性波段组成的数据进行异常检测。具体方法如图 7所示，首先对全谱段光谱影像数据利用CTBS（constrained-target band selection）方法选取代表性波段子集，之后构建实时异常和目标检测算法RTAD（real time anomaly detection）和RTCEM（real time constrained energy minimization），利用异常检测的历史信息优化检测效率[28]。最后对RTAD和RTCEM的检测结果进行阈值分割和图形学处理提取异常目标位置。
基于二维奇异谱分解的高光谱非监督分类算法用于大范围制图。其主要思路为先从高分辨率的彩色图像中获取同质区域，再对同质区域提取最佳波段，构建低维特征子空间。最后再利用SVM算法在特征子空间对高光谱影像数据同时采用2DSSA进行空间特征增强，提升数据提取质量。最佳波段选择可采用PSO（particle swarm optimization）方法筛选，得到低维度的特征子空间。最后在维度更低的特征子空间进行非监督分类[29]。算法架构如图 8所示。
![]()
图 8 基于二维奇异谱分解的高光谱非监督分类算法架构Figure 8. Architecture of hyperspectral unsupervised classification algorithm based on 2DSSA1.3.4 系统性能分析
根据光学设计和探测器指标参数，核算不同太阳天顶角和地表反射特性下轻小型高光谱成像仪在全谱段读取模式下的信噪比。其中成像仪入瞳辐亮度数据采用Modtran6.0大气辐射传输模型[30]中如表 3所示的成像条件下仿真。系统透过率和光栅衍射效率以研制的Offner分光模块和前置光学系统实测值作为输入，探测器量子效率曲线及其噪声特性由厂商的研制手册提供。信噪比核算结果表明，积分时间4 ms条件下，探测器满阱占比最大值超过70%。不同场景下，探测数据的信噪比如图 9所示，在植被反射率红谷位置（670 nm附近）处优于100，在红边位置（720 nm附近）反射率优于230。满足植被光谱特性和工业生产中产品检测的探测需求。
表 3 不同场景下的成像条件Table 3. Imaging conditions in different scenesSurface reflection characteristic Solar zenith angle/° Forest 30 Soil 30 Albedo0.3 30 Albedo0.3 45 Albedo0.3 60 2. 样机研制及性能测试
根据前文的光学系统设计，本文进行了系统光机总结构设计及加工。完成系统装调后，集成数据处理模块的高光谱成像仪外观如图 10所示，重量小于1.1 kg。利用积分球、靶标和平行光管进行MTF测量，测试结果如图 11所示。利用对比度传递函数（contrast transfer function）CTF法计算系统总MTF约为0.19。利用激光器、积分球对系统进行扫频测试，描绘出0.4 μm、0.7 μm和1.0 μm三个波长的光谱分辨率，并拟合定标方程。典型波长的光谱响应如图 12所示，光谱分辨率为3.5~5.4 nm。
3. 工业视觉实验及结果
为验证系统对实时目标检测的能力，本文分别对流水线中的白硬塑、绿片、蓝片、长纸壳、黄片、木块、灰泡沫、灰纸壳、红片、麻绳等10种进行了多次检测。实验场景如图 13所示。结合天底方向固定推扫成像模式和多谱段选择读取模式，利用8个典型波长对流水线物料进行高速杂质提取。为适配700 mm/s的传送带速度，空间分辨率0.5 mm，采用1400 fps的采集速度。由于本实验中传送带的物料宽度有限，数据采集中进行开窗处理，仅对物料成像，图像宽度为600 pixel，每500帧图像拼接为1景数据。每景数据尺寸为500×600×8。此外，采用每类杂质检出准确率与无杂质误检率作为性能评估指标，即每类杂质检出数与该杂质图像数比值以及无杂质图像被检测为含杂质图像所占比率。
实验针对未知杂质和已知目标两种条件在两种算法下开展对比验证。针对未知杂质的条件，采用10种样本杂质，每种杂质样本数超过9个。为验证杂质检测准确率，采用杂质数据130幅，检出114幅，平均检测准确率87%。为验证误检率，采用无杂质数据655幅，误检62幅，误检率10%。表 4统计了各杂质类型的检测准确率，图 14展示了未知杂质类型时杂质图像的检测结果。针对已知杂质检测准确率的验证总共检测10种杂质，每种杂质图像超过9幅，采用无杂质图像数据730幅，结果如表 5所示。杂质图像平均检测准确率96%。无杂图像误检率7.59%。
表 4 未知杂质类型时各杂质检测准确率Table 4. The detection accuracy of each impurity when the impurity type is unknownImpurity number Impurity type Impurity image Textual algorithm RXD Detection accuracy rate Detection accuracy rate 1 White plastic 14 14 100% 14 100% 2 Green plastic 13 12 92% 11 85% 3 Blue plastic 11 9 82% 10 91% 4 Cardboard 10 10 100% 9 90% 5 Yellow plastic 14 9 64% 10 71% 6 Bits of wood 12 10 83% 9 75% 7 Froth 17 15 88% 13 76% 8 Grey cardboard 14 13 93% 13 93% 9 Red plastic 14 13 93% 14 100% 10 Hemp rope 11 9 82% 8 73% Average - - 87% - 85% 表 5 已知杂质类型时各杂质检测准确率Table 5. The detection accuracy of each impurity when the impurity type is knownImpurity number Impurity type Impurity image No impurity image Textual algorithm RXD algorithm Detection Accuracy rate False detection Rate for error detection Detection Accuracy rate False detection Rate for error detection 1 White plastic 10 730 10 100% 68 9.32% 10 100% 59 8.08% 2 Green plastic 10 730 10 100% 11 1.51% 10 100% 34 4.67% 3 Blue plastic 10 730 10 100% 66 9.04% 10 100% 63 8.63% 4 Cardboard 10 730 10 100% 72 9.86% 9 90% 68 9.32% 5 Yellow plastic 10 730 10 100% 10 1.37% 10 100% 27 3.70% 6 Bits of wood 10 730 10 100% 69 9.45% 10 100% 65 8.90% 7 Froth 9 731 9 100% 67 9.17% 9 100% 56 7.67% 8 Grey cardboard 10 730 10 100% 68 9.32% 10 100% 75 10.27% 9 Red plastic 10 730 10 100% 43 5.89% 10 100% 31 4.25% 10 Hemp rope 10 730 6 60% 80 10.96% 7 70% 95 13.01% - Average - - - 96% - 7.59% - 96% - 7.85% 本文使用了RXD算法对杂物同样进行了检测，未知杂质的情况下部分杂物检出率虽优于本文算法，但在总体的平均精度85%是低于本文算法的。对于无杂质图像，误检率为7.85%。综上所述，针对未知杂物场景，系统的杂质检测精度可达87%，误检率10%，通过统计每景数据的读取时间和检出时间。对于500×600×8数据，光谱异常检测时间为0.05 s。针对每景2000×2000×8的设计数据尺寸，检测速度能够达到0.67 s/景，能够适配轻小型高光谱成像仪2000 fps的图像采集速度。针对已知杂物的场景，目标检测时间为0.045 s，设计尺寸数据的检测速度可达0.6 s/景，检测精度可达96%，误检率达7.59%。工业视觉实验证明，本文中研制的轻小型高光谱成像仪集成了光谱异常检测功能，在天底方向固定推扫成像+多谱段选择读取的工作模式下，可以实现精度超过87%，误检率10%的光谱异常检测，能在光谱探测的过程中实时提取光谱异常目标。
4. 总结
为解决目前无人机平台高光谱成像仪重量大、数据量大且不直观、探测效率低等问题，使光谱仪满足多种应用需求，实时生产多种数值产品，本文提出了一种具有多种图像采集模式和数据读取模式的多模轻小型高光谱成像仪。成像仪采用扫描模块通过改变物方扫描方式满足目标光谱特性研究、生产检测、区域调查等多个场景对探测模式的需求，尤其实现了高分辨率宽视场的光谱成像需求。通过低畸变、高通量、紧凑型分光光学系统设计满足无人机平台对光谱成像仪的重量要求和探测精度要求，根据仿真和性能测试，系统典型场景下系统的SNR满足探测需求，MTF达到0.19，光谱分辨率3.5~5.4 nm。通过设计不同的数据采集模式应对不同应用场景对数据量的要求，提升数据在生产应用中的使用效率。在低功耗紧凑型板卡上开发贯穿数据采集、预处理、数值产品生产的全流程数据处理系统，实现在高光谱探测过程中实时生产数据产品的目的，全面提升高光谱探测效率。实验表明，系统能够实现每秒2048 pixel×2048 pixel场景的高精度光谱异常目标探测，探测精度优于87%。未来笔者将继续分光系统的小型化、数据模块的加速和探测功能扩展。
-
表 1 传统偏振图像融合方法对比
Table 1 Comparison of traditional polarization image fusion methods
Methods Specificities Advantages Shortcomings Multi-scale transformation Important visual information can be extracted at different scales and provide better spatial and frequency resolution. It is capable of extracting multi-scale details and structural information to effectively improve the quality of fused images. The determination of decomposition levels and the selection of fusion rules usually depend on manual experience. Sparse representation The method uses a linear subspace representation of training samples and is suitable for approximating similar objects. The method captures sparse features and highlights target details, retaining unique source image information. Dictionary training has some computational complexity and is more sensitive to noise and pseudo-features. Pulse coupled neural network It is composed of various neurons, including reception, modulation, and pulse generation, and it is suitable for real-time image processing. It can effectively detect the edge and texture features of the image, and the edge information fusion effect is relatively good. The implementation requires multiple iterative computations. It has high operational coupling, many parameters, and is time-consuming. Pseudo-color-based methods The method maps the gray levels of a black-and-white or monochrome image to a color space or assigns corresponding colors to different gray levels. Images of different bands can be mapped to pseudo-color space, thus visually representing multi-band information for easy observation and understanding. The main function is to colorize the image, which cannot extract and fuse more information, and the ability to retain detailed information is relatively weak. 
下载: 导出CSV
表 2 基于CNN和GAN的偏振图像融合算法对比
Table 2 Comparison of polarization image fusion algorithms based on CNN and GAN
Methods Specificities Advantages Shortcomings CNN The complexity of the algorithm depends on the coding method and the design of fusion rules. The CNN-based fusion network has better feature learning and representation ability, which makes it more suitable for information extraction and feature fusion processes. CNN can automatically learn image features and patterns, which can simplify the process of algorithm design and implementation and greatly improve accuracy. It is widely used in the process of feature extraction and representation. The problem of overfitting may occur when training on small sample datasets. It may not be sensitive to the detailed information in the image and easily lose the details in the fusion process. Networks with deeper layers usually require a lot of computational resources and time. GAN The fusion process is modeled as an adversarial game between the generator and the discriminator. Through continuous learning optimization, the fusion result of the generator converges with the target image in terms of probability distribution. The feature extraction, fusion, and image reconstruction processes can be realized implicitly. The adversarial learning mechanism of the generator and discriminator enhances the realism and overall quality of the image fusion, better preserving the details and structural features of the source image. Training is performed unsupervised and usually does not require large amounts of labeled data. The training process is relatively unstable. The design and tuning process is relatively complex and requires reasonable selection and adjustment of the network architecture and loss function. It may lead to artifacts or unnatural problems in the generated images in some specific scenarios.
下载: 导出CSV
表 3 偏振图像数据集
Table 3 Polarization image dataset
Source Waveband Year Quantity Resolution Reference[60] Visible band(Grayscale) 2019 120 1280×960 Reference[61] Visible band(RGB) 2020 40 1024×768 Reference[62-63] Long-wave infrared band 2020 2113 640×512 Reference[47] Visible band(RGB) 2021 394 1224×1024 Reference[64] Visible band(RGB) 2021 66 1848×2048 Reference[65] Visible band(RGB) 2021 40 1024×1024
下载: 导出CSV
-
[1] LI S, KANG X, FANG L, et al. Pixel-level image fusion: a survey of the state of the art[J]. Information Fusion, 2017, 33: 100-112. DOI: 10.1016/j.inffus.2016.05.004
[2] ZHANG H, XU H, TIAN X, et al. Image fusion meets deep learning: a survey and perspective[J]. Information Fusion, 2021, 76: 323-336. DOI: 10.1016/j.inffus.2021.06.008
[3] 罗海波, 张俊超, 盖兴琴, 等. 偏振成像技术的发展现状与展望(特邀)[J]. 红外与激光工程, 2022, 51(1): 101-110. LUO Haibo, ZHANG Junchao, GAI Xingqin, et al. Development status and prospect of polarization imaging technology (Invited)[J]. Infrared and Laser Engineering, 2022, 51(1): 101-110.
[4] 周强国, 黄志明, 周炜. 偏振成像技术的研究进展及应用[J]. 红外技术, 2021, 43(9): 817-828. http://hwjs.nvir.cn/article/id/76230e4e-2d34-4b1e-be97-88c5023050c6 ZHOU Qiangguo, HUANG Zhiming, ZHOU Wei. Research progress and application of polarization imaging technology[J]. Infrared Technology, 2021, 43(9): 817-828. http://hwjs.nvir.cn/article/id/76230e4e-2d34-4b1e-be97-88c5023050c6
[5] 段锦, 付强, 莫春和, 等. 国外偏振成像军事应用的研究进展(上)[J]. 红外技术, 2014, 36(3): 190-195. http://hwjs.nvir.cn/article/id/hwjs201403003 DUAN Jin, FU Qiang, MO Chunhe, et al. Review of polarization imaging technology for international military application(Ⅰ)[J]. Infrared Technology, 2014, 36(3): 190-195. http://hwjs.nvir.cn/article/id/hwjs201403003
[6] 莫春和, 段锦, 付强, 等. 国外偏振成像军事应用的研究进展(下)[J]. 红外技术, 2014, 36(4): 265-270. http://hwjs.nvir.cn/article/id/hwjs201404002 MO Chunhe, DUAN Jin, FU Qiang, et al. Review of polarization imaging technology for international military application(Ⅱ)[J]. Infrared Technology, 2014, 36(4): 265-270. http://hwjs.nvir.cn/article/id/hwjs201404002
[7] 王霞, 赵家碧, 孙晶, 等. 偏振图像融合技术综述[J]. 航天返回与遥感, 2021, 42(6): 9-21. WANG Xia, ZHAO Jiabi, SUN Jing, et al. Review of polarization image fusion technology[J]. Aerospace Return and Remote Sensing, 2021, 42(6): 9-21.
[8] LI X, YAN L, QI P, et al. Polarimetric imaging via deep learning: a review[J]. Remote Sensing, 2023, 15(6): 1540. DOI: 10.3390/rs15061540
[9] YANG Fengbao, DONG Anran, ZHANG Lei, et al. Infrared polarization image fusion based on combination of NSST and improved PCA[J]. Journal of Measurement Science and Instrumentation, 2016, 7(2): 176-184.
[10] 杨风暴, 董安冉, 张雷, 等. DWT, NSCT和改进PCA协同组合红外偏振图像融合[J]. 红外技术, 2017, 39(3): 201-208. http://hwjs.nvir.cn/article/id/hwjs201703001 YANG Fengbao, DONG Anran, ZHANG Lei, et al. Infrared polarization image fusion using the synergistic combination of DWT, NSCT and improved PCA[J]. Infrared Technology, 2017, 39(3): 201-208. http://hwjs.nvir.cn/article/id/hwjs201703001
[11] 沈薛晨, 刘钧, 高明. 基于小波-Contourlet变换的偏振图像融合算法[J]. 红外技术, 2020, 42(2): 182-189. http://hwjs.nvir.cn/article/id/hwjs202002013 SHEN Xuechen, LIU Jun, GAO Ming. Polarization image fusion algorithm based on Wavelet-Contourlet transform[J]. Infrared Technology, 2020, 42(2): 182-189. http://hwjs.nvir.cn/article/id/hwjs202002013
[12] 张雨晨, 李江勇. 基于小波变换的中波红外偏振图像融合[J]. 激光与红外, 2020, 50(5): 578-582. ZHANG Yuchen, LI Jiangyong. Polarization image fusion based on wavelet transform[J]. Laser & Infrared, 2020, 50(5): 578-582.
[13] 王策, 许素安. 基于Retinex和小波变换的水下偏振图像融合方法[J]. 应用激光, 2022, 42(8): 116-122. WANG Ce, XU Suan. Underwater polarization image fusion method based on Retinex and wavelet transform[J]. Applied Laser, 2022, 42(8): 116-122.
[14] 陈锦妮, 陈宇洋, 李云红, 等. 基于结构与分解的红外光强与偏振图像融合[J]. 红外技术, 2023, 45(3): 257-265. http://hwjs.nvir.cn/article/id/379e87a8-b9c0-4081-820c-ccd63f3fe4f0 CHEN Jinni, CHEN Yuyang, LI Yunhong, et al. Fusion of infrared intensity and polarized images based on structure and decomposition[J]. Infrared Technology, 2023, 45(3): 257-265. http://hwjs.nvir.cn/article/id/379e87a8-b9c0-4081-820c-ccd63f3fe4f0
[15] LIU Y, LIU S, WANG Z. A general framework for image fusion based on multiscale transform and sparse representation[J]. Information Fusion, 2015, 24(C): 147-164.
[16] 朱攀, 刘泽阳, 黄战华. 基于DTCWT和稀疏表示的红外偏振与光强图像融合[J]. 光子学报, 2017, 46(12): 207-215. ZHU Pan, LIU Zeyang, HUANG Zhanhua. Infrared polarization and intensity image fusion based on dual-tree complex wavelet transform and sparse representation[J]. Acta Photonica Sinica, 2017, 46(12): 207-215.
[17] ZHU P, LIU L, ZHOU X. Infrared polarization and intensity image fusion based on bivariate BEMD and sparse representation[J]. Multimedia Tools and Applications, 2021, 80(3): 4455-4471. DOI: 10.1007/s11042-020-09860-z
[18] ZHANG S, YAN Y, SU L, et al. Polarization image fusion algorithm based on improved PCNN[C]//Proceedings of SPIE-The International Society for Optical Engineering, 2013, 9045.
[19] 李世维, 黄丹飞, 王惠敏, 等. 基于BEMD和自适应PCNN的偏振图像融合[J]. 激光杂志, 2018, 39(3): 94-98. LI Shiwei, HUANG Danfei, WANG Huimin, et al. Polarization image fusion based on BEMD and adaptive PCNN[J]. Laser Journal, 2018, 39(3): 94-98.
[20] 于津强, 段锦, 陈伟民, 等. 基于NSST与自适应SPCNN的水下偏振图像融合[J]. 激光与光电子学进展, 2020, 57(6): 103-113. YU Jinqiang, DUAN Jin, CHEN Weimin, et al. Underwater polarization image fusion based on NSST and adaptive SPCNN[J]. Laser & Optoelectronics Progress, 2020, 57(6): 103-113.
[21] 叶松, 汤伟平, 孙晓兵, 等. 一种采用IHS空间表征偏振遥感图像的方法[J]. 遥感信息, 2006, 21(2): 11-13. YE Song, TANG Weiping, SUN Xiaobing, et al. Characterization of the polarized remote sensing images using IHS color system[J]. Remote Sensing Information, 2006, 21(2): 11-13.
[22] 赵永强, 潘泉, 张洪才. 自适应多波段偏振图像融合研究[J]. 光子学报, 2007, 36(7): 1356-1359. ZHAO Yongqiang, PAN Quan, ZHANG Hongcai. Research on adaptive multi-band polarization image fusion[J]. Acta Photonica Sinica, 2007, 36(7): 1356-1359.
[23] 赵永强, 潘泉, 张洪才. 一种新的全色图像与光谱图像融合方法研究[J]. 光子学报, 2007, 36(1): 180-183. ZHAO Yongqiang, PAN Quan, ZHANG Hongcai. A new spectral and panchromatic images fusion method[J]. Acta Photonica Sinica, 2007, 36(1): 180-183.
[24] 周浦城, 韩裕生, 薛模根, 等. 基于非负矩阵分解和IHS颜色模型的偏振图像融合方法[J]. 光子学报, 2010, 39(9): 1682-1687. ZHOU Pucheng, HAN Yusheng, XUE Menggen, et al. Polarization image fusion method based on non-negative matrix factorization and IHS color model[J]. Acta Photonica Sinica, 2010, 39(9): 1682-1687.
[25] 周浦城, 张洪坤, 薛模根. 基于颜色迁移和聚类分割的偏振图像融合方法[J]. 光子学报, 2011, 40(1): 149-153. ZHOU Pucheng, ZHANG Hongkun, XUE Mogen. Polarization image fusion method using color transfer and clustering-based segmentation[J]. Acta Photonica Sinica, 2011, 40(1): 149-153.
[26] 李伟伟, 杨风暴, 蔺素珍, 等. 红外偏振与红外光强图像的伪彩色融合研究[J]. 红外技术, 2012, 34(2): 109-113. DOI: 10.3969/j.issn.1001-8891.2012.02.010 LI Weiwei, YANG Fengbao, LIN Suzhen, et al. Study on pseudo-color fusion of infrared polarization and intensity image[J]. Infrared Technology, 2012, 34(2): 109-113. DOI: 10.3969/j.issn.1001-8891.2012.02.010
[27] 孙晶. 多波段偏振图像融合方法研究[D]. 北京: 北京理工大学, 2019. SUN Jing. Research on Multi-band Polarization Image Fusion Method[D]. Beijing: Beijing Institute of Technology, 2019.
[28] 苏子航. 多波段偏振图像信息校正与增强技术研究[D]. 北京: 北京理工大学, 2021. SU Zihang. Research on Multi-band Polarization Image Information Correction and Enhancement Technology[D]. Beijing: Beijing Institute of Technology, 2021.
[29] HU J, MOU L, Schmitt A, et al. FusioNet: a two-stream convolutional neural network for urban scene classification using PolSAR and hyperspectral data[C]//Proceedings of the 2017 Joint Urban Remote Sensing Event (JURSE), 2017: 1-4.
[30] ZHANG J, SHAO J, CHEN J, et al. PFNet: an unsupervised deep network for polarization image fusion[J]. Optics Letters, 2020, 45(6): 1507-1510. DOI: 10.1364/OL.384189
[31] WANG S, MENG J, ZHOU Y, et al. Polarization image fusion algorithm using NSCT and CNN[J]. Journal of Russian Laser Research, 2021, 42(4): 443-452. DOI: 10.1007/s10946-021-09981-2
[32] ZHANG J, SHAO J, CHEN J, et al. Polarization image fusion with self-learned fusion strategy[J]. Pattern Recognition, 2021, 118(22): 108045.
[33] XU H, SUN Y, MEI X, et al. Attention-Guided polarization image fusion using salient information distribution[J]. IEEE Transactions on Computational Imaging, 2022, 8: 1117-1130. DOI: 10.1109/TCI.2022.3228633
[34] 闫德利, 申冲, 王晨光, 等. 强度图像和偏振度图像融合网络的设计[J]. 光学精密工程, 2023, 31(8): 1256-1266. YAN Deli, SHEN Chong, WANG Chenguang, et al. Design of intensity image and polarization image fusion network[J]. Optics and Precision Engineering, 2023, 31(8): 1256-1266.
[35] Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems, 2014: 2672-2680.
[36] MA J, YU W, LIANG P, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26. DOI: 10.1016/j.inffus.2018.09.004
[37] ZHAO C, WANG T, LEI B, Medical image fusion method based on dense block and deep convolutional generative adversarial network[J]. Neural Comput. & Applic., 2021, 33: 6595-6610.
[38] LIU Q, ZHOU H, XU Q, et al. PSGAN: a generative adversarial network for remote sensing image pan-sharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(12): 10227-10242. DOI: 10.1109/TGRS.2020.3042974
[39] MA J, XU H, JIANG J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995. DOI: 10.1109/TIP.2020.2977573
[40] LI J, HUO H, LI C, et al. Attention FGAN: infrared and visible image fusion using attention-based generative adversarial networks[J]. IEEE Transactions on Multimedia, 2021, 23: 1383-1396. DOI: 10.1109/TMM.2020.2997127
[41] MA J, ZHANG H, SHAO Z, et al. GANMcC: a generative adversarial network with multi-classification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14.
[42] WEN Z, WU Q, LIU Z, et al. Polar-spatial feature fusion learning with variational generative-discriminative network for PolSAR classi-fication[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(11): 8914-8927. DOI: 10.1109/TGRS.2019.2923738
[43] DING X, WANG Y, FU X. Multi-polarization fusion generative adversarial networks for clear underwater imaging[J]. Optics and Lasers in Engineering, 2022, 152: 106971. DOI: 10.1016/j.optlaseng.2022.106971
[44] LIU J, DUAN J, HAO Y, et al. Semantic-guided polarization image fusion method based on a dual-discriminator GAN[J]. Optic Express, 2022, 30: 43601-43621. DOI: 10.1364/OE.472214
[45] SUN R, SUN X, CHEN F, et al. An artificial target detection method combining a polarimetric feature extractor with deep convolutional neural networks[J]. International Journal of Remote Sensing, 2020, 41: 4995-5009. DOI: 10.1080/01431161.2020.1727584
[46] ZHANG Y, Morel O, Blanchon M, et al. Exploration of deep learning based multimodal fusion for semantic road scene segmen-tation[C]//14th International Conference on Computer Vision Theory and Applications, 2019: 336-343.
[47] XIANG K, YANG K, WANG K. Polarization-driven semantic segmentation via efficient attention-bridged fusion[J]. Optic Express, 2021, 29: 4802-4820. DOI: 10.1364/OE.416130
[48] 霍永胜. 基于偏振的暗通道先验去雾[J]. 物理学报, 2022, 71(14): 112-120. HUO Yongsheng. Polarization-based research on a priori defogging of dark channel[J]. Acta Physica Sinica, 2022, 71(14): 112-120.
[49] 孟宇飞, 王晓玲, 刘畅, 等. 四分暗通道均值比较法的双角度偏振图像去雾[J]. 激光与光电子学进展, 2022, 59(4): 232-240. MENG Yufei, WANG Xiaoling, LIU Chang, et al. Dehazing of dual angle polarization image based on mean comparison of quartering dark channels[J]. Laser & Optoelectronics Progress, 2022, 59(4): 232-240.
[50] 张肃, 战俊彤, 付强, 等. 基于多小波融合的偏振探测去雾技术[J]. 激光与光电子学进展, 2018, 55(12): 468-477. ZHANG Su, ZHAN Juntong, FU Qiang, et al. Polarization detection defogging technology based on multi-wavelet fusion[J]. Laser & Optoelectronics Progress, 2018, 55(12): 468-477.
[51] HUANG F, KE C, WU X, et al. Polarization dehazing method based on spatial frequency division and fusion for a far-field and dense hazy image[J]. Applied Optics, 2021, 60: 9319-9332. DOI: 10.1364/AO.434886
[52] 周文舟, 范晨, 胡小平, 等. 多尺度奇异值分解的偏振图像融合去雾算法与实验[J]. 中国光学, 2021, 14(2): 298-306. ZHOU Wenzhou, FAN Chen, HU Xiaoping, et al. multi-scale singular value decomposition polarization image fusion defogging algorithm and experiment[J]. Chinese Optics, 2021, 14(2): 298-306.
[53] 李轩, 刘飞, 邵晓鹏. 偏振三维成像技术的原理和研究进展[J]. 红外与毫米波学报, 2021, 40(2): 248-262. LI Xuan, LIU Fei, SHAO Xiaopeng. Research progress on polarization 3D imaging technology[J]. Journal of Infrared and Millimeter Waves, 2021, 40(2): 248-262.
[54] 王霞, 赵雨薇, 金伟其. 融合光学偏振的三维成像技术进展(特邀)[J]. 光电技术应用, 2022, 37(5): 33-43. WANG Xia, ZHAO Yuwei, JIN Weiqi. Overview of polarization-based three-dimensional imaging techniques(Invited)[J]. Opto-electronic Technology Application, 2022, 37(5): 33-43.
[55] 杨锦发, 晏磊, 赵红颖, 等. 融合粗糙深度信息的低纹理物体偏振三维重建[J]. 红外与毫米波学报, 2019, 38(6): 819-827. YANG Jinfa, YAN Lei, ZHAO Hongying, et al. Shape from polarization of low-texture objects with rough depth information[J]. Journal of Infrared and Millimeter Waves, 2019, 38(6): 819-827.
[56] 张瑞华, 施柏鑫, 杨锦发, 等. 基于视差角和天顶角优化的偏振多视角三维重建[J]. 红外与毫米波学报, 2021, 40(1): 133-142. ZHANG Ruihua, SHI Baixin, YANG Jinfa, et al. Polarization multi-view 3D reconstruction based on parallax angle and zenith angle optimization[J]. Journal of Infrared and Millimeter Wave, 2021, 40(1): 133-142.
[57] BA Y, Gilbert A, WANG F, et al. Deep shape from polarization[C]//Computer Vision–ECCV 2020: 16th European Conference, 2020: 554-571.
[58] 陈创斌. 基于偏振信息的表面法线估计[D]. 广州: 广东工业大学, 2021. CHEN Chuangbin. Surface Normal Estimation Based on Polarization Information[D]. Guangzhou: Guangdong University of Technology, 2021.
[59] 王晓敏. 融合偏振和光场信息的低纹理目标三维重建算法研究[D]. 太原: 中北大学, 2022. WANG Xiaomin. Research on Low Texture Target 3D Reconstruction Algorithm Integrating Polarization and Light Field Information[D]. Taiyuan: North University of China, 2022.
[60] ZENG X, LUO Y, ZHAO X, et al. An end-to-end fully-convolutional neural network for division of focal plane sensors to reconstruct S0, DoLP, and AoP[J]. Optic Express, 2019, 27: 8566-8577. DOI: 10.1364/OE.27.008566
[61] Morimatsu M, Monno Y, Tanaka M, et al. Monochrome and color polarization demosaicking using edge-aware residual interpolation [C]//2020 IEEE International Conference on Image Processing(ICIP), 2020: 2571-2575.
[62] LI N, ZHAO Y, PAN Q, et al. Full-time monocular road detection using zero-distribution prior of angle of polarization[C]//European Conference on Computer Vision (ECCV), 2020: 457-473.
[63] LI N, ZHAO Y, PAN Q, et al. Illumination-invariant road detection and tracking using LWIR polarization characteristics[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 180: 357-369. DOI: 10.1016/j.isprsjprs.2021.08.022
[64] SUN Y, ZHANG J, LIANG R. Color polarization demosaicking by a convolutional neural network[J]. Optic Letter, 2021, 46: 4338-4341. DOI: 10.1364/OL.431919
[65] QIU S, FU Q, WANG C, et al. Linear polarization demosaicking for monochrome and colour polarization focal plane arrays[J]. Computer Graphics Forum, 2021, 40: 77-89. DOI: 10.1111/cgf.14204
计量
- 文章访问数: 790
- HTML全文浏览量: 90
- PDF下载量: 260
