Review of Research on Low-Light Image Enhancement Algorithms
-
摘要:
低照度图像增强是图像处理领域的重要问题之一,近年来,深度学习技术的迅速发展为低照度图像增强提供了新的解决方案,且具有广阔的应用前景。首先,全面分析了低照度图像增强领域的研究现状与挑战,并介绍了传统方法及其优缺点。其次,重点讨论了基于深度学习的低照度图像增强算法,根据学习策略的不同将其分为五类,分别对这些算法的原理、网络结构、解决问题进行了详细的阐述,并按时间顺序将近6年基于深度学习的图像增强代表算法进行了对比分析。接着,归纳了当前主流的数据集与评价指标,并从感知相似度和算法性能两个方面对深度学习算法进行测试评估。最后,对低照度图像增强领域改进方向与今后研究作了总结与展望。
Abstract:Low-light image enhancement is an important problem in the field of image processing. The rapid development of deep learning technology provides a new solution for low-light image enhancement and has broad application prospects. First, the current research status and challenges in the field of low-light image enhancement are comprehensively analyzed, and traditional methods and their advantages and disadvantages are introduced. Second, deep learning-based low-light image enhancement algorithms are classified into five categories according to their different learning strategies, and the principles, network structures, and problem-solving capabilities of these algorithms are explained in detail. Third, representative deep learning-based image enhancement algorithms from the last six years are compared and analyzed in chronological order. Fourth, the current mainstream datasets and evaluation indexes are summarized, and the deep learning algorithms are tested and evaluated in terms of perceived similarity and algorithm performance. Finally, directions for improvement and future research in the field of low-light image enhancement are discussed and suggested.
-
Keywords:
- low-light images /
- image enhancement /
- deep learning /
- image processing /
- low-light dataset
-
-
表 1 传统低照度图像增强算法对比
Table 1 Comparison of conventional low-light image enhancement algorithms
Algorithms Key technology Advantage Disadvantage Histogram Equalization[2-6] Changing the histogram distribution of image grey scale intervals Enhanced brightness distribution and sharpness of images Grey scale merge, detail information lost Tone Mapping Algorithm[7-9] Convert high dynamic range images to low dynamic range images Improved image brightness and colour performance Limitations in image display quality Fusion-based Algorithm[10-12] Fusion of multiple image processing results Improved brightness and clarity has advantages Generates fusion artefacts and introduces image distortion Defogging-based Algorithm[13-16] Improve low-light image quality by removing haze effect from images Restore image detail and improve visual quality Partial loss of information, introduction of artefacts, noise Retinex Theory[17-20] Decomposition of the image into reflective and illuminated parts Improved colour balance and detail in images Complex calculations, more light-dependent 表 2 基于深度学习的低照度图像增强方法总结
Table 2 Summary of deep learning based low illumination image enhancement methods
Year Methodology Improvement Advantage Disadvantage 2019 EnlightenGAN[26] Attention-guided U-Net, an image-enhanced GAN network Processes and generates highly realistic multi-type images Introduces artefacts, amplifies noise; not stable enough ExcNet[31] Excludes backlighting and automatically adjusts brightness and contrast Automatically restores detail to backlit images; no need for extensive annotation data Subject to scene limitations, complex scenes may be affected; long training time 2020 Zero-DCE[32] Introduction of contrast loss function and feature network DCE-Net No need for a reference image; enhance the image while retaining detailed information Requires high input image quality; high training costs RRDNet[34] Retinex decomposes residual and salient images Zero sample learning; image details and color saturation are well preserved Relies on accurate reflection and light decomposition; requires high image quality DRBN[37] Recursive banding ensures detail recovery; band reconstruction enhances image Good reconstruction of detail enhancement results using paired and unpaired data training Simple experimental setup, not conducted on wider dataset 2021 Zero-DCE++[33] Accelerated and lightweight version of Zero-DCE Fast inference speed while maintaining performance Enhanced images appear over-enhanced or distorted TBEFN[25] Two-branch structure; adaptive residual block and attention module Handles exposure fusion and image enhancement simultaneously Requires large amount of labelled data; not effective in extreme lighting conditions RUAS[38] Constructing lightweight image enhancement based on Retinex theory Fast and requires few computational resources; very effective Performance and computational efficiency need further improvement 2022 LEDNet[39] Combined low light enhancement and dark de-blurring Handles both low light and blur; new dataset LOL-Blur The dataset does not contain other low-light environments and shooting scenes LEES-Net[27] Attention to mechanisms for positioning and dynamic adjustment Good generalization ability, robustness and visual effects Long training time and lack of real-time capability; artifacts and oversmoothing issues D2HNet[40] Joint denoising and deblurring using hierarchical networks High-quality image restoration for short and long duration shots Training data quality needs to be improved 2023 Literature[41] Introduction of a new framework for appearance and structural modelling Structural features guide appearance enhancements, producing lifelike results Simple model structure; no real-time enhancement Noise2Code[42] Projection based image denoising network algorithm Implementing joint training of denoising networks and VQGAN models Applications have some limitations; adaptability to complex degradation models NeRCo[28] Control of adaptable fitting functions to unify scene degradation factors Semantic Oriented Supervision for Improving Perceptual Friendliness and Robustness A priori limitations in text images; validity not verified in a wider range of scenarios DecNet[43] Decomposition and Tuning Networks and Self-Supervised Fine-Tuning Strategies Efficient and lightweight network with no manual tuning for good performance Degradation problems in images taken in real low-light conditions 2024 Multi-Channel Retinex[44] Multi-Channel Retinex Image Enhancement Network, Design Initialization Module Split channel enhancement is used to solve the problem of color deviation Higher model complexity, requires paired data training Retinexmamba[45] Integration of Retinexformer learning framework and introduction of light fusion state space models Improve the processing speed by using state space model and maintain the image quality during enhancement Lack of some non-reference image quality evaluation metrics LYT-NET[46] YUV separates luminance and chrominance and introduces a blending loss function Simplified light-color information separation reduces model complexity Model generalization ability to be verified, limited comparison methods 表 3 主流低照度图像增强数据集总结
Table 3 Comparison of conventional low-light image enhancement algorithms
Dataset No. of images Image formats Paired/unpaired Real/Synthetic SICE[47] 4413 RGB Paired Real LOL[48] 500 RGB Paired Real VE-LOL[49] 2500 RGB Paired Real+ Synthetic MIT-Adobe FiveK[50] 5000 RAW Paired Real SID[51] 5094 RAW Paired Real SMOID[52] 179 RAW Paired Real LIME[53] 10 RGB unpaired Real ExDARK[54] 7363 RGB unpaired Real 表 4 基于深度学习的低照度图像增强方法性能对比
Table 4 Performance comparison of deep learning based low illumination image enhancement methods
Year Methodology Network framework Dataset Image formats Evaluation metrics Operational framework PSNR/dB 2019 EnlightenGAN[26] U-Net NPE/MEF/LIME/DICM etc. RGB NIQE PyTorch 17.48 ExcNet[31] CNN IEpsD RGB CDIQA/LOD PyTorch 17.25 2020 Zero-DCE[32] U-Net type network SICE RGB PSNR/SSIM/MAE PyTorch 14.86 RRDNet[34] Retinex Breakdown NPE/MEF/LIME/DICM etc. RGB NIQE/CPCQI PyTorch 14.38 DRBN[37] Recursive network LOL RGB PSNR/SSIM PyTorch 20.13 2021 Zero-DCE++[33] U-Net type network SICE RGB PSNR/SSIM/MAE etc. PyTorch 16.42 TBEFN[25] U-Net type network LOL RGB PSNR/SSIM/NIQE TensorFlow 17.14 RUAS[38] Retinex Breakdown LOL/MIT-AdobeFiveK RGB PSNR/SSIM/LPIPS PyTorch 20.6 2022 LEDNet[39] Neural network LOL-Blur RGB PSNR/SSIM/LPIPS PyTorch 23.86 LEES-Net[27] CNN LOL-v2/LSRW Dark Face RGB PSNR/SSIM/LPIPS/LOE PyTorch 20.2 D2HNet[40] Pyramid network D2(synthetic dataset) RGB PSNR/SSIM/PR PyTorch 26.67 2023 Literature [41] End-to-end network LOL/SID RGB PSNR/SSIM PyTorch 24.62 Noise2Code[42] GAN model SIDD/DND RGB PSNR/SSIM PyTorch — NeRCo[28] Implicit network LOL/LIME/LSRW RGB PSNR/SSIM/NIQE/LOE PyTorch 19.84 DecNet[43] Retinex Breakdown LOL/NPE/MIT-AdobeFiveK RGB PSNR/SSIM/NIQE/LOE PyTorch 22.82 2024 Multi-Channel Retinex[44] Retinex Breakdown LOL/MIT-AdobeFiveK RGB PSNR/SSIM FSIM PyTorch 21.94 Retinexmamba[45] Retinex+Mamba LOL-v1 LOLv2-real RGB PSNR/SSIM/RMSE PyTorch 22.453 LYT-NET[46] YUV Transformer LOLv1/LOLv2-real/LOL-v2-syn YUV PSNR/SSIM Tensorflow 22.38 -
[1] LIU J, XU D, YANG W, et al. Benchmarking low-light image enhancement and beyond[J]. International Journal of Computer Vision, 2021, 129: 1153-1184. DOI: 10.1007/s11263-020-01418-8
[2] 郭永坤, 朱彦陈, 刘莉萍, 等. 空频域图像增强方法研究综述[J]. 计算机工程与应用, 2022, 58(11): 23-32. GUO Y K, ZHU Y C, LIU L P, et al. A review of research on image enhancement methods in the air-frequency domain[J]. Computer Engineering and Application, 2022, 58(11): 23-32.
[3] Jebadass J R, Balasubramaniam P. Low contrast enhancement technique for color images using intervalvalued intuitionistic fuzzy sets with contrast limited adaptive histogram equalization[J]. Soft Computing, 2022, 26(10): 4949-4960. DOI: 10.1007/s00500-021-06539-x
[4] 杨嘉能, 李华, 田宸玮, 等. 基于自适应校正的动态直方图均衡算法[J]. 计算机工程与设计, 2021, 42(5): 1264-1270. YANG J N, LI H, TIAN C W, et al. Dynamic histogram equalization algorithm based on adaptive correction[J]. Computer Engineering and Design, 2021, 42(5): 1264-1270.
[5] KUO C F J, WU H C. Gaussian probability bi‐histogram equalization for enhancement of the pathological features in medical images[J]. International Journal of Imaging Systems and Technology, 2019, 29(2): 132-145. DOI: 10.1002/ima.22307
[6] LI C, LIU J, ZHU J, et al. Mine image enhancement using adaptive bilateral gamma adjustment and double plateaus histogram equalization[J]. Multimedia Tools and Applications, 2022, 81(9): 12643-12660. DOI: 10.1007/s11042-022-12407-z
[7] Nguyen N H, Vo T V, Lee C. Human visual system model-based optimized tone mapping of high dynamic range images[J]. IEEE Access, 2021, 9: 127343-127355. DOI: 10.1109/ACCESS.2021.3112046
[8] 陈迎春. 基于色调映射的快速低照度图像增强[J]. 计算机工程与应用, 2020, 56(9): 234-239. CHEN Y C. Fast low-light image enhancement based on tone mapping[J]. Computer Engineering and Applications, 2020, 56(9): 234-239.
[9] 赵海法, 朱荣, 杜长青, 等. 全局色调映射和局部对比度处理相结合的图像增强算法[J]. 武汉大学学报, 2020, 66(6): 597-604. ZHAO H F, ZHU R, DU C Q, et al. An image enhancement algorithm combining global tone mapping and local contrast processing[J]. Journal of Wuhan University, 2020, 66(6): 597-604.
[10] 李明悦, 晏涛, 井花花, 等. 多尺度特征融合的低照度光场图像增强算法[J]. 计算机科学与探索, 2022, 17(8): 1904-1916. LI M Y, YAN T, JING H H, et al. Multi-scale feature fusion algorithm for low illumination light field image enhancement[J]. Computer Science and Exploration, 2022, 17(8): 1904-1916.
[11] 张微微. 基于图像融合的低照度水下图像增强[D]. 大连: 大连海洋大学, 2023. ZHANG W W. Low Illumination Underwater Image Enhancement Based on Image Fusion[D]. Dalian: Dalian Ocean University, 2023.
[12] 田子建, 吴佳奇, 张文琪, 等. 基于Transformer和自适应特征融合的矿井低照度图像亮度提升和细节增强方法[J]. 煤炭科学技术, 2024, 52(1): 297-310. TIAN Z J, WU J Q, ZHANG W Q, et al. Brightness enhancement and detail enhancement method for low illumination images of mines based on Transformer and adaptive feature fusion[J]. Coal Science and Technology, 2024, 52(1): 297-310.
[13] DONG X, PANG Y, WEN J, et al. Fast efficient algorithm for enhancement of low lighting video[C]//2011 IEEE International Conference on Multimedia and Expo, 2011: 1-6.
[14] HUO Y S. Polarization-based research on a priori defogging of dark channel[J]. Acta Physica Sinica, 2022, 71(14): 144202. DOI: 10.7498/aps.71.20220332
[15] HONG S, KIM M, LEE H, et al. Nighttime single image dehazing based on the structural patch decomposition[J]. IEEE Access, 2021, 9: 82070-82082. DOI: 10.1109/ACCESS.2021.3086191
[16] SI Y, YANG F, CHONG N. A novel method for single nighttime image haze removal based on gray space[J]. Multimedia Tools and Applications, 2022, 81(30): 43467-43484. DOI: 10.1007/s11042-022-13237-9
[17] LAND E H. The retinex theory of color vision[J]. Scientific American, 1977, 237(6): 108-129. DOI: 10.1038/scientificamerican1277-108
[18] WANG R, ZHANG Q, FU C W, et al. Underexposed photo enhancement using deep illumination estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 6849-6857.
[19] CAI Y, BIAN H, LIN J, et al. Retinexformer: one-stage retinex-based transformer for low-light image enhancement[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023: 12504-12513.
[20] REN X, YANG W, CHENG W H, et al. LR3M: robust low-light enhancement via low-rank regularized retinex model[J]. IEEE Transactions on Image Processing, 2020, 29: 5862-5876. DOI: 10.1109/TIP.2020.2984098
[21] Lore K G, Akintayo A, Sarkar S. LLNet: a deep autoencoder approach to natural low-light image enhancement[J]. Pattern Recognition, 2017, 61: 650-662. DOI: 10.1016/j.patcog.2016.06.008
[22] ZHANG Y, ZHANG J, GUO X. Kindling the darkness: a practical low-light image enhancer[C]//Proceedings of the 27th ACM International Conference on Multimedia, 2019: 1632-1640.
[23] ZHANG Y, GUO X, MA J, et al. Beyond brightening low-light images[J]. International Journal of Computer Vision, 2021: 1013-1037.
[24] LI C, GUO J, PORIKLI F, et al. LightenNet: a convolutional neural network for weakly illuminated image enhancement[J]. Pattern Recognition Letters, 2018, 104: 15-22. DOI: 10.1016/j.patrec.2018.01.010
[25] LU K, ZHANG L. TBEFN: a two-branch exposure-fusion network for low-light image enhancement[J]. IEEE Transactions on Multimedia, 2021, 23: 4093-4105. DOI: 10.1109/TMM.2020.3037526
[26] JIANG Y, GONG X, LIU D, et al. EnlightenGAN: deep light enhancement without paired supervision[J]. IEEE Transactions on Image Processing, 2021, 30: 2340-2349. DOI: 10.1109/TIP.2021.3051462
[27] LI X, HE R, WU J, et al. LEES-Net: fast, lightweight unsupervised curve estimation network for low-light image enhancement and exposure suppression[J]. Displays, 2023, 80: 102550. DOI: 10.1016/j.displa.2023.102550
[28] YANG S, DING M, WU Y, et al. Implicit neural representation for cooperative low-light image enhancement[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023: 12918-12927.
[29] YU R, LIU W, ZHANG Y, et al. Deepexposure: learning to expose photos with asynchronously reinforced adversarial learning[J]. Advances in Neural Information Processing Systems, 2018, 31: 7429-7439.
[30] 周腾威. 基于深度学习的图像增强算法研究[D]. 南京: 南京信息工程大学, 2021. ZHOU T W. Research on Image Enhancement Algorithm Based on Deep Learning[D]. Nanjing: Nanjing University of Information Engineering, 2021.
[31] ZHANG L, ZHANG L, LIU X, et al. Zero-shot restoration of back-lit images using deep internal learning[C]//Proceedings of the 27th ACM International Conference on Multimedia, 2019: 1623-1631.
[32] GUO C, LI C, GUO J, et al. Zero-reference deep curve estimation for low-light image enhancement[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 1780-1789.
[33] LI C, GUO C, CHEN C L. Learning to enhance low-light image via zero-reference deep curve estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(8): 4225-4238.
[34] ZHU A, ZHANG L, SHEN Y, et al. Zero-shot restoration of underexposed images via robust retinex decomposition[C]//2020 IEEE International Conference on Multimedia and Expo (ICME), 2020, DOI: 10.1109/ICME46284.2020.9102962.
[35] SOHN K, BERTHELOT D, CARLINI N, et al. Fixmatch: simplifying semi-supervised learning with consistency and confidence[J]. Advances in Neural Information Processing Systems, 2020, 33: 596-608.
[36] LIU Y, TIAN Y, CHEN Y, et al. Perturbed and strict mean teachers for semi-supervised semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 4258-4267.
[37] YANG W, WANG S, FANG Y, et al. From fidelity to perceptual quality: a semi-supervised approach for low-light image enhancement [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 3063-3072.
[38] LIU R, MA L, ZHANG J, et al. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 10561-10570.
[39] ZHOU S, LI C, CHANGE LOY C. LEDNet: joint low-light enhancement and deblurring in the dark[C]//European Conference on Computer Vision, 2022: 573-589.
[40] ZHAO Y, XU Y, YAN Q, et al. D2hnet: Joint denoising and deblurring with hierarchical network for robust night image restoration [C]//European Conference on Computer Vision, 2022: 91-110.
[41] XU X, WANG R, LU J. Low-Light Image Enhancement via Structure Modeling and Guidance[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 9893-9903.
[42] Cheikh Sidiyadiya A. Generative prior for unsupervised image restoration[D]. Ahmed Cheikh Sidiya: West Virginia University, 2023.
[43] LIU X, XIE Q, ZHAO Q, et al. Low-light image enhancement by retinex-based algorithm unrolling and adjustment[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 35(11): 2162-2388.
[44] 张箴, 鹿阳, 苏奕铭, 等. 基于多通道Retinex模型的低照度图像增强网络[J]. 信息与控制, 2024, 53(5): 652-661. ZHANG Z, LU Y, SU Y M, et al. Low-light image enhancement network based on multi-channel Retinex model[J]. Information and Control, 2024, 53(5): 652-661.
[45] BAI J, YIN Y, HE Q. Retinexmamba: retinex-based mamba for low-light image enhancement[J]. arXiv preprint arXiv: 2405.03349, 2024.
[46] Brateanu A, Balmez R, Avram A, et al. Lyt-net: lightweight yuv transformer-based network for low-light image enhancement[J]. arXiv preprint arXiv: 2401.15204, 2024.
[47] CAI J, GU S, ZHANG L. Learning a deep single image contrast enhancer from multi-exposure images[J]. IEEE Transactions on Image Processing, 2018, 27(4): 2049-2062.
[48] WEI C, WANG W, YANG W, et al. Deep retinex decomposition for low-light enhancement[J]. arXiv preprint arXiv: 1808.04560, 2018.
[49] LIU J, XU D, YANG W, et al. Benchmarking low-light image enhancement and beyond[J]. International Journal of Computer Vision, 2021, 129: 1153-1184.
[50] Bychkovsky V, Paris S, CHAN E, et al. Learning photographic global tonal adjustment with a database of input/output image pairs[C]//CVPR of IEEE, 2011: 97-104.
[51] CHEN C, CHEN Q, XU J, et al. Learning to see in the dark[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 3291-3300.
[52] JIANG H, ZHENG Y. Learning to see moving objects in the dark[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 7324-7333.
[53] GUO X, LI Y, LING H. LIME: Low-light image enhancement via Illumination Map Estimation[J]. IEEE Transactions on Image Processing, 2017, 26(2): 982-993.
[54] LOH Y P, CHAN C S. Getting to know low-light images with the exclusively dark dataset[J]. Computer Vision and Image Understanding, 2019, 178: 30-42.
[55] SARA U, AKTER M, UDDIN M S. Image quality assessment through FSIM, SSIM, MSE and PSNR——a comparative study[J]. Journal of Computer and Communications, 2019, 7(3): 8-18.
[56] Mittal A, Soundararajan R, Bovik A C. Making a "Completely Blind" image quality analyzer[J]. IEEE Signal Processing Letters, 2013, 20(3): 209-212.
[57] ZHANG R, Isola P, Efros A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 586-595.
[58] HU S, YAN J, DENG D. Contextual information aided generative adversarial network for low-light image enhancement[J]. Electronics, 2021, 11(1): 32.
[59] YANG S, ZHOU D, CAO J, et al. Rethinking low-light enhancement via transformer-GAN[J]. IEEE Signal Processing Letters, 2022, 29: 1082-1086.
[60] PAN Z, YUAN F, LEI J, et al. MIEGAN: mobile image enhancement via a multi-module cascade neural network[J]. IEEE Transactions on Multimedia, 2022, 24: 519-533.
[61] CHEN X, LI J, HUA Z. Retinex low-light image enhancement network based on attention mechanism[J]. Multimedia Tools and Applications, 2023, 82(3): 4235-4255.
[62] ZHANG Q, ZOU C, SHAO M, et al. A single-stage unsupervised denoising low-illumination enhancement network based on swin-transformer[J]. IEEE Access, 2023, 11: 75696-75706.
[63] YE J, FU C, CAO Z, et al. Tracker meets night: a transformer enhancer for UAV tracking[J]. IEEE Robotics and Automation Letters, 2022, 7(2): 3866-3873.
[64] Kanev A, Nazarov M, Uskov D, et al. Research of different neural network architectures for audio and video denoising[C]//2023 5th International Youth Conference on Radio Electronics, Electrical and Power Engineering (REEPE) of IEEE, 2023, 5: 1-5.
[65] FENG X, LI J, HUA Z. Low-light image enhancement algorithm based on an atmospheric physical model[J]. Multimedia Tools and Applications, 2020, 79(43): 32973-32997.
[66] JIA D, YANG J. A multi-scale image enhancement algorithm based on deep learning and illumination compensation[J]. Traitement du Signal, 2022, 39(1): 179-185.
-
期刊类型引用(2)
1. 赵恒谦,李坤恒,郭丽,康志娟,王雪飞,郝云昊,金倩. 基于便携式地物光谱仪的常见宝玉石材质反射率光谱特征分析. 矿物岩石. 2021(02): 1-12 . 百度学术
2. 史云胜,刘秉琦,杨兴. 石墨平台微结构的纳米级红外光谱表征. 红外技术. 2016(11): 914-919 . 本站查看
其他类型引用(5)