Abstract: The infrared detection of small targets is a prerequisite for realizing automated infrared intelligent security, surveillance, and guidance systems. However, the infrared detection of small targets suffers from high false-positive and false-negative rates, as well as severe information loss. We investigated a lightweight infrared detection algorithm based on YOLOv5 to achieve real-time high-precision infrared detection of small targets. First, targeting the feature information of small infrared targets, a dual-detection-head lightweight network structure (E-YOLO) was developed to address the issue of target information disappearing during feature extraction. In addition, a feature-space-based filter pruning algorithm was used to calculate the similarity between filters to prune redundant filters and improve the inference speed. Finally, a cosPIoU-varifocal loss function was developed, which used an adaptive penalty factor based on the target size and combined angle balancing to balance sample and overlap region losses, while performing positive sample weighting, thereby addressing the issue of redundant anchor box interference. Finally, validation was conducted on the SIRST and IDSAT datasets, achieving mAP values of 96.6% and 98.2%, respectively. After compression, the computational complexity and model size of the algorithm could be reduced to 0.5 GFLOPs and 0.19 MB, respectively, with an inference time of less than 3 ms. The developed algorithm comprehensively improved the average detection accuracy, parameter quantity, and model size, and could satisfy the requirements of real-time detection.