Abstract: Existing deep learning-based methods for infrared and visible image fusion often overlook the modal differences of source images. This leads to insufficient extraction and utilization of the structural and complementary features of multisource images, resulting in insufficient edge details and weak contrast in fusion results. To address this issue, a dual-adversarial mechanism fusion method for infrared and visible-light images under manifold structure constraints is proposed. First, a dual-branch network was employed in the generator to extract features from the two source images separately. A cross-modal feature transfer module is introduced between the two branches to enhance the extraction of complementary and common features. Two discriminators and generator adversarial games were designed for different modal images to help the network learn more accurate representations of different modal features. Finally, the geometric constraint relationship between the source and fused images on the manifold surface was computed. A manifold constraint term was utilized to optimize the parameters of the generator network, maintaining the correspondence between the manifold structures of the source and fused images. This enhanced the consistency between structures and features of the source and fused images reduce the influence of modal differences on the fused image. Experimental results demonstrate that the proposed method achieves superior performance in multiple evaluation metrics compared to several typical fusion algorithms. The fused images not only possess better details and contrast, but also conform to human visual perception.