Abstract: Multi-modal image fusion techniques aim to preserve the advantages of different modal images, such as texture details and prominent target subjects, for fused images. This study aimed to address the problems of insufficient extraction of cross-modal feature information, the complexity of cross-modal feature modeling, and the difficulty of processing shared information between different modalities that exist in the field of multi-modal image fusion. The properties of the Transformer and convolutional neural networks are combined, introducing two parallel branching networks. A new correlation-driven multi-modal feature-decomposition fusion network is proposed, accomplished by a two-stage training strategy. The first stage facilitates the extraction of cross-modal and shared features from infrared and visible images by modeling the inter-modal correlation and making full use of their information characteristics. In the second phase, a selective kernel network module was designed to adaptively adjust the weight assignment of different modal image features. Experimental results on three publicly available datasets show that the proposed method has advantages over other typical methods in quantitative and qualitative evaluations.