Breast cancer has become the leading cause of cancer-related deaths among women. Breast cancer diagnosis, particularly the classification of pathological subtypes, remains a critical challenge in medical imaging. While advancements have been made, there is still room to enhance diagnostic accuracy through multi-modal approaches incorporating diverse clinical data sources.

We propose the 3MT-Net (Multi-modal Multi-task Network) as a novel deep learning architecture to address this challenge. This model leverages multi-modal data, including clinical data, B-mode ultrasound, and color Doppler ultrasound, to improve the classification of benign and malignant breast tumors and the identification of their pathological subtypes. By validating the model across multicenter datasets, 3MT-Net demonstrates significant improvements in diagnostic accuracy. This research contributes new insights and methodologies to the field of AI-assisted breast cancer diagnosis.