A research achievement from the Intelligent Medical Computing Laboratory (IMCL, also known as GEMLab) at Macao Polytechnic University has been accepted by IEEE Transactions on Circuits and Systems for Video Technology (TCSVT). The study, led by doctoral researcher Qiang Huang as first author with Tao Tan as corresponding author, presents HoloPercept, a unified model for understanding ultrasound images and videos across multiple organs and tasks.
Ultrasound is widely used for cardiac assessment, breast screening, and thyroid examination. A single clinical examination may include both static images and dynamic videos, while requiring different operations such as lesion classification and organ segmentation. Existing deep-learning systems are often designed for one organ, one data format, or one task at a time, which limits information sharing and creates repeated training and deployment work.
Dual prompts for shared yet task-aware representations
HoloPercept is built on a Swin-UNet backbone and uses two complementary prompt types to regulate shared features. Task-mode prompts describe what the model is currently doing and provide global control over the task and input modality. Anatomy-aware prompts describe what the model is currently viewing by introducing anatomical location and structural information into local attention. Together, the prompts help one shared network support different task requirements without losing a unified representation.
The model supports four tasks in a common architecture: image classification, image segmentation, video classification, and video segmentation. This design allows knowledge to be shared across image and video inputs while preserving task-specific behavior where it is needed.
Memory-enhanced temporal reasoning
For task decoding, the study introduces the Prompt-conditioned Memory-enhanced Temporal Perception Head (PMTP-Head). It establishes information exchange between classification and segmentation branches and aggregates temporal features for video data. Its memory-augmented segmentation mechanism retrieves related structural information by comparing the current frame with historical frames, then fuses that information with the current representation to improve temporal consistency across consecutive predictions.
A unified multi-task ultrasound benchmark
To evaluate unified modeling, the team developed the Universal Multi-Task Ultrasound (UMT-US) data system. It combines public and private clinical data across eight anatomical regions and includes more than 21,000 expert-annotated ultrasound samples in both image and video formats. Patient-level data splits ensure that samples from the same patient do not appear in the training, validation, and test sets simultaneously.
Results across image and video tasks
HoloPercept achieves Dice similarity coefficients of 86.61% ± 1.63% for image segmentation and 90.80% ± 1.06% for video segmentation. Its image and video classification accuracies reach 80.02% ± 0.97% and 86.78% ± 1.02%, respectively. These results demonstrate the ability of a unified architecture to model multiple organs, tasks, and ultrasound data formats within one system.
More broadly, the study investigates a shareable and transferable representation space across ultrasound tasks. By combining dual-prompt control, cross-task information exchange, and temporal memory, it offers a technical route toward reducing duplicated modeling and building integrated ultrasound AI systems for images and videos. Future work will explore longer temporal contexts, richer multimodal clinical data, and automatic prompt generation.
The acceptance of this paper by IEEE TCSVT marks another research advance by IMCL in medical artificial intelligence, ultrasound image analysis, and multi-task learning.
Paper: HoloPercept: A Dual-Prompt-Guided and Memory-Enhanced General Ultrasound Model for Image-Video Understanding
Authors: Qiang Huang and the IMCL research team, with Tao Tan as corresponding author.