Bibliographic Details
| Title: |
Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretation |
| Authors: |
Lee, Changsun, Park, Sangjoon, Shin, Cheong-Il, Choi, Woo Hee, Park, Hyun Jeong, Lee, Jeong Eun, Ye, Jong Chul |
| Publication Year: |
2024 |
| Collection: |
Computer Science |
| Subject Terms: |
Electrical Engineering and Systems Science - Image and Video Processing, Computer Science - Computation and Language, Computer Science - Computer Vision and Pattern Recognition, Computer Science - Machine Learning |
| More Details: |
Recent medical vision-language models (VLMs) have shown promise in 2D medical image interpretation. However extending them to 3D medical imaging has been challenging due to computational complexities and data scarcity. Although a few recent VLMs specified for 3D medical imaging have emerged, all are limited to learning volumetric representation of a 3D medical image as a set of sub-volumetric features. Such process introduces overly correlated representations along the z-axis that neglect slice-specific clinical details, particularly for 3D medical images where adjacent slices have low redundancy. To address this limitation, we introduce MS-VLM that mimic radiologists' workflow in 3D medical image interpretation. Specifically, radiologists analyze 3D medical images by examining individual slices sequentially and synthesizing information across slices and views. Likewise, MS-VLM leverages self-supervised 2D transformer encoders to learn a volumetric representation that capture inter-slice dependencies from a sequence of slice-specific features. Unbound by sub-volumetric patchification, MS-VLM is capable of obtaining useful volumetric representations from 3D medical images with any slice length and from multiple images acquired from different planes and phases. We evaluate MS-VLM on publicly available chest CT dataset CT-RATE and in-house rectal MRI dataset. In both scenarios, MS-VLM surpasses existing methods in radiology report generation, producing more coherent and clinically relevant reports. These findings highlight the potential of MS-VLM to advance 3D medical image interpretation and improve the robustness of medical VLMs. |
| Document Type: |
Working Paper |
| Access URL: |
http://arxiv.org/abs/2412.13558 |
| Accession Number: |
edsarx.2412.13558 |
| Database: |
arXiv |
