Search

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in various multimodal tasks. However, their potential in the medical domain remains largely unexplored. A significant challenge arises from the scarcity of diverse medical images spanning various modalities and anatomical regions, which is essential in real-world medical applications. To solve this problem, in this paper, we introduce OmniMedVQA, a novel comprehensive medical Visual Question Answering (VQA) benchmark. This benchmark is collected from 73 different medical datasets, including 12 different modalities and covering more than 20 distinct anatomical regions. Importantly, all images in this benchmark are sourced from authentic medical scenarios, ensuring alignment with the requirements of the medical field and suitability for evaluating LVLMs. Through our extensive experiments, we have found that existing LVLMs struggle to address these medical VQA problems effectively. Moreover, what surprises us is that medical-specialized LVLMs even exhibit inferior performance to those general-domain models, calling for a more versatile and robust LVLM in the biomedical field. The evaluation results not only reveal the current limitations of LVLM in understanding real medical images but also highlight our dataset's significance. Our code with dataset are available at https://github.com/OpenGVLab/Multi-Modality-Arena.

OpenAI's latest large vision-language model (LVLM), GPT-4V(ision), has piqued considerable interest for its potential in medical applications. Despite its promise, recent studies and internal reviews highlight its underperformance in specialized medical tasks. This paper explores the boundary of GPT-4V's capabilities in medicine, particularly in processing complex imaging data from endoscopies, CT scans, and MRIs etc. Leveraging open-source datasets, we assessed its foundational competencies, identifying substantial areas for enhancement. Our research emphasizes prompt engineering, an often-underutilized strategy for improving AI responsiveness. Through iterative testing, we refined the model's prompts, significantly improving its interpretative accuracy and relevance in medical imaging. From our comprehensive evaluations, we distilled 10 effective prompt engineering techniques, each fortifying GPT-4V's medical acumen. These methodical enhancements facilitate more reliable, precise, and clinically valuable insights from GPT-4V, advancing its operability in critical healthcare environments. Our findings are pivotal for those employing AI in medicine, providing clear, actionable guidance on harnessing GPT-4V's full diagnostic potential.

Segment Anything Model (SAM) has achieved impressive results for natural image segmentation with input prompts such as points and bounding boxes. Its success largely owes to massive labeled training data. However, directly applying SAM to medical image segmentation cannot perform well because SAM lacks medical knowledge -- it does not use medical images for training. To incorporate medical knowledge into SAM, we introduce SA-Med2D-20M, a large-scale segmentation dataset of 2D medical images built upon numerous public and private datasets. It consists of 4.6 million 2D medical images and 19.7 million corresponding masks, covering almost the whole body and showing significant diversity. This paper describes all the datasets collected in SA-Med2D-20M and details how to process these datasets. Furthermore, comprehensive statistics of SA-Med2D-20M are presented to facilitate the better use of our dataset, which can help the researchers build medical vision foundation models or apply their models to downstream medical applications. We hope that the large scale and diversity of SA-Med2D-20M can be leveraged to develop medical artificial intelligence for enhancing diagnosis, medical image analysis, knowledge sharing, and education. The data with the redistribution license is publicly available at https://github.com/OpenGVLab/SAM-Med2D.

Although the Segment Anything Model (SAM) has demonstrated impressive performance in 2D natural image segmentation, its application to 3D volumetric medical images reveals significant shortcomings, namely suboptimal performance and unstable prediction, necessitating an excessive number of prompt points to attain the desired outcomes. These issues can hardly be addressed by fine-tuning SAM on medical data because the original 2D structure of SAM neglects 3D spatial information. In this paper, we introduce SAM-Med3D, the most comprehensive study to modify SAM for 3D medical images. Our approach is characterized by its comprehensiveness in two primary aspects: firstly, by comprehensively reformulating SAM to a thorough 3D architecture trained on a comprehensively processed large-scale volumetric medical dataset; and secondly, by providing a comprehensive evaluation of its performance. Specifically, we train SAM-Med3D with over 131K 3D masks and 247 categories. Our SAM-Med3D excels at capturing 3D spatial information, exhibiting competitive performance with significantly fewer prompt points than the top-performing fine-tuned SAM in the medical domain. We then evaluate its capabilities across 15 datasets and analyze it from multiple perspectives, including anatomical structures, modalities, targets, and generalization abilities. Our approach, compared with SAM, showcases pronouncedly enhanced efficiency and broad segmentation capabilities for 3D volumetric medical images. Our code is released at https://github.com/uni-medical/SAM-Med3D.

Although deep learning have revolutionized abdominal multi-organ segmentation, models often struggle with generalization due to training on small, specific datasets. With the recent emergence of large-scale datasets, some important questions arise: \textbf{Can models trained on these datasets generalize well on different ones? If yes/no, how to further improve their generalizability?} To address these questions, we introduce A-Eval, a benchmark for the cross-dataset Evaluation ('Eval') of Abdominal ('A') multi-organ segmentation. We employ training sets from four large-scale public datasets: FLARE22, AMOS, WORD, and TotalSegmentator, each providing extensive labels for abdominal multi-organ segmentation. For evaluation, we incorporate the validation sets from these datasets along with the training set from the BTCV dataset, forming a robust benchmark comprising five distinct datasets. We evaluate the generalizability of various models using the A-Eval benchmark, with a focus on diverse data usage scenarios: training on individual datasets independently, utilizing unlabeled data via pseudo-labeling, mixing different modalities, and joint training across all available datasets. Additionally, we explore the impact of model sizes on cross-dataset generalizability. Through these analyses, we underline the importance of effective data usage in enhancing models' generalization capabilities, offering valuable insights for assembling large-scale datasets and improving training strategies. The code and pre-trained models are available at \href{https://github.com/uni-medical/A-Eval}{https://github.com/uni-medical/A-Eval}.

The Segment Anything Model (SAM) represents a state-of-the-art research advancement in natural image segmentation, achieving impressive results with input prompts such as points and bounding boxes. However, our evaluation and recent research indicate that directly applying the pretrained SAM to medical image segmentation does not yield satisfactory performance. This limitation primarily arises from significant domain gap between natural images and medical images. To bridge this gap, we introduce SAM-Med2D, the most comprehensive studies on applying SAM to medical 2D images. Specifically, we first collect and curate approximately 4.6M images and 19.7M masks from public and private datasets, constructing a large-scale medical image segmentation dataset encompassing various modalities and objects. Then, we comprehensively fine-tune SAM on this dataset and turn it into SAM-Med2D. Unlike previous methods that only adopt bounding box or point prompts as interactive segmentation approach, we adapt SAM to medical image segmentation through more comprehensive prompts involving bounding boxes, points, and masks. We additionally fine-tune the encoder and decoder of the original SAM to obtain a well-performed SAM-Med2D, leading to the most comprehensive fine-tuning strategies to date. Finally, we conducted a comprehensive evaluation and analysis to investigate the performance of SAM-Med2D in medical image segmentation across various modalities, anatomical structures, and organs. Concurrently, we validated the generalization capability of SAM-Med2D on 9 datasets from MICCAI 2023 challenge. Overall, our approach demonstrated significantly superior performance and generalization capability compared to SAM.

Deep underground excavation of hard rock in a high-stress environment is prone to stress-induced brittle failure of the surrounding rock, usually as time-dependent spalling or slabbing of the sidewalls at a certain distance from the excavated face, seriously affecting construction safety and tunnel support. Triaxial testing of small-scale granite models with micro-camera recording of the damage under four different initial stress states showed spalling along the entire sidewall, the damage zone developing gradually in a horizontal direction towards the deeper part of the hole and finally forming a penetrating symmetrical arc-shaped groove along the horizontal direction, whereas the roof and bottom slab remained stable. The spalling process had four phases: quiescence, particle ejection at the upper and lower shoulders, sidewall crack propagation, and crack penetration spalling failure. The failure mode was static compared with the dynamic rockburst failure of a circular tunnel under the same 3-dimensional high-stress conditions, and the radial stress had a greater effect on failure than the axial stress; it was concluded that increasing the radial stress could improve stability by significantly reducing the degree of spalling failure., Deep underground excavation of hard rock in a high-stress environment is prone to stress-induced brittle failure of the surrounding rock, usually as time-dependent spalling or slabbing of the sidewalls at a certain distance from the excavated face, seriously affecting construction safety and tunnel support. Triaxial testing of small-scale granite models with micro-camera recording of the damage under four different initial stress states showed spalling along the entire sidewall, the damage zone developing gradually in a horizontal direction towards the deeper part of the hole and finally forming a penetrating symmetrical arc-shaped groove along the horizontal direction, whereas the roof and bottom slab remained stable. The spalling process had four phases: quiescence, particle ejection at the upper and lower shoulders, sidewall crack propagation, and crack penetration spalling failure. The failure mode was static compared with the dynamic rockburst failure of a circular tunnel under the same 3-dimensional high-stress conditions, and the radial stress had a greater effect on failure than the axial stress; it was concluded that increasing the radial stress could improve stability by significantly reducing the degree of spalling failure.

Tests using a 3D shaking table having a laminar frame overlay box and a geofoam buffer layer to study the mechanism and effect of seismic measures for a mountain tunnel employing a 1/40 scaled-down model based on a real tunnel (Huangcaoping highway tunnel No. 2, southwestern China) are described. Similarity relations and conditions between the test model and actual tunnel, seismic measure simulations, boundary conditions, sensor layout, modelling methods and ground motion input are discussed. Seismic measures of surrounding-rock reinforcement using anchors, increasing lining flexibility using steel wire mesh, setting flexible joints, and installing seismic insulation layer between reinforced surrounding rock and tunnel lining are presented. For a general mountain tunnel in hard-rock mass, the tunnel portal section from 0-50 m along the tunnel axis is suggested as the most important section for seismic reinforcement., Tests using a 3D shaking table having a laminar frame overlay box and a geofoam buffer layer to study the mechanism and effect of seismic measures for a mountain tunnel employing a 1/40 scaled-down model based on a real tunnel (Huangcaoping highway tunnel No. 2, southwestern China) are described. Similarity relations and conditions between the test model and actual tunnel, seismic measure simulations, boundary conditions, sensor layout, modelling methods and ground motion input are discussed. Seismic measures of surrounding-rock reinforcement using anchors, increasing lining flexibility using steel wire mesh, setting flexible joints, and installing seismic insulation layer between reinforced surrounding rock and tunnel lining are presented. For a general mountain tunnel in hard-rock mass, the tunnel portal section from 0-50 m along the tunnel axis is suggested as the most important section for seismic reinforcement.

The Mounigou highway tunnel in China crosses complex geological strata, with the surrounding rock mainly of poor quality. Failure in the form of collapses or squeezing deformation occurred on many occasions during tunnel construction. Deformation failure was most obvious in the soft and hard rock contact zone between phyllite and meta-sandstone. Field data and laboratory tests were used to develop a 3D-numerical model using the FLAC-3D software to investigate the stress and strain changes before and after excavation of the tunnel. Stress concentrations and differentiations occurred near the interface of the soft and hard rock strata after excavation. The stress transferred and was concentrated towards the hard rock stratum, while in the soft rock zone stress release occurred to some extent. The range of plastic deformation was larger in the soft rock than in the hard rock zone. Differences in principal stress and principal stress ratio increased after excavation. The increase in the difference in principal stress was larger than that of the principal stress ratio, and the principal stress increases were smaller in the soft rock than in the hard rock zone. The difference in principal stress increases and the far lower strength of the soft rock resulted in local, relatively high geo-stress effects in the soft rock, which was the main factor controlling the large deformation of soft rock in the tunnel. The large deformation in the soft and hard rock contact zone combined with deformation failure of the surrounding rock and damage to the supporting structure is attributed to a type of soft rock plastic-squeezing failure., The Mounigou highway tunnel in China crosses complex geological strata, with the surrounding rock mainly of poor quality. Failure in the form of collapses or squeezing deformation occurred on many occasions during tunnel construction. Deformation failure was most obvious in the soft and hard rock contact zone between phyllite and meta-sandstone. Field data and laboratory tests were used to develop a 3D-numerical model using the FLAC-3D software to investigate the stress and strain changes before and after excavation of the tunnel. Stress concentrations and differentiations occurred near the interface of the soft and hard rock strata after excavation. The stress transferred and was concentrated towards the hard rock stratum, while in the soft rock zone stress release occurred to some extent. The range of plastic deformation was larger in the soft rock than in the hard rock zone. Differences in principal stress and principal stress ratio increased after excavation. The increase in the difference in principal stress was larger than that of the principal stress ratio, and the principal stress increases were smaller in the soft rock than in the hard rock zone. The difference in principal stress increases and the far lower strength of the soft rock resulted in local, relatively high geo-stress effects in the soft rock, which was the main factor controlling the large deformation of soft rock in the tunnel. The large deformation in the soft and hard rock contact zone combined with deformation failure of the surrounding rock and damage to the supporting structure is attributed to a type of soft rock plastic-squeezing failure.

The tunnel is 4 400 m long, is at an altitude of about 3 400 m above sea level and has a buried depth of up to 1 000 m. It was excavated in carbon phyllite and slate. Large deformations occurred mainly within the phyllite, slate and a fault zone and resulted in subsidence of the tunnel arch, outward squeezing and buckling of the sidewalls and local collapse and deformation and fracture of the primary support structure. The maximum deformation was monitored to be 60 cm, with a maximum deformation rate of 39.3 mm/day, and 90% of the deformation occurred over a period of 60-120 days. A numerical model was developed to simulate the geo-stresses along the tunnel using FLAC-3D and the results showed that the major principal stress was about 25.5 MPa and could be divided into stress relaxation, increased stress and balanced stress zones. Three main deformation mechanisms were identified, plastic flow of soft rock, shear sliding of wedges and bending of thin-layered rock, while swelling of soft rock was found not to be a major factor., The tunnel is 4 400 m long, is at an altitude of about 3 400 m above sea level and has a buried depth of up to 1 000 m. It was excavated in carbon phyllite and slate. Large deformations occurred mainly within the phyllite, slate and a fault zone and resulted in subsidence of the tunnel arch, outward squeezing and buckling of the sidewalls and local collapse and deformation and fracture of the primary support structure. The maximum deformation was monitored to be 60 cm, with a maximum deformation rate of 39.3 mm/day, and 90% of the deformation occurred over a period of 60-120 days. A numerical model was developed to simulate the geo-stresses along the tunnel using FLAC-3D and the results showed that the major principal stress was about 25.5 MPa and could be divided into stress relaxation, increased stress and balanced stress zones. Three main deformation mechanisms were identified, plastic flow of soft rock, shear sliding of wedges and bending of thin-layered rock, while swelling of soft rock was found not to be a major factor.