Covered recently in the journal Nature Medicine, BiomedGPT is a new a new type of artificial intelligence (AI) designed to support a wide range of medical and scientific tasks. This new study, conducted in collaboration with multiple institutions, is described in the article as “the first open-source and lightweight vision-language foundation model, designed as a generalist capable of performing various biomedical tasks.”

“This work combines two types of AI into a decision support tool for medical providers,” explains Lichao Sun, an assistant professor of computer science and engineering at Lehigh University and a lead author of the study. “One side of the system is trained to understand biomedical images, and one is trained to understand and assess biomedical text. The combination of these allows the model to tackle a wide range of biomedical challenges, using insight gleaned from databases of biomedical imagery and from the analysis and synthesis of scientific and medical research reports.”

’16 state-of-the-art results’ for medical practitioners and patients

The key innovation described in the August 7 Nature Medicine article, “A generalist vision-language foundation model for diverse biomedical tasks,” is that this AI model doesn’t need to be specialized for each task. Typically, AI systems are trained for specific jobs, like recognizing tumors in X-rays or summarizing medical papers. However, this new model can handle many different tasks using the same underlying technology. This versatility makes it a “generalist” model?and a powerful new tool in the hands of medical providers.

“BiomedGPT is based on foundation models, a recent development in AI,” says Sun. “Foundation models are large, pre-trained AI systems that can be adapted to various tasks with minimal additional training. The generalist model described in the article has been trained on vast amounts of biomedical data, including images and text, enabling it to perform well across different applications.”

“By evaluating 25 datasets across 9 biomedical tasks and different modalities,” says Kai Zhang, a Lehigh PhD student advised by Sun who serves as first author of the Nature article, “BiomedGPT achieved 16 state-of-the-art results. A human evaluation of BiomedGPT on three radiology tasks showcased the model’s robust predictive abilities.”

Zhang says that he is proud that the open-source codebase is available for other researchers to use as a springboard to drive further development and adoption.

The team reports that the technology behind BiomedGPT may one day help doctors by interpreting complex medical images, assist researchers by analyzing scientific literature, or even aid in drug discovery by predicting how molecules behave.

“The potential impact of such technology is significant,” Zhang says, “as it could streamline many aspects of healthcare and research, making them faster and more accurate. Our method demonstrates that effective training with diverse data can lead to more practical biomedical AI for improving diagnosis and workflow efficiency.”

A team effort for clinical validation, and more

A crucial step in the process was validation of the model’s effectiveness and applicability in real-world healthcare settings.

“Clinical testing involves applying the AI model to real patient data to assess its accuracy, reliability, and safety,” Sun says. “This testing ensures that the model performs well across different scenarios. The outcomes of these tests helped refine the model, demonstrating its potential to improve clinical decision-making and patient care.”

Massachusetts General Hospital (MGH), a founding member of the Mass General Brigham healthcare system and teaching affiliate of Harvard Medical School, played a crucial role in the development and validation of the BiomedGPT model. The institution’s involvement primarily focused on providing clinical expertise and facilitating the evaluation of the model’s effectiveness in real-world healthcare settings. For instance, the model was tested with radiologists at MGH, where it demonstrated superior performance in tasks like visual question answering and radiology report generation. This collaboration helped ensure that the model was both accurate and practical for clinical use.

Other contributors to BiomedGPT include researchers from University of Georgia, Samsung Research America, University of Pennsylvania, Stanford University, University of Central Florida, UC-Santa Cruz, University of Texas-Health, Children’s Hospital of Philadelphia, and the Mayo Clinic.

“This research is highly interdisciplinary and collaborative,” says Sun. “The research involves expertise from multiple fields, including computer science, medicine, radiology, and biomedical engineering. Each author contributes specialized knowledge necessary to develop, test, and validate the model across various biomedical tasks. Large-scale projects like this often require access to diverse datasets and computational resources, along with access to skills in algorithm development, model training, evaluation, and application to real-world scenarios, as well as clinical testing and validation.

“This was a true team effort,” he says. “Creating something that can truly help the medical community improve patient outcomes across a wide range of issues is a very complex challenge. With such complexity, collaboration is key to creating impact through the application of science and engineering.”