In a recent investigation featured in the Radiology journal, radiologists have proven to outperform AI algorithms when it comes to discerning three prevalent lung diseases across a dataset of over 2,000 chest X-rays.
Radiologists have displayed superior accuracy in detecting three common lung diseases from chest X-rays, as unveiled in a study published in the Radiology journal. While AI tools exhibit sensitivity, they tend to generate more false positives, rendering them less dependable for autonomous diagnoses but valuable for offering second opinions.
Within the study, encompassing over 2,000 chest X-rays, radiologists demonstrated their prowess in precisely identifying the presence or absence of three prevalent lung diseases. This research was unveiled on September 26 in Radiology, a publication of the Radiological Society of North America (RSNA).
The Role of Radiography
“Chest radiography serves as a prevalent diagnostic tool, necessitating substantial training and experience for accurate interpretation,” highlighted Louis L. Plesner, M.D., the principal researcher and a resident radiologist and Ph.D. fellow in the Department of Radiology at Herlev and Gentofte Hospital in Copenhagen, Denmark.
While commercially available AI tools, approved by the FDA, are accessible to support radiologists, Dr. Plesner noted that the practical application of deep-learning-based AI tools in radiological diagnoses is still in its early stages.
“While AI tools are progressively receiving approval for use in radiology departments, there exists an unmet requirement to rigorously assess their performance in real-world clinical scenarios,” Dr. Plesner emphasized. “AI tools can aid radiologists in interpreting chest X-rays, but their real-world diagnostic accuracy remains uncertain.”
Study Findings
Dr. Plesner and a team of researchers conducted a comprehensive analysis, comparing the performance of four commercially available AI tools with a cohort of 72 radiologists in interpreting 2,040 consecutive adult chest X-rays collected over a two-year period at four Danish hospitals in 2020. The patient group had a median age of 72 years, and 32.8% of the chest X-rays displayed at least one target finding.
The chest X-rays underwent evaluation for three common findings: airspace disease, pneumothorax (collapsed lung), and pleural effusion (fluid buildup around the lungs).
AI tools achieved sensitivity rates ranging from 72% to 91% for airspace disease, 63% to 90% for pneumothorax, and 62% to 95% for pleural effusion.
“The AI tools exhibited moderate to high sensitivity, comparable to radiologists, in detecting airspace disease, pneumothorax, and pleural effusion on chest X-rays,” Dr. Plesner noted. “Nevertheless, they generated more false-positive results, predicting disease when none was present, compared to radiologists. Their performance also declined when multiple findings were present and for smaller targets.”
Comparative Predictive Values
Concerning pneumothorax, the positive predictive values—indicating the likelihood that patients with a positive screening test genuinely have the disease—ranged between 56% and 86% for AI systems, whereas radiologists achieved a 96% positive predictive value.
“AI exhibited its weakest performance in identifying airspace disease, with positive predictive values fluctuating between 40% and 50%,” Dr. Plesner observed. “In this challenging and elderly patient cohort, AI wrongly predicted the presence of airspace disease in 50% to 60% of cases. Such a rate is impractical for autonomous AI systems.”
Dr. Plesner underscored that radiologists aim to strike a balance between detecting and ruling out diseases, avoiding both substantial overlooked diseases and overdiagnosis.
“AI systems excel in disease detection but lag behind radiologists in affirming the absence of disease, particularly in intricate chest X-rays,” he stated. “Excessive false-positive diagnoses could lead to unnecessary imaging, radiation exposure, and heightened costs.”
Dr. Plesner mentioned that most studies tend to assess AI’s capacity to identify the presence or absence of a single disease, which is notably simpler than the multifaceted real-life scenarios where patients frequently present with multiple ailments.
“In many previous studies claiming AI’s superiority over radiologists, radiologists evaluated the image without access to the patient’s clinical history and prior imaging studies,” Dr. Plesner explained. “In everyday practice, a radiologist’s interpretation integrates these three data points. We speculate that the next generation of AI tools could substantially enhance their capabilities if they could also consider this synthesis, but such systems do not currently exist.”
Closing Remarks
“Our study underscores that radiologists generally outperform AI in real-world situations involving diverse patient profiles,” Dr. Plesner concluded. “While AI systems prove effective in recognizing normal chest X-rays, they should not be solely relied upon for making diagnoses.”
Dr. Plesner suggested that these AI tools could boost radiologists’ confidence in their diagnoses by offering a secondary assessment of chest X-rays.
Reference: “Commercially Available Chest Radiograph AI Tools for Detecting Airspace Disease, Pneumothorax, and Pleural Effusion” by Louis Lind Plesner, Felix C. Müller, Mathias W. Brejnebøl, Lene C. Laustrup, Finn Rasmussen, Olav W. Nielsen, Mikael Boesen, and Michael Brun Andersen, published on September 26, 2023, in Radiology.
DOI: 10.1148/radiol.231236
Table of Contents
Frequently Asked Questions (FAQs) about Radiologists Outperform AI in Diagnosing Lung Diseases
What were the key findings of the study comparing radiologists and AI in diagnosing lung diseases from chest X-rays?
The study found that radiologists outperformed AI algorithms in accurately identifying the presence or absence of three common lung diseases (airspace disease, pneumothorax, and pleural effusion) in a dataset of over 2,000 chest X-rays. While AI tools exhibited sensitivity, they generated more false-positive results compared to radiologists, particularly in complex cases.
What is the significance of the study’s findings for the field of radiology and healthcare?
The study highlights the importance of the human element in radiological diagnoses. Radiologists demonstrated their expertise in interpreting complex chest X-rays, not only in detecting diseases but also in ruling them out. This balance is crucial to avoid unnecessary imaging, radiation exposure, and increased costs. While AI tools have potential benefits, the study suggests that they should not replace radiologists but rather complement their work by providing a second opinion.
How did the study assess the performance of AI tools compared to radiologists?
The study compared the performance of four commercially available AI tools with 72 radiologists. They evaluated a diverse set of 2,040 chest X-rays collected over two years, assessing their sensitivity and positive predictive values for three common lung diseases. Sensitivity measures the ability to correctly identify true cases, while positive predictive value indicates the likelihood of true disease when a positive result is obtained.
What were the specific sensitivity and positive predictive value results for AI tools in the study?
For AI tools, sensitivity rates ranged from 72% to 91% for airspace disease, 63% to 90% for pneumothorax, and 62% to 95% for pleural effusion. However, positive predictive values varied with the disease. For pneumothorax, AI systems had values between 56% and 86%, while for airspace disease, positive predictive values ranged from 40% to 50%. Radiologists consistently achieved a higher positive predictive value, notably 96% for pneumothorax.
Are there limitations to the study’s findings?
One limitation is that the study primarily evaluated AI tools’ performance in single-disease scenarios, which may not reflect the complexity of real-life cases where patients often present with multiple ailments. Additionally, AI tools were assessed without access to patients’ clinical history and prior imaging studies, which can be crucial for radiological interpretation. The study emphasizes the need for AI tools to evolve to consider these factors.
What does the study recommend regarding the use of AI in radiology?
The study suggests that AI should not be used autonomously for making diagnoses in radiology, especially in situations involving diverse patient profiles and complex chest X-rays. Instead, AI tools can serve as valuable aids for radiologists by offering a secondary assessment of chest X-rays, potentially enhancing radiologists’ confidence in their diagnoses. The study underscores the importance of a collaborative approach between AI and radiologists in healthcare settings.
Is there any indication of future developments in AI tools for radiological diagnosis mentioned in the study?
The study speculates that future generations of AI tools could become significantly more powerful if they can incorporate patients’ clinical history and previous imaging studies into their analysis. However, as of now, such AI systems with this level of synthesis capability do not exist. This suggests a potential direction for the development of AI tools in radiology, aiming to better mimic the holistic approach of human radiologists.
More about Radiologists Outperform AI in Diagnosing Lung Diseases
-
Radiology Journal – The official journal where the study comparing radiologists and AI in diagnosing lung diseases was published.
-
Radiological Society of North America (RSNA) – The organization responsible for publishing the Radiology journal and promoting advancements in radiological science.
-
DOI: 10.1148/radiol.231236 – The digital object identifier (DOI) for the specific research paper discussed in the text.
-
AI in Healthcare – An article on the role of AI in healthcare, providing additional context for the study’s findings and implications.
-
Radiological Imaging – A resource discussing the importance of radiological imaging and the role of radiologists in healthcare.
8 comments
AI has potential, but radiologists r da real experts. Collaboration is key!
Great links provided. I’ll check out more about AI in healthcare & radiology. Thanks!
Imp. findings 4 healthcare! Radiologists & AI can work together 2 give patients da best care.
I wonder when we’ll c AI systems dat consider patient history & previous scans. Exciting possibilities!
wow, so radiologists beat AI in reading those lung xrays! dat’s sum good info, cud b a gamechanger.
da study highlights da complexity of medical imaging & da need 4 AI 2 evolve more. Maybe in da future, AI will b even bettr.
interesting study, shows radiologists r still very important, but AI can help as backup.
The study’s DOI is handy for referencing this research. Good to know.