sub-circles-media

MEDIA

Media and Related Feeds

Jul 19, 2021

Digital Health Dilemma: Artificial Intelligence Device Approvals

By: Alexander Fogel MD, MBA

Artificial Intelligence is believed by many to have the potential to revolutionize medicine, but a recent study highlights issues related to AI device approval.

How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals.

In any nascent field, the development of best practices is critical to ensure the reliability, reproducibility and safety of new devices. A recent analysis by Stanford researchers reviewed the evaluation process for all 130 FDA-approved medical AI devices. The authors found that 126 of 130 such devices underwent only retrospective studies, and that none of the 54 “high-risk” devices were tested in prospective studies. Furthermore, most of the computer-aided diagnostic devices did not include a side-by-side comparison of doctor performance with and without AI, which as the authors note is a critical factor in the devices’ intended use. Most (93/1130) devices did not undergo multi-site assessment, and 59/130 devices did not include a sample size of the studies used. The authors go on to demonstrate in a case study of a pneumothorax detection software that there is substantial variability in device performance when tested across multiple clinical sites. The authors call for more prospective studies of AI-enabled devices that are measured against standard-of-care, as well as increased post-market surveillance in this new field.

Wu E, et al. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med. 2021 Apr;27(4):582-584.

Figure from Wu et al. 2021. 130 FDA-approved AI medical devices characterized by body site, risk level, and study characteristics