How generalizable are radiology AI algorithms?

A team of researchers from Johns Hopkins University School of Medicine systematically evaluated 83 peer-reviewed studies of deep-learning algorithms that perform image-based radiologic diagnosis and had received external validation. Of the 86 algorithms described in the studies, over 80% had a decrease in performance on external datasets, and 24% experienced a substantial decline in performance.

“Our findings stress the importance of including an external dataset to evaluate the generalizability of [deep-learning] algorithms, which would improve the quality of future [deep-learning]…