Toward Reliable Algorithms

Article: AI algorithms detect diabetic eye disease inconsistently
Source: University of Washington Medicine, via NEI
Published: January 5, 2021

Relative performance of human grader compared with AI algorithms

Using artificial intelligence to screen retinal diseases is a promising line of research, with many software algorithms currently in development. The utility of this technology lies in its ability to accurately and reliably detect early disease states, a goal that is constantly improving. The current study examined the effectiveness of seven artificial intelligence (AI) algorithms in the detection of diabetic retinopathy, a common ocular complication of diabetes that could lead to vision loss. The algorithms were compared with human screeners in a teleretinal screening system, and both of these were compared to the standard of diagnostic expertise of retina specialists. A strength of the study is its sample size of nearly 24,000 images, from two VA health systems in this case. The researchers found much inconsistency between what is advertised of the algorithms and their performance in real-world clinical settings, with only one of the seven algorithms studied performing as well as the human screeners. This study is especially interesting for two reasons: (1) It highlights the importance of developing consistent methods in the diagnostic tool kit; conclusions can only be reliable if the methods are reliable. (2) Because this study is a kind of meta-analysis of several algorithms, it demonstrates the value of examining a broad range of information sources to arrive at the most robust method available. Note that through the examination of seven different algorithms, one was discovered to be as reliable as the human screeners, for the particular populations under study. 

My rating of this study: ⭐⭐⭐

Lee AY, Yanagihara RT, Lee CS, et al. "Multicenter, Head-to-Head, Real-World Validation Study of Seven Automated Artificial Intelligence Diabetic Retinopathy Screening Systems." Diabetes Care.  44(5): 1168-1175. Published online 5 January 2021. https://doi.org/10.2337/dc20-1877