key: cord-0453718-zb0bu5gm authors: Vaghefi, Ehsan; Yang, Song; Xie, Li; Han, David; Squirrell, David title: A multi-center prospective evaluation of THEIA to detect diabetic retinopathy (DR) and diabetic macular edema (DME) in the New Zealand screening program date: 2021-06-23 journal: nan DOI: nan sha: 31d7589fc200d78fafa5f381560e7db89260ac5c doc_id: 453718 cord_uid: zb0bu5gm Purpose: to assess the efficacy of THEIA, an artificial intelligence for screening diabetic retinopathy in a multi-center prospective study. To validate the potential application of THEIA as clinical decision making assistant in a national screening program. Methods: 902 patients were recruited from either an urban large eye hospital, or a semi-rural optometrist led screening provider, as they were attending their appointment as part of New Zealand Diabetic Screening programme. These clinics used a variety of retinal cameras and a range of operators. The de-identified images were then graded independently by three senior retinal specialists, and final results were aggregated using New Zealand grading scheme, which is then converted to referablenon-referable and Healthymildmore than mildvision threatening categories. Results: compared to ground truth, THEIA achieved 100% sensitivity and [95.35%-97.44%] specificity, and negative predictive value of 100%. THEIA also did not miss any patients with more than mild or vision threatening disease. The level of agreement between the clinicians and the aggregated results was (k value: 0.9881, 0.9557, and 0.9175), and the level of agreement between THEIA and the aggregated labels was (k value: 0.9515). Conclusion: Our multi-centre prospective trial showed that THEIA does not miss referable disease when screening for diabetic retinopathy and maculopathy. It also has a very high level of granularity in reporting the disease level. Since THEIA is being tested on a variety of cameras, operating in a range of clinics (ruralurban, ophthalmologist-ledoptometrist-led), we believe that it will be a suitable addition to a public diabetic screening program. Implementation of artificial intelligence (AI) in medicine and particularly in ophthalmology has a long history, but also accelerating rapidly in the past few years [1] [2] [3] [4] . So far, the most promising application of AI in ophthalmology is as a screening tool for Diabetic Retinopathy (DR) [5] [6] [7] [8] [9] . It is now well accepted that a comprehensive DR screening program can reduce the burden of diabetes related vision loss 4, 10, 11 . However, delivering large community-based programs can be a major challenge even in developed countries, such as including New Zealand which has both a high prevalence of diabetes 12 and a significant proportion of the population not being screened regularly 13 . AI based algorithms, that can reliably detect DR in retinal images and provide instantaneous reporting with high diagnostic accuracy, could significantly improve the earlier detection of DR. In addition, by enabling specialist-level diagnostics to be provided to multiple peripheral sites simultaneously these algorithms also have the potential to massively increase access to, and lower the cost of, screening for DR 14, 15 . In recent years there have been significant advances in development of AI algorithms to assist with diabetic eye screening programs across the world 1 . While the accuracy of AI-based models for detecting DR have been demonstrated in many previous studies [5] [6] [7] [8] [9] , most have failed to perform in the "real world" setting 16 . It has been shown that most research AIs for detection of retinopathy are not generalizable, as training datasets used are not representative of the wider society, obtained from relatively homogenous populations, limited in numbers or highly curated by clinicians, contain on image per eye, and very limited grade granularity (i.e. binary outcome for referable disease) 17 . Although there are several commercially available examples, very few AI products have thus far been integrated into a real-world clinical environment. Here in New Zealand, Toku Eyes® has partnered up with local District Health Boards (DHB) to develop THEIA™, a AI DR Screening tool that is: trained and tested locally, is clinic Artificial intelligence screening for diabetic retinopathy: the real-world emerging application Machine learning has arrived Automated screening for diabetic retinopathy-a systematic review Diabetic retinopathy: global prevalence, major risk factors, screening practices and public health challenges: a review THEIA ™ development, and testing of artificial intelligence-based primary triage of diabetic retinopathy screening images in New Zealand Towards implementation of AI in New Zealand national diabetic screening program: Cloud-based, robust, and bespoke Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes The value of automated diabetic retinopathy screening with the EyeArt system: a study of more than 100,000 consecutive encounters from people with diabetes Screening intervals for diabetic retinopathy and implications for care Cost-effectiveness of a national telemedicine diabetic retinopathy screening program in Singapore Global prevalence and major risk factors of diabetic retinopathy Cost-effectiveness analysis of an artificial intelligence-assisted deep learning system implemented in the national tele-medicine diabetic retinopathy screening in Singapore Health economic and safety considerations for artificial intelligence applications in diabetic retinopathy screening head-to-head, real-world validation study of seven automated artificial intelligence diabetic retinopathy screening systems Essentials of a Robust Deep Learning System for Diabetic Retinopathy Screening: A Systematic Literature Review Diabetic Retinal Screening, Grading, Monitoring and Referral Guidance Prospective evaluation of an artificial intelligenceenabled algorithm for automated diabetic retinopathy screening of 30 000 patients Artificial intelligence-enabled screening for diabetic retinopathy: a real-world, multicenter and prospective study Validation of automated screening for referable diabetic retinopathy with an autonomous diagnostic artificial intelligence system in a Spanish population Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning Performance of a deep-learning algorithm vs manual grading for detecting diabetic retinopathy in india Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy Diabetic retinopathy screening using deep neural network Prevalence of diabetic retinopathy in Type 2 diabetes in developing and developed countries Artificial intelligence for teleophthalmologybased diabetic retinopathy screening in a national programme: an economic analysis modelling study GROUND TRUTH \AI HEALTHY MILD MTMDR VISION THREATENING HEALTHY 1089 41 1 6 MILD 59 233 5 18 MTMDR 1 10 35 28 VISION THREATENING 0 2 5 169 Supplementary