The value of digital imaging in diabetic retinopathy Methodology The value of digital imaging in diabetic retinopathy PF Sharp S Wallace J Olson K Goatman F Strachan A Grant J Hipwell N Waugh A Ludbrook K McHardy M O’Donnell JV Forrester Health Technology Assessment 2003; Vol. 7: No. 30 HTAHealth Technology AssessmentNHS R&D HTA Programme Copyright notice © Queen's Printer and Controller of HMSO 2003 HTA reports may be freely reproduced for the purposes of private research and study and may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising Violations should be reported to hta@soton.ac.uk Applications for commercial reproduction should be addressed to HMSO, The Copyright Unit, St Clements House, 2–16 Colegate, Norwich NR3 1BQ How to obtain copies of this and other HTA Programme reports. An electronic version of this publication, in Adobe Acrobat format, is available for downloading free of charge for personal use from the HTA website (http://www.hta.ac.uk). A fully searchable CD-ROM is also available (see below). Printed copies of HTA monographs cost £20 each (post and packing free in the UK) to both public and private sector purchasers from our Despatch Agents. Non-UK purchasers will have to pay a small fee for post and packing. For European countries the cost is £2 per monograph and for the rest of the world £3 per monograph. You can order HTA monographs from our Despatch Agents: – fax (with credit card or official purchase order) – post (with credit card or official purchase order or cheque) – phone during office hours (credit card only). Additionally the HTA website allows you either to pay securely by credit card or to print out your order and then post or fax it. Contact details are as follows: HTA Despatch Email: orders@hta.ac.uk c/o Direct Mail Works Ltd Tel: 02392 492 000 4 Oakwood Business Centre Fax: 02392 478 555 Downley, HAVANT PO9 2NP, UK Fax from outside the UK: +44 2392 478 555 NHS libraries can subscribe free of charge. Public libraries can subscribe at a very reduced cost of £100 for each volume (normally comprising 30–40 titles). The commercial subscription rate is £300 per volume. Please see our website for details. Subscriptions can only be purchased for the current or forthcoming volume. Payment methods Paying by cheque If you pay by cheque, the cheque must be in pounds sterling, made payable to Direct Mail Works Ltd and drawn on a bank with a UK address. Paying by credit card The following cards are accepted by phone, fax, post or via the website ordering pages: Delta, Eurocard, Mastercard, Solo, Switch and Visa. We advise against sending credit card details in a plain email. Paying by official purchase order You can post or fax these, but they must be from public bodies (i.e. NHS or universities) within the UK. We cannot at present accept purchase orders from commercial companies or from outside the UK. How do I get a copy of HTA on CD? Please use the form on the HTA website (www.hta.ac.uk/htacd.htm). Or contact Direct Mail Works (see contact details above) by email, post, fax or phone. HTA on CD is currently free of charge worldwide. The website also provides information about the HTA Programme and lists the membership of the various committees. HTA The value of digital imaging in diabetic retinopathy PF Sharp1* S Wallace6 J Olson2 K Goatman3 F Strachan3,4 A Grant6 J Hipwell3 N Waugh7 A Ludbrook5 K McHardy8 M O’Donnell6 JV Forrester9 1 Department of Medical Physics, University of Aberdeen, UK 2 Department of Ophthalmology, Grampian Universities NHS Trust, UK 3 Department of Biomedical Physics and Bioengineering, University of Aberdeen, UK 4 Departments of Ophthalmology and Diabetes, Grampian Universities NHS Trust 5 Health Economics Research Unit, University of Aberdeen, UK 6 Health Services Research Unit, University of Aberdeen, UK 7 Scottish Health Purchasing Information Centre, Grampian Health Board, UK 8 Consultant, Grampian University Hospitals NHS Trust, UK 9 Department of Ophthalmology, University of Aberdeen, UK *Corresponding author Declared competing interests of authors: none Published November 2003 This report should be referenced as follows: Sharp, PF, Olson J, Strachan F, Hipwell J, Ludbrook A, O’Donnell M, et al. The value of digital imaging in diabetic retinopathy. Health Technol Assess 2003;7(30). Health Technology Assessment is indexed in Index Medicus/MEDLINE and Excerpta Medica/ EMBASE. NHS R&D HTA Programme The NHS R&D Health Technology Assessment (HTA) Programme was set up in 1993 to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS. Initially, six HTA panels (pharmaceuticals, acute sector, primary and community care, diagnostics and imaging, population screening, methodology) helped to set the research priorities for the HTA Programme. However, during the past few years there have been a number of changes in and around NHS R&D, such as the establishment of the National Institute for Clinical Excellence (NICE) and the creation of three new research programmes: Service Delivery and Organisation (SDO); New and Emerging Applications of Technology (NEAT); and the Methodology Programme. This has meant that the HTA panels can now focus more explicitly on health technologies (‘health technologies’ are broadly defined to include all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care) rather than settings of care. Therefore the panel structure was replaced in 2000 by three new panels: Pharmaceuticals; Therapeutic Procedures (including devices and operations); and Diagnostic Technologies and Screening. The HTA Programme will continue to commission both primary and secondary research. The HTA Commissioning Board, supported by the National Coordinating Centre for Health Technology Assessment (NCCHTA), will consider and advise the Programme Director on the best research projects to pursue in order to address the research priorities identified by the three HTA panels. The research reported in this monograph was funded as project number 94/18/05. The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health. The editors wish to emphasise that funding and publication of this research by the NHS should not be taken as implicit support for any recommendations made by the authors. HTA Programme Director: Professor Kent Woods Series Editors: Professor Andrew Stevens, Dr Ken Stein, Professor John Gabbay, Dr Ruairidh Milne, Dr Chris Hyde and Dr Rob Riemsma Managing Editors: Sally Bailey and Sarah Llewellyn Lloyd The editors and publisher have tried to ensure the accuracy of this report but do not accept liability for damages or losses arising from material published in this report. They would like to thank the referees for their constructive comments on the draft document. ISSN 1366-5278 © Queen’s Printer and Controller of HMSO 2003 This monograph may be freely reproduced for the purposes of private research and study and may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to HMSO,The Copyright Unit, St Clements House, 2–16 Colegate, Norwich, NR3 1BQ. Published by Gray Publishing, Tunbridge Wells, Kent, on behalf of NCCHTA. Printed on acid-free paper in the UK by St Edmundsbury Press Ltd, Bury St Edmunds, Suffolk. Criteria for inclusion in the HTA monograph series Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors. Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others. G iii © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Objectives: To assess the performance of digital imaging, compared with other modalities, in screening for and monitoring the development of diabetic retinopathy. Design: All imaging was acquired at a hospital assessment clinic. Subsequently, study optometrists examined the patients in their own premises. A subset of patients also had fluorescein angiography performed every 6 months. Setting: Research clinic at the hospital eye clinic and optometrists’ own premises. Participants: Study comprised 103 patients who had type 1 diabetes mellitus, 481 had type 2 diabetes mellitus and two had secondary diabetes mellitus; 157 (26.8%) had some form of retinopathy (‘any’) and 58 (9.9%) had referable retinopathy. Interventions: A repeat assessment was carried out of all patients 1 year after their initial assessment. Patients who had more severe forms of retinopathy were monitored more frequently for evidence of progression. Main outcome measures: Detection of retinopathy, progression of retinopathy and determination of when treatment is required. Results: Manual grading of 35-mm colour slides produced the highest sensitivity and specificity figures, with optometrist examination recording most false negatives. Manual and automated analysis of digital images had intermediate sensitivity. Both manual grading of 35-mm colour slides and digital images gave sensitivities of over 90% with few false positives. Digital imaging produced 50% fewer ungradable images than colour slides. This part of the study was limited as patients with the more severe levels of retinopathy opted for treatment. There was an increase in the number of microaneurysms in those patients who developed from mild to moderate. There was no difference between the turnover rate of either new or regressed microaneurysms for patients with mild or with sight-threatening retinopathy. It was not possible in this study to ascertain whether digital imaging systems determine when treatment is warranted. Conclusions: In the context of a national screening programme for referable retinopathy, digital imaging is an effective method. In addition, technical failure rates are lower with digital imaging than conventional photography. Digital imaging is also a more sensitive technique than slit-lamp examination by optometrists. Automated grading can improve efficiency by correctly identifying just under half the population as having no retinopathy. Recommendations for future research include: investigating whether the nasal field is required for grading; a large screening programme is required to ascertain if automated grading can safely perform as a first-level grader; if colour improves the performance of grading digital images; investigating methods to ensure effective uptake in a diabetic retinopathy screening programme. Health Technology Assessment 2003; Vol. 7: No. 30 Abstract The value of digital imaging in diabetic retinopathy PF Sharp,1* J Olson,2 F Strachan,3,4 J Hipwell,3 A Ludbrook,5 M O’Donnell,6 S Wallace,6 K Goatman,3 A Grant,6 N Waugh,7 K McHardy8 and JV Forrester9 1 Department of Medical Physics, University of Aberdeen, UK 2 Department of Ophthalmology, Grampian Universities NHS Trust, UK 3 Department of Biomedical Physics and Bioengineering, University of Aberdeen, UK 4 Departments of Ophthalmology and Diabetes, Grampian Universities NHS Trust 5 Health Economics Research Unit, University of Aberdeen, UK 6 Health Services Research Unit, University of Aberdeen, UK 7 Scottish Health Purchasing Information Centre, Grampian Health Board, UK 8 Consultant, Grampian University Hospitals NHS Trust, UK 9 Department of Ophthalmology, University of Aberdeen, UK *Corresponding author Health Technology Assessment 2003; Vol. 7: No. 30 v Glossary and list of abbreviations ............. vii Executive summary .................................... ix 1 Introduction ............................................... 1 Background ................................................ 1 Key features of diabetic retinopathy .......... 1 Early Treatment of Diabetic Retinopathy Study ........................................................... 1 Risk factors in development of retinopathy ................................................. 1 An overview of existing screening methods ...................................................... 2 Commission from NHS Health Technology Assessment Programme ............................. 2 2 Study design and patient recruitment ...... 5 Aim of study ............................................... 5 Study design ............................................... 5 Question 1: can a digital system detect the presence of diabetic retinopathy (of any sort/level)? .................................................. 5 Question 2: can a digital system detect progression of retinopathy? ....................... 7 Question 3: can a digital system determine when treatment is required? ...................... 7 Patient recruitment .................................... 7 Conclusions ................................................ 9 3 Photography ............................................... 11 Choice of digital fundus camera ................ 11 The EURODIAB protocol .......................... 11 Image artefacts due to camera dust ........... 12 Grading image quality ............................... 13 Failure to obtain gradable images .............. 14 Conclusions ................................................ 14 4 Automated detection of retinopathy ........ 15 Why automate? ........................................... 15 What lesions should we detect? .................. 15 Automated detection of retinopathy .......... 15 Microaneurysm detection ........................... 16 Automated detection of hard exudates ...... 17 Lesion location and maculopathy .............. 18 Turnover of microaneurysms ..................... 18 Conclusions ................................................ 19 5 Question 1: can a digital system detect the presence of diabetic retinopathy (of any sort/level)? .................................................. 21 Introduction ............................................... 21 Slit-lamp biomicroscopy ............................. 21 Photography and digital imaging .............. 21 Distribution of retinopathy amongst patients ....................................................... 22 Detection of retinopathy ............................ 22 Automated analysis in diabetic macular oedema ....................................................... 24 Sight-threatening retinopathy .................... 24 Is the nasal-disc field required? ................. 24 Comparison of optometrist with ophthalmologist ......................................... 25 Discussion ................................................... 25 6 Question 2: can a digital system detect progression of retinopathy? Question 3: can a digital system determine when treatment is required? .................................................... 29 Introduction ............................................... 29 Study protocol ............................................ 29 Can digital red-free photography monitor progression of retinopathy? ....................... 29 Can automated analysis of digital images follow progression of retinopathy? ............................................... 30 Problem of microaneurysm turnover ......... 31 Risk factors ................................................. 31 Discussion ................................................... 33 7 Costs and consequences of screening for diabetic retinopathy .................................. 35 Introduction ............................................... 35 Digital photography costs .......................... 35 Automated grading costs ........................... 35 Colour slide photography costs ................. 35 Optometrist screening costs ....................... 35 Results ........................................................ 36 Combining modalities – a modelling exercise ....................................................... 37 Sensitivity analysis ...................................... 37 Assumed life of equipment ........................ 38 Conclusions ................................................ 38 Contents vi 8 Conclusions ................................................ 41 Current study ............................................. 41 Relevance to the NHS ................................ 43 Research recommendations ....................... 43 The future .................................................. 43 Acknowledgements .................................... 45 References .................................................. 47 Appendix 1 Systematic literature review (completed 1998) ............................. 49 Appendix 2 Consent form, letter of invitation and patient information sheet ................... 117 Health Technology Assessment reports published to date ....................................... 121 Health Technology Assessment Programme ................................................ 129 Contents vii © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Health Technology Assessment 2003; Vol. 7: No. 30 Glossary Biomicroscopy A technique for examining the structures of the eye. Capital costs The costs associated with items that remain useful beyond the period in which these costs are incurred, for example, for equipment and buildings. Cost-effectiveness A comparison of the cost of an intervention with the effect on the patient. Creatinine A component of urine. Diabetic retinopathy A complication of diabetes in which the retina of the eye is affected by blocking off of its small blood vessels. Exudate Masses of macrophages containing lipids formed at the edges of where plasma has leaked from the capillaries. Field A single image taken of the retina. Fluorescein angiography A technique for visualising the blood flow through the vessels of the eye. A fluorescent dye is injected intravenously and images of the eye are taken as it flows through the blood vessels. Fovea A small pit or depression in the retina. The very centre of the macula. Fundus The portion of the interior of the eye visible through the ophthalmoscope. Fusiform Spindle shaped. Glycated haemoglobin A test to show how well controlled diabetes has been in the preceding months. Haematocrit The percentage of a blood sample occupied by cells. Haemorrhage Bleeding. HbA1c Glycated haemoglobin. See above. Ischaemia Reduction of blood supply. Laser photocoagulation The use of a highly focused laser to treat diseased tissue. Macula The area of the retina that is the centre of sight. Macular oedema Fluid in the macula. Maculopathy A pathological condition of the macula. Microalbuminuria Abnormally increased excretion of albumin in the urine. An early marker of diabetic kidney disease. Microaneurysms Localised dilations of retinal capillaries. They may leak, causing oedema and haemorrhage in the retina. Mydriasis The use of drops to dilate the pupil. Neovascularisation The formation of new blood vessels. Oedema A collection of fluid. Ophthalmoscopy The use of an optical instrument for examining the interior of the eyeball. Prognosis A forecast of the probable course and outcome of the disease. Proliferative retinopathy Diabetes can result in small blood vessels being blocked off, so depriving the retina of oxygen and nutrients. The eye tries to grow new vessels, proliferative retinopathy, that may bleed and detach the retina. Recurrent costs Costs which occur regularly. Retina The light-sensitive layer at the back of the eye. Risk factor A clearly defined occurrence that increases the probability that a person will get a disease. Sensitivity The probability of correctly identifying that a disease is present. Slit-lamp biomicroscopy A method of examining the eye using a special microscope. Glossary and list of abbreviations continued viii Glossary continued Specificity The probability of correctly identifying that a disease is not present. TIFF format Tag(ged) Image File Format – an image file popular owing to its platform independence, extendibility and great flexibility. Type 1 (insulin-dependent) diabetes The type of diabetes that develops when the body cannot produce any insulin. Type 2 (non-insulin-dependent) diabetes The type of diabetes that develops when the body can make some insulin but not enough, or when the insulin that is produced does not work properly. Visual acuity A measure of how well a person sees distant and close objects. VS2 Greater than or equal to Airlie House Photograph 2B in the equivalent of two Airlie House standard photographic fields. List of abbreviations BDR background diabetic retinopathy CCD charge-coupled device CI confidence interval CSMO clinically significant macular oedema or maculopathy ETDRS Early Treatment of Diabetic Retinopathy Study FAZ foveal avascular zone FD fractal dimension FROC free response receiver operating characteristic ICG indocyanine green IRMA intra-retinal microvascular abnormality MeSH medical subject headings NPDR non-proliferative diabetic retinopathy PC personal computer PDR proliferative diabetic retinopathy PIA perifoveal intercapillary area RAM random access memory RCT randomised controlled trial ROC receiver operating characteristic SLO scanning laser ophthalmoscope Glossary and list of abbreviations All abbreviations that have been used in this report are listed here unless the abbreviation is well known (e.g. NHS), or it has been used only once, or it is a non-standard abbreviation used only in figures/tables/appendices in which case the abbreviation is defined in the figure legend or at the end of the table. ix © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Health Technology Assessment 2003; Vol. 7: No. 30 Objectives To undertake a systematic literature review followed by a primary study to assess the performance of digital imaging, compared with other modalities, in screening for, and monitoring the development of, diabetic retinopathy. The study addressed three questions: 1. Can a digital imaging system detect retinopathy irrespective of sort or level? 2. Can a digital imaging system detect progression of retinopathy? 3. Can a digital imaging system determine when treatment is required? Design Question 1 All imaging was acquired at a hospital assessment clinic. Subsequently, study optometrists examined the patients in their own premises. Questions 2 and 3 In addition to the above, a subset of patients had fluorescein angiography performed every 6 months. The gold standard was clinical examination by an ophthalmologist. All questions were also addressed using automated analysis of digital red-free images. Subjects The study invited 1114 patients undergoing direct ophthalmoscopy at the diabetic clinic in Aberdeen; of these, 727 agreed and 387 declined. Of the former, 586 attended. Of these 103 patients had type 1 diabetes mellitus, 481 had type 2 diabetes mellitus and two had secondary diabetes mellitus; 157 (26.8%) had some form of retinopathy (‘any’) and 58 (9.9%) had referable retinopathy. Results Question 1: can a digital imaging system detect retinopathy irrespective of sort or level? Any retinopathy Manual grading of 35-mm colour slides produced the highest sensitivity (89%) and specificity (89%) figures, with optometrist examination recording most false negatives (sensitivity 75%). Manual and automated analysis of digital images had intermediate sensitivity. Referable retinopathy Both manual grading of 35-mm colour slides and digital images gave sensitivities of over 90% with few false positives (specificity 89 and 87%, respectively). Digital imaging produced 50% fewer ungradable images than colour slides. Question 2: can a digital imaging system detect progression of retinopathy? This part of the study was limited as patients with the more severe levels of retinopathy opted for treatment. There was an increase in the number of microaneurysms in those patients who developed from mild to moderate. There was no difference between the turnover rate of either new or regressed microaneurysms for patients with mild or with sight-threatening retinopathy. Question 3: can a digital imaging system determine when treatment is warranted? Since there was no definite answer to question 2, then the answer must be ‘no’ at present. Executive summary x Conclusions Implications for healthcare Digital imaging In the context of a national screening programme for referable retinopathy, digital imaging is an effective method. In addition, technical failure rates are lower with digital imaging than conventional photography. Digital imaging is also a more sensitive technique than slit-lamp examination by optometrists. Automated grading of digital images Automated grading can improve efficiency by correctly identifying just under half the population as having no retinopathy. Recommendations for future research 1. Is the nasal field required for grading? Our study would suggest not. Single-field imaging could potentially reduce the time taken to perform retinal screening and the number of technical failures. 2. Can automated grading safely perform as a first-level grader? Our study would suggest ‘yes’, but this needs to be confirmed in a large screening programme. 3. Does colour improve the performance of grading digital images? Although high- resolution colour digital images are now routinely available, their role in screening for diabetic retinopathy has yet to be assessed. 4. Can patient recruitment be improved? Future research is required to ensure effective uptake in a diabetic retinopathy screening programme. Executive summary Background Despite advances in diabetic care, visual impairment in diabetes remains a devastating complication, in terms of both personal loss for the patient and socio-economic costs to the community. It remains the commonest cause of blindness in the working population.1 It is potentially preventable, but this presents an immense challenge to the NHS, principally because the timing of treatment is critical. If laser photocoagulation is delayed until there is symptomatic visual loss, the outcome may be poor, but too early referral will overload ophthalmology departments, leading to inefficient practice. Two essential components of an effective and efficient system of diabetic care within the NHS are therefore first, regular retinal assessment, and second, a method of assessment that allows optimal timing of therapy. The current arrangements within the NHS for identifying diabetic eye disease are widely perceived as being costly and inefficient and this leads to pressure for change. The development of digital cameras is a promising step forward. As it may not need a skilled operator, then regular assessment is more feasible and its diagnostic performance may be as good as, or even superior to, current methods. It is ideally suited to quality assurance as it produces a hard copy. However, the introduction of this technology into the NHS will have major logistical and resource implications and it should be a prerequisite that it demonstrably performs better than existing systems. Key features of diabetic retinopathy Klein and colleagues2 were the first to suggest that baseline microaneurysm counts in patients with no other evidence of retinopathy can provide a useful predictor of long-term progression to proliferative retinopathy, independent of the effects of glycaemic control and blood pressure. Kohner and Sleightholm3 have provided evidence supporting the importance of microaneurysm detection and quantification from their analysis of fluorescein angiograms. They showed a significant correlation between microaneurysm number and the presence of haemorrhages and cotton-wool spots and, to a lesser extent, the severity of hard exudates and intra-retinal microvascular abnormalities (IRMAs). From examination of the natural progression of fundus changes in diabetics with at least moderate diabetic retinopathy, key features were identified as having a high predictive value in heralding a deterioration to a proliferative state. The severity of IRMA, venous beading and the number of haemorrhages and microaneurysms were thought to be of the greatest significance in identifying individuals who are likely to develop neovascularisation.4 Early Treatment of Diabetic Retinopathy Study The Early Treatment of Diabetic Retinopathy Study (ETDRS) was designed to provide answers as to how the devastating morbidity from visual loss could be reduced. The benefits to long-term visual outcome for patients with high-risk proliferative retinopathy undergoing laser panretinal photocoagulation have been recognised and adopted into standard clinical practice.5 However, the authors concluded that for those with mild to moderate retinopathy, and a low risk of progression to severe visual loss, early panretinal photocoagulation was inappropriate given its potentially detrimental effect on the peripheral visual field. They emphasised that the key to successful management was early identification of retinopathy with the meticulous monitoring of progression to allow optimum timing of intervention when a high-risk proliferative stage had been reached. Risk factors in development of retinopathy The Diabetes Control and Complications Trial6 has shown that strict metabolic control can both offset the development and slow the progression of diabetic retinopathy in type 1 diabetes. Whilst acknowledging the limitations of their cross- sectional study of 3250 European type 1 diabetes Health Technology Assessment 2003; Vol. 7: No. 30 1 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Chapter 1 Introduction patients, Sjolie and colleagues7 have also suggested that, in the later stages of retinopathy, the adjustment of blood pressure, fibrinogen and triglyceride levels may affect outcome. The cessation of cigarette smoking may also have a beneficial effect, although the evidence is conflicting.8–10 However, as intervention at both primary and secondary level can now be contemplated, the need for careful screening of diabetic populations and monitoring of established retinopathy becomes even more crucial. The recognition of very early diabetic fundal change may provide the impetus to patients to make positive lifestyle changes and tighten glycaemic control. Accurate grading of retinopathy and recognition of high- risk features are essential to providing appropriately timed laser therapy. An overview of existing screening methods Adhering to the principles of the St Vincent Declaration,11 the European Retinopathy Working Party defined a protocol of screening for diabetic retinopathy in 1991, advocating that all patients with diabetes should have an annual eye examination. This was broadly defined as ophthalmoscopy through pharmacologically dilated pupils or by retinal photography.12 In the absence of evidence to support a definitive method of performing retinal examination, a study of 3318 patients with diabetes was undertaken on behalf of the NHS by Buxton and colleagues. A comparison was made between direct ophthalmoscopy, performed by general practitioners (GPs), ophthalmic opticians and hospital physicians, and consultant ophthalmologist assessment of images acquired with a non-mydriatic Polaroid fundus camera.13 They concluded that all participants showed relatively poor sensitivities (hospital physicians 67%; GPs 53%; ophthalmic opticians 47%) although specificities were higher (hospital physicians 97%; GPs 91%; ophthalmic opticians 95%), indicating that relatively few inappropriate referrals to ophthalmology services would occur. However, it was apparent that direct ophthalmoscopy alone was likely to miss a significant proportion of patients with evidence of retinopathy, regardless of who performed the examination. Comparable results were obtained from analysis of fundus photographs, with sensitivities ranging from 35 to 67% and specificities marginally higher than those of the primary screeners at 95–98%. Although subsequent reports indicate that the quality of photography from a non-mydriatic camera may have been improved by use of mydriatic agents,14 the authors were unable to recommend a specific technique for routine screening of a general diabetic population. These results also reflect the outcome of work by previous authors15–19 and suggest that, at present, no individual method of routinely available fundus examination can be judged superior for clinical screening. Although the findings of Klein and colleagues15 seem encouraging, the wide confidence intervals calculated for their relatively small patient population warrant caution in interpretation of these results. Combining the modalities of ophthalmoscopy and retinal photography appears effective in detecting sight-threatening retinopathy according to O’Hare and colleagues;19 however, their technique does not achieve the recommended threshold of a minimum 80% sensitivity and 95% specificity for detecting mild retinopathy, suggested as an audit standard by the British Diabetic Retinopathy Working Group.20 Commission from NHS Health Technology Assessment Programme Aims of investigation Given the above background, there has been significant interest in developing screening techniques and in particular in exploring the potential of digital imaging. As there was no clear information as to how these techniques would fit into a clinical setting, the work in this report was commissioned in 1996 by the NHS Health Technology Assessment Programme. The intention was first to undertake a systematic literature review and then carry out a primary study to assess the performance of digital imaging in screening for, and monitoring the development of, diabetic retinal disease. The study aimed to show: (1) whether further technical development will be required, (2) what the diagnostic performance of current digital systems is in comparison with alternative approaches, (3) whether there is any evidence of clinical effectiveness and (4) whether it is likely to be cost-effective as a screening or disease monitoring (diagnostic) test. Overview of the study The purpose of the study was to address three questions: (1) can a digital imaging system detect retinopathy irrespective of sort or level?; (2) can a digital imaging system detect progression of Introduction 2 Health Technology Assessment 2003; Vol. 7: No. 30 3 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. retinopathy, in particular warranting further investigation?; and (3) can a digital imaging system determine when treatment is required? Diabetologists and GPs wish to know the answer to question 1 as they can use this information to motivate patients to improve their metabolic control and prevent, or slow, the progression of retinopathy and other microvascular complications of diabetes – nephropathy and neuropathy. Diabetologists, GPs and optometrists wish to know the answer to question 2, as this will help reduce the enormous workload of screening for sight- threatening retinopathy. Ophthalmologists wish to know the answer to question 3 as this will help them decide when to initiate laser treatment. We had previously developed software that could analyse digitised retinal images from fluorescein angiograms for the presence of the cardinal features of diabetic retinopathy, namely microaneurysms/ haemorrhages and exudates.21–24 This current project involves using a digital fundus camera, which would provide direct digital images, in conjunction with our software modified to analyse red-free images not only to detect retinopathy but to answer the three questions raised above. The study was preceded by a systematic review of the literature to assess the value of currently available digital imaging techniques and compare them with alternative methods. Layout of the report In the following chapter we will discuss the design of the study, centred around the three cardinal questions, and patient recruitment. The factors influencing the choice of the digital fundus camera system and the results of an assessment of the quality of the images produced by it will be presented in Chapter 3. In Chapter 4, the principles behind the software used to detect microaneurysms and hard exudates automatically will be set out. The study is divided into two branches. The role of digital imaging in screening will be looked at in Chapter 5, addressing question 1 and, in part, question 2. In Chapter 6 the value of digital imaging in monitoring progression of disease will be investigated. This will explore the answers to questions 2 and 3. Finally, costs and consequences will be examined in Chapter 7 and conclusions and proposals for further research are presented in Chapter 8. The systematic literature review is included as Appendix 1. Aim of study The systematic literature review (Appendix 1) concluded that digital photography offered a promising alternative to conventional photography and that its diagnostic performance was not impaired by the lower resolution of currently available systems. It confirmed that further studies were required to compare digital photography with conventional photography, when both sets of images are manually assessed by trained observers. In addition, it recommended that digital photography should be assessed against the performance of those currently providing retinopathy screening by direct ophthalmoscopy. As a result, the original proposal for investigating the value of digital imaging in diabetic retinopathy was followed. Three cardinal questions were to be addressed in this study: � Question 1: can a digital fundus camera system detect the presence of diabetic retinopathy (of any sort/level)? � Question 2: can a digital fundus camera system detect progression of retinopathy? � Question 3: can a digital fundus camera system determine when treatment is required? In pilot studies it was shown that the contrast of microaneurysms/haemorrhages in digitised red- free images was sufficient to allow accurate quantitative analysis. Hence a digital fundus camera would probably allow the detection of the presence of retinopathy, assuming that the patient has sufficiently clear media to give digital images of sufficiently high quality. The question is whether it is more effective than conventional photography or slit-lamp examination by non- ophthalmologists. Study design An outline of the study is shown in Figure 1. Patients were recruited whilst attending for their routine eye screening assessment at the diabetic clinic in Aberdeen. Routine screening usually involved direct ophthalmoloscopy with mydriasis. As direct ophthalmoloscopy is known to be an insensitive technique, the information from this procedure was not used in the study as it was not felt to be an acceptable screening method. Once patients had agreed to take part, they were all asked to attend a research clinic at the hospital eye clinic. There, patients had their visual acuity measured, their fundi photographed and fundi assessed by one of the study ophthalmologists. All patients were then asked to attend one of the study optometrists in the optometrists’ own premises. This appointment was arranged by the research registrar. The information from these first assessments was used to answer question 1, can a digital system detect the presence of diabetic retinopathy (of any sort/level)? The second question, can a digital system detect progression of retinopathy?, was to be answered by repeat assessment of all patients 1 year after their initial assessment. In addition to the above, a subset of patients who had more severe forms of retinopathy were monitored more frequently for evidence of progression. This subset had the following criteria when examined by the study ophthalmologists: � moderate non-proliferative or worse retinopathy � retinal thickening within a one disc diameter radius of the centre of the macula. This subset was also studied to answer question 3, Can a digital system determine when treatment is required? Question 1: can a digital system detect the presence of diabetic retinopathy (of any sort/level)? This question has two components. First, can a digital system detect the presence of any retinopathy, and second, can it be used to decide automatically that the disease has progressed to a stage where the patient should be transferred to the eye clinic for further action? As it has been estimated that ‘only’ 30% of all patients with Health Technology Assessment 2003; Vol. 7: No. 30 5 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Chapter 2 Study design and patient recruitment diabetes at any one time have retinopathy, then such a system could be of great benefit in reducing clinical workload.25 All patients were studied every 12 months. As shown in Figure 1, they underwent digital imaging on the Topcon™ system, where red-free images were acquired using two 45° fields, a macular field and a disc/nasal field. These digital images were transferred to a SUN™ system, where the programmes which had been developed for analysing for microaneurysms and haemorrhages were used. In addition, analogue red-free and colour images were taken. Conventional mydriatic retinal Study design and patient recruitment 6 Patients attending for routine eye screening using direct ophthalmoloscopy at the Diabetic Clinic were invited to enter the study by the research registrar All patients attended a research clinic at the Eye Department. • Colour photography • Red-free digital • Examination by study ophthalmologist + • Slit-lamp biomicroscopy by trained optometrists in their own premises Moderate or worse DR Exudates and/or haemorrhages/MA within 1 disc diameter of macula and retinal thickening No DR Or mild DR Every 6 months • Fluorescein angiography • Fundus photography • Medical assessment of risk factors (annual) Every 12 months • Imaging with the digital camera • Examination by optometrist with direct ophthalmoscope Outcome Comparison of performance of: • Analogue vs digital red-free • Colour vs digital • Optometrist vs digital (Question 2) Outcome • Assessment of effectiveness of digital system in following progression of retinopathy • Correlation of quantitative measures of retinopathy with clinical status of patient (Questions 2 and 3) Optometrist vs digital Optometrist vs colour Optometrist vs automated Outcome (Question 1) FIGURE 1 Experimental design. DR, diabetic retinopathy; MA, microaneurysm. photography was performed using a Topcon 50X fundus camera with 35-mm transparencies on Kodachrome 64 film. Patients were also examined at this visit by an ophthalmologist using slit-lamp biomicroscopy, this being the only widely agreed clinical gold standard for assessing severity of retinopathy. The use of two 45° fields allowed direct comparison with the proposed European grading system developed for the EURODIAB study. This method is directly comparable with the Airlie House grading system – the present gold standard for grading diabetic retinopathy – and has been proposed as a particularly suitable method for large epidemiological studies.26 This study is described in detail in Chapter 4. All patients also had an additional assessment by a study optometrist for fundus examination using slit-lamp biomicroscopy. This was performed on a separate occasion in the optometrist’s own premises. Question 2: can a digital system detect progression of retinopathy? Question 3: can a digital system determine when treatment is required? This arm of the study looked at the ability of the digital system to provide the ophthalmologist with a reliable, quantitative measure of retinal pathology which will allow him or her to monitor the progression of the disease. The crucial questions were whether a digital system could detect progression of retinopathy and whether it could be used to determine when treatment was required. As for the detection of any retinopathy, it was felt that the answer to the first question was probably ‘yes’. That to the second question may be ‘yes’ if the measurement of area of haemorrhage and number of microaneurysms at VS2 (greater than or equal to Airlie House Photograph 2B in the equivalent of two Airlie House standard photographic fields) is the end-point. An important facet of these two questions was to look at the natural history of retinopathy. Therefore, in addition to the subset of those patients initially assigned to this study, that is, those diagnosed on the initial screening study as having moderate proliferative retinopathy or worse, or clinically significant macular oedema (maculopathy), a group of patients with lesser degrees of retinopathy was included. All patients underwent fluorescein angiography and fundus photography every 6 months. Software to measure the number of microaneurysms and area of exudate from such images had previously been developed by our group.21–24 Finally, it was planned to correlate the changes reported from the computer measurements against clinical status. All patients were assessed medically once a year as various risk factors are known to be associated with the progression of retinopathy. Patients had body mass index, blood pressure, HBA1c and microalbuminuria measured.6,15,27–29 In addition to smoking and alcohol consumption, obstetric history, past medical history and drug history were also ascertained.30,31 This was done to allow the natural history of retinopathy to be put into clinical context and to help identify those patients needing more frequent screening than eye signs alone will suggest. Patient recruitment The numbers of patients required for the study were estimated as follows. For the community screening group, assuming that 30% (240) actually have retinopathy, the area under the receiver operating characteristic (ROC) curve will be estimated to ±2%. Differences in specificity of >6% could be identified (80% power, 2p = 0.05). Assuming 560 participants of whom 10% (56) progress, the area under the ROC curve will be estimated to ±4%. Comparisons with analogue image, colour slide and optometrist will identify differences in specificity of >7% (80% power, 2p = 0.05). For the hospital monitoring group, assuming 160 patients, of whom 10% (16) progress, the area under the ROC curve will be estimated to ±7%. Differences in specificity of >13% could be identified (80% power, 2p = 0.05). For the purposes of analysis, this group may be combined with those with mild diabetic retinopathy. Assuming the combined sample, giving 720 participants of whom 10% (72) may progress, the area under the ROC curve will be estimated to ±3%. Differences in specificity of >6% could be identified (80% power, 2p = 0.05). Health Technology Assessment 2003; Vol. 7: No. 30 7 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. The ability of digital imaging to identify progression to retinopathy that warrants treatment will be explored within the cohort of people found to have moderate/severe retinopathy at the first ‘inception’ examination. Assuming 80 participants of whom 40% (32) progress, the area under the ROC curve will be estimated to ±5%. Patients were recruited from those referred to the hospital diabetic clinic. At present 1.4% (7624) of the population of Grampian (528,100) attends these clinics (Table 1). Previous studies have shown that, for Aberdeen patients, almost 100% of those on insulin and 96% of all patients with diabetes are registered at the clinic.31 As a consequence, our study reflected the local diabetic population as a whole; in other parts of the country hospital diabetic medical clinics are mainly attended by those with complications or on insulin. It was estimated that over the 36-month period of this study, about 800 patients could be studied. It was expected that approximately 30% would have abnormal retinal pathology, depending upon duration and level of control.31 The study invited 1114 patients undergoing routine fundoscopy at the diabetic clinic to participate; 727 patients were recruited and 387 declined. Of the former, 586 patients attended an optometrist trained in slit-lamp biomicroscopy and also a hospital assessment clinic where digital and conventional photography was performed. Slit-lamp examination by an ophthalmologist was performed as the gold standard. These examinations were then repeated 1 year later. Patients failing to attend for an appointment were offered a second appointment, but not followed up further. Copies of the consent form, patient information sheet and letter of invitation are provided in Appendix 2. Of the patients, 103 had type 1 diabetes mellitus (17.6%), 481 had type 2 diabetes mellitus (82.1%) and two had secondary diabetes mellitus (0.3%). The male-to-female ratio was 398:214 (65:35%). The average age at the start of the study was 56.5 years (median 59.3 years) (Table 2). Table 3 summarises the alcohol consumption and smoking history. The definition of level of retinopathy is discussed in the section ‘Photography and digital imaging’ (p. 21). Study design and patient recruitment 8 TABLE 1 Patients attending Grampian diabetic medical clinics on 10 January 1995 Diet controlled 1852 Tablet controlled 2858 UK Prospective Diabetes Study 294 Insulin treated 2620 Total 7624 TABLE 2 Age distribution of patients entering the study Age range (years) Percentage <25 2.6 25–34 6.5 35–44 9.2 45–54 20.8 55–64 30.0 65–74 27.3 >75 3.6 TABLE 3 Smoking history and alcohol consumption of patients entering the trial Alcohol consumption (units per week) Smoking history Level of retinopathy (mean and range) Non-smoker Smoker Ex-smoker None 5.4, 0–60 (n = 427) 209 80 140 Mild 6.4, 0–42 (n = 107) 59 12 37 Moderate 5.3, 0–56 (n = 36) 17 6 13 Severe 10.2, 0–28 (n = 6) 3 1 2 Very severe – 0 0 0 Early PDR 4, 0–15 (n = 4) 2 1 1 High-risk PDR 8 (n = 1) 1 0 0 PDR, proliferative diabetic retinopathy. Health Technology Assessment 2003; Vol. 7: No. 30 9 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Conclusions For historical reasons, a larger number of patients attend the hospital diabetic clinic than might be expected in other areas. This is reflected in the demographics of the population studied, which has a higher prevalence of type 2 diabetes than might be expected in a hospital clinic. Although the majority of people in the study were of a working age, it is of concern that 528 of 1114 patients approached either declined to take part or failed to attend once recruited. It can be speculated that this is due to the fact that patients were already being screened and might have implications for any stand-alone retinopathy screening programme. Choice of digital fundus camera The choice of digital fundus camera is central to the project. At the time the study started, August 1996, there were few systems from which to choose. The defining parameters for selection were as follows: � A digital image resolution of at least 1024 × 1024 pixels. Previous work by the Aberdeen group on the automated detection of diabetic retinopathy from fluorescein angiograms had used 35-mm film digitised off-line using a Kodak Megaplus camera with a resolution of 1024 × 1024, a pixel size being equivalent to 13 �m. A coarser digitisation than this was unlikely to yield comparable results. � Monochrome image acquisition with a pixel depth of at least 8 bits. The project utilised red- free and fluorescein images. Although colour images might contain more information than monochrome images, at the time of the study commercial systems offered colour charge- coupled device (CCD) cameras with a coarser image digitisation, typically 640 × 480 pixels. � Images must not be compressed prior to storage. Some types of compression can result in loss of information. � Image file structure must be such that images can be extracted for analysis on a SUN workstation. Previously programs had been developed using the Unix™ environment on SUN workstations and time constraints meant that converting it to a personal computer (PC) environment was unrealistic. The only system able to meet those specifications was the Topcon retinal camera and IMAGEnet™ windows system (Topcon UK, Newbury, Berkshire, UK). The system, shown in Figure 2, consists of a Topcon TRC-50XT™ retinal camera with digital images being acquired with a Kodak Megaplus™ 1.4I CCD camera, with a digitisation of 1024 × 1024 pixels in monochrome with a pixel depth of 8 bits. A colour film back was available for the colour slides which were captured on to Kodak Kodachrome 64 film. Data were acquired on a PC for analysis, display and archiving under the IMAGEnet hardware and software. Acquired images were archived on CD-ROM and transferred to the SUN workstation for image analysis. The Topcon TRC-50XT camera was chosen for this study as it had the finest image digitisation of the digital fundus cameras commercially available at the time. The images from other manufacturers (such as Canon) were not comparable, principally because they used colour cameras which gave a more coarsely digitised image (e.g. 768 × 576 pixels). In our experience, the larger monochrome (red-free) images with their potentially higher resolution at a given field of view, and greater contrast, were more appropriate for computer analysis than smaller colour images. In addition to finer image pixellation from the Topcon camera, an earlier model photographic fundus camera from the same company was already being used in the eye clinic. The ophthalmic photographers were familiar with the operation of this camera and the quality of the slides they produced was consistently high. It was these slides, digitised using a Kodak Megaplus 1.4 CCD camera (the same CCD camera as fitted to the Topcon TRC-50XT digital fundus camera), which had been used to develop our automated techniques. Thus the ability to acquire the same images directly using the TRC-50XT camera was a logical progression. The EURODIAB protocol For this study we adopted the photographic protocol developed for the EURODIAB IDDM Health Technology Assessment 2003; Vol. 7: No. 30 11 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Chapter 3 Photography FIGURE 2 Topcon retinal camera with IMAGEnet system Complications Study in which two fields of each eye are obtained for each patient.26 The first, the macular field, extends from the optic disc to the temporal retina and the second, the nasal-disc field, covers the region from the disc to the nasal retina; 50° fields were obtained, as opposed to the 45° fields used in the EURODIAB study, because this field of view was closest to that available on the Topcon TRC-50XT fundus camera used (Figure 3). The difference, however, is negligible and will, if anything, increase the area of the retina examined. The EURODIAB protocol was validated against the recognised gold standard of the modified Airlie House system,32 with which it ‘compared favourably’. In the latter, seven pairs of 30° stereo photographs are obtained for each eye of each patient. Clearly the four images per patient of the EURODIAB scheme are vastly preferable to the 28 photographs of the modified Airlie House system if either is being considered for use in a large-scale screening programme. The same protocol was used for both colour slide photography and acquisition of the digital images, permitting their direct comparison. An initial concern was the quality of images produced by the digital fundus camera. In the following section is reported a study of the overall quality of the images with respect to the visibility of structures such as the nerve-fibre layer and the retinal vessels. Image artefacts due to camera dust Soon after patient photography got under way, it became apparent that the digital images acquired were being corrupted by the presence of dust inside the camera (Figure 4a) and that this dust threw shadows on the images which closely resembled small, dark microaneurysms. In the 5 months it took to identify and solve the problem, a total of 158 subjects had been photographed. In order to salvage the images collected and prevent the costly and time- consuming recall of subjects, a program was written to remove these dust artefacts from the images. Although the dust in the camera could be disturbed on a day-to-day basis, it was found that on any given day the pattern of dust was relatively stationary. Hence, if the average of all the images collected on a given day was calculated, then the superimposed dust would produce a large response whereas non-coincident features, such as the retinal vessels, would be suppressed (Figure 4b). From this average image, a ‘mask’ could be generated (by shade-correcting and thresholding the average image; Figure 4c). This mask could then be used to eliminate the dust artefacts in each image by replacing the image pixels at these locations with the average (actually the median) of its neighbours (Figure 4d). Photography 12 FIGURE 3 Two 50° red-free images of the right eye taken according to the EURODIAB IDDM Complications Study protocol. Left, macular view; right, nasal disc. FIGURE 4 Removal of camera dust artefacts from the region of interest in an image. (a) Part of the original image showing the presence of dust as small dark shadows. (b) The average of all images taken on the same day’s photography. After processing this average image a ‘mask’ is generated giving the locations of the image dust artefacts (c). This dust mask can then be used to remove the corrupting dust artefacts in the image (d). (a) (b) (c) (d) Grading image quality To assess the quality of the digital images a grading scheme was developed that was adapted from the scoring system for vitritis.33 The categories used to grade the quality of both the colour slides and digital images are listed in Table 4. Q1 indicates the highest quality images in which the fine structure of the nerve-fibre layer is clearly visible. In images of grade Q2, the nerve-fibre layer is not visible but the smallest vessels in the image are in sharp focus. In Q3 images, these small vessels are blurred but the larger vessels are well defined. Q4 indicates that the large blood vessels in the major arcades are just blurred and, finally, if the image is judged to be of insufficient quality for a classification of retinopathy to be made, then grade Q5 is assigned to the image. This last grade is also assigned to images for which there is significant blurring of major arcade vessels in one-third or more of the image, in the absence of visible referable retinopathy. This might occur, for instance, as a result of shadows falling across the image such caused by misalignment of the eye. For an overall grade of retinopathy to be assigned to a patient, quality grades between Q1 and Q4 were required for all four fields photographed. This ensures that lesions present in an obscured or degraded region of the retina are not overlooked, leading to potentially serious undergrading of the patient’s retinopathy. In practice, we would expect these patients whose images are graded as Q5 to be referred for separate examination using slit- lamp biomicroscopy by a trained individual such as an optometrist or an ophthalmologist. Grading of the digital images and colour slides was performed solely by the research fellow. As a result, direct comparison could be made between the grades assigned to each method of photography, avoiding concerns about inter- observer variability. The data in Table 5 are replotted in Figure 5 to show overall image quality for both fields. In Figure 6, the difference in image quality between colour slide and digital image is shown. Individual colour slides had a higher frequency of the highest quality grade Q1 slides (22%) than digital images (18%) (p < 0.001), but also a higher frequency of the lowest quality grade compared with digital images (5% versus 2%). If one examines the number of images falling into categories Q1, Q2 and Q3 then there is a higher proportion of digital images than colour slides (76 and 67% respectively, p < 0.001). Health Technology Assessment 2003; Vol. 7: No. 30 13 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. TABLE 4 Scheme used to grade 35-mm colour slide and digital image quality Grade Definition Q1 Nerve-fibre layer visible Q2 Nerve-fibre level not visible Q3 Small vessels blurred Q4 Major arcade vessels just blurred Q5 Significant blurring of major arcade vessels in over one-third of the image in the absence of visible referable retinopathy TABLE 5 Comparison of quality of the digital and 35-mm colour slide images Digital image Colour slide Q1 Q2 Q3 Q4 Q5 Total Q1 492 329 64 35 1 921 Q2 138 669 287 172 5 1271 Q3 41 249 159 156 2 607 Q4 55 354 242 458 28 1137 Q5 6 45 26 83 51 211 Total 732 1646 778 904 87 4147 0.00 Q1 Q2 Q3 Q4 Q5 Colour slide (67% grades Q1–Q3) Digital red-free (76% grades Q1–Q3) 0.10 0.20 0.30 0.40 Image quality Fr eq ue nc y FIGURE 5 Colour slide and digital image quality distributions (data as in Table 4) Failure to obtain gradable images In Figure 7, the number of patient visits for which one, two, three or four ungradable fields were obtained for the two modalities are compared. Of the 1041 patient visits for which both colour slides and digital red-free photographs were obtained, twice as many visits for colour slide photography (107 or 10%) generated one or more ungradable fields compared with digital photography (54 or 5%). This was statistically significant (p < 0.001). Colour slide photography would result in a large number of patients having to be recalled for photography and, of course, this has associated costs. Conclusions One of the main concerns with using digital images is that quality would be impaired by the digitising process. The choice of the Topcon fundus camera and IMAGEnet software was made primarily on the basis that the Kodak Megaplus CCD camera used by this system had the finest image digitisation available at that time, 1024 × 1024 pixel image array. To reduce possible problems with image quality further it was specified that the software should not utilise any form of image compression. Image quality can only be defined in terms of the use to which the image is to be put,34 in this case the detection of diabetic retinopathy. This will be explored further in later chapters. However as an initial step, a grading system was used that utilised the clarity of features such as nerve-fibre layer and vessels as a measure of overall image quality. This demonstrated that the finest quality photographs were produced by conventional photography. This is not surprising as the maximum resolution of conventional photographs is approximately 2–3 times better than the digital camera used in this study. However, if the first three categories of quality are regarded as acceptable, then the digital images were superior to the colour slides. When the image quality was analysed in terms of field, then the poorest quality field tended to be the nasal field, reflecting the need for a widely dilated pupil and the absence of peripheral cortical cataract, a common clinical finding, to obtain high-quality images. Digital photography produced fewer ungradable images, approximately half the number of visits for digital photography producing one or more ungradable images compared with colour slide photography. This was mainly due to the photographer having instant feedback on image quality by being able to view them instantaneously on the computer screen. If the image was not of sufficient quality the photographer simply repeated the photograph until one of sufficient quality could be obtained. This meant that approximately 50% fewer individuals would need to be recalled for repeat photography if a digital rather than a conventional photographic camera were used for image capture. Photography 14 –3 –2 –1 equal Difference in image quality Positive: red-free digital > colour slide Negative: colour slide > red-free digital 1 2 3 4–4 0.00 0.10 0.20 0.30 0.40 0.50 Fr eq ue nc y 1 2 3 4 No. of ungradable EURODIAB fields per patient visit Colour slide: 107 visits affected (10%) Digital red-free: 54 visits affected (5%) 0 10 20 30 40 50 N o. o f p at ie nt v is its FIGURE 6 Comparison of colour slide and digital image quality FIGURE 7 The number of patient visits for which one, two, three or four EURODIAB fields were ungradable Why automate? Establishing a photographic screening programme has workload implications. In addition to the time taken to photograph the diabetic population, trained personnel are then required to view and grade all the acquired images. The acquisition of digital images reduces the time previously associated with conventional photography, because it eliminates the overhead associated with the development of photographic slides. This also accounts for the higher proportion of digital images which are of reasonable quality; they may be viewed immediately and retaken if necessary without the need to recall the patient. There is clearly a case to be made for an automated analysis of images to eliminate the need for tedious and costly manual grading of the images. Digital images are uniquely suited to this task since a well-designed computer program is able to read these images and process them directly, without the need for human intervention. Automated analysis also offers a consistency and reliability of interpretation that is not found with the human observer who is prone to fatigue. What lesions should we detect? Microaneurysms Previous screening programmes have focused on the need to detect sight-threatening retinopathy; however, opinion has more recently shifted to the detection of any retinopathy. Microaneurysms are the earliest detectable signs of diabetic retinopathy and their numbers correlate closely with the severity of the disease.3,35,36 They are a logical choice for a screening programme, enabling patients with all degrees of disease severity to be detected. Furthermore, since the number of lesions increases with the severity of the disease, the likelihood of detecting patients with more severe retinopathy will increase, helping to ensure that those in need of referral are not missed. The process of manually identifying individual microaneurysms in a fundus image is tedious and prone to operator error. Although computer- assisted localisation of these lesions can help to reduce intra- and inter-observer errors,37 a fully automated and entirely objective approach is clearly preferable. Hard exudates Diabetic maculopathy (clinically significant macular oedema) is the most common cause of visual impairment in diabetic patients and is indicated by the presence of retinal thickening or hard exudates within one disc diameter of the fovea. Current screening methods favour non-stereoscopic photography owing to the increased complexity of acquiring and interpreting stereo photographs. These techniques, however, offer no information about the presence of retinal thickening. The reliable detection of patients with macular exudates is of increased importance, therefore, if patients with potential clinically significant macular oedema are not to be overlooked. The established means of manually quantifying exudate presence, via comparison with two standard photographs,38 is unavoidably subjective, has limited precision and, consequently, is inherently inaccurate. Computerised detection, on the other hand, offers the benefits of a fully automated, and therefore objective and repeatable, analysis with accurate quantification of the extent of the pathology. Automated detection of retinopathy We have previously developed programs for the automated detection and quantification of microaneurysms in retinal fluorescein angiographic images.22,24,39 It has been demonstrated that this automated system detects microaneurysms almost as well as clinicians.39 This program was used to analyse the fluorescein images taken in the disease monitoring part of the study (Chapter 6). For the screening study (Chapter 5), we needed to adapt the technique to the more challenging task of detecting microaneurysms in digital red-free images. The general philosophy of our approach is to construct a definition of the appearance of a Health Technology Assessment 2003; Vol. 7: No. 30 15 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Chapter 4 Automated detection of retinopathy microaneurysm or hard exudate. As will be described below, this is done on the basis of defining a number of features of the pathology, such as those relating to shape and intensity. By means of a training set of images, in which the pathology has been identified by an experienced observer, a set of rules is constructed for combining the values of these features in such a way as best to discriminate genuine from false pathology. As will be seen later in this chapter, the choice of an operating point for the program still requires a decision to be taken as to what constitutes an acceptable level of true and false- positive responses. Microaneurysm detection The definition of a microaneurysm We have defined microaneurysms as small (25–200 �m in diameter), circular features with an approximately Gaussian or ‘bell-shaped’ profile. In red-free images they appear dark against the background whereas in the frames of a fluorescein angiogram they are bright. In both cases, however, we have attempted to mimic the decisions of clinicians viewing these images by measuring each microaneurysm identified by them in a training set of diverse images. Thirteen measurements of each object are taken into account, including size, perimeter, circularity, aspect ratio, intensity and similarity to the ideal Gaussian profile. We do more than simply setting limits on these parameters, however. For instance, circularity is less significant for small microaneurysms that lie at the limit of the resolving power of the camera. The criteria for classifying an object as a microaneurysm will therefore vary with the size of the object. Operation of the program The initial processing of each image is performed in three stages. First the image is ‘shade-corrected’ to remove differences between images caused by changes in the illumination conditions under which each was obtained. Second, large features such as vessels and haemorrhages are removed by excluding all structures which are greater than a particular linear extent and, finally, only those remaining features which resemble the approximate shape and size of a microaneurysm are retained for subsequent classification. At this stage of the processing, the image contains a number of candidate microaneurysms only some of which will be genuine. For each of these candidates the program makes 13 measurements of intensity and shape, such as perimeter length, aspect ratio and circularity. These measurements are then tested using a set of rules which, if all are satisfied, identify the candidate as a microaneurysm. For example, a large object with insufficient average intensity might be rejected whereas a smaller, less prominent object is accepted owing to its greater circularity. At this stage, it was necessary to draw up the rules for the program by examining the performance of the program on a set of genuine microaneurysms identified by the consultant ophthalmologist. Unsurprisingly, it was not possible to classify correctly all the microaneurysms identified by the consultant without also detecting a large number of spurious objects. As a result, it was necessary to decide on the degree of sensitivity and specificity which would be clinically acceptable. Calibrating and testing the microaneurysm detector Figure 8 illustrates the performance of the program for the detection of individual microaneurysms. It is expressed in terms of a free response receiver operating characteristic (FROC) curve that plots the sensitivity of the program in detecting microaneurysms as a function of the number of false positives per image, that is, the average number of spurious microaneurysms reported as being present in the image. The data were generated by analysing a set of images consisting of 44 normals, 41 mild, 12 moderate, Automated detection of retinopathy 16 0.0 0.0 0.2 0.4 0.6 0.8 0.2 0.4 False positives per image 0.6 Se ns iti vi ty FIGURE 8 FROC curve comparing the performance of the microaneurysm detector (filled circles and dashed line) on digital red-free images with that of five clinical observers (filled and open triangles), with an ophthalmologist as the gold standard. The operating point of the detector is indicated by the open square. three severe and two early. By varying the program’s criteria for what constitutes a microaneurysm, by relaxing or tightening the rules used to classify each detected candidate microaneurysm, the effect on specificity of increasing the program’s sensitivity can be investigated. The outcome from the program is compared with the results from five observers looking at the same images: two ophthalmologists, two diabetologists and one physicist experienced in interpreting retinal images. The gold standard was generated by an experienced ophthalmologist. Clearly, the opinions of the human observers vary in sensitivity and specificity (number of false positives per image). For instance, the observer with the highest sensitivity has achieved this result at the expense of the greatest number of false positives (identification of objects that the ophthalmologist did not consider to be microaneurysms). The program’s FROC curve shows how changing the rules affects the sensitivity and specificity of microaneurysm detection. Obviously, it is necessary for the purpose of this study to select one operating point. Although a high sensitivity for a screening task is clearly desirable, specificity is also important. This is because the use of four fields per patient increases the likelihood of detecting a falsely positive microaneurysm and hence incorrectly concluding that a normal patient has retinopathy. By relaxing the rules, the number of genuine microaneurysms identified, and hence the sensitivity, could be increased. On the other hand, tightening the rules reduces the number of spurious objects erroneously classified as microaneurysms, so increasing specificity. An estimate of the best compromise between these two parameters was made, based on the analysis of the training set, and this version of the rules was chosen as the ‘operating point’ of the program shown in the Figure 8 by the open square. This corresponded to a sensitivity of 43% and 0.11 false positives per image. Automated detection of hard exudates Hard exudates are composed of lipoprotein and lipid-filled macrophages. This gives them a yellow, waxy appearance which is highly reflective and, as a consequence, they appear as regions of high local intensity in digital red-free photographs. Owing to their being formed by leakage from microvascular lesions, they have no characteristic shape but may be distinguished from similarly intense lesions such as cotton-wool spots or drusen by the strength of their boundary definition. In contrast, cotton-wool spots and drusen are diffuse structures with considerably less intense edges. This information has been used to classify bright objects in the images according to their size, perimeter length, overall intensity and edge definition. Once again the choice of appropriate classification rules is dictated by the exudates manually identified by an ophthalmologist in a training set of images. This set consisted of 42 images containing 479 exudate objects. Operation of the exudate detector In order to enhance those features with strong edge definition (i.e. potential exudates), the images are first ‘sharpened’ by application of an appropriate filter. This sharpening operation also helps to ensure that the detected features are accurately delineated prior to their measurement (see below). Shade correction is then performed as described above to eliminate the effects of illumination. The next stage in the processing is to locate the optic disc to enable it to be eliminated from subsequent analysis of the image. This is achieved by searching the image for the region which best matches a model disc. This model consists of a bright circle, representing the surface of the disc, bisected by a dark, diverging, vertical stripe which mimics the pattern of the dark blood vessels leaving and entering the eye via the central retinal artery and vein. Having identified the optic disc, the remaining area of the image is thresholded to identify any features which are brighter than the background and therefore could be hard exudates. Once again these candidates are classified by comparing their measurements with those of the exudates identified by a clinician in the set of training images. This comparison generates a number of rules based on the values, or pairs of values, of these parameters. Testing the hard exudate detector As for the microaneurysm detection program, the ability of the program to detect hard exudate objects was assessed using an FROC experiment, the results being shown in Figure 9. A test set of images was used consisting of 50 images in which a total of 215 individual exudates were identified by the clinical research fellow. The performance of the program is varied by relaxing or tightening the rules used to define an exudate in a manner Health Technology Assessment 2003; Vol. 7: No. 30 17 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. identical with that used for the microaneurysm program. Once again the performance of a number of clinicians asked to perform the same task was also included for comparison and the gold standard against which both are judged is provided by an ophthalmologist. The operating point is chosen to give an extremely low false-positive detection rate, 0.2 false-positives per image, so as to maximise the discrimination between exudates and cotton-wool spots or drusen. Consequently, the sensitivity will be low, 44%. However, it must be appreciated that the clinical significance of sensitivity and specificity will vary according to the total number of exudate objects which are actually present in a given image. For example, a single false positive detected in a normal image is much more serious than a single false positive detected in an image with 30 real lesions present. Lesion location and maculopathy Diabetic maculopathy (clinically significant macular oedema) is an important cause of visual impairment in patients with diabetes.40–42 It is indicated by a thickening of the retina, which can only be positively identified using stereo photography. In the absence of stereo photographs, however, a patient will be referred for further examination if he or she exhibits microaneurysms or hard exudates within one disc diameter of the fovea. By locating the fovea in the macular images obtained, we are able to define this region in the image and hence potentially identify patients with referrable macular lesions. The position of the fovea is identified in a similar manner to the optic disc position described above, finding the location in the image which best matches a model of the fovea. This model is simply a small, dark cone which mimics the increase in pigmentation in this area of the retina. Although an estimate of the macula size (one disc diameter or 1500 mm) can be made given the approximate resolution of the image, differences in optical power between patients means that this value is unreliable. A second estimate is therefore made based on a fraction (0.38) of the distance between the fovea and the centre of the optic disc. The average of these two measurements is then used to specify the size of the macular area. Because the locations of any microaneurysms or hard exudates detected in the image are known, we are able to calculate the numbers of each which lie within one disc diameter of the fovea. In Chapter 5 we will investigate whether this information enables us to detect patients with notable macular lesions and how many of the same were, on subsequent examination, found to have clinically significant macular oedema. Turnover of microaneurysms We have proposed investigating whether automated digital image processing techniques can follow the progression of retinopathy. Although the total number of microaneurysms has been shown to correlate with the severity and progression of early retinopathy,2,3,43 it has also been observed that these lesions are not static but appear and disappear over a period of time.44–46 This turnover of microaneurysms is not well understood but it has been suggested that the rate of their formation and regression might be related to the progression of retinopathy.46 In Chapter 6 we will calculate the turnover of microaneurysms detected in the three 6-monthly fluoresceins obtained from these patients and examine how these figures relate to the severity and progression of retinopathy present. Image registration In order to calculate turnover of microaneurysms, it is necessary to align or ‘register’ the images concerned. This allows corresponding microaneurysms which have been detected in the Automated detection of retinopathy 18 FIGURE 9 FROC curve comparing the performance of the hard exudate detector (filled circles and dashed line) with that of five clinical observers (filled triangles), with an ophthalmologist as the gold standard. The operating point of the detector is indicated by the open square. 0.8 0.6 0.2 0.4 0.0 2 4 6 82 False positive objects per image Se ns iti vi ty Health Technology Assessment 2003; Vol. 7: No. 30 19 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. different images to be identified according to whether they overlap (by >10%). Owing to the small size of these lesions, the registration is required to be very accurate and a cross- correlation algorithm47 with minor modifications has been found to give good results for these images. Once registered, each pair of the three 6-monthly fluorescein frames is analysed to give the number of static microaneurysms (those present in both images), new microaneurysms, (those on the second image only), and regressed microaneurysms (those on the first image only). Conclusions In this chapter we have described the algorithms for the automated detection of microaneurysms and hard exudates that have been used in this study. In the following chapters we will describe the results of our investigation into whether these automated analyses of digital red-free photographs may be used as a screening tool for diabetic retinopathy, and whether changes in the numbers of lesions present can be used to follow the progression of the disease. In addition, by identifying the locations of the fovea and optic disc in each image, the positions of the detected lesions can be related to the location of the macula. Using this information we are able to investigate whether the detection of macular microaneurysms or hard exudates or both provides useful markers for the referral of patients with suspected clinically significant macular oedema. Finally, we are able to calculate the turnover of microaneurysms in individual frames of consecutive fluorescein angiograms. Using this technique, we hope to show that the rate of new and regressed microaneurysms can provide useful information about the natural history of diabetic retinopathy, and that this information might allow the point when treatment is required to be determined. Health Technology Assessment 2003; Vol. 7: No. 30 21 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Introduction This study aims to look at the role of digital imaging in screening for diabetic retinopathy so that timely intervention, both medical and surgical, can be applied before significant visual impairment occurs. The patient population was described in the section ‘Patient recruitment’ (p. 7); to recap, 586 patients attended a hospital assessment clinic where digital and conventional photography was performed. Slit-lamp examination by an ophthalmologist was performed as the gold standard. Patients then attended a study optometrist who performed an examination of the fundi using slit-lamp biomicroscopy. These examinations were then repeated 1 year later. Slit-lamp biomicroscopy Six high-street optometrists were recruited through the auspices of the Aberdeen and North- east Scotland branch of the Association of Optometrists. Each underwent a specially devised training programme consisting of a day of lectures, a day of slide examination, practical demonstration of slit-lamp biomicroscopy and regular eye clinic attendance, culminating in a formal examination of their ability to recognise the features of diabetic retinopathy using slit-lamp biomicroscopy. The optometrists examined the subjects using slit- lamp biomicroscopy through dilated pupils in their own premises. A poster with examples of the various features of retinopathy was provided for reference. Patients were graded according to a modified interim ETDRS severity scale. This scale is based on the grading of features of diabetic retinopathy detected in seven-field stereoscopic 35-mm colour slides against standardised slides. The presence of four blot haemorrhages in any one quadrant was used as the definition of severe retinal haemorrhages, as this is a rough approximation to the reference slide used by the ETDRS. The definitions of non-proliferative diabetic retinopathy (NPDR) used in the study are given in Table 6. Proliferative diabetic retinopathy (PDR) was defined as the presence of new vessels in the fundus. Clinically significant macular oedema was defined as the presence of retinal thickening within a one-disc diameter radius of the centre of macula. The reference standard was taken as slit-lamp biomicroscopy performed by consultants or their specialist registrars with a special interest in medical retina. The clinical grading protocol was the same as the optometrists’. This is referred to as the ‘ophthalmologist gold standard’. Photography and digital imaging Conventional mydriatic retinal photography was performed using a Topcon 50X fundus camera with 35-mm transparencies on Kodachrome 64 film. High-resolution (1024 × 1024 pixels) red-free digital photography was performed on the same camera using the IMAGEnet 1.53 1024 digital image acquisition system as described in the section ‘Choice of digital fundus camera’ (p. 11). Chapter 5 Question 1: can a digital system detect the presence of diabetic retinopathy (of any sort/level)? TABLE 6 Definitions of NPDR grades Level Retinopathy features Mild At least one microaneurysm Moderate Severe haemorrhages (≥ 4 blot haemorrhages) in one quadrant and/or cotton-wool spots or venous beading or IRMA definitely present Severe Any one of the following: severe haemorrhages in four quadrants; venous beading in two quadrants; severe IRMA in one quadrant Very severe Any of the two ‘severe’ categories Photographs were taken according to the EURODIAB protocol (see the section ‘The EURODIAB protocol’, p. 11). All photographs and digital images were graded by a trained research registrar for image quality (see the section ‘Grading image quality’, p. 13) and for retinopathy severity according to the EURODIAB protocol (Table 7). This is referred to as ‘manual grading’. The research registrar was trained by attending weekly diabetic clinics for 6 months under consultant supervision. Her effectiveness at grading images was then formally evaluated by the consultant before any of the study images was graded. Digital images were stored in their original loss- less TIFF format on CD-ROMs and manually transferred to the Department of Bio-medical Physics, University of Aberdeen. The images were analysed using the software described in Chapter 4. The presence of any retinopathy and the quantity of retinopathy were recorded. The presence of exudates and/or haemorrhages within one disc diameter of the centre of the fovea in the macular field were used to infer the possible presence of diabetic macular oedema as retinal thickening cannot be visualised in monocular images. Distribution of retinopathy amongst patients The levels of retinopathy and clinically significant macular oedema detected by the ophthalmologists in patients on their first visit are given in Table 8. Figure 10 shows the distribution of retinopathy amongst eyes and fields. Detection of retinopathy The performance of the optometrists, the manual grading of red-free digital images and manual grading of 35-mm colour slides were determined by using the ophthalmologists’ clinical grading as the gold standard. Table 9 shows the sensitivity and specificity achieved for the different screening modalities in terms of whether or not an image was reported as having retinopathy. The actual agreement as to the level of retinopathy is shown in Figure 11. The results have been analysed for all retinopathy, that is, all patients irrespective of the degree of retinopathy, and for early retinopathy, which uses the results from those patients who have either mild or no retinopathy. Of course, the specificity will be the same in both cases. Question 1: can a digital system detect the presence of diabetic retinopathy (of any sort/level)? 22 TABLE 7 EURODIAB retinopathy grading protocol Level Retinopathy features Mild HMA and/or hard exudates Moderate Very severe HMA in one field, or HMA plus CWS and/or IRMA and/or VB Severe Very severe HMA in both fields, or severe HMA in one field plus very severe CWS and/or severe IRMA and/or severe VB Photocoagulated Photocoagulation scars Proliferative New vessels and/or fibrous proliferations, and/or pre-retinal haemorrhage, and/or vitreous haemorrhage HMA, haemorrhages and microaneurysms; CWS, cotton- wool spots; IRMA, intra-retinal microvascular anomalies; VB, venous beading; severe, lesion present ≥ standard photograph 1 but < standard photograph 2; very severe, lesion present ≥ standard photograph 2. TABLE 8 Frequency of retinopathy levels and clinically significant macular oedema (maculopathy) in patients at the first visit Retinopathy level No. Clinically significant macular oedema No. None 429 Absent 559 Mild 108 Present 24 Moderate 36 Severe 6 Very severe 0 Early PDR 5 High-risk PDR 2 Health Technology Assessment 2003; Vol. 7: No. 30 23 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. 0.80 0.60 0.40 Both eyes Right eye only Eye(s) Left eye only 0.20 0.00 1.0 0.8 0.6 0.4 0.2 0.0 Fr eq ue nc y Fr eq ue nc y Fr eq ue nc y 0.0 1.0 Mild (265) Moderate (71) Severe (13) Very severe (4) Overall (370) 0.8 0.6 0.4 0.2 0.0 0.2 Colour slides Digital red-free images Fr eq ue nc y 0.4 0.6 0.8 Both fields Macular field only Field(s) Nasal disc field only Right macular Right nasal disc Left macular Field No. of fields with retinopathy Left nasal disc One of 4 fields One of 3 fields One of 2 fields 1 field 1 2 3 4 (a) (b) (c) (d) FIGURE 10 (a) The proportion of patients with manually graded retinopathy present in the left eye only, the right eye only, or both. (b) The proportion of digital red-free eye-field pairs with manually graded retinopathy present in either the macular field only, the nasal-disc field, or both. (c) The distribution of retinopathy throughout the four fields obtained using the EURODIAB protocol. (d) The number of fields exhibiting retinopathy according to the level of retinopathy. TABLE 9 The detection of any retinopathy Screening modality All retinopathy Early retinopathy Sensitivity (%) Specificity (%) Sensitivity (%) Specificity (%) 35-mm colour slide 89 89 85 90 Manual grading of digital images 83 79 81 81 Optometrists 75 82 71 82 Automated grading of digital images 83 71 82 71 Automated analysis in diabetic macular oedema In this part of the study, 583 patients were entered; 24 cases of clinically significant macular oedema (retinal thickening within one disc diameter radius of the geometric centre of the eye) were diagnosed by the ophthalmologists (Table 8). The performance of the optometrists, the manual grading of red-free digital images and manual grading of 35-mm colour slides were determined using the ophthalmologists’ clinical grading as the gold standard (Table 10). Sight-threatening retinopathy Sight-threatening retinopathy was defined as that classed as moderate or worse on the retinopathy rating scale and/or the presence of clinically significant macular oedema. The performance of the optometrists, the manual grading of red-free digital images and manual grading of 35-mm colour slides at detecting retinopathy in such patients, irrespective of agreement in grading, is shown in Table 11. Once again, the ophthalmologists’ clinical grading is taken as the gold standard. In these results sight-threatening retinopathy for the automated analysis is defined as patients with microaneurysms or exudates in the macula. As such it does not grade the severity of the retinopathy. Is the nasal-disc field required? Many established screening protocols obtain only a macular view, chiefly because detection of sight- Question 1: can a digital system detect the presence of diabetic retinopathy (of any sort/level)? 24 1.0 0.8 Fr eq ue nc y 0.6 0.4 0.2 0.0 Normal Mild Moderate Retinopathy grade(a) (b) Severe Very severe Optometrists (548) Colour slide (657) Red-free manual (630) Red-free automated (688) 1.0 0.8 Fr eq ue nc y 0.6 0.4 0.2 0.0 Normal Mild Moderate Retinopathy grade Severe Very severe Optometrists (130) Colour slide (186) Red-free manual (186) Red-free automated (200) FIGURE 11 Comparison of grading produced by screening techniques with the ophthalmologist gold standard for patients with (a) no retinopathy and (b) mild retinopathy TABLE 10 The detection of clinically significant macular oedema by optometrists, by grading 35-mm colour slides and by grading digital images Screening modality Sensitivity Specificity (%) (%) 35-mm colour slides 83 84 Manual grading of digital images 83 83 Optometrists 46 92 Automated grading of digital images 76 85 TABLE 11 The detection of sight-threatening retinopathy by optometrists, by grading of 35-mm colour slides and by grading of digital images Screening modality Sensitivity Specificity (%) (%) 35-mm colour slides 96 89 Manual grading of digital images 93 87 Optometrists 73 90 Automated grading of digital imagesa 77 88 a Clinically significant macular oedema only. threatening retinopathy has been the main objective in the past. As Figure 10(b) shows, according to a clinician interpreting the 35-mm colour slides 8% (45/580) of eye–field pairs showed retinopathy in either nasal-disc field only and 14% (91/644) when digital photography was used. If the data are analysed in terms of patients, then if single-shot macula-only fields had been used, 22/322 (7%) patients with any retinopathy would have been undetected and 6/214 (3%) patients with sight-threatening retinopathy would have been undetected (Table 12). The corresponding results for manual analysis of the red-free images were 59/366 (16%) for any retinopathy and 1/217 (0.5%) for sight-threatening retinopathy. In terms of sensitivity and specificity (Table 13), the use of a single macula field has no significant effect for sight-threatening retinopathy. Comparison of optometrist with ophthalmologist The difference between optometrists and ophthalmologists in grading of retinopathy (gold standard) is shown in Figure 12. In 74% (531/718) of patients there is agreement over the grading whereas the optometrists have a sensitivity of 76% and a specificity of 82% for the task of deciding whether or not the images are abnormal (Table 9), levels comparable with the other modalities. For the detection of macular oedema, there is more of a problem. The overall agreement on grading is 90% (631/696 excluding three ‘did not attends’) but this largely reflects the 93% agreement over the 676 normals that were present. The optometrists failed to detect clinically significant macular oedema in over half of abnormals (11/22). This probably reflects the lack of experience in detecting macular oedema, as in everyday practice optometrists would see mainly normal eyes. Discussion For historical reasons, a larger number of patients attend the hospital diabetic clinic in Aberdeen than might be expected in other areas. This is reflected in the demographics of the population studied, which has a higher prevalence of type 2 diabetes than might be expected in a hospital clinic. Although the majority of people in the study were of working age, it is of concern that 528 of 1114 patients approached either declined Health Technology Assessment 2003; Vol. 7: No. 30 25 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. TABLE 12 Number of patient visits with retinopathy by field: analysis made using 35-mm colour slides Macular field Nasal field No retinopathy Mild retinopathy Sight-threatening Total No retinopathy 616 22 34 672 Mild retinopathy 22 64 84 170 Sight-threatening 0 6 90 96 Total 638 92 208 938 TABLE 13 Effect of using a single macular view for any retinopathy and sight-threatening retinopathy All fields Macula only Condition Screening modality Sensitivity Specificity Sensitivity Specificity (%) (%) (%) (%) Any retinopathy 35-mm colour slides 89 89 86 92 Manual grading of digital images 83 79 80 88 Automated grading of digital images 83 71 79 80 Sight-threatening retinopathy 35-mm colour slides 96 89 95 89 Manual grading of digital images 93 87 93 87 Automated grading of digital images 77 88 77 88 to take part or failed to attend once recruited. This probably reflects the fact that patients were already being screened and might have implications for any stand-alone retinopathy screening programme. The reference standard chosen was that of a consultant, or his or her specialist registrar, performing slit-lamp biomicroscopy. Although seven-field stereoscopic photography is often used as a reference in research programmes, it is too cumbersome for everyday clinical practice. Slit- lamp biomicroscopy is the standard tool used by ophthalmologists to examine for diabetic retinopathy and was felt to be the most relevant reference standard for screening. It is also potentially the most sensitive standard as more retina is visualised than by the two 50° fields of the EURODIAB photographic protocol or of the seven 35° fields of the ETDRS photographic protocol. Two grading protocols were used, one for the 35- mm colour slides and the digital images and the other for slit-lamp biomicroscopy. The EURODIAB grading protocol was used for the slides and the digital images as it has been shown to correlate with the interim ETDRS retinopathy severity grades. The interim ETDRS retinopathy severity scale was modified for slit-lamp biomicroscopy, thus allowing comparison between the two techniques despite the different fields of view. From a practical point of view the presence of four blot haemorrhages in any one quadrant was used as the definition of severe retinal haemorrhages as this is a rough approximation to the reference slide used by the ETDRS. In terms of sensitivity and specificity there was little to choose between conventional photography and digital photography for detecting any or early retinopathy. For sight-threatening retinopathy the optometrists had the lowest overall sensitivity (73%) although their specificity (90%) was comparable to that of the other modalities. When looking at clinically significant macular oedema (maculopathy), a sub-set of sight- threatening retinopathy, optometrists performed particularly badly with a sensitivity of only 46%. The specificity of 92% was comparable to the other modalities. Detection of macular oedema is difficult and with hindsight requires more practice than they received. Automated detection of diabetic retinopathy has progressed rapidly in the last decade, and commercial programs are now becoming available.48–50 Recently, Lee and colleagues49 have published their work with apparently similar good results. Unfortunately, no details of the methods used or the resolution of the images were Question 1: can a digital system detect the presence of diabetic retinopathy (of any sort/level)? 26 Optometrists (718) Colour slide (940) Red-free manual (921) Red-free automated (985) 1.0 0.8 0.6 0.4 0.2 0.0 Fr eq ue nc y –4 –3 –2 –1 equal Difference in retinopathy grades (positive: overgrade, negative: undergrade) 1 2 3 4 FIGURE 12 Comparison of retinopathy grades derived by optometrists with those produced by grading 35-mm colour slides or digital red-free images published as they wished to protect this information prior to commercialisation. Our own work started with fluorescein angiography, but for this study it was realised that this was not a practical modality for screening and the computer algorithms were modified to work with high- resolution red-free digital images instead. Automated techniques have the advantage of repeatability. Individual human graders tend to have their own varying internal reference standards that are difficult to make conform, despite training, leading to intra- and inter- observer variability. However, with the algorithm used in our automated detection program it is necessary to select the particular combination of sensitivity and specificity at which the algorithm operates, as discussed in the section ‘Calibrating and testing the microaneurysm detector’ (p. 16). Grading of photographs or digital images is a repetitive task and where two-thirds are expected to be normal this leads to fatigue and boredom. Computers suffer from none of these complaints, although they are heavily reliant on images being of a sufficient quality to permit analysis. Images will therefore have to be graded for image quality before being processed. This is, however, a far simpler and quicker task than actually grading the image. Also, as was shown in the section ‘Grading image quality’ (p. 13), it is easier to obtain good quality images with the digital system. We have been able to achieve significant sensitivity and specificity for the detection of the presence of any diabetic retinopathy. That in itself may at face value not seem a great achievement, but when one considers the vast numbers of patients with diabetes in the UK that require screening, then this is no mean achievement. In a previous study comparing the automated detection of retinopathy, using only microaneurysms, with 35-mm colour slides graded by a research fellow48 we were able to achieve a sensitivity of 85% and a specificity of 76% for detecting whether or not a patient had retinopathy. As pointed out in that paper, since the decision on the presence of retinopathy requires four images to be analysed, this task will have a higher sensitivity but lower specificity than for detecting retinopathy in a single image. Assuming a prevalence for diabetic retinopathy of 30%, this would have meant that 51% of this population would be correctly classified as having no retinopathy. In this current study, in which ophthalmologists using stereo biomicroscopy were used as the gold standard, then if we consider all grades of retinopathy we have achieved a sensitivity of 87% but a lower specificity of 71% with the automated system. Despite this, the automated grading would still correctly classify 48% of a theoretical diabetic population as having no retinopathy. Clearly, such results depend on the population examined. For sight-threatening retinopathy, the specificity rises to 88% whereas the sensitivity is lower than that achieved by other modalities at 77%. The problem is that we have not been able to detect all forms of sight-threatening retinopathy as we do not as yet have a computer algorithm to detect new vessel formation. Obviously, automated grading cannot be used on its own, but in the context of a manual grading system it will still greatly reduce the workload by correctly identifying just under half the population as having no retinopathy. In addition, it would appear that automated grading is able to detect the presence of clinically significant macular oedema (maculopathy) (using a combination of the microaneurysm detection program and the exudate program), with a specificity comparable to other modalities. This allows graders to target their attention to images with potentially sight-threatening retinopathy. The use of the automated hard exudate detector as a tool for detecting any retinopathy was not found useful. Used in conjunction with the microaneurysm detector program it reduced the specificity from 71 to 67%, a result not surprising given the specificity shown in Figure 9. As mentioned above, it does have an important role in detecting maculopathy where the analysis is limited to retinopathy within a one disc diameter radius of the centre of the fovea. We used the two-field protocol of the EURODIAB study as this has been validated against the seven- field stereoscopic protocol of the ETDRS. Although the EURODIAB retinopathy levels do appear to correlate with the interim ETDRS retinopathy levels, it should be remembered that it was only validated against 24 patients and thus it cannot be certain that a two-field protocol is sufficient to detect all sight-threatening retinopathy. Using only one field would have failed to detect a significant number of patients with any retinopathy. However, in the case of sight-threatening diabetic retinopathy, up to 3% of patients were missed. As the sensitivity and specificity changed little, then in some circumstances the use of macular fields only might be regarded as acceptable. Health Technology Assessment 2003; Vol. 7: No. 30 27 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Health Technology Assessment 2003; Vol. 7: No. 30 29 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Introduction The previous chapter examined the effectiveness of the digital system in providing the ophthalmologist with a reliable, quantitative measure of retinal pathology. In this chapter, the value of this information as a tool for monitoring the progression of the disease will be evaluated. The important clinical questions are whether a digital system can detect progression of retinopathy and whether it can be used to determine when treatment is required. An additional facet of this study is to look at the natural history of retinopathy. Therefore, in addition to those patients initially assigned to this study, that is, those diagnosed on the initial screening study as having moderate proliferative retinopathy or worse, or clinically significant maculopathy, a group of patients with no or mild retinopathy are included. Study protocol It had originally been planned to study only patients with sight-threatening retinopathy. It proved impossible to obtain sufficient patients, mainly because those with such retinopathy usually underwent treatment before progression could be measured and so the entry requirement was relaxed. Thirty-seven patients, having mild or no retinopathy, who were participating in the screening arm also agreed to take part in this study. A total of 81 patients attended for the first appointment; the spread of retinopathy is shown in Table 14. Of these, 38 patients, 16 of whom had sight-threatening retinopathy, completed the full series of three fluorescein angiograms and fundus photographs every 6 months over a period of 12 months. The fluorescein angiograms were analysed for the presence of microaneurysms and the number of new, static and regressed microaneurysms, as described in the section ‘Turnover of microaneurysms’ (p. 18). Patients also had a variety of potential risk factors measured, including body mass index, blood pressure, HBA1c and microalbuminuria. These data will be presented in the section ‘Risk factors’ (p. 31). Can digital red-free photography monitor progression of retinopathy? Crucial to using the digital fundus camera to measure change in retinopathy is an assessment of the reproducibility of retinopathy gradings. Figure 13(a) shows how the results from a clinician manually grading the digital red-free images varies when assessed against the gold standard; the results for 35-mm colour slides are shown in Figure 13(b). There can be significant variation in the grading given; between 10 and 18% of images classified as mild are graded as moderate and up Chapter 6 Question 2: can a digital system detect progression of retinopathy? Question 3: can a digital system determine when treatment is required? TABLE 14 Retinopathy of those patients entering the study No. of patients Those undergoing three fluorescein Level of retinopathy Totala studies None 1 0 Mildb 36 22 Moderate 31 13 Severe 6 3 Very severe 0 0 Early PDR 5 0 High-risk PDR 2 0 a Of these patients, 21 also had maculopathy. b Of these patients, eight also had maculopathy. to 30% of moderates are graded as mild. This intrinsic variability in grading reproducibility will provide a limit to the extent to which a change in retinopathy level can be measured. The optometrists’ results (Figure 13c) show a greater spread in gradings; for example, only 52% of the milds were correctly classified. Can automated analysis of digital images follow progression of retinopathy? It was postulated in the proposal for this study that a more objective measure of retinopathy, in Questions 2 and 3 30 (a) (b) (c) 1.0 0.8 0.6 0.4 0.2 0.0 Fr eq ue nc y Normal Mild Moderate Digital red-free retinopathy grade Severe Very severe Normal (630) Mild (186) Moderate (77) Severe (11) Very severe (1) Colour slide retinopathy grade Normal (657) Mild (186) Moderate (78) Severe (11) Very severe (1) 1.0 0.8 0.6 0.4 0.2 0.0 Fr eq ue nc y Normal Mild Moderate Severe Very severe Optometrist’s retinopathy grade Normal (548) Mild (130) Moderate (24) Severe (3) Very severe (0) 1.0 0.8 0.6 0.4 0.2 0.0 Fr eq ue nc y Normal Mild Moderate Severe Very severe FIGURE 13 The ability of a clinician grading (a) digital images and (b) 35-mm colour slides to monitor the progression of diabetic retinopathy. The x-axis indicates the reference grade assigned by the ophthalmologist viewing the patient’s retina directly and each set of bars indicates the corresponding spread of values assigned by a clinician grading the 35-mm colour slides of each retinopathy severity group. The numbers in parentheses indicate the population in each group. In (c) the results of optometrist grading are given. particular the number of microaneurysms, might provide a means for assessing progression of the disease. Figure 14 shows, for each patient retinopathy grade, the mean number of microaneurysms per patient and the variation in that number. Results are given for both red-free (a) and fluorescein angiograms (b). Although there is a general trend towards more microaneurysms with increasing retinopathy grade, there is considerable variability in the mean number per patient in any group. These results, of course, reflect the ability to use the average number of microaneurysms to measure differences in retinopathy grading. It can be argued that measuring the changes in the number of microaneurysms is likely to be more effective. Red-free digital images were taken at a 12-month interval and the data are shown (Figure 15) as a function of the change in grade of retinopathy between the two studies. In Figure 15(a) the mean number of microaneurysms is plotted and in Figure 15(b) the data are shown as a percentage change. Only two of the groups in which the retinopathy showed a deterioration had sufficient patients to do a statistical analysis. There was a statistically significant increase in the number of microaneurysms in those patients who developed from mild to moderate retinopathy (paired t-test, p = 0.02) and from none to mild (p = 0.01). Problem of microaneurysm turnover Figure 16 demonstrates the dynamic nature of microaneurysms in fluorescein angiograms. Taking all levels of retinopathy (Figure 16a), only 45% of the microaneurysms seen on the first visit are present 6 months later and 36% of those at the 6-month visit are still seen at the 12-month visit. Similar turnover is seen in Figure 16(b) for the sight-threatening retinopathy group (42 and 36%, respectively). As the overall total number of microaneurysms shows relatively little change, then the regressed microaneurysms are being balanced by new ones. If one examines the microaneurysm turnover rate for individual patients (Figure 17), where turnover rate is calculated as the ratio of (a) new or (b) regressed microaneurysms to the total number of microaneurysms, then a large spread of values is found. There is no statistically significant difference between the turnover rate of either new or regressed microaneurysms for patients with mild or with sight-threatening retinopathy. Risk factors As mentioned in the section on questions 2 and 3 Health Technology Assessment 2003; Vol. 7: No. 30 31 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Normal Mild Mod Sev V sev Early High-risk 150 100 50 0Normal (688) Mild (200) Mod (77) Sev (12) V sev (1) Early (5) High-risk (2) 100 80 60 40 20 0 M ea n no . o f m ic ro an eu ry sm s pe r pa tie nt M ea n no . o f m ic ro an eu ry sm s pe r pa tie nt v is it Ophthalmologist patient retinopathy grade Ophthalmologist patient retinopathy grade(a) (b) FIGURE 14 The mean number of microaneurysms per patient detected in (a) digital red-free and (b) fluorescein angiographic images. Each patient has been graded by an ophthalmologist according to the overall level of retinopathy present. Error bars are the standard deviations. (p. 7), all patients were assessed medically once a year as various risk factors are known to be associated with the progression of retinopathy. Patients had various parameters measured, including body mass index, blood pressure, HBA1c and microalbuminuria. Table 15 shows the results of a one-way analysis of variance to study the relationship between risk factors and level of retinopathy. As can be seen, only triglyceride approaches a level of statistical significance. The main reason for this is that, as mentioned earlier, the number of patients with the Questions 2 and 3 32 (a) (b) 150 100 50 0 –50 M ea n no . o f h ae m or rh ag es a nd m ic ro an eu ry sm s (s ta nd ar d de v. ) Pe rc en ta ge c ha ng e in t ot al n o. o f h ae m or rh ag es a nd m ic ro an eu ry sm s None Mild Mod Sev V sev None Mild Mod Sev V sev None Mild Mod Sev V sev None Mild Mod Sev V sev Retinopathy (12-monthly visit) grade None to none (290) None to mild (32) Mild to none (29) Mild to mild (44) Mild to moderate (12) Mild to severe (1) Moderate to mild (5) Moderate to moderate (10) Moderate to severe (1) Moderate to very severe (1) Severe to moderate (1) Severe to severe (1) Severe to early (1) None to none (290) None to mild (32) Mild to none (29) Mild to mild (44) Mild to moderate (12) Mild to severe (1) Moderate to mild (5) Moderate to moderate (10) Moderate to severe (1) Moderate to very severe (1) Severe to moderate (1) Severe to severe (1) Severe to early (1) 300 200 100 0 Retinopathy (12-monthly visit) grade FIGURE 15 (a) Change in the mean number of microaneurysms with progression of retinopathy, and (b) percentage change in the total number of microaneurysms with progression of retinopathy more severe levels of retinopathy is very limited as they would be treated rather than remain in the study. This can be seen from the graph showing triglyceride against level of retinopathy (Figure 18), where most patients are in the mild or moderate groups. Further analysis of this data therefore did not appear to be of value. Discussion Microaneurysms are a key lesion in studies on early diabetic retinopathy since the counts are said to correlate with severity of early retinopathy and its likely progression.2,3,43 However, if microaneurysm change is to be used on a patient by patient basis to make a clinical decision, there are a number of other factors to consider. First there is the question of the consistency in the grading of the level of retinopathy. It has been shown that there is a significant variation in the ratings produced by clinicians and optometrists from the gold standard of the ophthalmologist. Hence the actual definition of the degree of retinopathy has an error associated with it. Second, although the number of microaneurysms in general increases with the severity of the retinopathy, there is considerable variation in the mean number per patient. Thus the actual number of microaneurysms does not, in itself, Health Technology Assessment 2003; Vol. 7: No. 30 33 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. 800 700 600 500 400 300 200 100 0 1 2 Visit number(a) (b) 3N um be r of h ae m or rh ag es a nd m ic ro an eu ry sm s Static New 2000 1800 1600 1400 1200 1000 800 600 400 200 0 1 2 Visit number 3 N um be r of h ae m or rh ag es a nd m ic ro an eu ry sm s Static New FIGURE 16 Microaneurysm turnover as measured from fluorescein angiograms for patients with (a) any retinopathy (38 patients) and (b) sight-threatening retinopathy (16 patients) on their first visit FIGURE 17 Rate of turnover of microaneurysms, each point showing the number of (a) new microaneurysms and (b) regressed microaneurysms expressed as a percentage of the total. Each point shows the results for one patient and the patients are classified as having either mild (n = 32) or sight-threatening retinopathy (n = 40). 10 0 0 0.5 1 1.5 Grade: 1 = mild, 2 = worse than mild(a) (b) % n ew m ic ro an eu ry sm s 2 2.5 3 20 30 40 50 60 70 80 90 100 10 0 0 0.5 1 1.5 Grade: 1 = mild, 2 = worse than mild % r eg re ss ed m ic ro an eu ry sm s 2 2.5 3 20 30 40 50 60 70 80 90 100 allow the retinopathy to be graded. An analysis of how the average number of microaneurysms changed as the disease developed did show an increase in the number of microaneurysms in these groups of patients whose condition had changed from none to mild and from mild to moderate retinopathy. There were insufficient patients in the other groups. However, as can be seen from the error bars (standard deviations) in Figure 15(a), this was not sufficient to predict the change in retinopathy from microaneurysm counts alone. In keeping with the results of other studies, it was shown that on fluorescein angiography there is a significant turnover in the number of microaneurysms even over a period of 6 months, with some disappearing between visits and new ones being formed. Hence it is perhaps not surprising that the absolute number of microaneurysms does not provide an accurate measure of the level of retinopathy. The use of fluorescein angiograms raises the question of the time in the sequence that gives the best image for detecting microaneurysms. Jalli and colleagues51 have suggested that the actual phase of the angiogram studied has an important bearing on microaneurysm counts. They looked at microaneurysms in the arterial and late phases of fluorescein angiograms. Other microaneurysm counting studies have also used the arterial phase for counting microaneurysms.46,52 This approach is surprising as microaneurysms develop from capillaries and therefore to be sure that all microaneurysms are perfused with fluorescein then the arteriovenous/early venous phase, as we have used, seems more appropriate. Microaneurysms appearing only in the late phase probably contain thrombus and thus are partially occluded. Fully occluded microaneurysms do not fluoresce at all. In this study we only looked at microaneurysms present in the early venous/arteriovenous phase as we felt that this ensured that all microaneurysms present had been perfused. The mean and range of times (in seconds) at which the images were analysed were 37.2 (11.4–111.8) for visit 1, 32.8 (8.3–57.6) for visit 2 and 30.7 (17.7–52.8) for visit 3. For a particular patient the difference in the time (in seconds) of the analysed image was 4.5 (range 0.60–73.8) for visits 1 and 2 and 2.1 (range 0.10–32.2) between visits 2 and 3. In essence, this means that we only identified perfused microaneurysms, namely those of a fusiform or saccular nature. By doing so we have included the very beginnings of microaneurysm formation, namely fusiform capillary dilation, and excluded the very late stages where microaneurysms are occluded and no longer increasing in size. This is important as poorly perfused microaneurysms may exist in this state indefinitely but are no longer ‘active’. Given the dynamics of microaneurysms on fluorescein images, the potential of the rate of microaneurysm turnover as a marker of retinal progression was investigated. Although relatively small numbers of patients were available for analysis, there appeared to be no correlation between turnover rate and level of retinopathy over a 6-month period. Questions 2 and 3 34 TABLE 15 Significance of relationship between risk factors and level of retinopathy Risk factor p-Value No. of readings Body mass index 0.13 55 Creatinine 0.29 76 Systolic BP 0.25 38 Diastolic BP 0.5 41 Fasting cholesterol 0.92 66 Haematocrit 0.69 69 Haemoglobin 0.59 79 HbA1c 0.61 71 HDL 0.92 71 LDL 0.4 73 Microalbuminuria 0.21 52 Platelet count 0.33 73 White cell count 0.79 73 Triglyceride 0.076 73 BP, blood pressure; HDL, high-density lipoprotein; LDL, low-density lipoprotein. 1 2 5 4 Retinopathy grade Tr ig ly ce ri de le ve l 5 6 7 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 FIGURE 18 Triglyceride level as a function of level of retinopathy (n = 73) Health Technology Assessment 2003; Vol. 7: No. 30 35 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Introduction This chapter is concerned with the economic evaluation of three alternative methods of providing screening for diabetic retinopathy as used in this study: digital photography, analogue photography and direct ophthalmoscopy carried out by trained optometrists. The methods have already been described in Chapter 2. The requirements of the research programme were such that some costs were incurred that would not be part of a normal service setting (research costs) and where possible these have been adjusted for, or are explained in the text. The comparison of the screening methods does not include the costs of administering a call and recall system, and the equipment does not include visual acuity charts and light boxes, as both of these would be common to all screening programmes. Costs for the photographic screening methods were calculated as a cost per session and converted to a cost per patient using a baseline assumption of 10 patients per session. Digital photography costs Capital costs The photography was carried out in a Portacabin dedicated to the project, although it is possible for this technology to be provided on a mobile basis. The equipment consists of a basic fundus camera with a digital attachment and associated computer equipment and software. The building cost was discounted over 20 years and the equipment cost was discounted over 7 years, although it is recognised that technical obsolescence may set in much earlier. Other assumptions will be considered in the sensitivity analysis. Running costs Running costs for the building were allocated from hospital costs on the basis of the floor area. Staff costs per session took into account a nurse to carry out initial visual acuity checks and administer eye drops and the photographer and a senior registrar to read and report on the results. (The research registrar undertook some of these tasks but the grades of staff for the costing are those considered appropriate to the tasks.) The session time for the photographer allows for the carrying out of related administrative tasks. Consumables Consumables consisted of eye drops. Automated grading costs The costs for screening include the cost of medical staff time to read and report the results. This project developed the software for automated grading of digital images. The development costs consist of staff time and this cost has been discounted over 7 years, to be consistent with the equipment costs, and replaces the medical staff time in the cost of the screening. Colour slide photography costs Capital costs The photography was carried out in the same setting as the digital photography. Equipment consists of a basic fundus camera with a 35-mm slide attachment. The same fundus camera was used for both types of photography but the full cost was included both times as only one type of photography would be carried out in practice. The building cost was discounted over 20 years and the equipment cost was again discounted over 7 years, in the first instance. Running costs Running costs for the building and staff costs per session were the same as for the digital photography. Consumables Consumables consisted of eye drops, film and developing materials. Optometrist screening costs Optometrists carrying out screening in the project were given training at the start of the project as Chapter 7 Costs and consequences of screening for diabetic retinopathy described in the section ‘Slit-lamp biomicroscopy’ (p. 21) and a poster showing the various features of retinopathy was provided for reference. A standard fee of £18.00 per patient was paid for the screening. The optometrists were using facilities and equipment that would be used for normal sight testing. Results The costs for digital and analogue screening are shown in Table 16. The digital camera system is slightly cheaper and both camera systems were less costly than the payment to optometrists. However, these costs are simply for the initial screening visit and the reading of the results. The number of repeat visits, because of poor images, and the referral rate for assessment have to be taken into account. A cohort analysis has been carried out for this purpose and the results are shown in Table 17. Repeat screens are only relevant for photographic screening and the rates were 5% for digital and 10% for analogue photography. The number of patients called for an assessment visit depends upon the sensitivity and specificity of the screening test. Based on the figures given in the section ‘Detection of retinopathy’, and assuming a prevalence of sight-threatening diabetic retinopathy of 6%, the number of assessment visits would be 175 for digital (graded by research registrar), 159 for digital (automated grading), 171 for 35-mm colour slides and 138 for optometrists. These visits have been costed at £51 each, the average cost for an outpatient attendance. From Tables 16 and 17, it can be seen that digital screening with automated grading has the lowest cost per screen, and remains the least costly method when the additional factors of repeat visits and number of assessment visits are included. However, it is more expensive than digital imaging with medical staff reading the images or 35-mm colour slide photography in terms of cost per true positive detected. This is largely due to having a much lower sensitivity in reading the digital images. The assumptions for Table 16 were as follows: � Building area based on two standard rooms plus circulation space giving 33.8 m2. � Building cost discounted over 20 years gives an annual equivalent cost of £2653 (for comparison, the capital charge on the area assumed would be £2501). � Session costs based on 460 sessions per year to allow for holidays/sickness/equipment breakdowns, etc. � Cost per patient based on 10 patients per session. � Camera costs allocated between fundus/digital/analogue as £54,000 for the entire digital system and £35,000 for the analogue system. � Running costs cover cleaning/maintenance/rates and power. � Film costs £9.82 per film and has been assumed to include processing. Costs and consequences of screening for diabetic retinopathy 36 TABLE 16 Screening costs (£ 1998–9) Manual interpretation Automated grading of digital images of digital images 35-mm Colour slides Cost per Cost per Cost per Cost per Cost per Cost per session patient session patient session patient Building 5.77 0.58 5.77 0.58 5.77 0.58 Equipment digital 21.03 2.10 21.03 2.10 analogue 13.63 1.36 Running costs 4.41 0.44 4.41 0.44 4.41 0.44 Drops 0.50 0.05 0.50 0.05 0.50 0.05 Film and processing 24.55 2.46 Staff nurse 38.91 3.89 38.91 3.89 38.91 3.89 photographer 27.58 2.76 27.58 2.76 27.58 2.76 senior registrar 42.87 4.29 42.87 4.29 Automated grading of digital images 26.09 2.61 Total 141.07 14.11 124.29 12.43 158.22 15.82 � Nurse costed as grade E working 46 weeks per year and allocating 3.5 hours per session to allow for time getting ready or over-running. Salary based on scale average including employer’s on-costs. � Photographer cost based on MLSO-grade salary. � Senior registrar cost based on scale average including employer’s on-costs. One session taken as 0.1 of working week; reads 17.5 patient films; session cost calculated for 10 patients. � Development costs for software taken as the whole of 3 years’ salary for grade RA1. Combining modalities – a modelling exercise It is possible that automated grading could be used in conjunction with medical staff and this can be modelled using results from this study. The automated system would be used to report any retinopathy, as it has a better sensitivity rate of 87%, although specificity is worse, 71%. Medical staff carrying out a reading of the reported positives would eliminate some false positives. The medical staff would have to read images for 324 out of 1000 patients (based on a prevalence of 6% for sight-threatening diabetic retinopathy). It is assumed that in this model 48 true positives would be detected (automated grading would detect 52 and medical grading would identify 93% of these). It is further assumed that medical reading would continue to grade as false positive the same absolute number of patients (113) as were graded false positive when reading all 1000 cases. These are the least favourable assumptions to be made about the effect of combining the two modalities of reading. The results in Table 18 show that the cost per true positive remains higher than for digital imaging and medical staff grading alone. Altering the assumption about the combined sensitivity of the reading methods makes little difference. However, if the absolute number of false positives was reduced, this could affect the result. The most optimistic assumption would be that medical staff would continue to achieve 87% specificity when reading only the positives selected by automated grading. This would reduce the number of false positives to 36, giving a combined specificity of 96%, and would reduce the cost per true positive to £375. Hence it is the combined specificity of the two modalities that is important in estimating the relative cost-effectiveness of the screening. Sensitivity analysis Prevalence Increasing the assumed prevalence of sight- threatening diabetic retinopathy reduces the cost per true positive of all screening methods without affecting the ranking between methods (Table 19). Throughput per session The estimates in Table 16 are based on a throughput of 10 patients per session for both methods of photographic screening. The number of patients per session would have to fall below seven before all of the photographic methods Health Technology Assessment 2003; Vol. 7: No. 30 37 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. TABLE 17 Costs for a cohort of 1000 patients screened (£ 1998–9) Manual grading of Automated grading of 35-mm Colour Optometrist digital images digital images slides Screening cost 14110 12430 15820 18000 Repeat screens 706 621 1582 Assessment visits 8721 8109 8721 6477 Total 23537 21160 26123 24477 True positives detected (from 60) 56 46 58 42 Cost per true positive detected 420 460 450 583 TABLE 18 Model for a cohort of 1000 patients – automated grading combined with medical staff grading Screening cost – automated grading of digital images £12430 Regrading of positives by medical staff (324 × 4.29) £1390 Repeat screens £621 Assessment visits £8313 Total £22754 True positives detected (from 60) £48 Cost per true positive detected £474 became more expensive than optometrist screening, per patient screened (Table 20). The throughput per session would have to fall below six before it became more expensive than optometrist screening, per true positive detected. The ranking of alternative photographic methods would be the same unless different throughput rates were assumed for different methods. It may be that throughput would be lower for digital imaging because of the opportunity for patient education using the captured images. If throughput per session was eight or less for digital imaging, and remained at 10 for analogue imaging, then analogue imaging would be less expensive in terms of cost per screen and cost per true positive detected. Assumed life of equipment Equipment costs have been given as assumed life of 7 years, but replacement may occur much earlier because of developments in technology. This is more likely to affect the digital systems and the costs were recalculated using 5 years and 3 years as the assumed life (Table 21). If digital equipment is replaced after 3 years, the cost per screen becomes more expensive than analogue screening. The cost per true positive remains just below the figure for the analogue screening, at £447 compared with £450. Automated grading – software costs The development costs for the software have been treated as equipment costs and have been spread over 7 years. Reducing this time period would further increase the costs of automated reading. However, the costs have also been applied to screening in one location only. If the system were adopted more widely, the cost per screen would fall. In order for the cost per true positive to equate with the cost based on medical staff grading, the cost per screen for the software would have to fall to £0.24. This is equivalent to screening 50,000 patients per year. Conclusions The costing information presented above reflects those costs incurred in the trial. Their extrapolation to a more general context of a national screening programme needs to be taken cautiously. Costs and consequences of screening for diabetic retinopathy 38 TABLE 19 Costs for a cohort of 1000 patients screened (£ 1998–9) (prevalence of 12% in normal font and 8% in italics) Manual interpretation Automated grading 35-mm of digital images of digital images Colour slides Optometrist Screening cost 14110 12430 15820 18000 Repeat screens 706 621 1582 Assessment visits 11271 10098 11322 8364 9537 8772 9588 7089 Total 26087 23149 28724 26364 24353 21823 26990 25089 True positives detected 112 92 116 84 75 62 78 56 Cost per true positive detected 232 252 248 314 325 352 346 448 TABLE 20 Effect of throughput on cost per screen Cost per patient screened (£) Manual interpretation Automated grading No of patients per session of digital images of digital images 35-mm Colour slides 10 14.11 12.43 15.82 9 15.67 13.81 17.58 8 17.63 15.54 19.78 7 20.15 17.76 22.60 6 23.51 20.72 26.37 Health Technology Assessment 2003; Vol. 7: No. 30 39 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Optometrist analysis was the least cost-effective. As mentioned in the section ‘Comparison of optometrist with ophthalmologist’ (p. 25) to some extent this probably reflects the lack of practice in detecting macular oedema, as in everyday practice optometrists would see mainly normal eyes. Care was taken in training optometrists for this study, but performance would be expected to improve with further experience. The automated system, either on its own or in conjunction with medical staff, is more expensive per true positive detected than manual grading. To make it competitive, the cost per patient has to be reduced to £0.23, which, with the current cost implies an annual screening of 50,000 patients, a number that could be encountered in a national screening programme. The main problem was the relatively low sensitivity of the software for detecting sight-threatening diabetic retinopathy and, in part, this was because the software was unable to detect new vessels. Undoubtedly the sensitivity of software can be expected to improve as further development continues. Although manual grading appeared most cost- effective, the main problem, that has not been investigated, is the cost of setting up a manual screening service and ensuring that the quality of reporting remains consistent. Although it is not unreasonable to expect a trained grader to perform as well as the senior registrar used in this study, introducing quality control measures will have a cost implication. TABLE 21 Effect of alternative assumptions about replacement of digital equipment Cost per patient screened (£) Manual interpretation Automated grading 35-mm of digital images of digital images Colour slides Baseline (7 years) 14.11 12.43 15.82 5 years 14.79 13.11 3 years 16.40 14.72 Health Technology Assessment 2003; Vol. 7: No. 30 41 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Current study It is estimated that there are 1100 new cases of blindness every year secondary to diabetic retinopathy. Early detection of sight-threatening retinopathy enables laser therapy to be performed to prevent or slow the onset of visual loss. At present the commonest cause of visual impairment is ischaemic maculopathy, which is untreatable by surgical means. Early detection of retinopathy with aggressive management of metabolic control and blood pressure will therefore be the most effective way of meeting the St Vincent targets for visual impairment, thus reducing the social and economic toll of this complication. The European Retinopathy Working Party recommended a sensitivity of 80% and a specificity of 95% for screening programmes, but conventional screening modalities are failing to achieve these targets and, in addition, the provision of conventional screening modalities is inadequate and inequitable in the UK. The digital fundus camera is a promising development. It makes regular assessment more feasible since, unlike current approaches, it does not require a skilled assessor to see each patient. It also offers the potential for introducing automated screening of retinal images with the associated consistency in interpretation. It is ideally suited to quality assurance as it produces a hard copy that can be assessed, unlike subjective techniques such as slit-lamp biomicroscopy. One potential disadvantage is whether the quality of the digitised image is sufficiently high for the purpose of screening. Ergonomically, digital imaging was found to be more effective than colour slides. Digital photography produced fewer ungradable images and the number of individuals needing to be recalled for repeat photography if a digital rather than a conventional photographic camera was used for image capture was reduced by about 50%. This was mainly due to the fact that the photographer had immediate feedback on the quality of the images by being able to view them instantaneously on the computer screen. If the image was not of sufficient quality the photographer simply repeated the photograph until one of sufficient quality could be obtained. So far as image quality was concerned, a grading system, based on the clarity of features such as nerve-fibre layer and vessels, demonstrated that the finest quality photographs were produced by conventional photography. This is not surprising as the maximum resolution of conventional photographs is approximately 2–3 times better than the digital camera used in this study. However, in terms of producing images of a quality acceptable for screening, then the digital images were superior to the colour slides. When the image quality was analysed in terms of field, then the poorest quality field tended to be the nasal field, reflecting the need for a widely dilated pupil and the absence of peripheral cortical cataract, a common clinical finding, to obtain high-quality images. Caution must also be exercised over the effect of artefacts on image quality. The presence of dust on images simulating microaneurysms was one problem that we encountered. Previous publications on screening have tended to compare one modality with another but not all modalities together. In this study we have compared trained optometrists using slit-lamp photography against conventional mydriatic photography and digital photography. We did not use direct ophthalmoloscopy as this has been shown to be an insensitive technique, although it may have a role in sporadic screening or where no screening at all occurs. Red-free digital images were used as they provide the greatest contrast for red abnormalities on a red background. Although other clinicians may not be used to such monochrome images, it is the standard imaging modality for fluorescein angiography where the highest quality images are required. It was also felt important to attempt to train the optometrists to the same standard as the ophthalmologists in the clinic. In terms of sensitivity and specificity, there was little to choose between conventional mydriatic photography and digital photography for detecting any or early retinopathy. For sight- threatening retinopathy the optometrists had the lowest overall sensitivity (73%), although their specificity (90%) was comparable to that of the other modalities. Digital imaging and Chapter 8 Conclusions conventional photography are acceptable methods for screening. Screening by trained optometrists is not sufficiently sensitive for detecting either any retinopathy or sight-threatening retinopathy. In the context of a national screening programme, automated analysis techniques offer the advantages of repeatability and consistency. Individual human graders tend to have their own varying internal reference standards which are difficult to make conform, despite training, leading to intra- and inter-observer variability. In a previous study comparing the automated detection of retinopathy, using only microaneurysms, with colour slides graded by a research fellow,48 we were able to achieve a sensitivity of 85% and a specificity of 76% for detecting whether or not a patient had retinopathy. In this current study, in which ophthalmologists using stereo biomicroscopy was used as the gold standard, we have achieved a slightly higher sensitivity of 83% but a slightly lower specificity of 71%. Although the sensitivity exceeds the 80% recommended by the European Retinopathy Working Party, the specificity falls short of their value of 95%. Assuming a prevalence for diabetic retinopathy of 30%, this means that 48% of this population would be correctly classified as having no retinopathy. This is a significant achievement when one considers the large number of patients with diabetes in the UK who require screening. Clearly, such results depend upon the population examined. For sight-threatening retinopathy, the specificity rises to 88% whereas the sensitivity is lower than that achieved by other modalities at 77%. The problem is that we have not been able to detect all forms of sight-threatening retinopathy as we do not as yet have a computer algorithm to detect new vessel formation. In addition, it would appear that automated grading is able to detect the presence of clinically significant maculopathy (using a combination of the microaneurysm detection program and the exudate program), with a specificity of 85%, which is comparable to other modalities, thus enabling graders to target their attention on images with potentially sight-threatening retinopathy. We conclude that automated grading cannot, at present, be used on its own but in the context of a manual grading system it will still greatly reduce the workload by correctly identifying just under half the population as having no retinopathy. For this investigation, we used the two-field protocol of the EURODIAB study. Although more patients had retinopathy confined to the macular field compared with the nasal field, using the macular field only would significantly reduce the chances of detecting any retinopathy, with between 8 and 14% of cases being missed. In the case of sight-threatening diabetic retinopathy, however, the number of missed patients drops to 3% or less, with the sensitivity and specificity being almost the same for one as for two fields. This raises the possibility of adopting a screening programme based on taking only a macular field. Such a decision will depend on the likely incidence of more advanced retinopathy in the screened population. It should also be noted that the sensitivity and specificity achieved by the automated system will depend upon the number of images taken per patient; analysis of four images will lead to a higher sensitivity but lower specificity than for detecting retinopathy in two images. The role of automated grading for assessing the progression of disease was also explored. The value of this approach was fundamentally limited by two factors. First, there is an intrinsic variability in the grading of level of retinopathy by ophthalmologists and optometrists compared with the gold standard. Second, microaneurysms, as assessed on fluorescein angiography, are subject to a rapid turnover, with only about 45% of the microaneurysms being still seen 6 months later. There was a statistically significant increase in the number of microaneurysms in the group of patients whose condition changed from none to mild and the group who progressed from mild to moderate. However, when the actual individual rate of turnover was analysed, there was no significant difference between the group of mild and sight-threatening retinopathy. However, this aspect of the study was limited by the small number of patients completing it. At the start of the study (see the section ‘Overview of the study’, p. 2), three questions were posed. First, can a digital imaging system detect retinopathy irrespective of sort or level? The answer is that employing a manual analysis of the images, a sensitivity of 83% and a specificity of 79% can be achieved, and using an automated analysis, these becomes 83% and 71%. Second, can a digital imaging system detect progression of retinopathy? The answer is that although it can measure turnover, it has not been shown that this follows the progression of retinopathy. Third, can a digital imaging system determine when Conclusions 42 treatment is warranted? Since there was no definite answer to question 2, then that to the third question must be ‘no’ at present. Relevance to the NHS Digital imaging is already succeeding conventional photography in eye departments. This study has demonstrated that the digital fundus camera does indeed have a significant role to play in ophthalmology. The finest quality photographs were produced by conventional photography, in terms of producing images of a quality acceptable for screening, but the digital images were superior to colour slides. Most important, digital imaging was found to be more effective than colour slides, producing fewer ungradable images. With the development of higher resolution digital cameras, it is to be expected that conventional photography may eventually be superseded by digital imaging. Digital imaging and conventional photography are acceptable methods for screening; in terms of sensitivity and specificity there was little to choose between conventional mydriatic photography and digital photography for detecting any or early retinopathy. Screening by trained optometrists is, however, not sufficiently sensitive for detecting either any retinopathy or sight-threatening retinopathy. In the context of a national screening programme, automated analysis techniques offer the advantages of repeatability and consistency. The cost-effectiveness of the study showed that although the automated analysis had the lowest cost per screen, in terms of cost per true positive it was more expensive than clinical staff manually reading digital images or colour slides. However, this analysis did not take account of the cost of training screeners and maintaining the quality of their reporting. Also, the cost would drop to that of other approaches in a system which was screening 50,000 patients or more per year. The performance of the software developed for this project meant that, assuming a prevalence for diabetic retinopathy of 30%, 48% of this population would be correctly classified as having no retinopathy. This suggests a two-step procedure for any screening programme. Where no retinopathy is detected by the software, then the patient would be recalled in 1-year’s time for repeat screening. Where retinopathy is detected, the image would be manually graded. Thus automated first-level grading could considerably reduce the burden of manual grading. Research recommendations 1. The two-field protocol of the EURODIAB study was used, but analysis of the data shows that the detection of referable retinopathy was as reliable using only the macular field. Single- field imaging could potentially reduce time taken to perform retinal screening. This observation requires confirmation with a larger number of patients and different referral criteria. 2. The value of an automated grading system to assist in a screening programme has been demonstrated. Further work is required to improve the sensitivity and specificity of such programmes. In particular, it would be of value to develop software to detect new vessel formation and to investigate the potential of using colour information. 3. An insufficient number of patients was recruited to investigate the value of automated grading for the evaluation of disease progression. This should be studied further in conjunction with the development of automated systems. 4. Patient recruitment was poor. Future research is required to ensure effective uptake in a diabetic retinopathy screening programme. The future Any screening programme must be able to detect retinopathy with a significant level of accuracy and confidence. The European Retinopathy Working Party recommendations are achievable in terms of sensitivity, although the required specificity was not achieved by any of the modalities tested. The recommendations of 80% sensitivity and 95% specificity are not based on any scientific evidence and whether such high levels are actually required has yet to be determined. Perhaps the major advantage that digital imaging has over all other screening modalities is its suitability for quality assurance. For any screening programme this must be a major concern. Until the skills of slit-lamp biomicroscopy can be raised to those of the ophthalmologist then the role of high-street opthalmologists might be limited to digital retinal photography as part of a national screening network. Health Technology Assessment 2003; Vol. 7: No. 30 43 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Conclusions 44 The cost-effectiveness of the automated approach will be affected by developments in technology. On the hardware side, the major development is the arrival of digital cameras with a higher resolution than the 1024 × 1024 acquisition matrix of the Topcon system we used. For example, the current generation (2002) of cameras offer a resolution of 2160 × 1440 pixels. There will undoubtedly be improvements in automated analysis software and this has been identified as an area for further research. Health Technology Assessment 2003; Vol. 7: No. 30 45 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. The authors acknowledge the contributions ofMrs Alison Farrow and Ms Sandra McKay, who were responsible for the retinal photography. The assistance of Dr K Seipman with the translation of foreign language publications is gratefully acknowledged. The Health Services Research Unit is core funded by the Chief Scientist Office of the Scottish Office, Department of Health. Contributions of the authors PF Sharp (Professor of Medical Physics) was overall project coordinator. J Olson (Senior Registrar in Medical Ophthalmology) was coordinator for clinical research. F Strachan (Research Fellow) was a clinical researcher and lead for the systematic literature review. J Hipwell (Research Fellow) carried out software development and image analysis. A Ludbrook was responsible for health economics aspects. M O’Donnell (Research Assistant) and S Wallace (Research Assistant) provided assistance with the systematic literature review. K Goatman (Research Fellow) undertook data analysis. A Grant (Unit Director) gave advice on the systematic literature review. N Waugh (Director) provided expertise on purchasing perspectives of screening for diabetic eye disease. K McHardy (Consultant General Physician/Diabetologist) and JV Forrester (Professor) gave advice on clinical aspects of diabetic retinopathy screening. Acknowledgements 1. Ghafour IM, Allan D, Foulds WS. Common causes of blindness and visual handicap in the west of Scotland. Br J Ophthalmol 1983;67:209–13. 2. Klein R, Meuer SM, Moss SE, et al. Retinal microaneurysm counts and 10-year progression to diabetic retinopathy. Arch Ophthalmol 1985;113:1386–91. 3. Kohner E, Sleightholm M. Does microaneurysm count reflect the severity of early diabetic retinopathy? Ophthalmology 1986;93:586–9. 4. Early Treatment Diabetic Retinopathy Study Research Group. Fundus photographic risk factors for progression of diabetic retinopathy. ETDRS Report Number 12. Ophthalmology 1991;98:823–33. 5. Early Treatment Diabetic Retinopathy Study Research Group. Early photocoagulation for diabetic retinopathy. ETDRS Report Number 9. Ophthalmology 1991;98:766–85. 6. The Diabetes Control and Complications Trial Research Group. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. N Engl J Med 1993;329:977–86. 7. Sjolie AK, Stephenson J, Aldington S, et al. Retinopathy and vision loss in insulin-dependent diabetes in Europe. The EURODIAB IDDM Complications Study Group. Ophthalmology 1997;104:252–60. 8. Muhlhauser I. Cigarette smoking and diabetes: an update. Diabet Med 1994;11:336–43. 9. Chaturvedi N, Stephenson JM, Fuller JH. The relationship between smoking and microvascular complications in the EURODIAB IDDM Complications Study. Diabetes Care 1995;18:785–92. 10. Stratton IM, Kohner EM, Aldington SJ, Turner RC, Holman RR, Manley SE, et al. UKPDS 50: risk factors for incidence and progression of retinopathy in Type II diabetes over 6 years from diagnosis. Diabetologia 2001;44:156–63. 11. World Health Organization and the International Diabetes Federation. Diabetes care and research in Europe: the Saint Vincent Declaration. Diabet Med 1990;7:360. 12. Retinopathy Working Party. A protocol for screening for diabetic retinopathy in Europe. Diabet Med 1991;8:263–7. 13. Buxton MJ, Sculpher MJ, Ferguson BA, et al. Screening for treatable diabetic retinopathy: a comparison of different methods. Diabet Med 1991;8:371–7. 14. Jones D, Dolben J, Owens DR, et al. Non-mydriatric polaroid photography in screening for diabetic retinopathy: evaluation in a clinical setting. BMJ 1988;296:1029–30. 15. Klein R, Klein BEK, Neider MW, et al. Diabetic retinopathy as detected using ophthalmoscopy, a non-mydriatic camera and a standard fundus camera. Ophthalmology 1985;92:485–91. 16. Taylor R, Lovelock L, Tunbridge WMG, et al. Comparison of non-mydriatic retinal photography with ophthalmoscopy in 2159 patients: mobile retinal camera study. BMJ 1990;301:1243–7. 17. Harding SP, Broadbent DM, Neoh C, et al. Sensitivity and specificity of photography and direct ophthalmoscopy in screening for sight threatening eye disease: The Liverpool Diabetic Eye Study. BMJ 1995;311:1131–5. 18. Joannou J, Kalk WJ, Mahomed I, et al. Screening for diabetic retinopathy in South Africa with 60° retinal colour photography. J Intern Med 1996; 239:43–7. 19. O’Hare JP, Hopper A, Madhaven C, et al. Adding retinal photography to screening for diabetic retinopathy: a prospective study in primary care. BMJ 1996;312:679–82. 20. British Diabetic Association. Retinal photography screening for diabetic eye disease. London: British Diabetic Association; 1994. 21. Phillips RP, Spencer T, Ross PGB, Sharp PF, Forrester JV. Quantification of diabetic maculopathy by digital imaging of the fundus. Eye 1991; 5:130–7. 22. Spencer T, Phillips RP, Sharp PF, Forrester JV. Automated detection and quantification of microaneurysms in fluorescein angiograms. Graefes Arch Clin Exp Ophthalmol 1992;230:36–41. 23. Phillips R, Forrester J, Sharp PF. Automated detection and quantification of retinal exudates. Graefes Arch Clin Exp Ophthalmol 1993;231:90–4. 24. Spencer T, Olson JA, McHardy K, Sharp PF, Forrester JV. An image-processing strategy for the segmentation and quantification of microaneurysms in fluorescein angiograms of the ocular fundus. Comput Biomed Res 1996;29:284–302. 25. Scobie IN, MacCuish AC, Barrie T, Green FD, Foulds WS. Serious retinopathy in a diabetic clinic – Health Technology Assessment 2003; Vol. 7: No. 30 47 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. References 48 prevalence and therapeutic implications. Lancet 1981;2:520–1. 26. Aldington SJ, Kohner EM, Meurer S, Klein R, Sjolie AK. Methodology for retinal photography and assessment of diabetic retinopathy – The EURODIAB IDDM complications study. Diabetologia 1995;38:437–44. 27. Klein R, Klein BE, Moss SE, Davis MD, DeMets DL. Is blood pressure a predictor of the incidence or progression of diabetic retinopathy? Arch Intern Med 1989;149:2427–32. 28. Cruickshanks KJ, Ritter LL, Klein R, Moss SE. The association of microalbuminuria with diabetic retinopathy. The Wisconsin Epidemiologic Study of Diabetic Retinopathy. Ophthalmology 1993;100:862–7. 29. Wang PH, Lau J, Chalmers, TC. Meta-analysis of effects of intensive blood-glucose control on late complications of type 1 diabetes. Lancet 1993;341:1306–9. 30. Elman KD, Welch RA, Frank RN, Goyert GL, Sokol RJ. Diabetic retinopathy in pregnancy: a review. Obstet Gynecol 1990;75:119–27. 31. Wong JSK, Pearson DWM, Murchison LE, Williams MJ, Narayan V. Mortality in diabetes mellitus: experience of a geographically defined population. Diabet Med 1991;8:135–9. 32. Diabetic Retinopathy Study Research Group. A modification of the Airlie House classification of diabetic retinopathy. Report No. 7. Invest Ophthalmol Vis Sci 1981;21:210–26. 33. Nussenblatt RB, Palestine AG, Chan CC, Roberge F. Standardization of vitreal inflammatory activity in intermediate and posterior uveitis. Ophthalmology 1985;92:467–71. 34. Medical imaging – the assessment of image quality. ICRU Report 54. Bethesda, MD: International Commission on Radiation Units and Measurement (ICRU); 1996. 35. Kohner EM, Stratton IM, Aldington SJ, Turner RC, Matthews DR. Microaneurysms in the development of diabetic retinopathy (UKPDS 42). Diabetologia 1999;42:1107–12. 36. Matthews DR, Kohner EM, Aldington S, Stratton IM. Relationship of microaneurysm count to progression of retinopathy over 6 years in non- insulin-dependent diabetes. Diabetes 1995;9:117A. 37. Hellstedt T, Palsi VP, Immonen I. A computerized system for localization of diabetic lesions from fundus images. Acta Ophthalmol 1994;72:352–6. 38. Diabetic Retinopathy Study Research Group. A modification of the Airlie House classification of diabetic retinopathy. Report No. 7 Invest. Ophthalmol Vis. Sci. 1981;21:210–26. 39. Cree MJ, Olson JA, McHardy KC, Sharp PF, Forrester JV. A fully automated comparative microaneurysm digital detection system. Eye 1997;11: 622–8. 40. Klein R, Klein BE, Moss SE. Visual impairment in diabetes. Ophthalmology 1984;91:1–9. 41. Moss SE, Klein R, Klein BE. The incidence of vision loss in a diabetic population. Ophthalmology 1988; 95:1340–8. 42. Sparrow JM, McLeod BK, Smith TD, Birch MK, Rosenthal AR. The prevalence of diabetic retinopathy and maculopathy and their risk factors in the non-insulin-treated diabetic patients of an English town. Eye 1993;7(Pt 1): 158–63, 43. Klein R, Meuer SM, Moss SE, Klein BEK. The relationship of retinal microaneurysm counts to the 4-year progression of diabetic retinopathy. Arch Ophthalmol 1989;107:1780–5. 44. Kohner EM and Dollery CT. The rate of formation and disappearance of microaneurysms in diabetic retinopathy. Eur J Clin Invest 1970;1:167–71. 45. Feman SS. The natural history of the first clinically visible features of diabetic retinopathy. Trans. Am. Ophthalmol. Soc. 1994;92:745–73. 46. Hellstedt T, Immonen I. Disappearance and formation rates of microaneurysms in early diabetic retinopathy. Br J Ophthalmol 1996;80:135–9. 47. Cideciyan AV, Jacobson SG, Kemp CM, Knighton RW, Nagel JH. Registration of high resolution images of the retina. Proc SPIE 1992;1652:310–22. 48. Hipwell JH, Strachan F, Olson JA, McHardy KC, Sharp PF, Forrester JV. Automated detection of microaneurysms in digital red-free photographs: a diabetic retinopathy screening tool. Diabet Med 2000;17;588–94. 49. Lee SC, Lee ET, Kingsley RM, Wang Y, Russell D, Klein R, et al. Comparison of diagnosis of early retinal lesions of diabetic retinopathy between a computer system and human experts. Arch Ophthalmol 2001;119:509–15. 50. Ege BM, Hejlesen OK, Larsen OV, Moller K, Jennings B, Kerr D, et al. Screening for diabetic retinopathy using computer based image analysis and statistical classification. Comput Methods Programs Biomed 2000;62:165–75. 51. Jalli PYI, Hellstedt TJ, Immonen IJR. Early versus late staining of microaneurysms in fluorescein angiography. Retina 1997;17:211–15. 52. Badouin C, Maneschi F, Quentel G, Soubrane G, Hayes T, Jones G, et al. Quantitative evaluation of fluorescein angiograms – microaneurysm counts. Diabetes 1983;32:8–13. References Introduction Background Despite advances in diabetic care, visual impairment in diabetes remains a devastating complication, in terms of both personal loss for the affected individual and socio-economic costs to society. Of the 12 million citizens of the USA who are recognised as suffering from diabetes mellitus, it has been estimated that the prevalence of those with proliferative retinopathy is 700,000 (6% of the population) with an anticipated further 65,000 cases occurring per annum.1 For the insulin- dependent diabetic population of Europe, a cross- sectional study of patients attending 31 diabetes centres revealed a prevalence of pre-proliferative retinopathy of 35.6% (mild 25.8%; moderate-to- severe 9.8%) and proliferative retinopathy of 10.6%.2 The complications arising from diabetic retinopathy are an estimated 8000 new cases of blindness per year in the USA1 and 1100 new cases per annum in the UK,3 giving rise to a significant problem in the working-age population. Although it remains difficult to quantify the devastating effect of blindness with regard to personal loss, the economic cost to society in terms of unemployment and disablement benefits has been calculated as £3575 for each affected individual per annum in the UK.4 This was weighted against the estimated cost of treatment of £387 for each person at risk of blindness, indicating the economic advantages of successful treatment of those high-risk individuals. Pathogenesis and key features of diabetic retinopathy The natural history of retinopathy has been well defined, following a predictable course from the early stage of microaneurysm development to moderate retinopathy, with evidence of cotton- wool spots, intra-retinal microvascular abnormalities and venous beading indicating a deterioration in retinal blood supply. Hard exudates define areas where retinal capillary leakage is occurring in the presence of endothelial damage. Ultimately, in response to worsening ischaemia, growth of new blood vessels is stimulated. This proliferative stage of retinopathy poses a high risk for the patient as fragile new vessels have a tendency to haemorrhage, causing potentially significant visual impairment. Baseline microaneurysm counts in people with diabetes with no other evidence of retinopathy may provide a useful predictor of long-term progression to proliferative retinopathy, independent of the effects of glycaemic control and blood pressure.5 The importance of microaneurysm detection and quantification is supported by the analysis of fluorescein angiograms by Kohner and Sleightholm.6 This analysis has shown a significant correlation between microaneurysm number and the presence of haemorrhages and cotton-wool spots and, to a lesser extent, the severity of hard exudates and IRMAs. From examination of the natural progression of fundus changes in individuals with at least moderate diabetic retinopathy, it was found that the severity of IRMA, venous beading and the number of haemorrhages and microaneurysms present were of the most significance in identifying those people who are likely to develop subsequent neovascularisation.7 ETDRS The ETDRS was carried out to provide answers to the problem of reducing the devastating morbidity from visual loss secondary to diabetes.8 As a consequence of the trial, the benefits of laser panretinal photocoagulation for patients with high- risk proliferative retinopathy have been recognised and adopted into standard clinical practice. However, the authors concluded that early panretinal photocoagulation was inappropriate for those with mild to moderate retinopathy and a low risk of progression to severe visual loss, given its potentially detrimental effect on the peripheral visual field. They emphasised that the key to successful management was early identification of retinopathy with the meticulous monitoring of progression to allow optimum timing of intervention when a proliferative stage had been reached. Risk factors in development of retinopathy There are, however, other ways in which careful screening of diabetic populations and monitoring of established retinopathy may be beneficial. The Health Technology Assessment 2003; Vol. 7: No. 30 49 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Appendix 1 Systematic literature review (completed 1998) Appendix 1 50 Diabetes Control and Complications Trial has shown that strict metabolic control can both offset the development and slow the progression of diabetic retinopathy in type 1 diabetics.9 In addition to acknowledging the limitations of their cross-sectional study of 3250 European type 1 diabetics, Sjolie and colleagues have also suggested that, in the later stages of retinopathy, the adjustment of blood pressure, fibrinogen and triglyceride levels may affect outcome.2 The cessation of cigarette smoking may also have a beneficial effect although further prospective studies are required for clarification.10,11 The recognition of very early diabetic fundal change may provide the impetus to patients to make positive lifestyle changes and tighten glycaemic control. An overview of existing screening methods Based on the principles of the St Vincent Declaration,12 the European Retinopathy Working Party defined a protocol of screening for diabetic retinopathy.13 This advocated that all diabetic patients should have an annual eye examination by ophthalmoscopy through pharmacologically dilated pupils or by retinal photography. In 1991, the Department of Health commissioned a study of 3318 diabetic patients to establish the definitive method of performing fundal examination.14 A comparison was made between direct ophthalmoscopy, performed by GPs, ophthalmic opticians and hospital physicians, and an assessment of images acquired with a non- mydriatic Polaroid fundus camera by a consultant ophthalmologist. A fundal examination by an ophthalmologist with access to the Polaroid photographs was used as the ‘gold standard’. Buxton and colleagues concluded that direct ophthalmoscopy by all groups showed relatively poor sensitivities (hospital physician 67%; GP 53%; optician 47%), although specificities were higher (hospital physician 97%; GP 91%; optician 95%), indicating that relatively few inappropriate referrals to ophthalmology services would occur.14 However, it was apparent that direct ophthalmoscopy alone was likely to miss a significant proportion of patients with evidence of retinopathy, regardless of who performed the examination. Analysis of fundus photographs resulted in sensitivities ranging from 35 to 67% with specificities marginally higher than those of the primary screeners at 95–98%. Although it has been suggested that the performance of non- mydriatic cameras may be enhanced by the use of mydriatic agents,15 Buxton and colleagues’ findings are consistent with the findings of other comparative studies.16–20 These studies, summarised in Table 22, suggest that at present no individual method of routinely available fundus examination can be judged superior or indeed satisfactory for clinical screening. Although the findings of Klein and colleagues seem encouraging for a single non-mydriatric photograph, the wide confidence intervals calculated for their relatively small patient population warrant caution in interpretation of these results.16 Combining the modalities of ophthalmoscopy and retinal photography appears effective in detection of sight-threatening retinopathy for O’Hare and colleagues.20 However, even their technique does not achieve the recommended threshold of a minimum 80% sensitivity and 95% specificity for mild retinopathy as suggested for audit standards by the British Diabetic Retinopathy Working Group.21 An overview of digital technology Given the recognised limitations of existing retinopathy screening techniques, there has been a great deal of interest in exploring the potential role of digital photography and imaging techniques which can allow computer-assisted analysis of fundal images. Initially, digital techniques required the use of scanners to convert conventionally acquired images to a digital format. More recently, on-line direct acquisition of images has become increasing commonplace as technology advances. Within a digital camera system, photographic film has been replaced with CCD sensor, which is composed of a grid of individual picture elements or pixels.22 As reflected light from a fundus image falls on this sensor, each pixel generates a discrete numerical value that is representative of the level of luminance to which it is exposed. This numerical or digital signal is then relayed to the computer’s imaging system, and subsequently stored temporarily in random access memory (RAM). As monochrome and colour images require 1 and 3 MB of memory for storage, respectively, the number of images acquired before transfer to hard drive is necessary will be limited by computer RAM size. Therefore, there may be an advantage in systems that allow direct storage on to enhanced hard drives. As the number of pixels in the CCD is increased, image resolution is improved. Commission from the NHS Health Technology Assessment Programme It is against the above background that there has been significant interest in developing digital imaging techniques in the field of diabetic eye H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 5 1 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 22 Sensitivity and specificity of retinopathy detection techniques Screening test Reference/gold standard test Study population Definition of Sensitivity (%) Specificity (%) Reference positive test (95% CI) (95% CI) Non-mydriatic fundus cameras 45° non-mydriatic Polaroid Fundal examination by ophthalmic 3318 diabetic patients Referral with STDR 56 (48 to 63) 97 (96 to 97) Buxton, 199114 image read by ophthalmologist clinical assistant (technique not UK specified) with access to Polaroid images Single non-stereoscopic 45° 30° stereoscopic colour 99 diabetic patients A. Presence of any A. 100 (92 to 100) A. 91 (59 to 100) Klein, 199516 photograph centred between photographs according to – single eye examined retinopathy B. 93 (66 to 100) B. 98 (87 to 100) USA disc and fovea – 35-mm, ETDRS fields 1, 2 and 3 read B. Detection of non-mydriatic by trained ophthalmic graders PDR Non-mydriatic Polaroid Comparison with direct 2159 adults Presence of For new vessels For new vessels Taylor, 199017 photography – number of ophthalmoscopy examination by attending general retinopathy/ detected by either detected by either UK images taken not specified hospital physician participating diabetic clinics maculopathy by method: camera method: camera in diabetic clinic (overall sensitivity/ either test 65, ophthalmoscopy 60.3, specificity not calculated). Patients 77.5. For diagnosis ophthalmoscopy diagnosed with maculopathy exudative 39.7. For diagnosis referred for examination by maculopathy: exudative ophthalmologist camera 74.2, maculopathy: ophthalmoscopy camera 55.3, 57.4 ophthalmoscopy 67.0 Mydriatic fundus cameras Three 35-mm non-stereoscopic Examination by slit-lamp 395 diabetic patients Presence of STDR 89 (76 to 96) 86 (82 to 90) Harding,199518 45° overlapping images of each biomicroscopy by consultant UK fundus (Canon CR4-45NM ophthalmologist camera) Single non-stereoscopic 45° 30° stereoscopic colour 99 diabetic patients – A. Presence of any A. 98 (90 to 100) A 100 (77 to 100) Klein, 199516 photograph centred between photographs according to ETDRS single eye examined retinopathy B. 93 (68 to 100) B 100 (93 to 100) USA disc and fovea – 35-mm, with fields 1, 2 and 3 read by trained B. Detection of mydriatics ophthalmic graders PDR Single 60° colour slide Clinical examination by Patients attending Presence of Detection of any Detection of any Joannou, photography centred on the experienced ophthalmologist general diabetic clinic retinopathy and retinopathy 93 retinopathy 89 199619 macula – photographs graded (subgroup of 48 assessment made Detection of severe Detection of severe South Africa by clinicians patients underwent of grading accuracy retinopathy 100 retinopathy 75 ophthalmologist examination in addition to photography) continued Appendix 1 5 2 TABLE 22 Sensitivity and specificity of retinopathy detection techniques (cont’d) Screening test Reference/gold standard test Study population Definition of Sensitivity (%) Specificity (%) Reference positive test (95% CI) (95% CI) Direct ophthalmoscopy Direct ophthalmoscopy by Fundal examination by ophthalmic 3318 diabetic patients Referral with STDR i. 67 (50 to 84) i. 97 (96 to 99) Buxton, 199114 i. hospital physicians clinical assistant (technique not ii. 47 (23 to 71) ii. 95 (93 to 97) UK ii. opticians specified) with access to Polaroid iii. 53 (44 to 62) iii. 91 (90 to 92) iii. GPs non-mydriatic photographs Direct ophthalmoscopy by Fundal examination by staff-grade 493 patients examined A. Detection of A. i. 43 A. i. 94 O’Hare, 199620 i. opticians ophthalmologist (technique not by opticians; 517 background ii. 22 ii. 94 UK ii. GPs specified) with review of fundus patients examined retinopathy B. i. 75 B. i. 93 photograph by GPs B. Referral with ii. 56 ii. 98 STDR Direct ophthalmoscopy 30° stereoscopic colour 99 diabetic patients A. Presence of any A. 84 (72 to 93) A. 75 (51 to 91) Klein, 199516 through an undilated pupil by photographs according to ETDRS retinopathy B. 53 (37 to 69) B. 90 (79 to 96) USA experienced ophthalmic fields 1, 2 and 3 read by trained B. Detection of assistant ophthalmic graders PDR Combined techniques Direct ophthalmoscopy Fundal examination by staff-grade i. 493 patients A. Detection of A. i. 71 A. i. 94 O’Hare, 199620 combined with single photograph ophthalmologist (technique not examined by background ii. 65 ii. 92 UK centred on the macula (field size specified) with review of fundus opticians retinopathy B. i. 88 B. i. 99 not specified) photograph ii. 517 patients B. Referral with ii. 80 ii. 98 examined by STDR GPs CI, confidence interval; ETDRS, Early Treatment of Diabetic Retinopathy Study; PDR, proliferative diabetic retinopathy; STDR, sight-threatening diabetic retinopathy. disease. As no clear information was in existence as to how these techniques would fit into a clinical setting, we were commissioned in 1996 by the NHS Health Technology Assessment Programme to undertake both a systematic literature review and a primary study to assess the performance of digital imaging in screening for and monitoring the development of diabetic retinal disease. This document reports the findings of the systematic literature review. Ultimately, the principal interest of the NHS will be the clinical effectiveness and efficiency of digital imaging for screening for diabetic retinopathy and monitoring its progression. Assessing these qualities requires reliable information about the impact of digital imaging on clinical management of diabetic retinopathy and on later health (for example, sight-years saved), relating these to the resources used and comparing them with alternative policies for screening. However, it was our view that the technology had not yet reached the point of development where these parameters could be assessed reliably. Certainly, a search of MEDLINE using the standard Cochrane search strategy failed to identify any randomised controlled trials (RCTs) of digital imaging in this context. The review was therefore restricted to studies of the diagnostic performance of digital imaging in diabetic retinopathy, compared where possible with existing recognised techniques. In view of the difficulty in defining a gold standard in this field, we allowed the inclusion of comparative studies (studies comparing one ‘test’ with another, in this case, digital imaging). We had hoped to present a meta-analysis of results using ROC curves, thereby allowing consideration of performance of digital techniques against alternatives. Such curves allow direct comparisons of tests while varying the assumption on which normal and abnormal test results might be defined. However, owing to the diverse nature of the studies and the differing techniques under investigation, no meta-analysis of study results was possible and a qualitative analysis was carried out instead. Methods Aim and objectives of the review The overall aim of the systematic review was to assess the value of currently available digital imaging techniques and compare them with alternative methods. The objectives of the review were as follows: � to identify the number and quality of primary studies of digital imaging techniques in diabetic eye disease � to identify the range of available digital techniques applicable in this field � to determine whether current digital imaging techniques can detect early diabetic retinopathy when screening a population with no known retinopathy � to determine whether current digital imaging techniques can detect the progression of established retinopathy � to determine whether current digital imaging techniques can determine when patients require treatment of retinopathy � to evaluate ‘experimental’ techniques that may have a future clinical application � to compare current digital imaging techniques with alternative methods. Background The studies considered in this review investigated the diagnostic performance of digital imaging techniques in the field of diabetic retinopathy. The research methodologies of the studies considered therefore were different from those in reviews of effectiveness where the emphasis is solely on RCTs. The nature of the review was, however, systematic in the sense that there were explicit search strategies for identifying studies, explicit selection criteria for the studies that were considered, a systematic way of appraising the studies and a standard format was chosen for presentation of the data. Development of protocol A protocol was written at the start of the project that explicitly described the objectives of the review, the criteria required of studies for inclusion, the search strategy to be used for identification of studies and the methods of quality assessment, data abstraction and presentation of results. Systematic electronic bibliographic database searching The following electronic bibliographic databases were searched systematically: � MEDLINE (National Library of Medicine, electronic version of Index Medicus, USA) on OVID, CD PLUS. � EMBASE (Elsevier Science Publishers, electronic version of Excerpta Medica, Amsterdam) on BIDS and OVID. Health Technology Assessment 2003; Vol. 7: No. 30 53 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. � Science Citation Index (electronic version, paper version of same name, produced by ISI, Institute for Scientific Information, Philadelphia, PA, USA) on BIDS. � Ei Compendex Plus (Computerized Engineering Index, electronic version of The Engineering Index, produced by Engineering Information, Hoboken, NJ, USA) on BIDS. � Cochrane Library (database on disk and CD- ROM). The Cochrane Collaboration; 1997, Issue 3. Oxford: Update Software; 1997. Updated quarterly. � National Research Register (NRR), 14th consolidation, September 1996. The search strategy was first developed in MEDLINE as this is one of the best indexed and ‘user-friendly’ electronic bibliographic databases. Two sets of search terms were devised: one set to describe diabetic retinopathy and the other to describe digital imaging techniques. These were developed by the research team that included ophthalmologists, diabetologists, medical physicists and health services researchers experienced in literature searching. These two sets of terms were combined together using the Boolean operator ‘and’. (The use of a third set of study design terms was considered but was not included, in line with the decision to search for all possible study designs.) These sets of search terms were built up by investigating the medical subject headings (MeSH) terms using the MeSH tree with scope notes and Permuted Index as well as textword searching (searching for terms in the title and abstract). As new search terms were added to the search strategy, details of the first 50 titles and abstracts were scanned to assess their relevance to diabetic retinopathy and digital imaging. Terms that retrieved only irrelevant articles were further modified or rejected. The MEDLINE search strategy was modified for searching other databases. The modifications involved changing the syntax to suit that of the search software of the other databases and interrogating the thesaurus or indices of each database to identify equivalents of the MeSH terms (keyword system) used in MEDLINE. A more focused search was conducted on the other databases partly because they had less of a medical emphasis and partly because the strategy used on MEDLINE was fairly broad, resulting in many abstracts being assessed but few proving relevant. Searching the Internet We also searched the World Wide Web with the browser Internet Explorer and located relevant websites by using the advanced search options of the search engines Excite and Yahoo. The set of keywords entered into the query box were: digital imaging and diabetic and retinopathy. These keywords were combined together using the Boolean operator ‘and’. Handsearching Recent issues of three journals, British Journal of Ophthalmology (Volume 81, Issues 7–12, 1997), Graefe’s Archive for Clinical and Experimental Ophthalmology (Volume 235, Issues 7–12, 1997) and Investigative Ophthalmology and Visual Science (Volume 38, Issues 9–13, 1997), were handsearched for relevant publications that would not yet have appeared on the electronic databases owing to delays in indexing. Handsearching was performed by a health services researcher with an interest in the methodology of systematic reviews. This involved going through every page of each issue and reading through letters, editorials and conference reports in addition to published papers. Other methods of ascertainment of studies Reference lists of relevant studies The reference lists of relevant studies identified from the electronic databases were searched for references to other studies that might be relevant. Relevance was assessed from a hard copy of each article. We limited these searches to the ‘first- generation’ references only; in other words, we did not search the reference lists of studies originally identified from a previous reference list search. Contacting authors of key articles identified through the electronic searches Authors of key articles were contacted and asked if they were aware of any other relevant studies. A number of authors were also contacted for further information in relation to their publications. Other In addition the Proceedings of the British Diabetic Association’s Education and Care Section Annual Conference, 8–10 October 1997, and the British Diabetic Association’s Medical and Scientific Section Autumn Meeting, 9–10 October 1997, and the Spring Meeting, 25–27 March 1998, were searched for relevant abstracts or poster presentations. Appendix 1 54 Identification of possible studies All possibly relevant studies were electronically imported or manually entered into the reference managing software package Reference Manager (Version 7.01N; Research Information Systems, Carlsbad, CA, USA). Subject keywords and source of article were added. Register of possible studies Initially, all electronically derived abstracts and study titles were read by a diabetologist and a health services researcher with an interest in the methodology of systematic reviews to assess subject relevance. However, because of the high degree of concurrence and the greater speed at which abstracts could be assessed by the diabetologist, it was decided that the diabetologist alone should assess the abstracts. All relevant studies were assigned specific topic keywords on Reference Manager and the full published paper was obtained. Assessment of studies for inclusion Hard copies of studies were assessed for subject relevance and eligibility by a diabetologist. The assessor was not blinded to author, institution or journal. Studies were included if they reported on the use of either direct or indirect digital imaging techniques in the field of diabetic retinopathy and involved patients with either type 1 or 2 diabetes. Owing to the early stage of development of digital technology, poster and abstract presentations were included in the review. Given the original remit of the review, early experimental techniques which were applied to animal models only were not included. Studies used to monitor response to treatment were also not included as we were primarily interested in digital techniques which could be applied to diabetic retinopathy screening and monitoring of retinopathy progression. Quality assessment of studies to be included Comparative studies where digital imaging was assessed against an alternative technique were graded on the following methodological criteria proposed by Carruthers and colleagues:23 � use of a recognised gold reference standard � independent assessment of test under review and gold reference standard � test applied to an appropriate study population, that is, diabetic patients suspected but not known to have retinopathy, diabetic patients with established retinopathy and non-diabetic control groups � avoidance of verification bias, that is, reference standard applied to all patients under study � reproducible description of both the test and the reference standard given. Studies were graded depending on the number of these criteria met, with a score of V being awarded when a study met all five criteria and a score of I being given when only one of the above criteria was met. Posters and abstract presentations generally received a low grading owing to a lack of available information on study methodology. It is accepted that this grading may not reflect the value of future publications derived from this work. The difficulty in defining a gold standard for evaluation of techniques in this field is recognised by the reviewers; in this context, it is accepted to be either biomicroscopic examination by an experienced ophthalmologist or the use of the seven-field Airlie House photography protocol. Grading of results section in studies to be included In addition to the above methodological criteria, an assessment was made of the interpretation of study results by authors based on the following criteria adapted from those proposed by Carruthers and colleagues:23 � that sensitivity and specificity could be correctly calculated from comparison with a recognised gold standard as defined above � test reproducibility calculated by the authors to determine the consistency of results obtained � appropriate statistical method of analysis applied to results � data presented to allow confirmation of authors’ findings. Those studies meeting the above four criteria for grading of results were given a grading of IV, those meeting three of the four a grading of III, etc. The grading was carried out by a diabetologist and then, independently, by two expert in health services research. No significant difference was found between the graders. Posters and abstract presentations generally received a low grading owing to a lack of available information on the authors’ interpretation of results. It is accepted that this grading may not reflect the value of future publications derived from this work. Data abstraction The following information was extracted from the individual studies: technique under review; population studied; study aim; description of gold standard or comparative method used; summary of results; summary of reviewer’s comments; Health Technology Assessment 2003; Vol. 7: No. 30 55 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. reference; and methodological and results grading. This information has been presented in tabular form for ease of reference. Data analysis The original research proposal was for a quantitative analysis of digital imaging techniques as applied to diabetic retinopathy assessment. In view of the early stage of evolution of digital technology in this field, no statistical meta-analysis of study results was possible owing to their diverse nature and differing techniques. Therefore, only a qualitative analysis was possible. The results of the individual studies were summarised systematically and the consistency of similar studies was then considered formally. Derivation of included studies Results of systematic literature review Around 2767 published abstracts and poster presentations were considered for inclusion. A total of 40 studies met the criteria for inclusion; 28 of these were comparative studies comparing digital imaging techniques with alternative methods and 12 were studies describing previously evaluated digital techniques being used in research. Twenty-five studies were found in MEDLINE after reading 679 abstracts, a further five were found among 1327 abstracts generated by the systematic searches in the other electronic databases and the remaining nine were found in other ways. Table 23 summarises how studies were first identified, how many were judged to be possibly relevant to the review and how many were confirmed suitable for inclusion. Reporting the findings of the systematic review The following three sections will provide a qualitative description of the results of the search strategy, broadly discussed under the categories of digital fundus photography, digital angiography and the scanning laser ophthalmoscope. Each section contains a descriptive text, outlining the development of the digital technique and its application to the detection of diabetic retinopathy. The role of the search strategy was to identify papers where a digital imaging technique had been compared with existing screening and diagnostic techniques or, once validated, used to further our understanding of the pathogenesis of Appendix 1 56 TABLE 23 Results of systematic literature review Source No. of reports No. of published No. of studies No. of studies identifieda abstracts/posters possibly relevant included in assessedb to review final review Electronic searches MEDLINE 733 679 72 25 EMBASE 1273 771 35 3 Science Citation Index 626 368 15 2 Ei Compendex 215 156 3 0 Cochrane Library 32 32 0 0 National Research Registerc 12 N/A 3 0 Subtotal 2891 2006 128 30 Other Proceedings 371 371 6 5 Experts contacted 38 N/A 2 3 Bibliographies checked 36d N/A 19 2 WWW search: Excite 50 N/A 0 0 Yahoo 155 N/A 0 0 Handsearching 390 390 0 0 Subtotal 1040 761 27 10 Total 3931 2767 152 40 a Some reports were identified from more than one source. b The number of abstracts assessed after duplicates had been removed by the Reference Manager database. This is only applicable to searches carried out on the electronic databases. c A national database of information about ongoing research currently taking place in, or of interest to, the NHS. d Number of reference lists checked. diabetic retinopathy. These papers have been summarised in tabular form as techniques under evaluation or validated techniques used in research. Where digital imaging has been compared with alternative techniques, an assessment of the study methodology and the authors’ interpretation of results is presented. Publications identified by the review that did not fulfil the above criteria but were felt to be of importance for their contribution to the overall development of digital imaging in this field have been referred to in the text only. Papers relating to the treatment of retinopathy or confined to animal models have not been included in the final review. Digital fundus photography Number and quality of eligible studies Seventeen relevant publications were identified relating to digital fundus photographic techniques, using both indirect and on-line image acquisition. Four of the 17 included publications were in poster or abstract format. These studies are summarised in Table 24 and evaluated in Table 25. Manual evaluation of images Digital imaging versus colour slide photography Indirect digitisation of original colour slides to facilitate easier storage and computer analysis does not seem to affect image quality adversely to a degree affecting clinical use, as demonstrated by George and colleagues24 (Table 24). Diagnosis from digitised images displayed on a high-resolution video monitor was in agreement with that from the original colour slides in 95% of cases of sight- threatening retinopathy and 100% of non-sight- threatening retinopathy. It was noted that cotton- wool spots were underdiagnosed in the digital images although the authors concluded that the decision to refer for specialist advice was unaffected. Realising the potential for improved storage and retrieval of images, Friberg and colleagues (Table 24) attempted an early evaluation of the quality of directly acquired images against conventional imaging modalities.25 Ten diabetic patients, with fundal appearances ranging from mild background retinopathy to proliferative retinopathy, were included in an assessment of 50 patients with retinal pathology. Using a digital system capable of generating images composed of 512 × 512 pixel array, giving rise to a quarter of a million pixel elements per picture, the authors conceded that picture resolution was less than that of conventional slide photography (standard 35- mm transparency film may contain up to 4 million pixel elements per image). However, a correct clinical diagnosis was made for each diabetic patient from reviewing digital images alone and the authors concluded that the system appeared to give sufficient detail for clinical purposes. As they predicted, the technology has advanced rapidly since the time of this early study and commercial cameras delivering an image of 1024 × 1024 pixels are now readily available. Cameras that can image 2036 × 3060 pixel array are beginning to enter the market but are likely to be limited in their use at present owing to their higher cost.26 A Canon CR5 45NM-based digital imaging system has been compared against the results of 35-mm colour slide photography by Ikram and colleagues (Table 24), using slit-lamp biomicroscopy to provide a reference standard for validation of the techniques.27 From a population of 66 diabetic patients attending a mobile screening unit, clinical examination confirmed the presence of background diabetic retinopathy (BDR) in 48%, pre- or proliferative diabetic retinopathy (PDR) in 3% and clinically significant macular oedema or maculopathy (CSMO) in 15%. The results of photography were graded by a consultant diabetologist and found to be comparable for detection of retinopathy using both techniques (35-mm colour slide, BDR 38%, PDR 3% and CSMO 13%; digital imaging system, BDR 44%, PDR 3% and CSMO 14.5%). All patients with sight-threatening retinopathy were identified by each photographic technique. Although there has been no direct calculation of sensitivity and specificity, the authors conclude that digital imaging is at least comparable to colour slide photography from their study. A similar conclusion was reached by George and colleagues (Table 24) using a Canon CR5 retinal camera in an extension of their earlier work on retinopathy detection.28 Using a study group of 40 patients with a wide spectrum of diabetic retinopathy, the results of directly acquired digital images were compared with conventional slide photography. An exact agreement in grading was achieved in 93.3% of eyes. When the images of those undergraded as non-sight-threatening retinopathy were evaluated, this discrepancy was attributed to the lower resolution of digital photography. In two of the three cases, this was due to the failure to capture IRMAs, clearly visible on colour slide images. In one case, cotton-wool Health Technology Assessment 2003; Vol. 7: No. 30 57 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Appendix 1 5 8 TABLE 24 Evaluation of digital photography techniques Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Digital imaging in comparison with colour slide photography Digitisation of 45° macular view colour transparencies 150 retinal images from diabetic patients, covering a spectrum of retinopathy and including those with normal fundi To compare the detection of DR from digitised images displayed on a high-resolution video monitor with diagnosis made from colour slide examination Comparative method Assessment made against diagnosis from original slide Low intra-observer variation of 7% noted when assessing quality control images from both techniques. Of patients diagnosed with STDR on colour slide, 95% similarly diagnosed from digital images (84/88; 95% CI 88 to 99%); for those with NSTDR, 100% (62/62; 95% CI 93 to 100%) diagnosed from digital images. 4 cases of STDR undergraded to NSTDR by digital image analysis. (Difficulties in identification of CWS and macular hard exudates masked by light reflection noted in digital images. Initial poor quality slide prevented macular haemorrhage being identified on corresponding digitised image) Within a clinical setting, the authors report close agreement between the interpretation of images from both techniques. The under-reporting of STDR is likely to be overcome by direct digital image acquisition, which may have prevented the problems with image quality reported in these particular patients. 100% agreement for patients with NSTDR suggests both photographic techniques would be equally attractive for screening programmes, although validation against existing techniques has not been performed George, 199724 Methodology Grade IV Results Grade III continued H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 5 9 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Direct acquisition of digital fundus photographs and fluorescein angiograms using a TOPCON TRC 50 camera interfaced with a PAR IS2000 imaging system via a high- resolution video camera 10 diabetic patients included in a preliminary study of 50 consecutive patients attending retinal specialists clinics To evaluate the role of directly acquired digital imaging in comparison with conventional photography in diagnosis of retinal disease, including DR Comparative study Comparison of diagnosis made by retinal specialist from digital images with that of colleague using conventional photographs – any discrepancies reviewed by third specialist using both sets of images and patient history In all 10 patients with DR under study, both specialists were in agreement – 1 diagnosed with mild NPDR; 7 diagnosed with exudative retinopathy; 2 diagnosed with neovascularisation (1 neovascularisation elsewhere; 1 neovascularisation of disc) Although patient numbers under study are inadequate to influence clinical practice, the authors have demonstrated agreement of both photographic techniques across a spectrum DR. Despite the limitations of the relatively low- resolution system under review, lesions including neovascularisation were detectable. The study population was biased towards those with exudative or PDR; these results may not reflect the outcome for a general screening population. However, they illustrate that digital imaging has significant advantages over conventional photography in terms of instant availability of images with ready correction of alignment and focusing problems Friberg, 198725 Methodology Grade III Results Grade I continued Appendix 1 6 0 TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Direct acquisition of digital images using a Canon CR5 45NM fundus camera and electronic imaging system 66 diabetic patients To evaluate the role of digital imaging in comparison with standard 35-mm colour slide photography in evaluation of DR Gold standard Fundal examination by slit-lamp biomicroscopy Clinical examination was used to define the prevalence of BDR at 48%, PDR at 3% and CSMO at 15%. Detection of retinal disease was comparable using both camera techniques. Digital imaging: BDR 44%, PDR 3% and CSMO 14.5%. 35-mm colour slides: BDR 38%, PDR 3% and CSMO 13% The authors concluded that digital imaging is at least comparable to conventional 35-mm slide photography in detection of retinopathy. No patients with STDR were undetected using either photographic technique. Although not presented by the authors, sensitivity was calculated from available data as: Digital imaging: BDR 91.7%; PDR 100%; CSMO 96.7% 35-mm colour slide: BDR 79.2%; PDR 100%; CSMO 86.7% Unfortunately, data were not presented in a manner which clarified whether specificity was also comparable with both techniques. This may be available in subsequent publication but would clearly have implications for clinical application. However, given the initial results, further assessment against current screening methods should be supported Ikram, 199727 (Poster and personal correspondence with author) Methodology Grade V Results Grade I continued H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 6 1 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Direct acquisition of images using a Canon CR5 retinal imaging system (2 X 45° fields) 40 diabetic patients with wide spectrum of established retinopathy including normal controls (5 patients) To compare the results of directly acquired digital images with conventional colour slide photography Gold standard EURODIAB photographic protocol of 2 × 45° overlapping fields used in this study has been validated as comparable to gold standard Airlie House 7 × 35° fields for 35-mm colour slides Exact agreement in grading in 93.3% of eyes (95% of STDR and 100% of NSTDR correctly identified) Undergrading of STDR in 3 cases (IRMA not identified in 2 cases; CWS misclassified in 1 case). System limited by current pixel density of 768 × 512 with loss of fine detail structures. IRMA clearly evident in colour slide film Using the data presented by the authors, digital imaging sensitivity and specificity can be calculated: detection of NSTDR; sensitivity 96.3%; detection of STDR, sensitivity 93.8%. Although the 5 patients without retinopathy were correctly identified, the patient population is biased towards those with established retinopathy. Therefore, results obtained are not applicable to a screening programme. However, given the limitations of lower resolution, the digital system has performed well in comparison with 35-mm colour slide photography. The detection rate of 95% of STDR is superior to that of alternative screening techniques in current use George, 199828 Methodology Grade V Results Grade II continued Appendix 1 6 2 TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Digital imaging in comparison with Polaroid photography On-line acquisition of digital fundus images – electronic imaging system attached to a Canon CR5 45NM fundus camera 107 diabetic patients photographed after mydriasis – 213 images obtained To evaluate the diagnosis of STDR from digital images compared with Polaroid Comparative study The overall prevalence of retinopathy was determined from examination of both sets of images. No other independent clinical examination appears to have been performed from the information available DR was present in 58 eyes, of which 55/58 (95%) were detected on digital images and 49/58 (84%) were detected on Polaroid. Thirty-four eyes showed retinopathy meriting ophthalmologist referral – 34/34 (100%) were evident with digital imaging and 24/34 (71%) evident on Polaroid Digital images were superior to Polaroid in the detection of retinopathy and identification of those patients requiring ophthalmologist referral. The authors have also highlighted the advantages of electronic imaging systems in terms of patient comfort, enhanced storage and retrieval of images and the potential for transfer for off-site analysis. Future comparison against other screening modalities using a gold standard for validation would be required prior to a recommendation for clinical practice Ryder, 199629 Methodology Grade IV Results Grade I Direct acquisition of digital images using a TOPCON non-mydriatic camera and Imagenet system or Frost Medical Systems Ris-Lite system 118 patients photographed after mydriasis To determine the effective- ness of digital imaging in detecting DR against Polaroid photography performed with a Canon CR4 camera – for both techniques single 45° field centred between macula and disc analysed Gold standard Standard seven- field 35° fundus photography Detection of any retinopathy Digital imaging sensitivity 74% and specificity 96% (Polaroid, sensitivity 72% and specificity 88%) Detection of retinopathy meriting referral for ophthalmologist review: digital imaging sensitivity 85% and specificity 98% (Polaroid sensitivity 90% and specificity 98%) For single image comparison, digital images appear as effective as Polaroid systems Taylor, 199831 (Poster and personal correspon- dence with author) Methodology Grade V Results Grade III continued H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 6 3 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Digital imaging in comparison with retinal examination On-line digital photographs – three 30° fields. Diagnosis made by retinal specialist from examination of photographs 11 consecutive patients with diabetic retinopathy attending an ophthalmologist clinic To determine value of digital fundus camera as a screening tool for DR Comparative study Clinical examination by another retinal specialist in masked fashion Of 22 eyes examined, classification agreed on in 19. Digital image detected one case new vessels missed on examination; examination confirmed one case of new vessels less distinct on photograph. One patient defined diabetic maculopathy by image and background DR on examination The study population presented is inadequate to allow implications for clinical practice to be determined. Although the authors suggest that the study provides support for the use of digital imaging as a screening tool, the population studied were all known to have retinopathy and do not fit the criteria for a general diabetic population. The potential advantages of electronic image transfer are recognised Gupta, 199633 (Poster and personal correspon- dence with author) Methodology Grade IV Results Grade I Direct acquisition of digital fundus images 611 consecutive patients attending a general diabetic clinic randomised for assessment by either protocol To determine whether digital imaging enhanced detection of retinopathy when added to routine screening Comparative study Results of direct ophthalmoscopy and visual acuity assessment alone compared with results of routine screening plus digital image analysis For the detection of early background retinopathy: detection rate in type 1 DM 42.6% with digital imaging vs 26.7% with routine screening; type 2 DM 33.2% with digital imaging vs 20.3% with routine screening The addition of digital imaging significantly increased the detection rate of mild retinopathy in the group under review compared with those undergoing only routine screening. The authors concluded that the detection of early DR is enhanced by fundus photography. These findings did not apply to other degrees of retinopathy. Although the study was weakened by the lack of a gold standard for validation of results, this preliminary report shows a role for digital photography in early detection of retinopathy. As advances are made in the prevention of progression of retinopathy, this finding will be of significance should the emphasis in screening move from detection of referable eye pathology to detection of retinopathy at onset Lindsay, 199834 (Poster) Methodology Grade III Results Grade II continued Appendix 1 6 4 TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Automated detection of diabetic retinopathy 60° red-free fundus photographs of posterior pole digitised using a Nikon Coolscan slide reader to provide 700 × 700 pixel images. Prospective study 200 images from patients with diabetic retinopathy and 101 images from non- diabetics with normal fundi Evaluation of the ability of an artificial neural network to detect DR after optimisation of computer protocols Gold standard Evaluation of conventional images by an experienced retinal specialist For detection of DR, system achieved sensitivity of 88.4% and specificity of 83.5% in comparison with specialist. Ability to differentiate exudates or haemorrhage/microaneurysms from normal retina (or normal retinal image containing vessels only) 93% and 73.8%, respectively. To ensure that all patients referred by specialist for evaluation are also detected by the system would require increasing sensitivity to 99% at the expense of reducing specificity to 69%. At present, authors feel the system is comparable to results achieved by optometrists, Polaroids from non- mydriatic cameras and diabetologists using direct ophthalmoscopy Although the neural network can differentiate blood vessels, hard exudates and haemorrhages/microaneurysms with relative accuracy from background retina, the basis for subsequent decision to refer patients remains unclear. No features of moderate (CWS, IRMA, VB) or proliferative (neovascularisation) are being detected using the system described. A comparison of the technique with an established screening method would be valuable to determine the role in future clinical management. At present, the system remains untested in a clinical setting Gardner, 199635 Methodology Grade V Results Grade II continued H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 6 5 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Directly acquired on- line digital fundus photographs or digitised colour photographs – seven 35° fields according to Airlie House criteria 100 diabetic patients To develop an automated method of lesion identification and quantification of severity of DR Gold standard Diagnosis by retinal specialist from seven 35° colour fundus photographs For microaneurysms, dot-and-blot haemorrhages and striate haemorrhages, computer showed 99% specificity compared with retinal specialist; sensitivity better than retinal specialist. For CWS, exudates, IRMA and neovascularisation, sensitivity was 93% and specificity 90%. The authors believe that the use of computer-based analysis is comparable to retinal specialists for the interpretation of fundus photographs The authors have attempted to develop a fully automated computer-driven analysis system, with detection of lesions across the whole spectrum of DR. We have been unable to identify subsequent formal publication of this presented abstract to provide clarification of the methodology used in this study. The authors have stated an intention to develop a commercially available system in the near future. While highlighting this work as an area of future interest, no comment can be made at present regarding implications for patient management based on the available data Sinclair, 199636 (Abstract and personal correspond- ence with author) Methodology Grade IV Results Grade II continued Appendix 1 6 6 TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Automated detection of hard exudates Direct acquisition of digital fundus images 134 images taken from those routinely acquired at a general diabetic screening clinic To develop an automated DR screening programme, applying a statistically based pattern recognition program Not stated in report Pattern recognition allowed correct identification of optic disc and fovea in 78.4% of images. Microaneurysms, haemorrhages, exudates and CWS were identified with success rates of 66, 89, 97 and 63%, respectively Authors found difficulty differentiating between microaneurysms and haemorrhages and between exudates and CWS at this preliminary stage. They suggest a successful role for their novel statistical approach to pattern recognition in future automated screening programmes. Further development and clinical assessment will be required before this is likely to impact on patient management Ege, 199737 (Abstract) Methodology Grade II Results Grade I Digitisation of colour transparency slides to yield black and white image with 512 × 512 pixel array Standard photographs used in the ETDRS for classification of retinal exudates The semi- automated detection of retinal exudates Gold standard Exudate area determined by program assessed against standard photograph grading of severity With shade correction and contrast enhancement, program could identify 3 distinct grades of severity which correlated with those used in the EDTRS grading. With serial analysis of individual images, standard deviations of calculated exudate area were significantly lower in grade 1 compared with grade 3 images, i.e. reproducibility of technique decreased as number of exudates increased Ward, 198939 Methodology Grade II Results Grade II continued H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 6 7 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Indirect digitisation of 30° or 50° field colour slides centred on macula after projection through a red- free filter Diabetic patients with exudative retinopathy. No normal controls were included in this paper Automated detection and quantification of retinal exudates Gold standard Manual estimation of false positive and negative by experienced ophthalmologist Sensitivity 87% (range 61–100%) Coefficient of variation for reproducibility 3% for confluent areas of exudate and 17% for small scattered areas Authors have suggested that results may be improved by using directly acquired digital images. The region of interest was manually delineated prior to processing. It is anticipated that program refinement could enable this function to be automated. The speed of data acquisition and analysis and lack of user interaction will lead to enhanced objectivity in the assessment of exudative retinopathy. This has potential in both screening for retinopathy and in the monitoring of response to treatment. The program presented appears robust and well validated. Evaluation of its performance in a study population with and without exudative retinopathy would be of interest Phillips, 1991,40 199341 Methodology Grade IV Results Grade IV continued Appendix 1 6 8 TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Secondary digitisation of colour transparency photographs using filters to give images in three colour planes (red, green and blue) with 512 × 480 pixel resolution Retrospective analysis of images from 30 patients, specifically chosen to contain both the lesion under investigation plus a random distribution of other variables such as background pigmentation. 10 images each with either CWS, exudates or drusen were identified To determine whether fundus lesions can be identified on the basis of colour alone. The effect of luminance on colour was removed to overcome variability in exposure. This allowed a 2- dimensional vector to be used to assign position of individual lesions on a chromaticity scatter diagram Gold standard Examination of colour transparencies by a retinal specialist Different lesions appeared to occupy distinct regions of the chromaticity scatter diagram, with a degree of overlap between CWS and drusen. This was reflected in the greatest error in discrimination between these lesions after application of the Mahalanobis classifier and jackknife technique for assessment of separability of lesions into correct groups (exudates, sensitivity 70%, specificity 95%; CWS, sensitivity 70%, specificity 65%; drusen, sensitivity 50%, specificity 85%; calculated from data presented by the authors) The authors propose that their technique alone is insufficient as a discriminator when lesions are of similar colour. They propose to investigate the use of additional features of size, shape, edge sharpness and texture to aid lesion recognition. At this stage in development, it is unlikely that this program will enhance automated image analysis Goldbaum, 199042 Methodology Grade IV Results Grade I continued H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 6 9 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Automated detection of VB Colour transparencies processed by digital slide scanner. Regions of interest corresponding to 64 × 64 pixels manually identified for analysis Prospective study of patients attending an ophthalmology department for assessment of diabetic retinopathy. 54 vessel segments from 18 sets of photographs processed for further evaluation Automated assessment of VB Gold standard Assessment by professional photographic graders, using adaptation of the Airlie House criteria for classification and studying slide reproductions of the digitised vessel segments 51 slides considered assessable by 2 graders, achieving exact match of clinical grade in 76% of cases. For remaining images, grading differed by one level of severity with 7 of 12 being in the ‘questionable vs definite’ category and definitive grading made by a senior colleague. Computer-based VB index was able to differentiate significantly advanced beading from each of the other 3 categories, definite beading from both advanced beading and normal vessels, and normal vessels from both definite and advanced beading (p = 0.05, Tukey’s non-parametric test) The application of Fourier analysis to the measurement of variation of vessel diameter allows quantitative assessment of VB which has been shown to reflect clinical grading. It is recognised that VB is a difficult lesion to differentiate clinically and a method of reducing subjectivity is welcomed. Integration with additional lesion detection programs would be beneficial for use in clinical practice. Validation in a clinical setting would assist definition of its role in future use Kozousek, 199244 Methodology Grade V Results Grade I continued Appendix 1 7 0 TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Automated detection of neovascularisation Low-angle fundus photographs from the optic disc enlarged to allow manual tracing of vessel patterns. Images digitised and density–density correlation method used to calculate fractal dimensions 10 retinal vessel patterns from patients with known new vessels fulfilling criteria of NVD ≥ EDTRS Grade 3; images from 14 healthy fundi as control group To use the principles of fractal geometry to differentiate normal retinal vasculature from the development of neo- vascularisation Gold standard Professional photographic graders using modified Airlie House criteria for the definition of neovascularisation Assuming 100% accuracy by graders, FD = 1.8 yields a sensitivity for detection of NVD ≥ EDTRS Grade 3 of 90%. With these conditions, specificity is 93% with false-positive value of 7% and false-negative value of 10% The method of deriving digital images of vessel patterns is cumbersome and may be improved by the use of direct acquisition digital images. The technique is limited by an inability to identify NVD Grade 2, which also pose a risk of haemorrhage, and the presence of vitreous haemorrhage which obscures fundal detail. The area under review is a low-angle 10° view centred on the optic disc, which will not allow detection of early peripheral neovascularisation. The technique may have implications for automated screening strategies but will require further clinical validation Daxer, 199347 Methodology Grade V Results Grade III continued H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 7 1 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 24 Evaluation of digital photography techniques (cont’d) Technique Population studied Study aim Gold standard Results Reviewers’ comments Reference Methodology under review or comparative and results method grading Automated detection of vessel diameter CWS, cotton-wool spots; DM, diabetes mellitus; DR, diabetic retinopathy; NPDR, non-proliferative diabetic retinopathy; NSTDR, non-sight-threatening diabetic retinopathy NVD, new vessels disc; VB, venous bleeding. Digitisation of serial 30° 35-mm photographs via high-resolution video camera and Context Vision GOP-302 image-analysing computer 53-year-old normotensive male type 1 diabetic with mild background retinopathy and normal renal function Evaluation of a semi- automated computer- driven method of assessing vessel diameter – vessel segment manually selected. Vessel diameter analysis by calculating average grey profile across vessel at 12 neighbouring parallel cut lines adjacent to region of interest Comparative study Comparison with observer- dependent evaluation using graphics-digitising table and manual definition of vessel diameters Co-efficient of variation of single image 1.5–7.5% for semi- automated method; 6–34% for observer method (noted to be dependent on vessel size). Standard diameter of variation from the mean 4.2 µm for automated method (cf. 18.9 µm for observer), with 95% CI being 3.2 to 6.0 and 14.6 to 27.1 µm, respectively (p <0.001) Image analysis method appears more reproducible and accurate, particularly with small-diameter vessels. ECG- triggered photography may reduce variability in serial photography analysis by minimising effect of cardiac cycle on perfusion pressure. This technique will enhance research in retinal blood flow parameters and the pathogenesis of retinopathy. It is unlikely to provide a useful tool in general retinopathy screening and management Newsom, 199255 Methodology Grade III Results Grade II Appendix 1 7 2 TABLE 25 Assessment of study methodology and results – evaluation of digital photography techniques Methodology Results Use Appropriate No Correct gold Independent study verification Reproducible sensitivity/ Reproducibility Statistics Data Reference standard assessment population bias description Grading specificity calculated appropriate presented Grading George, 199724 � � � � IV � � � III Friberg, 198725 � � � III � I Ikram, 199727 � � � � � V � I George, 199828 � � � � � V � � II Ryder, 199629 � � � � IV � I Taylor, 199831 � � � � � V � � � III Gupta, 199633 � � � � IV � I Lindsay, 199834 � � � III � � II Gardner, 199635 � � � � � V � � II Sinclair, 199636 � � � � IV � � II Ege, 199737 � � II � I Ward, 198939 � II � � II Phillips, 1991,40 199341 � � � � IV � � � � IV Goldbaum, 199042 � � � � IV � I Kozousek, 199244 � � � � � V � I Daxer, 199347 � � � � � V � � � III Newsom, 199155 � � � III � � II spots among panretinal photocoagulation scars were misdiagnosed as laser burns. However, while accepting the limitation of their system operating at a pixel density of 768 × 512, the digital camera shows promise as an additional tool in retinopathy management with the potential to improve its performance as technology advances. Digital imaging versus Polaroid photography A direct comparison of instant electronic imaging systems against Polaroid photography has been presented by Ryder and colleagues (Table 24).29 After administration of mydriatics, 213 eyes were imaged. Diabetic retinopathy was found to be present in 58 eyes, and identifiable in 55 of the digital images but only 49 of the Polaroid photographs. Thirty-four eyes were deemed appropriate for referral to an ophthalmologist: evidence of retinopathy requiring referral was present in the digital images in all cases but only seen in 24 (71%) of the Polaroid photographs. In addition to the superiority of lesion detection demonstrated, the patients preferred the electronic imaging system as the less intense photographic flash was perceived to be more comfortable. This reflects the findings of Taylor and colleagues (Table 24), who found that 96% of patients surveyed following fundus photography found the electronic systems as comfortable or better (44%) than Polaroid photography.30 From an educational aspect, 93% of patients agreed that the ability to view their own fundus images was important, with 91% also finding value in an immediate explanation of the images. The same group have also performed an assessment of the effectiveness of digital imaging in comparison with Polaroid photography (Table 24).31 They demonstrated a sensitivity of 74% (Polaroid 72%) and a specificity of 96% (Polaroid 88%) for the detection of any retinopathy; the detection of retinopathy meriting referral to an ophthalmologist showed a sensitivity of 85% (Polaroid 90%) and a specificity of 98% (Polaroid 97%). These results were obtained with the use of seven-field stereo photography as a recognised gold standard. Therefore, while retaining the advantages of an instantly acquired image, it would appear that the potential for enhanced picture storage and manipulation from a digital system can be gained without any loss of picture quality. St Thomas’ Hospital, London has an established Diabetic Eye Complication Screening programme, allowing open access for GPs who manage their own diabetic clinics.32 Although photography was traditionally performed using a non-mydriatic Polaroid camera, this technique has recently been replaced by digital photography. The authors confirm that an on-going evaluation of their new system in comparison with conventional imaging is in progress but are unable to provide preliminary results at this stage. Digital imaging versus retinal examination In an evaluation of digital cameras in a clinical setting, images were directly acquired from 11 consecutive patients using three 30° fields of view (Table 24).33 The results of image analysis and clinical examination by a retinal specialist showed agreement of classification in 19 of the 22 eyes. Of those where there was disagreement, one case of new vessels was determined by photography alone; one case of questionable new vessels on photography was clearly evident on examination; and in one eye, diabetic maculopathy was classified on photographic screening with background retinopathy only being present on examination. This latter finding may relate to the difficulty in determining the presence of macular oedema from non-stereoscopic images. The numbers reviewed in this presentation are small and the authors’ claim that digital imaging will provide an efficient screening tool is premature without the benefit of a large-scale clinical trial. However, they do highlight the possibility of using the digital camera to acquire images at the place of delivery of diabetic care with subsequent electronic transmission elsewhere for further analysis. With perhaps more relevance to the practical management of diabetic patients within a hospital setting in the UK, the detection rate of retinopathy was noted to be higher in consecutive patients attending a general diabetic clinic when digital imaging was added to routine screening by direct ophthalmoscopy (type 1 diabetes mellitus, prevalence of background retinopathy with digital imaging 42.6% versus 26.7% with routine screening only; type 2 diabetes mellitus, prevalence of background retinopathy with digital imaging 33.2% versus 20.3% with routine screening only) (Table 24).34 It was assumed that the prevalence of retinopathy would be equal in both groups under study, although details of their randomisation protocol are not available. Although there was no significant difference in detection rates for other degrees of retinopathy, the data presented suggest that the use of digital photography can enhance the detection of mild retinopathy when early lesions may be difficult to detect with direct ophthalmoscopy alone. Health Technology Assessment 2003; Vol. 7: No. 30 73 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Automated analysis of images Detection of general retinopathy Retinal microaneurysm counts have been shown to be a useful indirect marker for grading the severity of retinopathy. These correlate with the presence of both haemorrhages and cotton-wool spots and, to a lesser degree, with hard exudates and intraretinal microvascular abnormalities.6 In addition, based on a study monitoring the increase in total microaneurysm count over a 4-year period, it is suggested that this change from baseline may be used as a surrogate measure for identifying those patients likely to develop significant diabetic retinopathy in the future.5 This prompted Gardner and colleagues (Table 24) to apply automated computer analysis techniques to digitised colour slides for the detection of early retinopathy.35 In a prospective study of 301 posterior pole fundal images from both diabetic patients with retinopathy and non-diabetic controls with healthy fundi, a neural network system was able to detect retinopathy with a sensitivity of 88.4% and a specificity of 83.5% in comparison with examination of the original fundus photographs by an experienced ophthalmologist. The ability to differentiate lesions from normal background retina was 73.8% for microaneurysms and haemorrhages and 93.0% for hard exudates. Adjustment of the sensitivity thresholds to 99% would allow all cases detected by the specialist to be identified by the system but would have led to a corresponding reduction in specificity to 69%. Their initial results are, however, comparable to those being achieved by optometrists and diabetologists using direct ophthalmoscopy and Polaroids from non-mydriatic cameras (Table 22). In practice, the application of widespread retinal photography on the scale required to undertake regular screening assessment of the diabetic population will be limited by the availability of trained photographic graders. Therefore, there is pressure to develop a fully automated screening programme, taking advantage of the potential for computer-aided analysis of digital images. Sinclair and colleagues (Table 24) are currently developing an automated diabetic retinopathy screener with early results being presented at the American Diabetic Association Meeting in June 1996.36 From analysis of the images of 100 patients, using direct acquisition or secondary digitisation of colour slides in seven standard 35° fields, their findings appear encouraging. In comparison with an assessment of the original colour fundus photographs by a retinal specialist, the computer showed a 99% specificity in detection of microaneurysms, blot and striate haemorrhages. Although the sensitivity was quoted to be better than that of the retinal specialist, exact values were not given. With regard to the detection of the more advanced lesions of nerve-fibre layer infarcts, exudates, intraretinal microvascular abnormalities and neovascularisation, the computer achieved a sensitivity of 93% and a specificity of 90%. Clearly, the need for validation of their program in a larger trial is recognised but these preliminary results suggest that automated systems may be able to play a role not only in the detection of retinopathy but also in monitoring patient progression to a stage where referral to an ophthalmologist is warranted. Using a statistical approach to pattern recognition, Ege and colleagues (Table 24) report preliminary data on the ability of their automated system to analyse digital images taken as part of the routine monitoring of retinopathy in their clinic setting.37 Based on analysis of 134 images, the optic disc and fovea were correctly identified in 78.4% of cases. Microaneurysms, haemorrhages, exudates and cotton-wool spots were detected with sensitivities of 66, 89, 97 and 63%, respectively. Problems were identified in the ability to differentiate microaneurysms and haemorrhages and to distinguish cotton-wool spots from hard exudates. The authors recognise that their system is at an early stage of development and further refinement is required before it may provide a clinically useful tool. However, their presentation underlines the need for the development of reliable automated screening programmes if digital photographic services are to be more widely offered as a routine part of diabetic care. Detection of hard exudates Hard exudates have been noted to be a consistent feature as retinopathy progresses, indicating areas of retinal vascular leakage.38 As manual counting of exudates and calculation of their area is both time consuming and unsatisfactory, several authors have applied computer analysis to this problem area (Table 24).39–42 As exudates reflect more light than the background retina, they are represented by pixels of a higher grey level value in digital images, that is, they appear ‘whiter’ than the surrounding retina. This property allows their identification by a process of thresholding, where pixels equal to or greater than a chosen grey level are selected by the computer. After a series of shade correction programs on secondary digitised images, both Ward and colleagues39 and Phillips Appendix 1 74 and colleagues40,41 were able to apply this principle successfully in an effort to provide a quantitative method of detecting exudates. By monitoring both number and area, this could provide a method of determining response to treatment such as focal laser therapy. Although the above techniques do not distinguish between other objects with a similar grey threshold such as cotton-wool spots or drusen, Goldbaum and colleagues (Table 24) have attempted discrimination of hard exudates from these lesions on the basis of colour alone.42 A training set of images was employed, where groups of pixels from representative lesions were selected manually and used to calculate colour estimates for each lesion type. Using this software program, the authors found a limited ability to discriminate between these lesions, although hard exudates were most likely to be correctly identified (hard exudate, sensitivity 70% and specificity 95%; cotton-wool spot, sensitivity 70% and specificity 65%; drusen, sensitivity 50% and specificity 85%). Further refinement of the technique has been proposed with the addition of measures of size, shape, edge sharpness and texture to improve accuracy. Detection of venous beading The relative importance of individual features of diabetic retinopathy for predicting future progression of disease has been determined by the ETDRS.7 In addition to the severity of intra- retinal microvascular abnormalities and haemorrhages/microaneurysms, venous beading was noted to be a powerful predictor for the future development of proliferative retinopathy. Several authors have used a technique of Fourier analysis to determine variation in the diameter of blood vessels, developing a computer-assisted method of detection and quantification of venous beading.43–45 Kozousek and colleagues (Table 24) have compared their program with assessment of the original colour slides by professional photographic graders, using the Airlie House criteria for classification.44 An exact agreement with photographic graders was achieved in 76% of the 51 digitised slides suitable for analysis. For the remainder, grading differed by one level of severity only, with seven out of 12 cases being in the ‘questionable versus definite’ category. The calculated venous beading index could be used to differentiate between gradings of severity and to distinguish normal vessels from both definite and advanced beading. Although the regions of interest were manually identified for computer analysis in this initial study, the authors suggested that the technique could prove a valuable addition to future automated general retinopathy screening programmes. Detection of neovascularisation The ETDRS has indicated that patients who have progressed to the development of high-risk retinopathy should receive panretinal laser photocoagulation.8 They recommended that those with less severe retinopathy should have deferral of laser therapy and close monitoring of their progress. At this earlier stage, the adverse effects of laser radiation on visual field and central vision outweigh the benefit gained in reduced likelihood of future development of high-risk retinopathy. However, this strategy is dependent on the consistent recognition of advancing eye disease. Therefore, there is clearly a need to develop techniques that will allow the detection of neovascularisation at the earliest possible stage for appropriate timing of laser therapy. In his overview of fractal analysis applied to the retinal vasculature, Mainster describes the application of a fractal dimension (FD) to describe the properties of a branching vessel pattern.46 For example, a straight line is attributed an FD of 1, an area is given an FD of 2 and a structure filling three- dimensional space is given an FD of 3. Therefore, the FD of a branching structure lying flat on the retina will lie somewhere between 1 and 2. As it is an indirect measure of how completely a structure fills space, the relative value attributed to a branching structure will increase as its nature becomes more convoluted and it effectively covers a surface more completely, that is, the FD will become closer to 2. Using this principle, Daxer (Table 24) has performed computer analysis of digitised images to determine if fractal properties can confirm the presence of neovascularisation.47 In a comparison of images from 10 diabetic patients with proliferative retinopathy and 14 healthy controls, the FDs of the retinal vasculature were generally higher in the diabetic group. Using a threshold FD value of 1.8, Daxer was able to predict neovascularisation at the optic disc equivalent of the ETDRS grade 3 with a sensitivity of 90% and a specificity of 93%, using trained photographic graders as a reference standard. The original work required manual delineation of vessels patterns from colour slide photographs, which then underwent secondary digitisation for analysis. This process could be simplified significantly with direct acquisition of digital fundus images. Health Technology Assessment 2003; Vol. 7: No. 30 75 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. The technique used in this study was based on the evaluation of a low-angle 10° field centred on the optic disc and does not address the detection of peripheral new vessels elsewhere. However, as optic disc neovascularisation is recognised as the most high-risk form of proliferative retinopathy for subsequent retinal or vitreous haemorrhage, this novel technique remains promising for future detection of those most at need of urgent laser therapy. In subsequent research, Daxer has demonstrated that FD values could reflect the clinical evidence of development of new vessels in an adult with type 2 diabetes mellitus, and their regression following panretinal laser therapy.48 Although preliminary results were confined to the study of a single patient, it is possible that this technique may also provide a quantitative measure of the effectiveness of laser photocoagulation. Applications of digital photography in research In 1986, Brinchmann-Hansen and Engvold49 described a technique for the calculation of vessel diameter from digitised images, which has been applied as a research tool by several authors in the investigation of the pathogenesis of diabetic retinopathy (see Table 26).50–54 Together with the use of laser Doppler velocimetry, an indirect calculation of blood flow through the retinal circulation has been made possible. However, as the technique is dependent on the manual assessment of vessels’ edges which may induce observer error, Newsom and colleagues (Table 24) have developed a semi-automated technique of image analysis.55 Using digitised 35-mm retinal images, the vessel region of interest was identified by cursor and the computer calculated the average grey-level profile across the vessel from an average of 12 serial measurements. With the ability to perform automatic vessel edge detection, the computer-driven program showed a lower coefficient of variation than the observer-driven method (1.7–7.5% versus 6–34%). The standard deviation of variation from the mean was lower in the semi-automated analysis program at 4.2 �m (95% CI 3.2 to 6.0 �m) compared with 18.9 �m for the observer method (95% CI 14.6 to 27.1 �m), suggesting lower variability of the technique. Therefore, in summary, the authors propose that their computer-assisted analysis program will provide a useful research tool which is likely to be improved by the study of directly acquired digital images. Other aspects of evaluation – telemedicine The application of telemedicine to screening for retinopathy has already been adopted in areas where, for economic or geographical reasons, access to ophthalmic services has been limited. There is an on-going collaboration between local physicians and the Ophthalmology Faculty at the University of Texas, San Antonio, TX, USA to improve the delivery of eye care in the community.56 With funding from the South Texas/Border Health Initiative, a bus has been provided to allow a mobile ophthalmic evaluation service to operate. This includes digital fundus photography with direct transmission of images to a grading team at the San Antonio campus, who provide an immediate diagnosis for the examining doctor to discuss with the patient. Given the high prevalence of diabetes mellitus in the local Hispanic population, this strategy has the potential to revolutionise the delivery of diabetic eye screening although, at present, no formal evaluation of the success of this technique appears to have been carried out. However, even at this early stage of development, digital technology has been adopted as a way forward in diabetes care in the USA, with several ophthalmic imaging networks already in place to provide central image interpretation for patients in locations as diverse as California and Puerto Rico.57 In response to the St Vincent Declaration12 and the low rate of retinopathy screening being achieved with traditional programmes in Germany, Mann and colleagues have presented early work outlining a central grading centre support for images transmitted from diabetes centres.58 Although accepting that images are of lower resolution than 35-mm slides, this has not been deemed to obviate their use for population screening in the opinion of experienced ophthalmologists. At this stage, no data on a formal evaluation of the system are available. Within the UK, a similar strategy has been employed to allow rural GPs in remote areas access to specialist ophthalmic services by the direct transmission of fundus images of their diabetic population for specialist grading. In Powys, Wales, GPs are linked to Telemed, described as Europe’s most advanced medical telecommunications project.59 This system allows integration of patients, primary care physicians and hospital specialists with applications not only in retinopathy screening but also extending to cover dermatology referrals and video-linked consultations for physiotherapy assessment and asthma care. In practice, the convenience of these Appendix 1 76 H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 7 7 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 26 Fundus photography – applications in research Technique under review Population studied Study aim Research technique Results Authors’ conclusions Reference Secondary digitisation of conventional fundus photographs Digitisation of 30° red-free fundus photographs to determine vessel diameter in conjunction with laser Doppler velocimetry of the superior temporal vein to calculate flow rates 24 healthy volunteers and 76 patients, the latter attending retinopathy clinic (63 with type 1 DM, 13 with type 2 DM). Diabetic patients comprised 12 with no retinopathy; 27 with background retinopathy; 13 with pre- proliferative and 12 with proliferative retinopathy Determination of rate of retinal blood flow in healthy and diabetic fundi Technique of vessel diameter assessment validated elsewhere by Brinchmann-Hansen and Engvold.49 Average of three readings of diameter from three photographs, corrected for patient’s refraction, gives coefficient of variation of 0.26% Hyperperfusion of retinal circulation noted in all patients compared with normal; in comparison with patients with no retinopathy, retina flow was 33.2% higher with background, 69.4% with pre-proliferative and 50.1% with proliferative retinopathy. Panretinal photocoagulation led to a significant reduction in flow in comparison with all other groups (4.48 compared with 9.52 µl/min in non-diabetic controls) Flow rates calculated to be independent of age, sex, BP, blood glucose concentration, HbA1c, intraocular pressure and type and duration of diabetes. The authors propose that the increased shear stress induced by hyperperfusion is an important factor in the pathogenesis of DR Patel, 199250 Digitisation of 30° monochromatic fundus photographs to determine vessel diameter in conjunction with laser Doppler velocimetry to calculate flow rates. Photography taken in mid- diastole to eliminate effect of pulsatility from cardiac cycle 10 normal volunteers; 12 type 1 DM – 10 studies performed with blood glucose maintained >10 mmol/l and 10 studies with blood glucose <10 mmol/l (i.e. 8 patients studied twice). All diabetics had mild NPDR. Elevation of MAP above baseline was achieved using a tyramine infusion To investigate the effect of hypertension on retinal vascular autoregulation in diabetic and non- diabetic subjects via estimation of blood flow velocity Techniques validated in earlier work by Brinchmann-Hansen and Engvold49 In non-diabetics, retinal blood flow showed significant increase only at MAP 40% above baseline (p = 0.012). In contrast, increased flow in diabetics with low glucose significant at both 30% and 40% elevation above MAP (p <0.05). For diabetics with high blood glucose, only 15% elevation of MAP required to give significant increase in flow (p <0.03) By calculation of the coefficient of autoregulation, significant impairment of response to increased perfusion pressure was evident in all diabetics, most marked in those with elevated blood glucose >10 mmol/l. The authors conclude that the resultant increase in flow rate will exacerbate endothelial damage, providing a potential mechanism for accelerating the progression of DR Rassam, 199551 continued Appendix 1 7 8 TABLE 26 Fundus photography – applications in research (cont’d) Technique under review Population studied Study aim Research technique Results Authors’ conclusions Reference Secondary digitisation of 60° red-free negatives centred on the fovea Retrospective analysis of images from 45 diabetic children with type 1 DM aged 9.4–18.4 years – images from 74 eyes studied To evaluate the effect of long-term glucose control on retinal vessel diameter in children and young adults and to determine whether this may be a predictor of future retinopathy Technique validated by earlier work by Brinchmann-Hansen and Engvold.49 Manual calculation of vessel width directly on monitor using mouse-controlled cursor (micrometry) Coefficient of variation for repeated measurements 1.8–2.0% for veins and 1.8–2.1% for arteries using densitometry, cf. 3.0–3.1 and 3.6–4.1% respectively using micrometry. Linear regression analysis showed association between HbA1c value on date of second photograph (p = 0.049) and average HbA1c in year prior to photography (p = 0.051) with average increase in venous calibre. No association with changes in arterial calibre noted, although no correction possible to compensate for potential effect of perfusion pressure from cardiac cycle on overall appearance. No correlation found with sex, age, duration or age of onset of diabetes, stage of puberty or BP Despite limitations of a retrospective study using small number of patients, the authors conclude that the observed increase in venous diameter in association with prolonged hyperglycaemia supports previous research into the hyperperfusion model of DR. Clinically visible venous congestion may herald the development of capillary changes in young diabetics. Improvements in digital image analysis may allow presented technique to gain application in clinical use for rapid evaluation of vessel diameter Falck, 199552 continued H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 7 9 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 26 Fundus photography – applications in research (cont’d) Technique under review Population studied Study aim Research technique Results Authors’ conclusions Reference Digitisation of 30° monochromatic fundus photographs to determine vessel diameter in conjunction with laser Doppler velocimetry to calculate flow rates 4 groups of 15 patients studied: normotensive non- diabetics; normotensive diabetics (12 type 1 and 3 type 2); hypertensive non- diabetics; hypertensive diabetics (5 type 1; 10 type 2). Diabetics were studied under conditions of relative normoglycaemia (<10 mmol/l) and hyperglycaemia (>15 mmol/l); hypertensive patients were studied before and after control achieved. Diabetics had mild NPDR To determine whether DM can alter the autoregulatory vasoconstrictive response to 60% oxygen breathing under hypertensive and hyperglycaemic conditions via measurement of retinal blood flow Technique validated in previous work by Brinchmann-Hansen and Engvold49 Oxygen reactivity reduced in hypertensive and normotensive diabetics in hyperglycaemic conditions compared with normotensive volunteers (p < 0.005). Also reduced in controlled and uncontrolled hypertensive diabetics at hyperglycaemia compared with respective normoglycaemic group. No statistical difference between hyperglycaemic and normoglycaemic normotensive diabetics. For non-diabetics, oxygen reactivity reduced in hypertensive compared with normotensive volunteers (for controlled hypertension, p = 0.026; for uncontrolled hypertension, p < 0.059) The authors conclude that retinal vascular reactivity to hyperoxygenation stimuli is significantly reduced in the hypertensive diabetic patient, regardless of blood glucose control. However, the deleterious effect of hyperglycaemia is thought to outweigh that of hypertension. The need for improved glycaemic control and control of hypertension to prevent accelerated progression of retinopathy is discussed Patel, 199453 continued Appendix 1 8 0 TABLE 26 Fundus photography – applications in research (cont’d) Technique under review Population studied Study aim Research technique Results Authors’ conclusions Reference BP, blood pressure; MAP, mean arterial pressure. Analysis of 30° red-free fundus photographs using a Context Vison GOP-302 digital image analysis system; laser Doppler velocimetry used to allow estimation of retinal blood flow rate 10 diabetic patients, both type 1 and 2, recruited from the diabetic retinopathy clinic with NDPR (1 patient ETDRS grade 20; 4 grade 30; 6 grade 41); 16 non-diabetic controls To determine the effect of alcohol on retinal autoregulation in response to the vasoconstrictive challenge of 60% oxygen inhalation in diabetic and non- diabetic subjects Technique validated in previous work by Brinchmann-Hansen and Engvold49 Both groups responded significantly to oxygen challenge before and after dosing with ethanol 0.5 g/kg, which was thought adequate to reflect social blood alcohol levels – reduced maximum red cell velocity, vessel diameter and retinal blood flow were demonstrated in each group in comparison with baseline measurements. No statistical difference was noted between diabetic subjects and non- diabetic subjects in their response In the normotensive normoglycaemic diabetic patients with mild non- proliferative retinopathy under review, the autoregulatory response to hyperoxygenation was comparable to that of non- diabetic subjects. It remained unaffected by acute ethanol consumption in both groups, leading the author to suggest that the retinal circulation may be superior over other circulatory systems in its ability to autoregulate in the presence of ethanol. However, the limited number of patients under review and absence of more advanced grades of retinopathy suggest caution in interpretation of these findings Dhasmana, 199454 techniques for the patient are readily apparent although as yet no formal study of the efficacy or effectiveness of digital screening in diabetic retinopathy against formal clinical examination has been identified in this setting. Digital angiography Introduction The use of a fluorescent emission from circulating dye to highlight features of the human retinal vasculature was initially described by Novotny and Alvis in 1961.60 However, the technique was limited in practice owing to the need to capture images on ciné film, requiring subsequent development and projection for viewing. An advance on conventionally acquired serial photographic images was the introduction of the low-light TV camera in 1986 by Korber, which allowed results to be viewed immediately by the clinician.61 The data obtained could then be stored by use of a conventional video cassette recorder. The recording of real-time data allowed the author to develop a quantitative method of assessment of retinal blood flow by the examination of individual still pictures from a sequence using computer-controlled image analysis. This method of estimating arteriovenous passage time, arm-to-retina time and mean plasma flow velocity was subsequently adopted widely.62,63 Using high-quality video recordings, which were then slowed to one-quarter of real-time speed for analysis courtesy of Scotland Yard, Jacobs and colleagues (see Table 29) were able to study the vascular origins of optic disc new vessels.64 In a study involving 10 people, including seven diabetic patients with proliferative retinopathy, variable origin of new vessel filling was identified. Within the diabetic sub-group, three patients derived flow from the retinal venous circulation, three patients demonstrated filling of new vessels simultaneously with the central retinal artery and one patient with diabetes showed filling that appeared to derive from two cilioretinal arteries, suggesting a choroidal origin. Although the authors were unable to determine the clinical significance of their findings, their work illustrated the superiority of videofluoroscopy with an image capture rate of 25 frames per second over conventional fluorescein photography in dynamic investigation of neovascularisation. The later development of PC-based video frame grabbers allowed conversion of analogue images to a digital format, permitting computerised image analysis. In 1989, OIS and Topcon marketed the first commercially available system capable of direct acquisition of analogue-to-digital images within the camera control unit itself, designed specifically for use with fluorescein angiography. Studies evaluating fluorescein angiography are summarised in Table 27 and their study methodologies and results evaluated in Table 28. Indirect digitisation of angiograms An early comparative study to demonstrate that the conversion to digital format did not cause degradation of conventional fluorescein angiograms which would adversely affect clinical information was carried out by Seeley and colleagues in 1989 (Table 27).65 Sixty-eight images were selected, giving a representative selection of common retinal pathologies (29 images included evidence of diabetic retinopathy) and an equal number of normal images. After conversion to a 512 × 480 pixel digital format, images were presented on a cathode-ray tube display. Clinical grading was compared with that obtained by examination of back-illuminated photographic transparencies, without projection. Using ROC curves for analysis, no appreciable difference between the information derived from the two sets of images was identified. The inter-observer differences in performance were attributed to varying clinical experience. Therefore, from their preliminary study, the authors supported the concept of development of a digitised angiogram facility, recognising the future potential for easier image storage and retrieval and image enhancement. Other authors have applied indirect digitisation techniques to both conventionally acquired photographic angiogram sequences and videoangiograms in order to provide a more quantitative assessment of background diabetic retinopathy, as summarised in Table 27. Digital image analysis Automated assessment of microaneurysms In response to the difficulty in achieving reliability and reproducibility in the manual counting of microaneurysms, Phillips and colleagues (Table 27) applied computerised image analysis to the assessment of digitised angiogram negatives.40 Microaneurysms are visible as round hyperfluorescent objects, with a higher grey level than the background retina, providing a method of differentiation. However, as they are similar in threshold to neighbouring structures such as blood Health Technology Assessment 2003; Vol. 7: No. 30 81 © Queen’s Printer and Controller of HMSO 2003. All rights reserved. Appendix 1 8 2 TABLE 27 Evaluation of digital fluorescein angiography Technique Population Study aim Gold standard Results Authors’ conclusions Reference Methodology under review studied or comparative and results method grading Comparison of digitised angiograms with original 35-mm images Automated assessment of microaneurysms 68 conventional FA images selected for digitisation via video format camera and frame grab board to give 512 × 480 pixel images. Images viewed on CRT display 29 out of 36 patients under review had DR To evaluate whether conversion of conventional photographic angiograms to digital images leads to loss of resolution causing failure in diagnosis of DR. Both sets of images assessed by 3 experts in retinal diagnosis Gold standard Opinion of expert in retinal angiography responsible for selection of images Using ROC curve analysis to evaluate diagnosis and certainty responses, no appreciable difference between the two types of images was noted. Observer 1, AUC for CRT 0.80 and for slides 0.79; observer 2, 0.87 and 0.86, respectively; observer 3, 0.91 and 0.91, respectively. Inter-observer differences attributed to past experience but did not reflect on methods under review The chosen digital technique does not cause loss of diagnostic information. The perceived advantages of the digital system are ease of image archiving and retrieval; immediate review of examinations, and the potential for transmission of images to other centres while retaining the original on file Seeley, 198965 Methodology Grade V Results Grade IV Indirect digitisation of 30° or 45° macular views from FA negatives performed by Panasonic WV-CD 20 CCD video camera, Data Translation DT-2861 frame grabber with IBM-compatible computer Macular region of eight angiograms from diabetic patients assessed Automated assessment of MAs Gold standard Manual counting of MAs by a single observer using digitised angiographic frames under evaluation MAs: sensitivity 82–93% in comparison with manual counting, with specificity of 93%. Coefficient of variation 7–14.5% The authors conclude that their results for detection of MAs compare favourably with previously published work by other authors using automated methods Phillips, 199140 Methodology Grade IV Results Grade IV continued H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 8 3 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 27 Evaluation of digital fluorescein angiography (cont’d) Technique Population Study aim Gold standard Results Authors’ conclusions Reference Methodology under review studied or comparative and results method grading Fluorescein angiograms converted to digitised 512 × 512 array using a monochrome CCD camera and Data Translation DT-2861 frame grabber linked to an IBM- compatible PC Development of an automated method for the detection and quantification of MAs Assessed against results obtained by manual counting by team of 5 experienced ophthalmologists working independently, and grading both analogue and digitised images Gold standard Correct location of the lesions on analogue prints established by two of the authors Using FROC curves to evaluate performance, there was no difference between computer and clinician on the analysis of digitised images. Significantly higher true- positive rate (sensitivity) was achieved by the clinicians using analogue images for any given false-positive rate The authors suggest that direct on-line acquisition of digital images may reduce the effect of image degradation inherent in the secondary digitisation technique described. Increasing image resolution to the currently available 1024 × 1024 array will also improve sensitivity. With refinement, the automated technique may prove valuable in the detection and monitoring of DR Spencer, 199267 Methodology Grade IV Results Grade IV Fluorescein angiogram negatives were converted into digital format using a Kodak Megaplus CCD camera and Series-151 image- processing system, achieving images with a resolution of 1024 × 1024 pixels 13 images from diabetic patients reviewed, with 7 displaying evidence of MAs An automated method for identification and quantification of MAs from digital images compared with manual assessment of MA counts from high-quality analogue prints of the original angiogram negatives made by 5 opthalmologists (2 consultants and 3 registrars) Gold standard Authors assessment of MA counts from the angiogram prints Analysis performed using FROC curves, revealing that the computer’s performance was satisfactory in comparison with that demonstrated by the clinicians. A maximum sensitivity of 82% was reached by the automated system, imposed by limitations of the present program in recognising lesions leaking fluorescein or conglomerations of MAs, calculated as too large for inclusion Digital image processing can offer an objective and easily repeatable quantification of retinal MAs. The authors feel that the method has significant advantages over previously described techniques although no direct comparisons have been made Spencer, 199668 Methodology Grade V Results Grade IV continued Appendix 1 8 4 TABLE 27 Evaluation of digital fluorescein angiography (cont’d) Technique Population Study aim Gold standard Results Authors’ conclusions Reference Methodology under review studied or comparative and results method grading Indirect digitisation of standard 35° fluorescein angiogram negatives Patients with evidence of DR (severity of retinopathy and number of patients not stated) Development of a fully automated method of MA detection – comparison made with manual assessment of computerised images by clinicians Gold standard Manual identification of MAs from test images by an experienced study ophthalmologist and medical physicist The automated detector achieved 82% sensitivity with 5.7 false positives per image. Data presented as a FROC curve suggest the automated method is comparable in practice to results from clinicians The authors outline a fully automated strategy, with image registration by computer allowing analysis of serial images from an individual. The system can be adjusted to allow greater sensitivity at the expense of specificity or vice versa depending on the characteristics of the population under study. In comparison with clinicians, the system appears robust enough for use in clinical practice Cree, 199769 Methodology Grade IV Results Grade III Conventional angiogram negatives digitised to give 25 overlapping fields for analysis, each comprising 256 × 256 pixels 25 angiograms chosen randomly from diabetic patients participating in a prospective trial of the effect of antiplatelet agents in micro- angiopathy (DAMAD study). Type 1 and 2 represented, with at least background retinopathy and ≥ 5 MAs in the posterior pole of the eye Automated detection of MAs from fluorescein angiograms Comparative study 1. Manual counting of MAs from projected negatives by ophthalmologists and a trained technician. Each angiogram assessed twice. 2. A second validation was done by technician marking digital image on screen directly with a light-pen to overcome influence of magnification on manual and automated methods When computer assessment validated by the technician using method 2, the authors attribute the automated method with 70% sensitivity and a predictive value of 86%. The technique did not allow calculation of specificity The authors conclude that the method of automated detection under review provides a method of MA detection comparable to that achieved by a trained technician, both in accuracy and performance time (30–40 minutes per angiogram) Baudoin, 198470 Methodology Grade II Results Grade II continued H ealth Technology Assessm ent 2003; Vol. 7: N o. 30 8 5 © Q ueen’s Printer and C ontroller of H M SO 2003. A ll rights reserved. TABLE 27 Evaluation of digital fluorescein angiography (cont’d) Technique Population Study aim Gold standard Results Authors’ conclusions Reference Methodology under review studied or comparative and results method grading Automated assessment of maculopathy Indirect digitisation of 30° or 45° macular views from fluorescein angiogram negatives performed by Panasonic WV-CD 20 CCD video camera, Data Translation DT- 2861 frame grabber with IBM-compatible computer 10 angiograms analysed, of which 3 demonstrated diabetic retinopathy (1 central serous retinopathy; 1 age-related macular degeneration; 1 branch retinal vein occlusion; 4 normal controls). Comparison made of the intensity of fluorescence in early and late images to give an indication of the degree of vessel leakage Automated method of performing an assessment of macular leakage – if this is present, the calculation of the area of an angiogram showing persistence or increase in fluorescence levels is under evaluation as an indirect method of quantitative assessment Comparative study No definitive gold standard available. Threshold gradient value calculated using normal angiograms to minimise false-positive results. Assessments made of repeatability and robustness after calibration against test angiograms. Repeatability – same images digitised and assessed on 5 consecutive occasions. Robustness improved by analysing 3 early frames, counting only positive pixels appearing in 2 out of 3 images, against those from a single late image High degree of repeatability and robustness for single and multiple areas of leakage >1/5 of disc area (coefficient of variation 6%) and for single areas <1/5 of disc area. System performed less well with multiple small areas with total area <1/5 of disc area (coefficient of variation 27%) The authors propose that their method provides a simple method of quantifying macular leakage which is superior to manual assessment (although results of direct comparison not quoted). This has implications for both diagnosis and monitoring response to treatment. At this stage in development, potentially misleading lesions such as drusen need to be excluded by prior clinical examination Phillips, 199140,66 Methodology Grade III Results Grade III continued Appendix 1 8 6 TABLE 27 Evaluation of digital fluorescein angiography (cont’d) Technique Population Study aim Gold standard Results Authors’ conclusions Reference Methodology under review studied or comparative and results method grading Assessment of perifoveal and macular microcirculation 30° fluorescein angiograms centred on the macula obtained – digitised to 512 × 512 pixel resolution. Manual identification of the FAZ performed using a cursor and area and longest diameter calculated. Vitreous fluorophotometry also performed in diabetics as an indirect assessment of dye leakage Prospective study of 18 diabetic patients with diabetes duration of ≥ 5 years and retinopathy