key: cord-0944598-5eybcv5b
authors: Calcagno, Simone; BiondiZoccai, Giuseppe; Stankovic, Tatjana; Szabo, Erzsebet; Szabo, Aniko Berta; Kecskes, Istvan
title: Novel tech throws knock-out punch to ECG improving GP referral decisions to cardiology
date: 2022-02-21
journal: Open Heart
DOI: 10.1136/openhrt-2021-001852
sha: d48ce807484234659eb86db86b67395777d96d92
doc_id: 944598
cord_uid: 5eybcv5b

PURPOSE: In a comparator study, designed with assistance from the Food and Drug Administration, a State-of-the-Art (SOTA) ECG device augmented with automated analysis, the comparator, was compared with a breakthrough technology, Cardio-HART (CHART). METHODS: The referral decision defined by physician reading biosignal-based ECG or CHART report were compared for 550 patients, where its performance is calculated against the ground truth referral decision. The ground truth was established by cardiologist consensus based on all the available measurements and findings including echocardiography (ECHO). RESULTS: The results confirmed that CHART analysis was far more effective than ECG only analysis: CHART reduced false negative rates 15.8% and false positive (FP) rates by 5%, when compared with SOTA ECG devices. General physicians (GP’s) using CHART saw their positive diagnosis rate significantly increased, from ~10% to ~26% (260% increase), and the uncertainty rate significantly decreased, from ~31% to ~1.9% (94% decrease). For cardiology, the study showed that in 98% of the cases, the CHART report was found to be a good indicator as to what kind of heart problems can be expected (the ‘start-point’) in the ECHO examination. CONCLUSIONS: The study revealed that GP use of CHART resulted in more accurate referrals for cardiology, resulting in fewer true negative or FP—healthy or mildly abnormal patients not in need of ECHO confirmation. The indirect benefit is the reduction in wait-times and in unnecessary and costly testing in secondary care. Moreover, when used as a start-point, CHART can shorten the echocardiograph examination time.

In the last decade, cardiovascular disease (CVD) has become one of the factors which threaten human health 1 and is the main cause of death in the world. [2] [3] [4] According to a report presented by the Global Burden of Disease Study, these diseases were responsible for 31.8% of all deaths worldwide. 5 Representing a high impact in morbidity and cost to society, there is an urgent need to detect early signs of CVD including heart failure (HF) one of the most difficult to diagnose. 6 In one multiperspective study 'the difficulties with diagnosis, unclear illness perception, and management disparity' were identified as critical aspects that may impact management of CVD which put in evidence the problem of the effective diagnosis of such life-threatening diseases. 7 Misdiagnosis or delay in diagnosis of CVD is correlated with lengthened time to treatment, increased hospital stays, including high percentage of readmission, and a higher death rate which brings us to the point that 'the early diagnosis is imperative'. 8 Primary care is the initial contact point 9 with the patient where CVD is first presented

What is already known about this subject? ► In primary healthcare, there is a need for the early detection of cardiovascular disease including heart failure, one of the most difficult to diagnose. However, the standard rule-based ECG interpretation provides limited diagnostic evidence for common heart problems.

What does this study add?

► The study indicates that Cardio-HART (CHART) assists the medical professional to determine if the symptoms may be related to cardiac abnormalities or establish some new risks that can lead to cardiac disease if not treated properly. This is especially valuable for patients with recurrent symptoms, which can be difficult to catch with the traditional ECG, or during the regular medical examination.

How might this impact on clinical practice?

► As a result of more accurate and effective diagnostic capabilities, general physicians using CHART were more confident and certain of their decisions both in referrals to secondary care and, just as importantly, knowing when a patient can safely remain in primary care. ► Consequently, patients are more likely to get timely access to care instead of being subjected to further testing and clinical visits to determine their cardiac status and prognosis.

and suspected but all too often can go unnoticed or underdiagnosed, sometimes for many months. Underdiagnosis is not uncommon, as studies have found high prevalence rates of CVD in high-risk populations in primary care, such as elders short of breath, type 2 diabetes or chronic obstructive pulmonary diseases (COPD), make the process of identification of CVD all the more difficult. 10 A review of malpractice claims in the USA revealed that failure to diagnose, which included failure, delay and wrong diagnosis, was responsible for over 41% of all claims against general physicians (GPs) and 48% against nurse practitioners. 11 Further review of claims unmasked a possible common factor, an 'inconclusive ECG'. Compouding the situation, every day, a large number of people with chest pain refer to heart centres with half of them without a real cardiac problem. The economic burden of CVD on healthcare systems is significant. Hence, correct early diagnosis and appropriate treatment of these patients on initial presentation to their primary care clinic makes it a challenge not only for physicians and hospitals but also for governments, health-insurance companies and health maintenance organisations. 12 Focusing on this goal, the demand for innovative diagnostic methods for CVD is continuously developing, particularly challenging is the diagnosis of HF, especially in primary care. Therefore, the current diagnostic approach in adults that relies mainly on the use of ECG as the primary diagnostic tool in primary care could be further refined and improved.

Addressing this last objective and to reduce the economic burden from false positive (FP) and false negative (FN) patients, a clinical study was conducted using a new breakthrough medical device that has shown evidence related to improving diagnostic effectiveness in clinical situations, including primary care. Improved diagnostics are achieved through the clever use of novel biosignals processed by Artificial Intelligence techniques to assist GPs in the diagnosis of CVD. It is a direct substitute for ECG devices in the standard of care.

In each of the study locations, the State-of-the-Art (SOTA) 12-Lead Standard ECG devices that was in daily use was used as the comparator. As Cardio-HART (CHART) includes automated diagnostic processing, the local ECG devices lacking such functionality, were also augmented with the leading automated diagnostic algorithms to assist GPs in interpretation of ECG signals. The resulting output is in the form of a diagnostic report.

CHART uses novel technology in the analysis of the heart biosignals. The CHART system differs from the SOTA ECG devices in that it combines the functionality of three heart devices into one, including ECG, Echocardiography (ECHO) and Auscultation. As a result, it can detect 143 heart parameters and 47 heart findings associated with ~95% of all common heart diseases prevalent in clinical situations. The resulting output is in the form of an easy-to-grasp report for use by clinicians.

This study demonstrates GP diagnostic effectiveness through a direct clinical comparison between the comparator ECG devices, representing the current standard of care and the CHART device, a breakthrough technology.

The study was conducted as a multicentre clinical study to confirm the functional and operational aspects between two medical devices competing head-to-head against each other, CHART versus SOTA ECG, in clinical practice to determine which provided the most beneficial clinical decision support for GPs and best outcomes for patients. The results for each patient were verified and validated in accordance with its intended performance by a separate team of cardiologists to ensure an accurate ground truth.

Each patient willingly gave their consent, in writing. Both devices are non-invasive and were considered low risk.

The goal was to determine the utility of CHART and its usability for its intended use, in a clinical environment, by its intended users, in a study population representative of the target population, in direct comparison to a SOTA ECG device used within the Standard of Care for Primary Care. The study was designed to collect the data to confirm the safety and effectiveness of the CHART system when compared with ECG. A total of 550 patients were recruited into the CUUS, and their clinical results evaluated to measure the diagnostic and decision support capability of the SOTA ECG report when directly compared with the CHART report.

The results confirm that in many ways CHART analysis is far more effective than ECG only analysis. FN rates are significantly decreased (CHART produced a 15.8% decrease in FN) and FP rates decreased by 5% (FP in the patient referral decision by GP as compared with ECGonly based decisions). The data collected would confirm the hypothesis that CHART analysis is more effective than ECG only analysis in assisting the general practitioner (GP) in determining their referral decision (Send/Don't Send) and the basis for it.

The setup for the study was to assign two independent GPs to each patient. Each GP would independently examine the same patient then render their diagnosis based on the device they were assigned in the protocol. One GP would be using CHART and the other GP would be using their local SOTA ECG device augmented with automated algorithmic assessment. Each patient would first be tested using the ECG device then the CHART device. Two diagnostic reports were then produced, one for each protocol. Each GP, based on their protocol device, was to diagnose the patient's cardiac status and determine a course of action. In this way the patient is the control, being the same patient for both GPs. At the mid-point of the study, the GPs were switched to the other device/protocol.

The first decision by the GP's was to render a diagnosis based on their understanding of the patient's cardiac status. From this they were to determine whether the patient was to be referred to cardiology or not. This was the 'Send'/'Don't Send' decision. If Send, they were to indicate with what priority, whether urgent, immediate or routine. If Don't Send, they were to indicate whether patient was normal, or be indicated for a control visit, at 3, 6 or 12 months.

See online supplemental file 1 for more details.

The goal was to directly compare two cardiac diagnostic systems to each other to characterise their effectiveness and utility in providing diagnostic assistance to GPs. The referral decision was used to measuring the effectiveness of the decision, see table 1. Referral decisions were made by two different GP's, based on either ECG-only or CHART analysis, for the same patient and evaluated through statistical analysis of comparison to establish the certainty of the decisions and understanding of the diagnosis.

The study involved both male and female patients, distributed equally, with at least three risk factors for heart disease, 13 attending a primary care clinic for health reasons, including annual physical examination, that is, a typical primary care setting. The minimum target sample size was 500 patients of which at least 15% should be considered as healthy patients (no risk factors). The age distribution is: 61 patients between 20 and 40 years, 130 between 41 and 55 years, 156 between 56 and 65 years, 148 between 66 and 75 years and 52 above 76 years. The obesity distribution is: 6.2% underweight, 35.8% normal, 34.7% overweight and 23.3% obese.

The reference referral decision defined by cardiologyconsensus was based on three factors, risk, ECG and an ECHO examination (transthoracic echocardiogram). Each of the ground truth decisions was established by a minimum of three independent cardiologists. The average consensus performance of the binary referral decision (cardiologist's agreement for Send/Don't) was 87.5% and for priority/watch detailed decision was 60.6%. These numbers were expected and comparable with the known measurement variability of ECHO assessment 14 15 or ECG. 16 The overall reference send rate was 43.0%, which is an important cardiac condition property of the included patient population. This referral to cardiologist care rate is similar to that of other studies, confirmed by the literature, with a typical range of 30%-50%. [17] [18] [19] [20] Referral decision results The GP decision referral performance was based on either the ECG or CHART reports, presented in table 2.

In the statistical calculation for the reference decision was the consensus-based ground truth.

The significance is derived from confidence intervals: if the observed value is higher than the upper confidence value, then it considered a significant increase. Less significant increase means there is an increase in the observed value, but it is not higher than the upper confidence value. Figure 1 illustrates the performance evaluation and comparison of GP referral decision between ECG-based and CHART-based answers. CHART-based decisions showed higher accuracy compared with ECG-based decisions, especially its sensitivity which CHART showed a big increase. CHART also decreased the FP rate from 21% to 16%, and FN rate from 50% to 34%. The significant increase of positive predictive value, negative predictive value, kappa, area under the curve and LR+ and decrease of LR− confirms the benefit of using the CHART report over the ECG only report.

The positive rate (PR) or send rate by GPs is similar between ECG and CHART reports (PR%=33% by ECG and PR%=36% by CHART), however CHART's send rate is closer to the ground truth (PREV%=43%). This means that the send rate of the GP referral decision using CHART met with the cardiologist expectations as confirmed by the ground truth. This includes a reduction in FP but also an increase in TP from FNs that were missed in ECG-based decisions.

Interestingly, the overall number of patients referred didn't really change, as normal patients are replaced with patients with potentially abnormal cardiac function.

The CHART assessment was performed twice, first in primary care (CHART#1 and ECG reports), and again in cardiology care (CHART#2 report), prior to the ECHO examination. The typical delay between the two tests is 3-15 days.

The relevant CVD diagnosis was compared between four cases: (1) GP diagnoses on ECG report, (2) GP diagnosis on CHART#1 report, (3) the over-reading cardiologist (ORC) diagnosis that included ECG, then CHART#1, and (4) the referral cardiologist (RC) diagnoses that included ECG, CHART#1 and CHART#2 report. Table 3 lists the diagnostic findings based on either ECG or CHART reports. Figure 2 summarises the three types of diagnosis, where the three colours represent the rate of 'absence', 'presence' and 'not-sure' answers. The GP's positive (number of abnormal or presence) diagnosis rate shows a significant increase, from 9.8%-23.6% to 28.8% by CHART compared with ECG based assessment. The uncertainty rate measured by the number of 'not sure' answersshow a significant decreased, from 30.9-1.7% to 2.1%.

The RC's uncertainty rate on CHART#2 is small, but significantly, is similar to that of the GP on CHART#1. The RC's positive diagnosis rate on CHART#2 is (23.6%) which is in line to GP's positive diagnosis rate on CHART#1 (28.8%), which is significantly more in agreement when compared with ECG at (9.8%).

A notable outcome in table 3, between arrhythmia and ischaemia categories is that being typically ECG-based Heart failure and cardiomyopathies findings, they showed similar results between ECG and CHART, but with CHART showing much less uncertainty when used. In the structural abnormality category the results are mixed, when using ECG there was high uncertainty, but when using CHART there was high certainty (decreased uncertainty). These results are not surprising as ECG has low sensitivity for these conditions, that are typically confirmed with ECHO. The rest of the diagnostic points (valve disease, murmurs and systolic time interval findings) are only supported by CHART, which dispels all the uncertainties observed with use of ECG.

In the agreement statistics, the GP to ORC and GP to the RC decision were compared on the same patient, based on their ECG and CHART reports, see table 4. In the case of CHART, the decision agreement between GP and ORC is significantly higher compared with ECG ( figure 3A ). This suggest that when CHART is available to GPs, the over-reading role might be less necessary. Notably also, the agreement comparison between GP and ORC is also similar to that of GP and RC (figure 3B). In clinical practice, the consequences of this means that RC's will get fewer Normal patients, patient that don't need echocardiology.

Comparison between ORC and RC ORC and RC were asked to compare ECG and CHART reports. The following results were observed based on 522 answers set on 8 questions, see table 5. The results for ORC and RC are very similar in all the questions, they confirm each other.

The benefit of CHART is twofold, on the one hand, it provides better cardiac status diagnostics, and on the other hand it provides more reliability due to its wider range of parameters and findings. As such, CHART increases the certainty (reduces uncertainty) of decisions aided by its findings.

RC were asked to evaluate the CHART report to show how useful it could be in providing an appropriate start point to an ECHO examination, thereby avoiding a 'cold start'. The following results were observed based on 391 answers set on 3 questions, see table 5. CHART-findings based on ECHO-findings are a good indicator as to what kind of heart problems, comorbidities, that can be expected (the 'start point') during the ECHO examination. Aware of the big picture and range of comorbidities can help shorten examination times and reduce the probability of undiagnosed or misdiagnosed conditions.

GPs in Primary Care can be meaningful contributors to diagnosing heart disease when patients first present, enabling earlier detection of (CVD) onset and reduction in unnecessary and costly referrals to Cardiology. This will have significant benefits in terms of reduced waittime, and more timely access to treatment.

The value of CHART stems from its HART/ECHOfindings as they bring to primary care morphological and functional characteristics to heart disease able to identify cardiac conditions that are typically only diagnosed by ECHO. This effectively bridges the technological diagnostic gap between use of ECG in primary care and use of ECG and ECHO in secondary care. Part of the GP's tasks was to indicate a priority when a patient was referred, 'send', to cardiology. A beneficial consequence of the use of ECHO-findings by CHART, is that they serve to establish a starting point for prioritisation of patients based on medical justification based on disease severity, urgency. Typically, referrals to secondary care are on a first come first serve basis, with patients in greater need provided a higher priority only when symptoms have become overt or acute. Such is the case of acute HF, where over 79% of all cases are typically diagnosed in Hospital. Yet, in the 6 months prior to their hospital diagnosis, 41% of these patients had visited Primary Care having at least one of three key HF symptoms that should have triggered an earlier assessment. 21 The surprise consistency in diagnostic accuracy between the GP and the ORC (positive per cent agreement, PPA=67%, negative per cent agreement, NPA=81%), compared with ECG-based assessment (PPA=52%, NPA=77%) suggests that the over-reading role might be reconsidered when CHART is present. The pandemic has shown that change is needed given the budget pressures placed on healthcare, further study is warranted.

In the study, RCs performed a second CHART examination on initial patient presentation to cardiology. This allowed cardiologists to determine whether ECHO confirmation as indicated by the GP was in fact warranted. Replacing the standard ECG with CHART on patient presentation to cardiology would have several benefits. First, it would help identify patients that don't need ECHO, currently a massive contributor to increased wait-times and backlogs. Second, it would help avoid a 'cold-start', as it is not uncommon for patients to present to cardiology with insufficient referral information and so help focus the cardiologist on a specific region of interest. For example, unless the referring GP specifically suggests HF in the referral, Cardiologists do not Heart failure and cardiomyopathies routinely consider HF during the Echo examination. As CHART can provide an understanding of both left heart side abnormalities (left ventricular hypertrophy, left atrial enlargement, mitral regurgitation and wall motion abnormality) and right heart side abnormalities (right ventricular enlargement, right atrial enlargement, tricuspid rergurgitation and pulmonary hypertension), a first use on initial patient presentation to cardiology can help identify areas of concern, including HF. Accordingly, CHART is not a substitute for ECHO, nor is it intended as a surrogate to cardiologists but rather as a valid support for GPs in clinical settings. Yet when ECHO is not yet available, it can assist GP's with determining patient pathway with more confidence and immediacy.

The right side of the heart is less represented by the included ECHO-parameters and even less so by ECG, as such their place in the statistical results is limited. Nonetheless, right-side heart disease and its structural, functional abnormalities and related pulmonary disease comorbidities (COPD, COVID-19) have attracted growing attention. Right-heart side study through a biosignal approach shows strong potential. Further research is warranted to investigate in more detail the relationship between HF categories and right-side HF through the window of biosignals, risk factors, detectable abnormalities, COPD and other comorbidities and symptoms.

Originally designed with extensive inputs and specific design criteria from the Food and Drug Administration, 22 23 the results of the CUUS study showed that the beneficial implications of this novel tech in primary care are significant, both from a clinical and economic perspective. The study results shows that CHART has the potential to enable effective widespread screening of patients for the early detection of CVD onset and resolve many 'inconclusive ECG' results thereby reducing both FN and FP in referral decisions.

Twitter Giuseppe BiondiZoccai @gbiondizoccai

Contributors SC and GBZ unrelated and independent of the clinical study, completed an independent review of the results of the clinical study. TS was the principal investigator and assisted with data and verification of ground truth. ES and ABS assisted with data and verification of ground truth. IK assisted in the statistical assembly and data preparations. IK is the guarantor, who is responsible for the overall content. Heart failure and cardiomyopathies ORCID iD Istvan Kecskes http://orcid.org/0000-0002-9688-8125

Exposure rate of cardiovascular risk factors among clients of health-care clinics in Kashan, Autumn

Global, regional, and national age-sex-specific mortality for 282 causes of death in 195 countries and territories, 1980-2017: a systematic analysis for the global burden of disease study 2017

Biosensors for cardiac biomarkers detection: a review

ESC guidelines for the diagnosis and management of chronic coronary syndromes

World Health Organization. The top 10 causes of death

ACC/AHA guidelines for the evaluation and management of chronic heart failure in the adult: executive summary. A report of the American College of Cardiology/ American heart association Task force on practice guidelines (Committee to revise the 1995 guidelines for the evaluation and management of heart failure)

Understanding the management of heart failure with preserved ejection fraction: a qualitative multiperspective study

Different effects of SGLT2 inhibitors according to the presence and types of heart failure in type 2 diabetic patients

Practical guidance on heart failure diagnosis and management in primary care: recent EPCCS recommendations

What the general practitioner needs to know about their chronic heart failure patient

Nurse practitioner closed claim study, the doctors company, thedoctors

Chest pain units: a modern way of managing patients with chest pain in the emergency department

ACC/AHA guideline on the assessment of cardiovascular risk

Variability in ejection fraction measured by echocardiography, gated single-photon emission computed tomography, and cardiac magnetic resonance in patients with coronary artery disease and left ventricular dysfunction

Variability in echocardiographic measurements of left ventricular function in septic shock patients

Observer variability in ECG interpretation for thrombolysis eligibility: experience and context matter

Interpretations of and management actions following electrocardiograms in symptomatic patients in primary care: a retrospective dossier study

Palpitation referrals from primary care to a secondary care cardiology outpatient clinic: assessing adherence to guidelines

Shifting hospital care to primary care: an evaluation of cardiology care in a primary care setting in the Netherlands

Routes to diagnosis of heart failure: observational study using linked data in England

Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data-Premarket Approval (PMA) and Premarket Notification 510(k) Submission

Addendum: Estimands and Sensitivity Analysis in Clinical Trials, to determine the suitable statistical analysis we are going to use in our CS