key: cord-0965213-xoudee7t
authors: Schilling, William HK; Callery, James J.; Chandna, Arjun; Hamers, Raph L; Watson, James A; White, Nicholas J
title: The WHO guideline on drugs to prevent COVID-19: small numbers- big conclusions
date: 2021-03-30
journal: Wellcome Open Res
DOI: 10.12688/wellcomeopenres.16741.1
sha: db1ddff8d87eefd6abccfd544920a81bc01de2e4
doc_id: 965213
cord_uid: xoudee7t

The World Health Organization living guideline on drugs to prevent COVID-19 has recently advised that ongoing trials evaluating hydroxychloroquine in chemoprophylaxis should stop. The WHO guideline cites “high certainty” evidence from randomised controlled trials (RCTs) that hydroxychloroquine prophylaxis does not reduce mortality and does not reduce hospital admission, and “moderate certainty” evidence of poor tolerability because of a significantly increased rate of adverse events leading to drug discontinuation. Yet there is no such evidence. In the three pre-exposure chemoprophylaxis RCTs evaluated in the guideline there were no deaths and only two COVID-19-related hospital admissions, and there was a mistake in the analysis of the number of discontinuations (after correction there is no longer a statistically significant difference between those taking the drug and the controls). Guidelines on the prevention and treatment of COVID-19 should be based on sufficient verified evidence, understanding of the disease process, sound statistical analysis and interpretation, and an appreciation of global needs.

Tropical Medicine Research Unit (MORU) gateway.

Invited Reviewers 

The views expressed in this article are those of the author(s). Publication in Wellcome Open Research does not imply endorsement by Wellcome.

On 2 March 2021 the World Health Organization (WHO) issued a guideline on COVID-19 prophylaxis which included unusual judgements with far reaching implications derived from small numbers of observations 1, 2 . There are several analogies between the legal system and clinical investigations, and in particular the interpretation of clinical trials and the production of treatment guidelines. Both seek the truth, both review the strengths and weaknesses of evidence, and both end in a judgement. The WHO guideline uses standard methodological approaches to evaluate and grade clinical research outputs and to generate guidelines 3 . In general, this is a conservative process requiring a substantial quality evidence base for definitive recommendations. This reduces uncertainty in the assessments, and it helps ensure that the consequent guidelines are robust. We argue that this has not happened for the WHO guideline on drugs to prevent COVID-19.

The WHO guideline provides a strong recommendation against hydroxychloroquine in COVID-19 pre-or post-exposure prophylaxis 1,2 . In many ways, hydroxychloroquine has become the COVID-19 pariah, but it still deserves a fair trial. It is true that hydroxychloroquine for both treatment and prevention was intensely politicised, and it was recommended prematurely by many governments, institutions and prominent individuals early in the COVID-19 pandemic. It should not have been. It is also true that definitive large randomised controlled trials (RCTs) have shown unequivocally that hydroxychloroquine is not life-saving in hospitalised patients 4 . This is the stage of the disease when anti-inflammatory drugs, such as dexamethasone or interleukin 6 (IL-6) receptor antagonists, but not antiviral drugs (notably remdesivir), have proved lifesaving 5,6 . However, in contrast to the large randomised controlled trials (RCTs) in hospitalised patients, relatively few patients have been enrolled in studies of hydroxychloroquine in early treatment, or in post-exposure (PEP) or pre-exposure prophylaxis (PrEP). In the three published, or posted, pre-exposure prophylaxis RCTs there were very few endpoints 1,2 . Nevertheless, the WHO guideline has concluded definitively that hydroxychloroquine does not provide useful benefit in any of these situations. The WHO guideline group has also taken the unusual step of advising funders and researchers that they should reconsider the initiation and continuation of ongoing trials, i.e. they should stop. So, although it is described as a "living guideline", without further evidence it is unlikely that this particular guideline will live much longer. Case closed.

From a statistical perspective both the justice system, and the institutions which issue disease prevention and treatment regulatory approvals and guidelines, focus primarily on demonstrating proof beyond reasonable doubt. Trials that lead to conviction have proved guilt beyond reasonable doubt. Preregistration RCTs aim to prove efficacy of a drug beyond reasonable doubt, in addition to showing that the cost of this efficacy is not too high in terms of tolerability and safety. In the WHO COVID-19 prophylaxis guideline the opposite is being done. A definitive statement about lack of clinical utility is being made. According to the guideline the highest efficacy estimate compatible with the data (lower end of the confidence interval) is not a clinically useful effect. Despite the small number of endpoints, the guideline claims that hydroxychloroquine, taken for prevention, results in no important differences in mortality, admission to hospital, or laboratory confirmed COVID-19 1,2 . It also claims that adverse events (AEs) leading to drug discontinuation is a significant problem for hydroxychloroquine prophylaxis. Both contributed to the WHO judgement that hydroxychloroquine should not be used, and should not be evaluated further in COVID-19 prophylaxis trials 1,2 .

There is certainly not enough evidence to recommend hydroxychloroquine for COVID-19 prophylaxis (there never has been), but is this small and heterogeneous evidence base enough to state conclusively, as the WHO guideline has done, that hydroxychloroquine does not provide a modest but worthwhile benefit? Does it justify the implicit recommendation that ongoing RCTs in the prevention of COVID-19 should stop now? Nearly all the evidence used to generate this strong recommendation has been in the public domain for several months. It comprises three RCTs in post-exposure prophylaxis (PEPwhich is close to early treatment) and three in pre-exposure prophylaxis (PrEP-true prevention) 1,2 . Two studies used confirmed or suspected COVID-19 (mainly suspected) as their primary endpoints, and the other four used laboratory-confirmed COVID-19. Dosages differed -notably, the largest PrEP study (>75% of the PrEP data) used a much lower hydroxychloroquine dose, closer to that used in antimalarial chemoprophylaxis rather than the more widely used rheumatoid arthritis doses used in other COVID-19 prophylaxis trials 7 . Another PrEP study had only a single case of COVID-19 8 ; and none of the data in any of the included studies were collected outside of North America or Europe. There were other differences which overall may be summarised as "substantial heterogeneity" between studies.

These six randomised controlled comparisons enrolled 6,059 participants, but they generated relatively few endpoints (suspected or confirmed COVID-19, hospital admission or death). In the three PrEP trials there were only 26 confirmed COVID-19 cases in total (15 out of 1,197 randomised to hydroxychloroquine, 11 out of 687 randomised to placebo). With so few events and considerable heterogeneity in design, the metaanalysis is sensitive to the methods employed. A previous meta-analysis chose to use the appropriately adjusted primary endpoints reported in each hydroxychloroquine prevention study (e.g. one study was a cluster randomised trial 9 so adjustment for cluster was necessary) 10 . This estimated a metaanalytic risk ratio of 0 In the PrEP studies only two admissions were for COVID-19. These data clearly do not exclude modest but clinically significant differences in the two "most important" outcomes, and most certainly do not equate to "high certainty evidence".

These evaluations should be contrasted with the earlier assessment by the WHO guideline group of dexamethasone and hydroxychloroquine in hospitalised COVID-19 patients 11, 12 ( Figure 2 ). The evidence that dexamethasone reduces mortality in hospitalised patients with COVID-19 receiving respiratory support was reviewed by the WHO guideline group in September 2020 11 . Their judgement was based on the very large platform RCT (RECOVERY), in which there were 980 deaths. The odds ratio for death in dexamethasone recipients receiving respiratory support was 0.82 (95%CI: 0.72 to 0.92) 4 . For hydroxychloroquine, lack of efficacy was concluded from the outcomes of 10,859 mainly hospitalised patients (almost half from the RECOVERY trial) with over 2,000 deaths. The meta-analysis of virological effect, without intra-study adjustments, resulting in a meta-analytic odds ratio of 1.03 (95% C.I. 0.80 to 1.32) 1,2 . Figure 1 compares the two analyses.

The WHO guideline development panel decided that "Mortality would be the outcome most important to individuals, followed by need for hospital admission, laboratory confirmed SARS-CoV-2 infection, and adverse effects leading to discontinuation" 1,2 . The review determined that there was no important difference in mortality, admission to hospital, or laboratory confirmed COVID-19, and that the evidence quality to support these statements was high. The four-star "High GRADE rating" is defined as: "the authors have a lot of confidence that the true effect is similar to the estimated effect". Yet there were only 13 deaths in total in the six prophylaxis trials, and they were all from one cluster-randomised non-blinded PEP trial 9 . Five were in subjects allocated hydroxychloroquine (one of whom took no drug) and eight were in subjects allocated to no drug. So, without a single death in the three PrEP RCTs, and a highly unstable odds ratio of 0.67 for mortality in subjects allocated to hydroxychloroquine versus those who were not in the PEP RCTs (95% C.I. 0.22 to 2.05), the panel were able to conclude that this provided "high certainty evidence" that hydroxychloroquine pre-exposure prophylaxis does not stratified meta-analytic estimate for mortality when combining the RECOVERY 4 and SOLIDARITY 6 trials (which used the same hydroxychloroquine dosage) resulted in a 95% confidence interval for the risk ratio of between 0.98 and 1.21 6 . Both of these results were graded as "moderate certainty evidence" (defined as: "the true effect is probably close to the estimated effect") with serious risk of bias ( Figure 2 ) 11 . So somehow these effect estimates (and thus the certainty of the treatment recommendations) based on thousands of deaths in well conducted RCTs are considered less certain (i.e. less reliable) than an estimate derived from 13 deaths.

Toxicity and tolerability are critical considerations for prophylaxis. In justifying their strong negative recommendation, the WHO guideline states that hydroxychloroquine "probably increases the risk of adverse effects leading to discontinuation of the drug (moderate certainty)" 1,2 . Aside from (i) whether it is correct to pool toxicity assessments across different dose regimens, (ii) whether the more subjective measure of discontinuation should be evaluated rather than standardised severity gradings for AEs, and (iii) whether non-placebo-controlled evidence should be included, there is an important mistake in the calculations. The WHO meta-analysis of AEs leading to study drug discontinuation miscoded the number of AE in the study by Grau-Pujol et al. 8 . There were more discontinuations in the placebo group (n=5) than in the hydroxychloroquine group (n=1). Figure 3 shows the incorrect forest plot claiming a significant difference, with a corrected version below. After correction for the miscoded AEs in the Grau-Pujol study, there is no longer a significant difference and the 95% confidence interval for the odds ratio now ranges from 0.83 to 3.29. This emphasises the danger of issuing "strong" recommendations on the basis of limited and unstable data.

By issuing a judgement on the basis of scanty evidence (some of which is incorrect), and recommending that trials should stop, the WHO committee has decided that if efficacy were to be shown by continuing current trials (a 30% reduction in the risk of COVID-19 is compatible with the results from these trials 1,2 ), then hydroxychloroquine should still not be used.

What are the implications of this judgement? Any recommendation from the WHO must be taken very seriously. Such recommendations are very influential, and these may well stop all ongoing studies. Once closed, clinical trials will not reopen. More data will come from recently completed trials 15 but, if these are not decisive, we may never know the truth.

It is reasonable to conclude already that hydroxychloroquine does not provide high prevention or early treatment efficacies. Vaccines are rightly the priority. They give high levels of protective efficacy, and must be deployed as widely as possible.

promising drugs to prevent COVID-19". Recently registered trials are not proposing to evaluate hydroxychloroquine, so the main purpose of this recommendation seems to be to stop ongoing trials. We appreciate the urgency of COVID-19 and the need to accelerate research and share research outputs so that responsible guidance can be provided rapidly. Guidelines on the prevention and treatment of COVID-19 should be based on sufficient verified evidence, understanding of the disease process, sound statistical analysis and interpretation, and an appreciation of global needs.

No data are associated with this article.

For therapeutics the preliminary evidence to date suggests that monoclonal antibodies may be more effective than small molecule repurposed antiviral candidates. But limited resource settings are unlikely to have high vaccine coverage for many months or even years, and global access to antibody therapies is very uncertain. An inexpensive, well established, widely available and relatively well tolerated drug providing moderate preventive efficacy would still be valuable-particularly in situations where there are outbreaks of vaccine escape mutations. Hydroxychloroquine is still being recommended by several countries so solid and convincing evidence of benefit or lack of benefit is still needed. The guideline group recommended that "resources should rather be oriented to evaluate other more Figure 3 . Comparison of the forest plots for adverse effects leading to discontinuation of hydroxychloroquine with the miscoding for the Grau-Pujol study (above) 11 and the correct coding (below).

World Health Organization: Drugs to prevent COVID-19: A WHO living guideline

A living WHO guideline on drugs to prevent covid-19

GRADE: an emerging consensus on rating quality of evidence and strength of recommendations

Effect of Hydroxychloroquine in Hospitalized Patients with Covid-19

PubMed Abstract | Publisher Full Text | Free Full Text 6. WHO Solidarity Trial Consortium

PubMed Abstract | Publisher Full Text | Free Full Text

Hydroxychloroquine as preexposure prophylaxis for COVID-19 in healthcare workers: a randomized trial

Pre-exposure prophylaxis with hydroxychloroquine for COVID-19: initial results of a double-blind, placebo-controlled randomized clinical trial. PREPRINT (Version 1) available at Research Square

A Cluster-Randomized Trial of Hydroxychloroquine for Prevention of Covid-19

PubMed Abstract | Publisher Full Text | Free Full Text

Systematic review and metaanalysis of randomized trials of hydroxychloroquine for the prevention of COVID-19

Prophylaxis for covid-19: living systematic review and network meta-analysis

World Health Organization: Corticosteroids for COVID-19

World Health Organization: Therapeutics and COVID-19. Accessed 11

Guidelines should not pool evidence from uncomplicated and severe COVID-19

PubMed Abstract | Publisher Full Text | Free Full Text

Centered Outcomes Research Unit: Healthcare Worker Exposure Response and Outcomes of Hydroxychloroquine (HERO-HCQ)

Reference Source

Saw Swee Hock School of Public Health, National University of Singapore, Singapore, SingaporeThe authors have put forth a cogent argument that the WHO guideline lacked sufficiently strong evidence to support the positions that it had taken. They have, however, also acknowledged that hydroxychloroquine (HCQ) does not provide high prevention or post-exposure efficacy. Based on the trials published/ongoing, it might be possible that HCQ might yet be found to provide "moderate efficacy" in prophylaxis against COVID-19. It is equally if not more likely that the results would demonstrate lack of significant efficacy, while subjecting the participants to a small but definite risk of harm.I think it might be helpful if the authors clarified their position at their end. Should all ongoing HCQ prophylaxis trials be continued, or should there be some attempt at coordination so that only one or a minimum number of definitive trials be run? Taken to the extreme, this could mean that trials could continue to be run for different dosages of HCQ, or in combination with other agents. The issue of focusing resources is an important one, particularly during this ongoing pandemic.Finally, I could not find in the letter any statement on whether some or all authors are involved in HCQ prophylaxis trials. It might be helpful for purposes of transparency to state this.

Are all factual statements correct, and are statements and arguments made adequately supported by citations? Yes

Where applicable, are recommendations and next steps explained clearly for others to follow? Yes 

Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore This is a very important open letter that raises significant questions about a "strong" recommendation from the WHO living guideline. The points are made well but there are some areas which could be clearer.The analogy with a legal judgement is good but there are differences. Most decisions in clinical medicine are not as clear cut as "guilty or not guilty" and they offer a balance of risks vs benefits. This is important especially in this case as the risks have been significantly distorted by a retracted paper which had a material effect on recruitment on some of the larger studies cited.

In a similar vein, it would have been good to have a couple of lines at least about the Surgisphere fiasco and its impact on recruitment for major ongoing clinical trials at the time. While the fiasco is still fresh in the minds of many of us now, it will be long forgotten before some time and it would be good to have a critical record of the impact of the rush to publication by some of the major medical journals who may not have exercised the same degree of due diligence that would have been exercised pre-pandemic.

While the forest plots are interesting especially to statisticians, for most clinicians, they are not that easy to understand and by separating them, it is easy to misunderstand whether the authors of this open letter are endorsing the views of the WHO guideline authors.

It would be good to have a table listing the trials that were included in the WHO guideline and to list the strengths and weaknesses of each study (e.g. sample size, end point, etc.).

It would also be interesting to point out the direction and magnitude of the impact of HCQ which seems to be quite consistent in the studies. Small but not negative and similarly for the adverse effect profile.

It would also be good to add in the studies which are still ongoing or have been published recently -perhaps as a supplementary appendix if this is done with this journal.

Is the rationale for the Open Letter provided in sufficient detail? Yes

Are all factual statements correct, and are statements and arguments made adequately supported by citations? Yes

Where applicable, are recommendations and next steps explained clearly for others to follow? PartlyCompeting Interests: I have no direct competing interests but I have met and interacted with Prof Sir Nicholas White about our own prophylaxis study as well as numerous other areas in infectious diseases over the last 20 years Reviewer Expertise: Clinical trials, emerging infectious diseases I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.