key: cord-1048933-s2i60ajr
authors: Zhang, Guanran; Kuang, Sirui; Zhang, Xiaoli
title: Assessing the reporting quality of randomized controlled trials on COVID-19 vaccines: a systematic review
date: 2022-02-17
journal: Hum Vaccin Immunother
DOI: 10.1080/21645515.2022.2031453
sha: 3f2314ea80457835964e5eebe2567a273e052c9a
doc_id: 1048933
cord_uid: s2i60ajr

This systematic review evaluated the reporting quality of COVID-19 vaccine randomized controlled trials (RCTs). Relevant RCTs published between July 20, 2020 and June 11, 2021 were identified in the PubMed database by two independent reviewers. Study quality was evaluated with the 2010 AND 2001 Consolidated Standards of Reporting Trials (CONSORT) adherence scores. A total of 22 RCTs were included. The median CONSORT adherence score according to the 2010 criteria was 21 (range, 12–25), thus indicating that 75% of the items in more than half of the RCTs had clear reports. Univariate analysis showed that CONSORT adherence scores were not predicted by category; analysis of variance also showed no significant difference between groups. Our results indicated that the overall quality of COVID-19 vaccine RCTs was very good. Current evidence indicates that a variety of COVID-19 vaccines are effective. No RCTs have reported serious adverse effects such as mortality.

Evidence-based medicine is fundamentally dependent on the quality of available clinical evidence. The results of randomized controlled trials (RCTs) provide the highest level of primary evidence, and the use of large sample sizes improves the power of statistical tests and reduces the risk of bias. Due to the lack of targeted drugs for COVID-19, many countries began to concurrently develop COVID-19 vaccines in the early stages of the pandemic. 1 Vaccination is considered to be the most effective measure for preventing the further spread of COVID-19. 2 Vaccines stimulate the body's immune system to produce antibodies against a specific virus, thus reducing the probability of future infection. Vaccinations prevent 2-3 million deaths from infections annually. 3 The effectiveness and breadth of COVID-19 vaccination will be the main determinant of how long the pandemic will last. 4 The first approved COVID-19 vaccine was produced by Pfizer-BioNTech and has been widely administered in the UK. The need for the rapid development of vaccines to combat the COVID-19 pandemic has necessitated the introduction of temporary regulations to expedite the authorization of their use in humans. 5 As a result, the risk of side effects (e.g., serious disease, mortality) have only been based on experimental research results from the first three stages of vaccine development; epidemiological research results typically available in the fourth stage are currently lacking. 6 By April 2020, approximately 100 different COVID-19 vaccines had been developed by research and development departments in different countries all over the world, with some having proceeded to the human trial stage. 7 If sufficient protection can be obtained after the first vaccine dose, the second dose can be delayed; this would ensure that a greater number of people in regions with limited access to vaccines can receive the first dose. 8 Recently, an increasing number of RCTs have investigated the effectiveness of COVID-19 vaccines by comparing infection rates between vaccinated (experimental group) and unvaccinated (control group) individuals. 9 The purpose of this systematic review was to evaluate the quality of these RCTs and to summarize the effectiveness and adverse effects of currently available COVID-19 vaccines. The overarching aim was to provide a frame of reference to facilitate vaccination selection.

This review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines that are in the supplemental materials. Relevant articles were identified by using the search terms "vaccines" and "COVID 19" in the PubMed database. The authors of the present review were not involved in the conduct of any previous RCTs pertaining to this topic.

A comprehensive search was conducted in PubMed using the following search terms: ("COVID 19 vaccines"[MeSH Terms] OR ("COVID 19"[All Fields] AND "vaccines"[All Fields]) OR "COVID 19 vaccines"[All Fields] OR ("COVID19"[All Fields] AND "vaccine"[All Fields]) OR "COVID19 vaccine"[All Fields]) AND (randomized controlled trial [Filter] ). Only studies published in English were included. All identified RCTs pertaining to COVID-19 vaccines were published between July 20, 2020 and June 11, 2021 ( Figure 1 ).

All extracted data were independently compiled by two reviewers. The reporting quality of each study was evaluated using the 19-item 2001 Consolidated Standards of Reporting Trials (CONSORT) statement (Supplemental Table S1 ) and the 28-item 2010 CONSORT standardized evaluation checklist ( Table 1 ). The overall report quality score was referred to as the CONSORT adherence score.

The inclusion criteria comprised the following: (a) evaluation of a COVID-19 vaccine using a randomized controlled design; (b) use of a COVID-19 vaccine in the experimental group; and (c) articles published in English. Studies were excluded if they did not report safety or effectiveness data, or were duplicate publications or secondary reports of previously published RCTs. If the results of a single RCT were reported in multiple publications, the one with the most complete data was selected.

The difference between the level 1 screening (titles and abstracts), two reviewers were resolved through discussion.

Two independent reviewers extracted the following data: first author name; year of publication; whether or not the term "RCT" was used in the study title; use of a structured or nonstructured abstract; experimental design and allocation ratio to the intervention and control groups; specific content recorded in the article or protocol; study setting; place of the data collection; drug information; primary and secondary outcomes; measurement information; methods used for sample size calculation and randomization; allocation concealment; blinding method; whether or not an intent-to-treat (ITT) analysis and subgroup analysis were performed; flowchart; recruitment and follow-up time; results for vaccine efficacy; experimental registration number; and source of funding. Any discrepancies were resolved by consensus between the two reviewers.

The main purpose of this study was to assess the quality of RCTs that have evaluated COVID-19 effectiveness and safety. Using CONSORT criteria, we assigned 1 point for each criterion and calculated the total score of each item. SPSS Statistics 25 was used to analyze the collected data, and descriptive statistics were used to calculate the median and mean. The linear regression coefficient generated by the CONSORT adherence scores was used as the dependent variable to obtain the regression coefficient and P value. The difference between the groups and whether the classification could predict the dependent variable were evaluated.

As shown in the flow chart in Figure 1 , a total of 11053 articles were retrieved from PubMed. Title and abstract screening excluded 10999 non-RCTs. Full texts of the remaining 54 studies were evaluated according to our predefined inclusion and exclusion criteria. Thirty-two studies were excluded, as they were either secondary reports of previous RCTs (29 articles) or duplicate studies (3 articles). Thus, a total of 22 studies were included in our analysis ( Table 2) 10-31 and their characteristics are summarized in Table 3 ; data are expressed as absolute counts and proportions.

The majority of the RCTs were conducted in countries in Europe and North America. The impact factor of most of the journals in which the studies were published (73%, n = 16) exceeded 30. Over half (63%, n = 14) of the studies included more than 500 participants. Most of the studies were either phase I or II vaccine trials; only 12% (n = 4) were phase III trials.

Inter-rater agreement for the 2010 CONSORT standardized evaluation checklist were classified via Cohen's κ statistic as substantial, good, or perfect (Table 1) . CONSORT adherence scores ranged from 0 to 28. As the reported reference median CONSORT adherence score was 21 (range, 12-25), this indicated that 75% of the items in more than half of the RCTs in the present review had clear reports. The results of the descriptive analysis showed that less than 10% of the studies had a score of <14, thus indicating that study quality was "very good" ( Table 1 ). The results showed the positive number of CONSORT in the frequency distribution diagram ( Figure 2 ). All studies provided detailed scientific background information and reported baseline participant characteristics in both the experimental and control groups. Clinical outcomes after vaccination were summarized and presented in the form of tables. Since univariate analysis did not show significant differences between categories, CONSORT adherence scores could not be predicted according to category. Analysis of variance showed that there was no significant difference between the groups ( Table 4 ).

The median CONSORT adherence score according to the 2001 CONSORT statement criteria was 16 (range, [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] ; this indicated that 84% of the RCTs had clear reports. The results of the descriptive analysis showed that less than 10% of the RCTs had a score of <9. This reflected a "very good" study quality and was consistent with the results obtained using the 2010 CONSORT standardized evaluation checklist (Supplementary  Table S1 ).

We evaluated adverse event report scores based on the rating of the hazardous recommendations. Our analysis indicated that inactivated vaccines, nucleic acid vaccines, adenovirus vector vaccines, protein subunit vaccines, and other types of COVID-19 vaccines had good efficacy. These vaccines were also safe, as no serious adverse reactions such as death were reported (Supplementary Table S2 ).

The use of allocation concealment and blinding across the included studies is shown in Supplementary Table S3 . Over half (59%, n = 13) of the studies used a centralized randomization method, and 82% (n = 18) used the blind method. Only 27% (n = 6) of the studies performed an ITT analysis. Studies that did not report the use of ITT were assumed to have not used this analysis method. Supplementary Table S4 summarizes the endpoints used in the 22 trials. Most trials ended with adverse reactions or immunogenicity; safety and adverse reactions were the most commonly reported outcomes (77%, Therefore, CONSORT adherence scores were not predicted by category.

The results of our review indicate that the quality of RCTs on COVID-19 vaccines was not affected by the specific stage of vaccine development that was under investigation. Indeed, the CONSORT adherence scores indicated that the reporting quality of these RCTs was very good. This finding is pertinent for governments worldwide, as they are responsible for the majority of funding for vaccine research and development.

Vaccines are one of the most effective and safest means for preventing the further spread of COVID-19. 32 A number of different factors may affect the quality of research reports. For example, study quality is often significantly associated with the type of funding source. Journals with more published papers have higher impact factors, which are often associated with increased study quality. Studies in such journals are more likely to have a large sample size and include a wide range of age groups, from adolescents to the elderly.

At present, the incidence of new COVID-19 cases has not plateaued in many countries. In addition, some countries have even reported mutated variants with increased transmissibility. The emergence of COVID-19 variants indicates that a second vaccine dose is necessary, as previous studies have found that vaccine effectiveness after the first dose decreases after a period of time. 33 A third dose can further maintain effectiveness over the long-term and should be considered in countries where the proportion of the population with both first and second doses has reached a certain threshold. 34 Different vaccine types can be selected by countries according to their actual situation. 35 The results of the present review indicate that the majority of the investigated vaccine types are very effective. From the conclusion that there is no significant difference in univariate analysis, it can be seen that the literature quality of different categories with different characteristics is similar. It can be concluded that the quality of these RCTs is very good; the reporting was very specific and detailed, regardless of journal impact factor, funding source, region in which trials were conducted, and sample size.

Some RCTs did not provide details regarding the random allocation of study participants, as well as whether allocation concealment was performed. Some RCTs did not report whether researchers or patients were blinded to treatment allocation. The current stage of COVID-19 vaccine research has mainly focused on outcomes pertaining to adverse reactions, immunogenicity, and vaccine effectiveness; the latter outcome has been limited by the inability to mass produce experimental vaccines for evaluation in clinical trials. Nevertheless, a plethora of studies on COVID-19 vaccines are planned or in progress, and their results will provide important data on actual vaccine effectiveness. To date, the completed RCTs on COVID-19 vaccines have been of very good quality; this may be attributed to the individual efforts of research personnel, as well as the large amount of invested human, material, and financial resources. High-quality RCTs not only provide a greater reference value for future studies, but also contribute more to the global efforts to combat the COVID-19 pandemic. This is pertinent, as greater challenges for vaccine development are expected with the continuous emergence of COVID-19 variants. Long-term studies are required to determine whether existing vaccines can effectively and safely prevent infection by different variant strains. 36, 37 Nevertheless, we found that some studies omitted certain CONSORT checklist items, such as declaring that the study was a RCT in the title; this criterion was only satisfied in 59% of the included studies. The implementation of randomization was only described in 41% of the studies. Thus, the majority of studies did not adhere to the principles of randomization; alternatively, they may have followed these principles but failed to report it. This resulted in a reduced study quality to some degree.

Some limitations are acknowledged in the present review. For example, we did not perform a detailed analysis of study follow-up duration and specific types of adverse reactions. Furthermore, the included studies did not provide detailed data on participant race, sex, age, or other differences. In addition, as the univariate analyses did not yield statistically significant associations with CONSORT adherence scores, we were unable to conduct a multivariate analysis to adjust for confounding factors.

In conclusion, based on the use of the CONSORT criteria, we determined that the RCTs on COVID-19 vaccines that have been published to date are of very high quality. This may be attributed to not only the adherence of study authors to established research reporting guidelines, but also the strict evaluation of manuscripts by referees during the peer review stage.

No potential conflict of interest was reported by the author(s). 

Data are available upon reasonable request.

Distinct durability of IgM/IgG antibody responses in COVID-19 patients with differing severity

The politics of vaccination: a global history

Diagnosis of COVID-19 for controlling the pandemic: a review of the state-of-the-art

The potential neurological effect of the COVID-19 vaccines: a review

SARS-CoV-2 vaccines in development

What defines an efficacious COVID-19 vaccine? A review of the challenges assessing the clinical efficacy of vaccines against SARS-CoV-2

The COVID-19 vaccine development landscape

Effectiveness of the first component of Gam-COVID-Vac (Sputnik V) on reduction of SARS-CoV-2 confirmed infections, hospitalisations and mortality in patients aged 60-79: a retrospective cohort study in Argentina

Vaccine epidemiology: efficacy, effectiveness, and the translational research roadmap

Safety and immunogenicity of S-Trimer (SCB-2019), a protein subunit vaccine candidate for COVID-19 in healthy adults: a phase 1, randomised, double-blind, placebo-controlled trial

Leav B; mRNA-1273 Study Group. A preliminary report of a randomized controlled phase 2 trial of the safety and immunogenicity of mRNA-1273 SARS-CoV-2 vaccine

Immunogenicity and safety of a severe acute respiratory syndrome coronavirus 2 inactivated vaccine in healthy adults: randomized, double-blind, and placebo-controlled phase 1 and phase 2 clinical trials

COVE Study Group. Efficacy and Safety of the mRNA-1273 SARS-CoV-2 Vaccine

Efficacy of NVX-CoV2373 Covid-19 Vaccine against the B.1.351 Variant

COVID-19 Genomics UK consortium; AMPHEUS Project; Oxford COVID-19 Vaccine Trial Group. Efficacy of ChAdOx1 nCoV-19 (AZD1222) vaccine against SARS-CoV-2 variant of concern 202012/01 (B.1.1.7): an exploratory analysis of a randomised controlled trial

Interim results of a Phase 1-2a trial of Ad26.COV2.S Covid-19 vaccine

Oxford COVID Vaccine Trial Group. Safety and immunogenicity of the ChAdOx1 nCoV-19 vaccine against SARS-CoV-2: a preliminary report of a phase 1/2, single-blind, randomised controlled trial

Gam-COVID-Vac Vaccine Trial Group. Safety and efficacy of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine: an interim analysis of a randomised controlled phase 3 trial in Russia

Safety, tolerability, and immunogenicity of an inactivated SARS-CoV-2 vaccine in healthy adults aged 18-59 years: a randomised, double-blind, placebo-controlled, phase 1/2 clinical trial

Phase I/II study of COVID-19 RNA vaccine BNT162b1 in adults

Oxford COVID Vaccine Trial Group. Safety and immunogenicity of ChAdOx1 nCoV-19 vaccine administered in a prime-boost regimen in young and old adults (COV002): a single-blind, randomised, controlled, phase 2/3 trial

Safety and immunogenicity of two RNA-based Covid-19 vaccine candidates

Effect of an inactivated vaccine against SARS-CoV-2 on safety and immunogenicity outcomes: interim analysis of 2 randomized clinical trials

C4591001 Clinical Trial Group. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine

Safety and immunogenicity of an inactivated SARS-CoV-2 vaccine, BBIBP-CorV: a randomised, double-blind, placebo-controlled, phase 1/2 trial

Phase 1-2 trial of a SARS-CoV-2 recombinant spike protein nanoparticle vaccine

Efficacy of the ChAdOx1 nCoV-19 Covid-19 vaccine against the B.1.351 variant

The safety and immunogenicity of an inactivated SARS-CoV-2 vaccine in Chinese adults aged 18-59 years: a phase I randomized, double-blinded, controlled trial

Immunogenicity and safety of a recombinant adenovirus type-5-vectored COVID-19 vaccine in healthy adults aged 18 years or older: a randomised, double-blind, placebo-controlled, phase 2 trial

Safety and immunogenicity of an inactivated SARS-CoV-2 vaccine, BBV152: a double-blind, randomised, phase 1 trial

Aging in COVID-19: vulnerability, immunity and intervention

Vaccine effectiveness of the first dose of ChAdOx1 nCoV-19 and BNT162b2 against SARS-CoV-2 infection in residents of long-term care facilities in England (VIVALDI): a prospective cohort study

Effectiveness of mRNA BNT162b2 COVID-19 vaccine up to 6 months in a large integrated health system in the USA: a retrospective cohort study

Efficacy and Safety of COVID-19 vaccines in Phase III trials: a meta-analysis. Vaccines (Basel)

New variant of SARS-CoV-2 in UK causes surge of COVID-19

Introduction of the South African SARS-CoV-2 variant 501Y.V2 into the UK