key: cord-0971278-h9iaiiaf authors: Luo, Chongliang; Jiang, Ying; Du, Jingcheng; Tong, Jiayi; Huang, Jing; Lo Re, Vincent; Ellenberg, Susan S.; Poland, Gregory A.; Tao, Cui; Chen, Yong title: Prediction of post‐vaccination Guillain‐Barré syndrome using data from a passive surveillance system date: 2021-02-23 journal: Pharmacoepidemiol Drug Saf DOI: 10.1002/pds.5196 sha: 43cceaac5347baeaa629189e7b9488f2accd0c5b doc_id: 971278 cord_uid: h9iaiiaf PURPOSE: Severe adverse events (AEs), such as Guillain‐Barré syndrome (GBS) occur rarely after influenza vaccination. We identify highly associated AEs with GBS and develop prediction models for GBS using the US Vaccine Adverse Event Reporting System (VAERS) reports following trivalent influenza vaccination (FLU3). METHODS: This study analyzed 80 059 reports from the US VAERS between 1990 and 2017. Several AEs were identified as highly associated with GBS and were used to develop the prediction model. Some common and mild AEs that were suspected to be underreported when GBS occurred simultaneously were removed from the final model. The analyses were validated using European influenza vaccine AEs data from EudraVigilance. RESULTS: Of the 80 059 reports, 1185 (1.5%) were annotated as GBS related. Twenty‐four AEs were identified as having strong association with GBS. The full prediction model, using age, sex, and all 24 AEs achieved an area under the receiver operating characteristic (ROC) curve (AUC) of 85.4% (90% CI: [83.8%, 86.9%]). After excluding the nine (e.g., pruritus, rash, injection site pain) likely underreported AEs, the final AUC became 77.5% (90% CI: [75.5%, 79.6%]). Two hundred and one (0.25%) reports were predicted as of high risk of GBS (predicted probability >25%) and 84 actually developed GBS. CONCLUSION: The prediction performance demonstrated the potential of developing risk‐prediction models utilizing the VAERS cohort. Excluding the likely underreported AEs sacrificed some prediction power but made the model more interpretable and feasible. The high absolute risk of even a small number of AE combinations suggests the promise of GBS prediction within the VAERS dataset. vaccines, many diseases such as diphtheria, tetanus, Haemophilus influenzae type b (Hib) disease, poliomyelitis, measles, mumps, congenital rubella, and smallpox have been dramatically reduced or even eradicated worldwide. 1 Vaccines, like other biological products can also cause various side effects. As vaccines are usually administrated to healthy persons, adverse events (AEs) after vaccination may arouse suspicion about the safety of the vaccines and cause vaccine hesitancy or refusal in certain populations. 2, 3 Post-marketing surveillance is needed in the general population in order to identify and evaluate AEs for vaccine safety studies. The Vaccine Adverse Event Reporting System (VAERS) was established by the Food and Drug Administration (FDA) and the Centers for Disease Control and Prevention (CDC) to collect reports about AEs after vaccination. Since its creation in 1990, VAERS has been used to continually monitor reports following vaccination to determine whether a vaccine has a higher than expected rate of AEs, 4 especially for those rare AEs that are difficult to evaluate in clinical trials during the vaccine development stage. 1 Rare AEs of certain vaccines (i.e., safety signals) are commonly detected by disproportionality analysis, which compares their rates with certain background rates. 5 These statistically significant signals offer hypotheses that can be further studied to assess causality. Due to the well-known limitations of the spontaneous reporting data, the use of VAERS data for risk prediction has been limited. VAERS is also subject to reporting bias, such as underreporting of AEs, especially for common and mild events. 6 In addition, there is no control group or validation data that would allow a generalizable conclusion. As a result, any predictive model should be conducted and interpreted with caution. There is limited research on risk prediction using VAERS. One such study is by Pellegrino et al., who systematically reviewed the pharmacogenetic studies related to VAERS and provided recognized genetic risk factors. 7 In the era of big data, machine-learning techniques and data-driven methods are being increasingly applied to medical and healthcare areas 8, 9 and have achieved important successes. In the context of vaccine pharmacovigilance, an applicable risk-prediction model makes early intervention possible, which could prevent or mitigate some severe AEs. In this study, we studied the occurrence of Guillain-Barré syndrome (GBS) after influenza vaccination. GBS, an acute immunemediated peripheral neuropathy, is suspected as one of the most common acute paralytic neuromuscular disorders and one of the most severe AEs following immunization (AEFIs) in adults. 10, 11 The occurrence of GBS among the general population is rare, and estimates of incidence range from 0.8 to 1.9 cases per 100 000 person-years. 12 Influenza vaccines have long been suspected to increase the risk of GBS. Some studies have found associations between influenza vaccination and GBS, [13] [14] [15] while other studies have not, [16] [17] [18] and the association between seasonal influenza vaccine and GBS can vary from season-to-season. 19 Thus, there remains doubt over the causative nature of the influenza vaccines with GBS. Despite the effort devoted to studying the association between the risk of GBS and influenza vaccination, it is important for the public to know that getting seasonal influenza vaccines is the best way to prevent flu infection and complications. 19 Meanwhile, potential risk prediction for GBS onset is also critical as it may provide early alerts to the rare population who might at the risk of getting GBS. Little research has been conducted to identify the AEs that are related to GBS among post-vaccination subjects, which can be used to develop risk-prediction models. It is difficult to accurately diagnose GBS at an early stage due to its diverse causes and clinical presentations; however, those patients with a high risk of contracting GBS following influenza vaccination identified by our model could be more alert to the fluctuation of symptoms and seek medical treatment before the possible onset of GBS. The purpose of our investigation is to identify novel risk factors and generate a novel risk prediction model, which needs to be further validated by prospective studies. Once validated, the risk prediction models could lead to useful insights for clinical decision making. At the end of 2018, the VAERS database contained more than 400 000 vaccine-associated AE reports. Each report had been manually annotated at the preferred-term (PT) level in the Medical • Spontaneous reporting system such as the US Vaccine Adverse Event Reporting System (VAERS) contains massive records of adverse events (AEs) after vaccination, which could be used to develop prediction model for severe AEs. • We identify highly associated AEs with GBS and develop prediction models for GBS using 80 059 VAERS reports following trivalent influenza vaccination (FLU3) between 1990 and 2017. • Some AEs (e.g., pruritus, rash, injection site pain) are mild and common, but negatively associated with GBS. They are likely to be underreported when GBS occurs simultaneously. • After excluding the nine likely underreported AEs, the final prediction AUC is 77.5% (90% CI: [75.5%, 79.6%]). Paraesthesia and apnea, together with age (49-64) and sex (male) are high-risk factors for GBS. The analysis is independently validated using European influenza vaccine AEs data from EudraVigilance. • The risk prediction model could help the reporter or healthcare professionals to monitor the vaccine's condition and take early intervention when certain high-risk early AEs are observed. Dictionary for Regulatory Activities (MedDRA) by domain experts. According to the CDC, the VAERS reports had been screened to remove duplicate reports. 20 We extracted all of the VAERS reports submitted after FLU3 vaccination from 1990 to 2017. The detailed flow chart of the data processing and the analysis procedure was in Figure 1 . VAERS reports typically include more than one AE, with the median number of AEs being 3, and 20% of reports contains only one AE. After quality control (e.g., exclusion of reports with age < 0.5 or missing age or sex, removal of AEs due to the investigation related MedDRA terms), we obtained 80 059 reports and 2977 unique AEs other than GBS. The total number of GBS-related reports was 1185. Cohort characteristics, such as age, sex, and onset interval distribution were displayed in Table 1 . Additionally, we also processed the European EudraVigilance data and used them for validation purpose. The European data were obtained from European Medicines Agency (EMA) at 2016 and included influenza vaccines AE reports from 2003 to 2016. We filtered out the reports where the occurrence was outside European area. 13 550 reports were extracted, of which 327 reports were GBS-related. Due to data access limitation, the European data contained all the influenza vaccines. We constructed a 2-by-2 table for each AE and measured its association with GBS by the odds ratio (OR) and tested the significance by Chi-squared test. To enhance the reproducibility of our findings, we applied a conservative Bonferroni correction for multiple testing, with the overall nominal significance level α = 0.05. We screened the identified AEs for clinical interpretation and further risk prediction. We first screened out extremely rare AEs with a prevalence of less than 0.05%. After consultation with a neurologist, we also screened out those AEs that are actually typical of GBS treatments (as MedDRA also contains medical and health-related concepts beyond AEs), very severe and typical GBS symptoms, or known to happen after GBS treatment. The identified AEs were validated using the European data by comparing their prevalence and ORs in both data sets. We built logistic regression models to predict the occurrence of GBS after FLU3 vaccination, using age, sex and associated AEs. All models were fit using the same set of training data (80% of all cohorts), and performance was measured by the AUC 21 value of predicting the same set of testing data (20% of all cohorts). The first naïve model, which included only age and sex, was presented for baseline comparison. The second model was fit using age, sex, and the identified AEs. The second model may be questionable, as it involves the negatively associated AEs which are more likely to be underreported when GBS occurs and as a result, the association may be distorted. To obtain an applicable predictive model, we further excluded this group of AEs and used only the positively associated AEs, as well as sex and age, to build the final model. The absolute risk of each factor and the AE combinations were also presented by refitting the final model using the full US data. Similar prediction was also conducted in the European data, using the same set of predictors as in the US data. The population characteristics in the extracted reports were summarized in Table 1 . Age was categorized into four groups: 0.5-17, 18-49, 50-64, and 65+ years. For the US population, the median age was 50 years, with the interquartile range (IQR) from 29 to 66. Nearly 70% of the cohort was female. Most reports that were not related to GBS had an onset time within 1 week, whereas more than 70% of the reports that were GBS related had an onset time that exceeds 1 week. This was consistent with results from Haber et al. 22 To show how multiple AEs were temporally ordered in one report, we manually reviewed and annotated 15 US reports and presented them in Figure S1 of the supporting information. The first six reports ended with GBS as the last reported event. Compared to non-GBS reports, GBS tends to occur later than non-GBS AEs, consistent with the pathophysiology of GBS. Most of the AEs were very rare. For example, 90% of the AEs were reported fewer than 90 times throughout the 28 year timeframe we examined. We identified 83 AEs from the US data, among which 24 were kept after further screened by clinical experts. The detail of the screening procedure was deferred to the supporting information. The identified AEs and the validation using the European data were listed in Table 2 . We evaluated the identified AEs in terms of their association with GBS and prevalence among the VAERS FLU3 cohort. Nine AEs (pyrexia, chills, nausea, pruritus, rash, urticaria, injection site pain, injection site swelling and injection site erythema) were negatively associated with GBS, and their prevalence was high. Thirteen AEs (muscle spasms, hypertension, dysphagia, hyperglycaemia, diabetes mellitus, dysuria, depression, apnea, fecal incontinence, constipation, urinary incontinence, dysuria, urinary tract infection and urinary retention) were positively associated with GBS, but their prevalence was low (<1%). Back pain and paraesthesia were two AEs that were both positively associated with GBS and have a relatively high prevalence. Interestingly, the associations and prevalence of these AEs in the US data are highly consistent with those in the European data. Specifically, the same nine AEs were also negatively associated with GBS and have relatively high prevalence in the European data. The remaining 15 AEs were also positively associated with GBS in the European data. As noted above, the nine AEs that are negatively related to GBS are more common and mild and thus more likely to be underreported, especially when a severe AE such as GBS occurs. This underreporting could alter the direction of their association with GBS and make the prediction not applicable. In contrast, for the other 15 AEs that are less common and mild, the bias is expected to be relatively small. Figure 2A shows the ROC curves of GBS prediction using three nested models with different predictors in the US data. for GBS. The predicted absolute risk is presented in Figure 2B ,C. The absolute risk of all single AEs and combinations of two AEs are plotted in Figure 2B . Even for only two AEs together, the absolute risk can be as high as 30% and above. For example, the combination of apnea and paraesthesia has a predictive risk of 66%. This combination consists of seven subjects, of which four report GBS. Since usually only a few of the identified AEs are reported, this combination table can be used as a quick reference tool for identifying the risk of GBS when certain AEs are observed. We plot the absolute risk versus the number of AEs in Figure 2C , where 201 (0.25%) reports were predicted as of high risk of GBS (e.g., risk > 25%) and 84 actually developed GBS, the PPV is thus 41.8%. Vaccination is one of the most successful public health interventions ever implemented. It is important to study vaccine safety issues in order to maintain high levels of public trust in vaccines, and thereby mitigate vaccine hesitation. In this paper, we developed a riskprediction model for GBS using associated AEs identified from VAERS data. To the best of our knowledge, this approach is the first attempt to utilize VAERS data for risk prediction. We demonstrate the potential to develop a GBS "alarm signal" based solely on reported VAERS AEs. The application of the prediction model can be valuable for persons receiving a FLU3 vaccination, for clinicians as a means to understand the likelihood of developing GBS, for regulators or companies to detect a signal of GBS early in the use of a vaccine, and for scientists who seek to understand mechanisms for GBS. The purpose of this study was to determine whether certain AEs are associated with GBS, rather than the association of GBS with influenza vaccination. Our study thus differed from existing signal detection studies that use VAERS data, 23 The data-driven association analysis identifies AEs that can motivate further etiology studies, after excluding those AEs with established GBS causality. The nine AEs that are negatively associated with GBS are of the highest prevalence. They are mild and more likely to occur shortly after vaccination. A possible explanation is that either they are protective for GBS or they are subject to underreporting, especially when GBS happens. Although the underreporting pattern is likely the case, 6 we cannot rule out the possibility of a protective effect. On the other hand, some of the 15 identified AEs that are positively associated with GBS, have been reported to be related to GBS during its early stage in the literature. For example, paraesthesia may be an initial presenting symptom, and bilateral facial weakness with paraesthesias is typically involved in GBS. 24 In addition, autonomic dysfunction, including hypertension and urinary retention, can be a presenting sign of GBS in children, 25 and urinary incontinence should be included in prognostic models for GBS. 17 Some of the identified AEs are likely to be preexisting conditions, which are also useful for predicting GBS. For example, Kaplan et al. suggest that diabetes mellitus exacerbates the clinical and electrophysiological features of GBS and influences long-term disability. 16 Besides, it's well known that the depression, hyperglycaemia and diabetes mellitus may also relate to immunological abnormality. The findings from our investigation have potential to impact clini- A key issue in risk prediction is the temporal order between the predictor AEs and the outcome GBS, that is, the AEs need to happen before GBS to be valid for prediction. However, manual chart review and annotation for temporal information using VAERS reports at large scale are challenging as most temporal information related to GBS progression are stored in unstructured narrative symptom texts. Based on our investigation of a small subset of reports through manual chart review (see Figure S1 in the supporting information), as well as the literature, 13 The vaccine adverse event reporting system (VAERS) Time trends in autism and in MMR immunization coverage in California Parents who refuse or delay HPV vaccine: differences in vaccination behavior, beliefs, and clinical communication preferences. Hum Vaccines Immunother Prevention; Food and Drug Administration. VAERS User Guide The VAERS Working Group. Intussusception among recipients of rotavirus vaccine: reports to the vaccine adverse event reporting system Safety monitoring in the vaccine adverse event reporting system (VAERS) The first steps towards the era of personalised vaccinology: predicting adverse reactions A call for deep-learning healthcare Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning Influenza vaccination and Guillain Barre syndrome Safety of trivalent inactivated influenza vaccines in adults: background for pandemic influenza vaccine safety monitoring Population incidence of Guillain-Barré syndrome: a systematic review and meta-analysis The Guillain-Barre syndrome and the 1992-1993 and 1993-1994 influenza vaccines Association between Guillain-Barré syndrome and influenza A (H1N1) 2009 monovalent inactivated vaccines in the USA: a meta-analysis Guillain-Barré syndrome following vaccination in the national influenza immunization program, United States, 1976-1977 Guillain-Barré syndrome in the United States, 1979-1980 and 1980-1981: lack of an association with influenza vaccination Investigation of the temporal association of Guillain-Barré syndrome with influenza vaccine and influenzalike illness using the United Kingdom general practice research database Surveillance for Guillain-Barré syndrome after 2015-2016 and 2016-2017 influenza vaccination of Medicare beneficiaries Vaccine Safety Concerns: Guillain-Barré Syndrome Centers for Disease Control and Prevention. Vaccine Adverse Event Reporting System (VAERS) Fact Sheet The meaning and use of the area under a receiver operating characteristic (ROC) curve Guillain-Barré syndrome following influenza vaccination A signal detection method for temporal variation of adverse effect with vaccine adverse event reporting system data Mimics and chameleons in Guillain-Barré and Miller Fisher syndromes Guillain-Barré syndrome presenting with urinary retention and hypertension Vaccine Safety Datalink project: a new tool for improving vaccine safety monitoring in the United States. The Vaccine Safety Datalink Team CLAMP-a toolkit for efficiently building customized clinical natural language processing pipelines Prediction of post-vaccination Guillain-Barré syndrome using data from a passive surveillance system Because this study did not involve human subjects it was outside of the purview of institutional review boards. https://orcid.org/0000-0003-3682-9454