key: cord-0844964-761xx2p1 authors: Lutalo, Tom; Nalumansi, Aminah; Olara, Denis; Kayiwa, John; Ogwang, Bernard; Odwilo, Emmanuel; Watera, Christine; Balinandi, Stephen; Kiconco, Jocelyn; Nakaseegu, Joweria; Serwanga, Jennifer; Kikaire, Bernard; Ssemwanga, Deogratius; Abiko, Brendah; Nsereko, Christopher; Cotten, Matthew; Buule, Joshua; Lutwama, Julius; Downing, Robert; Kaleebu, Pontiano title: Evaluation of the performance of 25 SARS-CoV-2 serological rapid diagnostic tests using a reference panel of plasma specimens at the Uganda Virus Research Institute date: 2021-09-15 journal: Int J Infect Dis DOI: 10.1016/j.ijid.2021.09.020 sha: ef3868763e6443fa9d953fb5312d4946d6135906 doc_id: 844964 cord_uid: 761xx2p1 Introduction Serological testing is needed to better understand the epidemiology of the SARS-CoV-2 virus. Rapid Diagnostic Tests (RDTs) have been developed to detect specific antibodies, IgM and IgG, to the virus. We evaluated the performance of 25 of these RDTs. Methods A serological reference panel of 50 positive and 100 negative plasma specimens was developed from SARS-CoV-2 PCR and antibody positive patients and pre-pandemic SARS-CoV-2-negative specimens collected in 2016. Test performance of the 25 RDTs was evaluated against this panel. Results A total of 10 RDTs had a sensitivity ≥98% while 13 RDTs had a specificity ≥98% to anti-SARS-CoV-2 IgG antibodies. Four RDTs; Boson, MultiG, Standard Q and ViviaDiag had both sensitivity and specificity of ≥98% to anti-SARS-CoV-2 IgG antibodies. Only 3 RDTs had a sensitivity ≥98% while 10 RDTs had a specificity ≥98% to anti-SARS-CoV-2 IgM antibodies. Three RDTs; Autobio, MultiG and Standard Q, had sensitivity and specificity of ≥98% to combined IgG/IgM. The RDTs that performed well also had perfect or almost perfect inter-reader agreement. Conclusions This evaluation identified three RDTs with a sensitivity and specificity to IgM/IgG antibodies of ≥98% with the potential for widespread antibody testing in Uganda. Serological testing is needed to better understand the epidemiology of the SARS-CoV-2 virus. Rapid Diagnostic Tests (RDTs) have been developed to detect specific antibodies, IgM and IgG, to the virus. We evaluated the performance of 25 of these RDTs. A serological reference panel of 50 positive and 100 negative plasma specimens was developed from SARS-CoV-2 PCR and antibody positive patients and pre-pandemic SARS-CoV-2-negative specimens collected in 2016. Test performance of the 25 RDTs was evaluated against this panel. A total of 10 RDTs had a sensitivity ≥98% while 13 RDTs had a specificity ≥98% to anti-SARS-CoV-2 IgG antibodies. Four RDTs; Boson, MultiG, Standard Q and ViviaDiag had both sensitivity and specificity of ≥98% to anti-SARS-CoV-2 IgG antibodies. Only 3 RDTs had a sensitivity ≥98% while 10 RDTs had a specificity ≥98% to anti-SARS-CoV-2 IgM antibodies. Three RDTs; Autobio, MultiG and Standard Q, had sensitivity and specificity of ≥98% to combined IgG/IgM. The RDTs that performed well also had perfect or almost perfect inter-reader agreement. This evaluation identified three RDTs with a sensitivity and specificity to IgM/IgG antibodies of ≥98% with the potential for widespread antibody testing in Uganda. Coronavirus disease 2019 was declared a pandemic by the World Health Organization (WHO) on March 11, 2020. Globally as of September 1 st , 2021, there have been nearly 218 million cases reported to WHO with over 4.5 million deaths while in Uganda 99,408 cases with over 3000 deaths have been recorded (WHO 2021) . Since this evaluation, the number of new cases and deaths continued to rise. Standard laboratory confirmation of severe acute respiratory syndrome-2 (SARS-COV-2) is based on the detection of unique viral sequences in nasopharyngeal samples by NAAT (WHO 2020b) . Although the priority intervention from a public health perspective is to identify those with acute infection and to quarantine them and their immediate contacts in order to control the spread of infection, it quickly became apparent that it is also important to identify convalescent cases through antibody testing in order to better understand the epidemiology of the virus and thereby to introduce effective control measures. Antibody testing has traditionally been conducted using enzyme-linked immunosorbent assays (ELISAs) or more recently with RDTs. ELISAs facilitate the testing of large numbers of specimens per run while RDTs are lateral-flow devices for individual specimens. RDTs typically give results in less than 30 minutes and are therefore ideal for use at the point-of-care (POC). During the early days of the pandemic, commercial ELISAs and RDTs were hard to come by and none were approved for use in Uganda. Most people infected with SARS-CoV-2 have an incubation period of 3 -7 days before the appearance of symptoms. IgM seroconversion occurs within 10 -14 days and IgG seroconversion within 12 -14 days after symptom onset (Long et al. 2020; Lou et al. 2020; To et al. 2020; Zhao et al. 2020 ) and can be detected in less than 40% of infected people within one week of symptom onset and in 100% by day 15 (Batra et al. 2020; Zhao et al. 2020) . Antibodies can take much longer to develop in those with sub-clinical or mild infection (WHO 2020b) . The strength of the antibody response depends on a number of factors including age, nutritional status and disease severity amongst others (WHO 2020a). IgM antibodies start to disappear by week 5 and by week 7 are no longer detectable -IgG antibodies persist beyond week 7 (Xiao et al. 2020) . It is not clear if antibodies confer immunity to re-infection though recurrence of COVID-19 illness appears to be very uncommon (CDC 2020a). As a result of global shortages of reagents for molecular testing, a number of groups (Lassaunière et al. 2020; Zhao et al. 2020 ) investigated the potential use of antibody tests, particularly those for IgM, either singly or in combination, to diagnose acute COVID-9 infection. However, a reliable diagnosis of infection by antibody testing is only possible in the recovery phase when the possibility of intervening has passed and consequently serological diagnosis is not recommended for informing clinical management or contact tracing(WHO 2020b). In Uganda as in many other countries, there is a sense of urgency to understand the epidemiology of the virus in order to implement effective control measures. This requires mass screening of the population for anti-SARS-CoV-2 antibodies to determine among other things; how many people have been infected with the virus and how this changes over time; the risk factors for infection such as age, ethnicity, domicile or underlying health issues; the proportion of infected people with mild or asymptomatic infection and how long antibodies can be detected in individuals who have been infected (CDC 2021). Many countries are now testing for SARS-CoV-2 antibodies at the population level or in specific groups, such as health workers, close contacts of known cases, or within households (WHO 2020c). The tools to conduct mass serological screening including ELISA and RDT kits with emergency use approval slowly became available in mid-2020 and plans were made to evaluate their performance. Laboratory tests, including RDTs, that detect antibodies to SARS-CoV-2 in people need validation to determine their accuracy and reliability. Inaccurate RDT results would have serious consequences and would affect pandemic control efforts. In Uganda, all new diagnostic assays that are introduced into the market must undergo in-country laboratory validation at the Uganda Virus Research Institute (UVRI), which is a designated WHO and Africa CDC SARS-CoV-2 reference laboratory, before being recommended to the Ministry of Health for use in the country. Both CDC and WHO also advise that diagnostic and antibody tests should be validated in appropriate populations and settings before they are recommended (CDC 2020b; WHO 2017) . At UVRI, the normal practice to validate new antibody test kits is to evaluate their performance on a reference panel of well-characterized plasma specimens. In mid-2020 no commercial nor WHO reference panels were available and consequently UVRI developed its own reference panel. Presumptive anti-SARS-CoV-2 antibody-positive specimens were selected from available qRT-PCR-confirmed, symptomatic and asymptomatic SARS-CoV-2 cases -the number of days post symptom onset was reported for some of the symptomatic cases. Presumptive SARS-CoV-2 antibodynegative specimens were selected from the UVRI repository of specimens collected during an HIV national serosurvey conducted in 2016, long before the COVID-19 pandemic. Specimens from qRT-PCR-confirmed SARS-CoV-2-positive cases were tested on the 6 assays below following the manufacturer's Information for Use (IFU) instructions; Architect SARS-CoV-2 IgG CMIA (nucleocapsid protein) Architect SARS-CoV-2 IgM CMIA (spike protein) Euroimmun Anti-SARS-CoV-2 ELISA (IgG) (spike protein) EDI Novel Coronavirus COVID-19 IgG ELISA (nucleocapsid protein) EDI Novel Coronavirus COVID-19 IgM ELISA (nucleocapsid protein) InBios SCoV-2 Detect IgM ELISA (spike protein) A positive sample was defined as a sample reactive on at least two IgG ELISA/CMIA targeting the spike protein (Euroimmun Anti-SARS CoV-2 ELISA-IgG) and the nucleocapsid protein (Architect SARS-CoV-2 IgG CMIA) and also reactive on at least two IgM ELISA/CMIA targeting the spike protein (InBios SCoV-2 Detect IgM ELISA and Architect SARS-CoV-2 IgM CMIA). EDI Novel Coronavirus COVID-19 IgG and IgM ELISA had low sensitivity and hence results with these assays were not considered during selection of specimens for the positive panel. A negative sample was defined as a sample, non-reactive on at least three of the four assays. One hundred samples with the above profile were included in the SARS-CoV-2negative reference panel (UVRI 2021). A total of 25 serological RDTs (Appendix B) were evaluated against the characterized serological reference panel. The antigen(s), spike and/or nucleocapsid protein, targeted by the RDTs was not disclosed in the IFUs for most of the RDTs. The majority of RDTs had a single reading window with a control line, an IgM test line and an IgG test line while one (Biocredit) had two separate cassettes. Two RDTs had a reading window for IgM/IgG (Sino Care and Wondfo), while one had reading windows for IgG and IgA/IgM and not IgM (Antai). The manufacturer's IFUs were followed, and results read by 2 technicians blinded to each-others results. The statistician conducting the data analysis was blinded to the RDT identity. Only concordant results between technicians were used to evaluate the performance of the RDTsinter-reader variability was also documented. The sensitivity was calculated as the number of specimens determined as positive by the two technicians for each RDT under evaluation divided by the number of specimens tested from the positive panel and expressed as a percentage. The specificity was calculated as the number of specimens determined as negative by the two technicians for each RDT under evaluation divided by the number of specimens tested from the negative panel and expressed as a percentage. The accuracy was calculated as the proportion of RDT test results that agreed with the panel source (positive and negative panels) and expressed as a percentage. The sensitivity, specificity, and accuracy calculations were performed using the proportion command in STATA 15 and confidence intervals produced with the Wilson score method (Newcombe 1998) . Sensitivity, specificity, and accuracy were also determined for combined IgM/IgG (either or both IgM and IgG). The observed proportion and level of agreement between the two technicians were generated using the Cohen's Kappa statistic. This was generated for each isotype (IgM The evaluation protocol was reviewed and approved by UVRI's Research Ethics Committee and the Uganda National Council for Science and Technology (UNCST). The panels were unlinked to personal identifiers and results could not be traced to individual patients. Consent to participate and to store samples for future use was also sought. Results by individual RDT performance are summarized (in alphabetical order) in Table 1 . Most RDTs showed poor performance with none showing both anti-SARS-CoV-2 IgM and IgG antibody sensitivity and specificity ≥98%. Many RDTs that performed well in at least one reading window showed good reactivity to anti-SARS-CoV-2 IgG antibodies with 10 having a sensitivity ≥98% while 13 had a specificity ≥98% (Table 1) . Only three RDTs had an anti-SARS-CoV-2 IgM antibody sensitivity ≥98% while ten RDTs had a specificity ≥98% for anti-SARS-CoV-2 IgM antibodies. Three RDTs; Autobio, MutltiG and Standard Q had a sensitivity and a specificity ≥98% to combined IgM/IgG. There were seven RDTs that had an accuracy ≥98% for anti-SARS-CoV-2 IgG antibodies with three (Boson, Standard Q and ViviaDiag) having an accuracy of 100%. There were four RDTs (Boson, MultiG, Standard Q, ViviaDiag) where both anti-SARS-CoV-2 IgG antibody sensitivities and specificities were ≥98%. There was almost perfect agreement between the two technicians for determination of IgG in 20/23 (87.0%) RDTs (where there was a reading window for IgG). Four of the RDTs had perfect agreement for IgG with Kappa statistic of 100% (Biocredit, BTNX, MultiG, Standard Q, ViviaDiag). Agreement in determining IgM was much lower with only 11/23 (47.8%) tests where there was almost perfect agreement (Table 2 ). The WHO continues to review the evidence on antibody responses to SARS-CoV-2 infection and has published guidance on adjusting public health and social measures for the next phase of the COVID-19 response (WHO 2019). The development of accurate RDTs for the diagnosis of anti-SARS-CoV-2 IgM and IgG antibodies will benefit epidemiological and surveillance studies in identifying past COVID-19 symptomatic and asymptomatic infections including those in 'hotspots'. This will serve as an aid in determining the extent of herd immunity, though for how long immunity will last, especially with the appearance of SARS-CoV-2 variants is not yet known (Aschwanden 2021) . Accurate RDTs detecting the relevant antibodies, will benefit vaccine studies in identifying SARS-CoV-2 vaccine responders and for how long one remains immune to the virus. This evaluation of 25 RDTs showed significant variation in performance emphasizing the need for more input in research and development in order to come up with more accurate tests. There were only four RDTs that had a sensitivity and specificity ≥98% for anti-SARS-CoV-2 IgG i.e. Boson, MultiG, Standard Q and ViviaDiag. Of these four RDTs, only Boson had a sensitivity ≥98% for anti-SARS-CoV-2 IgM (sensitivity was 100% and corresponding specificity was 87.0 for anti-SARS-CoV-2 IgM; There are other published evaluation reports that have similarly shown poor serological RDT performance compared to that reported by the manufacturer (Deeks et al. 2020; Jacobs et al. 2020; Vauloup-Fellous et al. 2021) . Since some of these other studies used samples from any qRT-PCR positive individuals, they show lower sensitivity in the first week post symptom onset with improved performance at later time points. Few studies have evaluated RDTs using samples taken beyond a month post symptom onset. Furthermore, there is limited information on the performance of these RDTs in asymptomatic participants (Deeks et al. 2020 ). While a serial or orthogonal (Xu et al. 2020 ) testing approach has been recommended for surveillance especially when using ELISAs (CDC 2020b), here we propose that parallel testing with RDTs could also be a viable approach. With that approach, two RDTs with ≥ 98% sensitivity and specificity could be used together with an equally accurate RDT as a tie-breaker for discrepant results. Our study evaluated only 25 RDTs but we are aware that many SARS-CoV-2 RDT kits are on market. With the well characterized panel of samples we now have and with WHO serological reference standards now available from the National Institute for Biological Standards and Control (NIBSC), validation of additional RDTs will be quicker. The study had some limitations. The selection of the RDTs to evaluate was dictated by what was availed to us by the local distributors in the country. Some distributors provided test kits for evaluation, less than the desired 150 samples. Another limitation was that we were not able to procure an IgM ELISA based on the nucleocapsid protein. Some of the ELlSA kits we procured did not perform well and hence results with these kits were excluded from the development of the serological reference panel. Subsequent to the completion of this evaluation, it was discovered that authorization for the use of some of the RDTs had been revoked and removed from the FDA EUA notification list as of 23.2.21. TL BO, handled the data and analysis. 2021) Five reasons why COVID herd immunity is probably impossible A comparative evaluation between the Abbott Panbio™ IgG/IgM rapid test device and Abbott Architect™ SARS CoV-2 IgG assay CDC's Diagnostic Test for COVID-19 Only and Supplies CDC (2020b) Interim guidelines for COVID-19 antibody testing in Clinical and Public Health Settings CDC (2021) COVID-19 Serology Surveillance Strategy Antibody tests for identification of current and past infection with SARS-CoV-2 Cochrane Database of Systematic Reviews IDSA (2020) IDSA COVID-19 Antibody Testing Primer Implementing COVID-19 (SARS-CoV-2) Rapid Diagnostic Tests in Sub-Saharan Africa: A Evaluation of nine commercial SARS-CoV-2 immunoassays medRxiv:2020 Antibody responses to SARS-CoV-2 in patients with COVID-19 Serology characteristics of SARS-CoV-2 infection since the exposure and post symptoms onset medRxiv:2020 Interrater reliability: the kappa statistic Two-sided confidence intervals for the single proportion: comparison of seven methods Temporal profiles of viral load in posterior oropharyngeal saliva samples and serum antibody responses during infection by SARS-CoV-2: an observational cohort study The Lancet UVRI (2021) COVID-19 Panel Raw Data Performance of 30 commercial SARS-CoV-2 serology assays in testing symptomatic COVID-19 patients European journal of clinical microbiology & infectious diseases : official publication of the Bulletin:A guide to aid the selection of diagnostic tests Considerations in adjusting public health and social measures in the context of COVID-19 WHO (2020a) Advice on the use of point-of-care immunodiagnostic tests for COVID-19 WHO (2020b) Diagnostic testing for SARS-CoV-2: interim guidance WHO (2020c) Unity Studies: Early Investigation Protocols WHO (2021) WHO Coronavirus (COVID-19) Dashboard Profile of specific antibodies to SARS-CoV-2: The first report Evaluation of Orthogonal Testing Algorithm for Detection of SARS-CoV-2 Antibody Responses to SARS-CoV-2 in Patients With Novel Coronavirus Disease We thank the individuals who provided the specimens including those at the different COVID-19 treatment centres. We acknowledge the Uganda Government funding to UVRI. The procurement of the SARS-CoV2 qRT-PCR kits was funded mainly through the WHO and Africa All authors declare no competing interests. All authors declare no competing interests. No specific funding was received from funding bodies in the commercial, public or not-for-profit sectors.