key: cord-1053342-t2a64gd7
authors: Ablona, Aidan; Falasinnu, Titilola; Irvine, Michael; Estcourt, Claudia; Flowers, Paul; Murti, Michelle; Gómez-Ramírez, Oralia; Fairley, Christopher K.; Mishra, Sharmistha; Burchell, Ann; Grennan, Troy; Gilbert, Mark
title: Validation of a Clinical Prediction Rule to Predict Asymptomatic Chlamydia and Gonorrhea Infections Among Internet-Based Testers
date: 2020-12-14
journal: Sex Transm Dis
DOI: 10.1097/olq.0000000000001340
sha: 184971e3dbe7178e49de52a3fbaeea24fbe5d7bd
doc_id: 1053342
cord_uid: t2a64gd7

BACKGROUND: Clinical prediction rules (CPRs) can be used in sexually transmitted infection (STI) testing environments to prioritize individuals at the highest risk of infection and optimize resource allocation. We previously derived a CPR to predict asymptomatic chlamydia and/or gonorrhea (CT/NG) infection among women and heterosexual men at in-person STI clinics based on 5 predictors. Population differences between clinic-based and Internet-based testers may limit the tool's application across settings. The primary objective of this study was to assess the validity, sensitivity, and overall performance of this CPR within an Internet-based testing environment (GetCheckedOnline.com). METHODS: We analyzed GetCheckedOnline online risk assessment and laboratory data from October 2015 to June 2019. We compared the STI clinic population used for CPR derivation (data previously published) and the GetCheckedOnline validation population using χ(2) tests. Calibration and discrimination were assessed using the Hosmer-Lemeshow goodness-of-fit test and the area under the receiver operating curve, respectively. Sensitivity and the fraction of total screening tests offered were quantified for CPR-predicted risk scores. RESULTS: Asymptomatic CT/NG infection prevalence in the GetCheckedOnline population (n = 5478) was higher than in the STI clinic population (n = 10,437; 2.4% vs. 1.8%, P = 0.007). When applied to GetCheckedOnline, the CPR had reasonable calibration (Hosmer-Lemeshow, P = 0.90) and discrimination (area under the receiver operating characteristic, 0.64). By screening only individuals with total risk scores ≥4, we would detect 97% of infections and reduce screening by 14%. CONCLUSIONS: The application of an existing CPR to detect asymptomatic CT/NG infection is valid within an Internet-based STI testing environment. Clinical prediction rules applied online can reduce unnecessary STI testing and optimize resource allocation within publicly funded health systems.

was recommended as a composite outcome, as clinically both infections are typically tested for in the same sample. Recommending screening only for clients with a minimum CPR-predicted risk score of 6 maintained a 91% sensitivity in detecting infections (168/184 infections detected) while reducing potentially unnecessary tests by 32% (3361/10,437 tests not needed).

Internet-based STI testing services have expanded in recent years. 11, 12 Notably, the COVID-19 pandemic has led to significant closures of publicly funded sexual health clinics, leading to the necessary expansion of Internet-based services in response to restricted access to in-person care. 13, 14 The application of tools like CPRs to these settings is an emerging area of interest. 15, 16 Existing CPRs that have been developed within in-person clinic settings may not be directly applicable to Internet-based testing services because of population differences among those accessing either setting. 17, 18 Thus, validation studies applying existing CPRs to Internet-based testing environments are necessary before their implementation. GetCheckedOnline, an Internet-based testing service for STIs and blood-borne infections in BC, Canada, currently uses a universal approach for CT/NG screening, wherein all individuals are routinely recommended CT/NG nucleic acid amplification test (NAAT) urine screening, regardless of reported risk. 19 The primary objective of this study was to assess the validity, sensitivity, and overall performance of our existing clinic-based CPR for recommending CT/NG screening within the GetCheckedOnline testing environment.

The TRIPOD checklist for reporting the validation of a multivariable prediction model for individual prognosis or diagnosis was used to inform the description of our methods. 20 

GetCheckedOnline is available in 6 cities across the province of BC, Canada, with approximately 1000 STI tests completed monthly and an overall test positivity rate of 6% (oral communication with Heather Pedersen, Online Sexual Health Services Manager, BC Centre for Disease Control, December 2019). Use of GetCheckedOnline is free of charge and confidential, with formal enrollment in a publicly funded health program not required. The model for testing is described elsewhere. 19 In brief, clients must first create an account online and provide demographic information, including mandatory provision of gender and date of birth (used to derive age), with optional provision of race/ethnicity. GetCheckedOnline assesses gender using sex and gender terms interchangeably (e.g., "man/male" and "woman/female"); for clarity, "men" and "women" are used herein. To initiate a test episode, clients complete a 14-question risk assessment questionnaire. This includes mandatory questions about the gender of sex partners, the presence of STI-related symptoms, and having a sex partner (s) with a diagnosed STI, in addition to optional questions about types of sex (past 3 months), number of sex partners (past 3 months), previous chlamydia diagnosis (past year), and previous gonorrhea diagnosis (past year). Recommendations for any urine, oropharyngeal, and rectal NAAT for chlamydia and gonorrhea, as well as serology for syphilis, HIV, and hepatitis C, are tailored based on risk assessment questionnaire responses. All women and heterosexual men, categorized at a test episode level by sexual behavior, are automatically recommended chlamydia and gonorrhea urine NAAT, with additional rectal swabs recommended for clients who report being a receptive anal sex partner. GetCheckedOnline clients download their laboratory requisition forms to bring directly to participating laboratory locations for specimen collection. All STI testing is conducted centrally by the provincial Public Health Laboratory, and results are recorded in each client's electronic medical record; results are provided online (if negative) or by telephone (if positive or indeterminate). Compared with clients of in-person sexual health services, GetCheckedOnline clients are more likely to be older, identify as a sexual minority, and have had previous experience accessing STI testing services. 18 

The existing clinic-based CPR used in this study predicts CT/NG infection among asymptomatic women and heterosexual men (excluding transgender individuals because of low sample sizes). 9 Clinician-recorded predictor variables were extracted from electronic medical records of STI clinic attendees in Vancouver, BC, Canada, from 2000 to 2006. Multivariable logistic regression modeling was used to quantify relationships between predictors and CT/NG infection. The CPR assigns numeric scores to each test episode, with total risk scores ranging from −2 (negligible risk) to 27 (very high risk; Table 1 ). CT/NG screening was recommended by authors for clinic attendees with a minimum total risk score of 6, as this detected CT/NG infection with 91% sensitivity while reducing the fraction screened by 32%, close to the recommended performance benchmarks of 90% sensitivity and 40% reduction in fraction screened. 21 The area under the receiver operating characteristic curve (AUC) was 0.74 (95% confidence interval [CI], 0.70-0.77). 9

We extracted clinical data from the GetCheckedOnline program database and laboratory data from electronic medical records between October 2015 and June 2019. Client demographics, risk assessment questionnaire responses, and testing results were linked at a test episode level using a unique identifier. We included all CT/NG test episodes (urine and/or swabs) completed within the study period among clients who were at least 14 years old. We excluded test episodes completed by nonheterosexual men or men who did not report the gender of their sexual partners, and by clients who identified as transgender. Test episodes where clients reported experiencing STI-related symptoms or contact with a partner with a diagnosed STI were excluded because testing would be clinically indicated, obviating the need to apply a CPR. Our primary outcome of interest was CT/NG infection diagnosed within a GetCheckedOnline test episode. Risk of infection was predicted by the clinic-based CPR. Responses for the 5 CPR predictor variables, captured in client demographics and risk assessment questionnaires, were categorized to match those in the CPR. Variables were considered to have missing data if no response was provided or if clients selected "Prefer not to answer," "Do not know," or "Not applicable." A complete case analysis was conducted, excluding any testing episodes with missing data for any of the 5 CPR predictor variables, with the justification that, in implementation, any testing episode with missing data would be recommended automatically for CT/NG testing within GetCheckedOnline.

We conducted bivariate analyses to compare CT/NG infection prevalence and the distribution of CPR predictor variables between the STI clinic derivation population (data previously published) and the GetCheckedOnline validation population using χ 2 tests, significant at α < 0.01. We described CT/NG infection prevalence for each predictor variable and calculated univariate odds ratios (ORs; with 95% CI) to assess the direction and magnitude of association between predictor variables and CT/NG infection. Predicted probabilities of CT/NG infection were calculated per test episode by assigning a score for each predictor variable according to the CPR and calculating a total risk score (see CPR scoring in Table 1 ). We assessed calibration by conducting the Hosmer-Lemeshow goodness-of-fit test using the score as its discretization, with P < 0.05 indicating poor model fit, and by describing CT/NG prevalence within total risk score categories. The model's ability to discriminate between CT/NG-positive and CT/ NG-negative episodes was assessed by calculating the AUC, with 95% CI estimated using bootstrap samples on 1000 resamples of the data. Sensitivity, specificity, the fraction of clients screened, and the positive predictive value were quantified. Cutoff scores meeting the same performance benchmarks as used in the CPR derivation (≥90% sensitivity) were determined.

We conducted sensitivity analyses to explore the impact of excluding missing data from the 5 CPR predictor variables. Missing data were first described per CPR predictor variable and then assessed for associations with CT/NG infection. We then imputed missing data using Multiple Imputation with Chained Equations using the mice package in R. 22 Furthermore, CPRs may be modified or updated with the addition or removal of predictor variables to improve performance in novel settings. 23 We updated the CPR by removing any predictor variables having different associations with the outcome within the GetCheckedOnline and STI clinic populations, keeping all other variables constant, and assessed the updated CPR's sensitivity, fraction screened, and AUC within the complete case data set. Absolute total risk scores per testing episode would change; however, relative scores were used to calculate the AUC. Analyses were performed using R version 3.5.2 

From October 2015 to June 2019, 14,967 CT/NG test episodes were completed using GetCheckedOnline. We excluded 4815 test episodes among behaviorally nonheterosexual men and transgender-identified individuals and further excluded 2603 test episodes where individuals reported symptoms or a partner with an STI. Of the resulting 7549 episodes among asymptomatic women and heterosexual men, 2071 episodes (27.4%) were excluded from the analysis because of missingness for any CPR predictor variable. A total of n = 5478 CT/NG testing episodes remained for our complete case analysis, representing 36.6% of all completed CT/NG testing episodes during the study period (Fig. 1) .

The overall prevalence of CT/NG infection was slightly higher in the GetCheckedOnline validation population (n = 5478; 2.4% prevalence) than in the STI clinic derivation population (n = 10,437; 1.8% prevalence), and this was significant (P = 0.007). The GetCheckedOnline testers included greater proportions of women, those aged 30 to 39 years, those who reported more than 3 recent sexual partners, and those without previous histories of either CT or NG diagnosis, but had a similar distribution by race/ethnicity ( Table 2) . CT/NG infection prevalence within levels of each CPR predictor variable also differed between populations ( Table 3 ). The direction of association between most predictors and CT/NG infection was the same in the GetCheckedOnline and STI clinic populations; however, the magnitude of these relationships differed. Notably, in the GetCheckedOnline population, there was no association between race/ethnicity and CT/NG infection (OR, 1.14; 95% CI, 0.76-1.67).

Total CPR-predicted risk scores for the GetCheckedOnline population ranged from −2 to 25. The CPR model fit the GetCheckedOnline data well (Hosmer-Lemeshow statistic = 3.43, P > 0.90), and similar to trends within the STI clinic population, CT/NG prevalence within the GetCheckedOnline population increased with increasing total risk score categories (Supplementary Fig. 1 , http://links.lww.com/OLQ/A591). The CPR had a reasonable discriminative ability between those with and without CT/ NG infection (AUC, 0.64; 95% CI, 0.60-0.69; Supplementary  Fig. 2 , http://links.lww.com/OLQ/A591). Applying the minimum cutoff score of 6 (recommended in the derivation study) to the GetCheckedOnline population resulted in a sensitivity of 82% while screening 66% of the population (Table 4 ). However, because a sensitivity of at least 90% is recommended, the screening threshold would need to be lowered to a minimum total risk score of 4 or greater, which would yield a sensitivity of 97% and a greater proportion of clients screened (86%). The prevalence of CT/NG infection among those eligible for screening at a risk score cutoff of 4 was 2.7% compared with 0.5% among those not recommended for screening.

Proportions of missing data in GetCheckedOnline varied by predictor variable, with ethnicity having the highest proportion of missing data (23.0%), followed by number of sexual partners in the past 3 months (3.9%), past-year chlamydia diagnosis (3.5%), and past-year gonorrhea diagnosis (3.5%). Prevalence of CT/NG infection among observations with missing data for any predictor variable was 2.2%. Missing data for any predictor variable were not associated with positivity (OR, 0.90; 95% CI, 0.63-1.25). In applying the CPR to a multiply imputed data set (n = 7549), the CPR Given that there was no association between race/ethnicity and CT/NG infection in the GetCheckedOnline population (Table 3) , an updated CPR excluding race/ethnicity as a predictor variable was also applied to the complete case data set. Compared with the performance of the original CPR, this modified CPR had a similar overall discrimination (AUC, 0.66; 95% CI, 0.62-0.71) and a similar sensitivity (94%) and fraction screened (83%) at a minimum cutoff score of 4 (Supplementary Table 1 , http://links.lww.com/OLQ/A591).

The application of an existing CPR to detect asymptomatic CT/NG infection is valid within an Internet-based STI testing environment. The distribution of predictor variables, overall CT/NG infection prevalence, and unadjusted associations between predictor variables and CT/NG infection differed between the GetCheckedOnline and STI clinic populations. The CPR had a lower discriminative ability when applied to the GetCheckedOnline validation population (AUC, 0.64) compared with the STI clinic derivation population (AUC, 0.74). To maintain a minimum sensitivity of ≥90%, testing would be reduced by 14%. Sensitivity analyses demonstrated little difference in the CPR's performance by proportions of missingness in data collected online.

Population differences in in-person, clinic-based, and Internet-based testers are important considerations comparing this CPR's performance across settings and time. In considering STI services specifically, population dynamics (e.g., shifts in incidence within specific sexual networks or population subgroups) may influence risk over time beyond individual-level factors used in a prediction tool created at one time point. Temporal differences in transmission and prevalence between the period of CPR derivation (2000) (2001) (2002) (2003) (2004) (2005) (2006) and validation (2015-2019) may explain performance differences; similar to our findings, a lower discrimination (AUC, 0.64) was observed when the CPR was validated in a clinic-based population at a later period. 9 Also similar to our findings, external validation of this CPR to STI clinic settings beyond Vancouver (the site of the CPR's derivation) demonstrated lower discriminative power (AUC, 0.69) within a higher CT/NG prevalence (5.3%) sample. 25 Regional differences in underlying CT/ NG prevalence, as can be observed within GetCheckedOnline's provincial scope, may have impacted the CPR's performance. 26 Although we observed differences in the distribution of gender between the STI clinic and GetCheckedOnline populations, discrimination on GetCheckedOnline was similar between women (AUC, 0.63; 95% CI, 0.54-0.69) and men (AUC, 0.64; 95% CI, 0.57-0.71; data not shown). The lower AUC may be explained by the nonassociation between non-White race/ethnicity and CT/NG infection, and CPR performance was similar when race/ethnicity was removed as a predictor variable. Given that GetCheckedOnline addresses physical, psychosocial, and sociocultural barriers in accessing in-person STI testing services, 18, 27 our findings support the notion that the population of Internet-based testers may be distinct from those testing in-person.

To our knowledge, this is the first study validating the use of a clinic-based tool online, with several important strengths and limitations. Our validation study uses a large sample size of Internet-based testers to identify population differences compared with clinic-based testers. Similar methods could be used to develop or validate existing tools specific to other priority populations, such as men who have sex with men, and for other STIs, such as syphilis. One limitation of our analyses may be outcome ascertainment bias and underestimating undiagnosed infections, particularly among women, as GetCheckedOnline did not offer oropharyngeal CT/NG swab testing for women during the study period. The complete case analysis may have introduced selection bias into our analytic sample, particularly when data are not missing completely at random. 28 However, predictor missingness being unassociated with CT/NG positivity and multiple imputation analyses demonstrated that missing responses may have been missing at random. Requiring mandatory responses to avoid missingness in GetCheckedOnline data may be unnecessary and may infringe upon service users' personal autonomy and right to skip or refuse answering questions. That being said, misclassification of sexual orientation by behavior was possible if men did not report the gender of their sexual partners, in which case they were excluded from this analysis and the number of behaviorally heterosexual men using GetCheckedOnline may be underestimated. We also recognize the limitation of using a combined CT/NG outcome, and this may limit generalizability; in future research, we will be assessing the utility of separate CPRs for these infections. Clinical prediction rules can be used as tools in Internet-based STI testing platforms for streamlining resource allocation through restricting testing to those at highest risk, reducing both client-and systems-level testing-related burdens. These tools could also be used as self-assessment educational tools to raise awareness of STI infection risk, or to direct clients to different types of STI supports, such as broader psychosocial and preventative interventions, based on predicted infection risk. Deriving a novel CPR using data from clients of Internet-based testing services may address performance differences and improve yields in reducing testing volumes, as variables not included in this CPR (e.g., condomless sex) may be relevant in predicting infections online. 15 However, missing data characteristic of online environments may limit the yield of applying CPRs because of the need for complete data at the time tests are recommended.

Further research is needed before determining whether or how to fully integrate CPRs into Internet-based testing services. For example, evaluating the clinical impact (such as earlier diagnosis) and cost-effectiveness of using this CPR in an online setting, compared with not applying the CPR, would help inform the use of these tools. 23 From the service user perspective, it will be important to study the acceptability of using CPRs to guide STI test recommendations, particularly for clients of Internet-based testing services, who may or may not be recommended testing that matches their own estimate of sexual risk or situation. This is especially pertinent given the absence of a direct clinical consultation in Internet-based testing and could lead to some clients answering assessment questions in ways that allow them to access tests they would otherwise be denied. Finally, we recognize the need for further research among individuals holding socially marginalized identities who may be impacted by these tools, including co-developing effective and appropriate messaging as to why testing may or may not be recommended. Before implementation, meaningful consideration is also necessary regarding racialized and transgender communities who are or are not represented in these data and broader health services. 29, 30 As Internet-based STI testing services become more integrated in publicly funded healthcare systems, it is critical to consider the translation of existing selective screening tools to these contexts. Our findings provide strong evidence of the validity of using a clinic-based CPR in optimizing Internet-based testing services such as GetCheckedOnline. However, further work is needed both in considering the practicalities of implementing tools like CPRs and in mitigating unintended consequences of their use within these services.

Chlamydia, gonorrhoea, trichomoniasis and syphilis: Global prevalence and incidence estimates

Improving care for sexually transmitted infections

US public sexually transmitted disease clinical services in an era of declining public health funding

A critical appraisal of risk models for predicting sexually transmitted infections

Deriving a clinical prediction rule to target sexual healthcare to women attending British General Practices

Prediction of Chlamydia trachomatis infection to facilitate selective screening on population and individual level: A cross-sectional study of a population-based screening programme

Clinical prediction rules

Performance and cost-effectiveness of selective screening criteria for Chlamydia trachomatis infection in women. Implications for a national chlamydia control strategy

Deriving and validating a risk estimation tool for screening asymptomatic chlamydia and gonorrhea

Screening and treatment of Chlamydia trachomatis infections

The eSexual Health Clinic system for management, prevention, and control of sexually transmitted infections: Exploratory studies in people testing for Chlamydia trachomatis

Internet-based self-sampling for Chlamydia trachomatis testing: A national evaluation in Sweden

The potential impact and availability of sexual health services during the COVID-19 pandemic

Joint submission by the British Association for Sexual Health and HIV (BASHH) and the British HIV Association (BHIVA). Health and Social Care Inquiry on delivering core NHS and care services during the pandemic and beyond

Assessing association between IWantTheKit risk quiz tool and sexually transmitted infection positivity in male users for sexually transmitted infection screening

The development of an online risk calculator for the prediction of future syphilis among a high-risk cohort of men who have sex with men and transgender women in Lima

A cohort study comparing rate of repeat testing for sexually transmitted and blood-borne infections between clients of an Internet-based testing programme and of sexually transmitted infection clinics in Vancouver

Differences in experiences of barriers to STI testing between clients of the Internet-based diagnostic testing service GetCheckedOnline.com and an STI clinic in Vancouver

Get checked... where? The development of a comprehensive, integrated Internet-based testing program for sexually transmitted and blood-borne infections in British Columbia

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement

Region X Infertility Prevention Project. Re-evaluating selective screening criteria for chlamydial infection among women in the U S Pacific Northwest

Multivariate imputation by chained equations in R

Risk prediction models: II. External validation, model updating, and impact assessment

R: A Language and Environment for Statistical Computing

A validation study of a clinical prediction rule for screening asymptomatic chlamydia and gonorrhoea infections among heterosexuals in British Columbia

Regional differences in chlamydia and gonorrhoeae positivity rate among heterosexual STI clinic visitors in the Netherlands: Contribution of client and regional characteristics as assessed by cross-sectional surveillance data

Qualitative analysis of the experiences of gay, bisexual and other men who have sex with men who use GetCheckedOnline.com: A comprehensive Internet-based diagnostic service for HIV and other STIs

Principled approaches to missing data in epidemiologic studies

I don't think this is theoretical; this is our lives": How erasure impacts health care for transgender people

Dissecting racial bias in an algorithm used to manage the health of populations