key: cord-0711359-01vo6q23
authors: Knottnerus, J. André; Tugwell, Peter
title: It's always about numerators and denominators (N/D)
date: 2020-09-23
journal: J Clin Epidemiol
DOI: 10.1016/j.jclinepi.2020.09.013
sha: 5cb2a747c5ab924cd2f6e3aea2c521b5046c141b
doc_id: 711359
cord_uid: 01vo6q23

nan

It's always about numerators and denominators (N/D) A both fundamental and seemingly simple methodological requirement in clinical epidemiology, and more generally in research and care, is the adequate handling of numerators, denominators, and their interrelationship N/ D. This implies not just good arithmetic, but primarily conceptual understanding. This is self-evident in the analysis of quantitative data. Estimators of occurrence, such as frequencies and rates, can only be meaningfully calculated if the correct numerators are related to the correct denominators. And when analysing comparisons between groups expressed as, for example, relative risks or odds ratios, we need the appropriate, and in case of unbalance adjusted, numerators and denominators for each group. Also in N of 1 trials [1] the point is clear, with the nominators being events or scores during exposure versus non-exposure, and the denominators being time of exposure versus non-exposure, respectively.

But also in design development consistent 'N/D thinking' is essential to avoid bias. For example, in effectiveness or etiological studies this implies ensuring that, when comparing the occurrence of events in two or more groups, the denominators are comparable, for example, by randomization in trials or adjustment in observational studies. In addition, N/D thinking is crucial is designing interventions and measurements and in preventing and handling missing data: is there any indication that one or more of such building blocks of good research is at risk of causing selection or observation bias by affecting the comparison, and how can this be addressed?

Not always recognized is the importance of N/D thinking in (more) qualitative clinical research. Also this is basically studying numerators (certain characteristics or perceptions of individuals) in relation to denominators (those who are included in the full sample; already published samples; oremore implicitlyeprevious clinical experience). Also here the question must be asked to what extent N/D results could be influenced by the way the study is designed.

In healthcare, N/D awareness can help physicians understand that differences in clinical results can sometimes be explained by differences between practice populations, or recognize that patients who return to say they feel better may not be representative of the full denominator of the patients treated. N/D thinking can also help understand patients' misconceptions of proportions, and making clear to them, for example, that hearing at age 55 about having a 2/3 risk of myocardial infarction does not refer to today's risk but to the average risk throughout further life [2] . Such insights are fundamental for health communication with patients and the public [3] . They are also highly relevant for (educating) public media or politicians, that often have difficulty in correctly presenting N/D issues. A topical example is talking about the most affected country in the COVID-19 pandemic without mentioning whether this applies to the absolute number of cases or deaths, incidence rate, or death rate, which of course makes a huge difference [4] .

A basic N/D issue in research is whether, in studying phenomena of interest, representative denominators are used. This was addressed by Antequera c.s., who, in a bibliometric study, assessed female (under)representation in primary studies underpinning clinical guideline recommendations and systematic reviews for sepsis treatment. They found that female participation in these primary studies is below their representation in the sepsis population. ''Sex'' and ''gender'' terminology was used properly in less than half of studies used, and no more than around a fifth of studies reported by sex. According to the authors, sex-based participation disparities and lack of sex-related analyses and reporting limit the generalizability of research findings and hamper the external validity of the effectiveness of interventions. They make recommendations to all stakeholders in research to report disaggregated data, discuss the influence of sex and gender on research findings, and address diversity of patient populations.

A good example of the need to make an appropriate match between numerators and denominators is subgroup analysis, which is intended, in the end, to better reflect individual patients in practice. In a systematic review Yu et al. evaluated the degree of personalization of benefit and harm results of RCTs of pharmacological therapy published in the McMaster Premium LiteratUre Service (PLUS) database [5] , with the proportion of trials reporting subgroup analyses of a combined benefit-harm outcome as primary outcome and the proportion of trials reporting subgroup analyses or clinical prediction guides for benefits or harms as secondary outcomes. The authors found that very few RCTs reported a combined benefit-harm outcome, and only one included a clinical prediction guide. While in aImost 60% of the RCTs subgroup analyses were performed, their overall methodological quality was low. It was concluded that, despite great interest in the personalization of treatments, this is rarely reported in high-profile trials, and that there is a need for methods development for personalization of research results. Webster-Clark c.s., considering that effect estimates from prespecified subgroups may not apply to corresponding subgroups in the source population, studied whether systematic or structural sources of misleading subgroup estimates could play a role here by using directed acyclic graphs to evaluate selection bias. They also provide a hypothetical example illustrating how erroneous conclusions can result, and a tool for readers to explore additional cases. According to the authors it can be misleading to assume that subgroups within a trial are random samples of corresponding subgroups in the wider population. They recommend that researchers examine multiple potential modifiers of treatment effect at once, rather than conduct multiple crude subgroup analyses, to identify heterogeneous treatment effects. A form of very focused subgroup analysis, evaluated by Swift and co-authors, is relating prognosis to tumor heterogeneity (TH) in order to promote precision medicine. They conducted a systematic review to identify guidelines to assist systematic reviewers or clinical researchers in identifying sampling bias due to tumor heterogeneity, and performed a post hoc analysis of prostate tumor somatic mutation data from a previous systematic review to evaluate reporting on TH. The investigators found no formal guidelines, and methods or results to address tumor heterogeneity are insufficiently reported by most authors. They conclude that formal guidelines are required to help evidence-based decision makers to understand TH-related sample bias, and that primary authors need to clearly report tissue sample pathology methods, results, and tumor purity.

Another N/D challenge is whether a comprehensive or representative set of primary studies is retrieved in systematic reviews or meta-analyses. Gurung and co-authors investigated the performance of the Emtree term 'diagnostic test accuracy [DTA] study' in Embase. For a random selection of studies they compared the DTA studies found by researchers with those that were identified by using this term. The Emtree term turned out to have a sensitivity of 42.4% and a specificity of 99.5%, while 30% of the studies were incorrectly labeled as being a DTA study. The conclusion was that the Emtree DTA label failed to find most of the DTA studies and is not helpful to retrieve DTA studies accurately. Page c.s. point at the enormous importance of synthesizing the continuous and rapidly expanding volume of research literature in relation to the COVID-19 outbreak. Rapid reviews [6] can inform policy making under urgent circumstances while living systematic reviews [7] will provide up-to-date evidence synthesis. To help identify, map, and synthesize the denominator of relevant research output as it emerges, online collaborative tools and virtual workspaces are crucial, but this faces major challenges such as: duplication of efforts leading to research waste, inefficiency in conducting research, and missing the opportunity to address important questions. To address these challenges, open science ensuring more collaborative, transparent, and rigorous research is crucial. The authors therefore elaborate and apply the principles of Open Synthesis to highlight its potential benefits, particularly in global crises such as the current COVID-19 pandemic. In line with this, Tricco et al. emphasize that for the conduct of rapid reviews the COVID-19 pandemic created unique challenges, including the urgency of the request from decision-makers, identification of and access to sources of evidence, extrapolation of results from indirect evidence, and wide dissemination of results. Also these authors make a plea for international coordination to reduce the risk of duplication, and to effectively use global collective evidence synthesis resources. They outline several relevant methodological challenges to the conduct of rapid reviews that have become apparent during the COVID-19 pandemic, using an 8-step framework that follows the knowledge synthesis process and is also useful for future research.

DeMets and Fleming make clear that independent oversight of (the denominator of) the numerous clinical trials being concurrently conducted, as provided by Data Monitoring Committees (DMCs), is particularly important in public health emergencies such as the coronavirus epidemic. One of the issues they address is concurrently conducted trials on closely related scientific questions in related clinical settings, often individually underpowered for safety and having separate DMCs. For such trials, processes should be implemented enabling the DMCs to share with each other emerging confidential evidence to better assess risks and benefits. As the authors say, ideally a single DMC would monitor a portfolio of clinical trials or a trial with multiple arms, such as a platform trial [8] .

In a commentary on the methodological challenges regarding the testing of COVID-19 tests for identification of infected persons and for revealing a previous infection, Bossuyt addresses several key problems, such as: the absence of an independent reference standard for the clinical evaluation of tests for viral RNA, lack of rigor in the recruitment of study participants, insufficiently informative study reports with failure to adhere to reporting guidelines, difficulty to assess the applicability of study findings, and single estimates of clinical sensitivity despite relevant heterogeneity. He calls for action, in collaboration between clinical epidemiologists, clinical researchers, and laboratory medicine experts, and makes recommendations to address the identified challenges. Among these N/D issues are prominent, such as the requirement that participants and samples should be collected in the target population and that test performance should not be summarized in a single number, as that can be misleadingly ignore the variability in relation to, for example, target population and setting.

In an overarching commentary on evidence-based medicine in times of crisis, Djulbegovic and Guyatt stress that, in the face of the COVID-19 pandemic threat and the related challenges in research, society and policy making, applying evidence-based medicine (EBM) and associated GRADE [9] principles is crucial. This is in fact a matter of differentiating the full denominator of available studies according to quality, as GRADE classifies evidence from high quality to very low quality with categories of moderate and low in-between. When, in response to COVID-19, making recommendations regarding issues such as use of masks or hydroxychloroquine, implementing interventions based on high-quality evidence of substantial effectiveness can be expected to result in net benefit, whereas adopting interventions based on very low quality evidence runs a high risk of net harm. Indeed, as these authors write, 'appropriate application of EBM and GRADE is never more important than in times of health crisis affecting millions of people'.

A clinician's guide for conducting randomized trials in individual patients

Sex differences in lifetime risk and first manifestation of cardiovascular disease: prospective population based cohort study

Numeracy, ratio bias, and denominator neglect in judgments of risk and probability

COVID-19 dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU). Available at

Premium literature Service (PLUS): an evidence-based medicine information service delivered on the web

The art and science of knowledge synthesis

Living systematic review: 1. Introduction-the why, what, when, and how

An overview of platform trials with a checklist for clinical readers

GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology

Peter Tugwell E-mail address: anneke.germeraad@maastrichtuniversity.nl (J.A. Knottnerus)