key: cord-278297-twiye6jv
authors: Lourenco, Stella F.; Tasimi, Arber
title: No Participant Left Behind: Conducting Science During COVID-19
date: 2020-05-11
journal: Trends Cogn Sci
DOI: 10.1016/j.tics.2020.05.003
sha: 
doc_id: 278297
cord_uid: twiye6jv

Abstract Cognitive scientists have ramped up online testing in response to the coronavirus pandemic. Although research conducted online solves the problem of data collection, a lack of internet access among low-income and minority communities may reduce the diversity of study samples and, thus, impact the generalizability of scientific findings.

When her children's school district closed its doors to curb the spread of the coronavirus, Tamara Solis had to make a difficult decision [1] . She could pay for internet access so that her kids could complete their virtual coursework, but doing so would limit her ability to afford food and rent.

The COVID-19 pandemic has upended life in more ways than one. People like Tamara Solis, for example, are being confronted with the harsh reality of how to pay for food and housing in a time when millions and millions of people have lost their jobs [2] . Students graduating from college will not experience the ceremonial milestone they cherish. And cognitive scientists who study human populations have been forced to close their labs in order to uphold social distancing measures.

Among the challenges facing scientists today is how to collect the data needed for publications, grant submissions, dissertations, promotion materials, and so on. Online testing offers a potential solution to this problem, even for fields like developmental psychology where in-person testing is paramount. Indeed, many child development laboratories are going online for the first time, a shift that is being facilitated by resources such as the Parent and Researcher Collaborative (https://childrenhelpingscience.com), a single, crowd-sourced platform where researchers from different labs can post their studies for families to participate in. And as more and more cognitive scientists are conducting their research online, we find ourselves wondering who is (and isn't) participating in this research.

Online testing offers numerous benefits--it is fast, efficient, and inexpensive [3] . One might even argue that platforms such as Amazon Mechanical Turk (MTurk) and TurkPrime have revolutionized behavioral data collection. As excited as we are about the promises of online testing (e.g., in fields like developmental psychology where data collection is typically slow and expensive), we are also concerned about how the demographics of online participants may shift during COVID-19. In particular, we worry that online testing may reduce the diversity of participants--especially those from low-income and minority households-whose participation in scientific research has been essential in understanding all sorts of phenomena, from language proficiency [4] and spatial reasoning [5] to academic achievement [6] and brain development [7] .

At first glance, such a concern may seem unwarranted. After all, studies conducted online seem to be more diverse than studies conducted in laboratories [3, [8] [9] . In fact, with respect to household income, some estimates suggest a greater number of lower-income workers on MTurk than expected given the general population of workers [10] . Following this point, the monetary compensation associated with these studies could very well increase the diversity of participants, especially given that Turkers participate in numerous online experiments to supplement their income [10, 11] . So why are we worried about less diverse online samples during an economic downturn? J o u r n a l P r e -p r o o f Participation in online studies requires internet access, a staple among many, though not all, households. In the U.S., there is a well-known 'digital divide', in which Americans in rural and poor communities are less likely to have internet in their homes than urban and more affluent communities [12] . The Federal Communications Commission estimates that more than 21 million Americans do not have broadband connection with download speeds of at least 25 megabits per second, and nearly half of all households making less than $30,000 per year have no connection whatsoever. Moreover, Black and Hispanic Americans lag behind their White counterparts in internet adoption even when controlling for income. Nevertheless, in normal times, individuals without home internet access might be able to get online in public places such as schools, libraries, coffee shops, fast food restaurants, and even parking lots.

But these are not normal times. Amenities such as the internet may become a luxury as more and more people continue to lose their jobs and struggle to pay their bills. What is more, people who would normally gain internet access in public places may now be unable to do so because of stay-at-home orders and/or fear of contracting the coronavirus. Accordingly, it is possible that participants in online studies may now consist of those who can afford to weather the economic storm and/or have the time to participate in research studies given new demands on family care while working from home. Thus, the push to online testing at this time may exclude a large segment of the population, thereby limiting the generalizability of findings and posing challenges to reproducibility.

To be clear, we are not claiming that online testing should stop. If anything, we commend our colleagues for the dedication and generosity they have shown in promoting resources for online testing (especially within our own field of developmental psychology where such resources were not widely available). At the same time, we think the time is now ripe for a discussion about how science can progress in a way that is mindful of access and inequality, a problem that strikes us as particularly acute these days.

There are several steps we can take to ensure and promote the diversity of study samples. Below we sketch out some recommendations aimed at advancing demographic diversity during and after the COVID-19 pandemic. We recognize that this list is not exhaustive, and we also recognize that some of these recommendations will be difficult and costly to implement. Nevertheless, we hope that what follows serves as a launching pad for further consideration.

First, we recommend that studies conducted online collect and report detailed demographic information of study participants. One potentially efficient way to mandate demographic data collection is to set it as a default for all studies that are posted online. In this vein, we also encourage journals to require authors to report the characteristics of their study samples. For example, the Society for Research in Child Development (SRCD) enacted a sociocultural policy across all of its journals earlier this year, in January 2020. We recommend that other journals follow suit. Although the simple reporting of demographics J o u r n a l P r e -p r o o f will not itself increase diversity within samples, it nevertheless ensures transparency and promotes future reproducibility.

Second, we urge scientists to make efforts to provide temporary internet connection to low-income participants. There are mobile hotspots available for purchase, which could be mailed to participants or dropped off at their homes to allow for participation in online studies. We recognize that the financial burden here is likely to be disproportionate across researchers, some of whom will have funding to support such a move and some who may not.

Relatedly, as a longer-term goal that transcends the issue of online testing, scientists and grant-funding agencies might consider lobbying government for subsidies related to internet costs, and perhaps even advocate for universal availability of internet access, which is essential for living and operating in contemporary times. Some internet companies have already reduced prices for qualifying customers and have provided free access to online educational resources [13] . This is a promising start, but much more will be needed to ensure availability to those who need but cannot afford internet access.

Finally, we recommend that scientists who have the resources to implement off-site testing consider using such an approach to reach participants in low-income and minority communities. Some universities (e.g., University of Illinois at Chicago, University of Iowa, and Indiana University) have developed mobile laboratories with portable technology. These laboratories may be especially useful in the years to come, particularly if the necessary precautions are taken to ensure the safety of experimenters and participants, including the use of personal protective equipment (PPE) and regular disinfection of study materials.

Even if we succeed in flattening the curve, resurgences in infection are likely until (or if) a vaccine for the coronavirus can be developed. Thus, we may be on the precipice of a new normal, in which threats of disease may require long-term social distancing practices and may differentially impact those in low-income and minority communities. At a time when access to a large segment of the population may be hampered, we must not forget the value and necessity of diverse populations in uncovering the human experience in terms of its universality and its variability [14] [15] . After all, this is what cognitive science is all about.

Getting free internet is hard for poor students despite provider promises, survey finds. The Los Angeles Times

Nowhere to hide' as unemployment permeates the economy. The New York Times

An evaluation of Amazon's Mechanical Turk, its rapid rise, and its effective use

Language learning, socioeconomic status, and child-directed speech

Socioeconomic status modifies the sex difference in spatial skill

Socioeconomic status and child development

The neuroscience of socioeconomic status: Correlates, causes, and consequences

Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing

Lookit (part 1): A new online platform for developmental research

Research in the crowdsourcing age, a case study

Social heuristics shape intuitive cooperation

The Digital Divide

To fight coronavirus, millions more Americans need internet access. The New York Times

The weirdest people in the world?

The persistent sampling bias in developmental psychology: A call to action