key: cord-0058395-s4dv2fok
authors: Verma, Nitin; Shiroma, Kristina; Rich, Kate; Fleischmann, Kenneth R.; Xie, Bo; Lee, Min Kyung
title: Conducting Quantitative Research with Hard-To-Reach-Online Populations: Using Prime Panels to Rapidly Survey Older Adults During a Pandemic
date: 2021-02-17
journal: Diversity, Divergence, Dialogue
DOI: 10.1007/978-3-030-71305-8_32
sha: 2dd3ba9d6d49fe545a6cd5eab97beccbc4740bfa
doc_id: 58395
cord_uid: s4dv2fok

Vulnerable populations (e.g., older adults) can be hard to reach online. During a pandemic like COVID-19 when much research data collection must be conducted online only, these populations risk being further underrepresented. This paper explores methodological strategies for rigorous, efficient survey research with a large number of older adults online, focusing on (1) the design of a survey instrument both comprehensible and usable by older adults, (2) rapid collection (within hours) of data from a large number of older adults, and (3) validation of data using attention checks, independent validation of age, and detection of careless responses to ensure data quality. These methodological strategies have important implications for the inclusion of older adults in online research.

During the COVID-19 pandemic, online data collection has become crucial to safeguard the health of participants, researchers, and society at large. However, online data collection presents challenges for researchers who study hard-to-reach populations such as older adults (65 + years). Traditional sampling methods such as using university subject pools or crowdsourcing platforms (e.g., Amazon's Mechanical Turk (MTurk)) are of limited use in reaching older adults [1, 2] . Another challenge is to design online studies that are comprehensible and usable by older adults.

In this paper, we outline methodological strategies for an online survey study with a large sample of older and younger adults recruited using Prime Panels. This paper is part of a multiphase research project funded by the National Science Foundation (NSF) that aims to explore factors that influence both younger and older adults' trust in public health information during a pandemic. Empirical data from our NSF project will be reported elsewhere. The strategies outlined here have implications for researchers who study hard-to-reach-online populations such as older adults.

As the population and proportion of older adults continue to grow in the U.S., it is increasingly important to collect data that represent this growing population properly [3] [4] [5] . In traditional in-person studies, researchers rely on relationships with community partners (e.g., senior centers, healthcare organizations, public libraries) to recruit older adults for data collection [6] [7] [8] [9] . Given the high cost of personnel and financial resources to sustain such recruitment, researchers need alternative methods. Additionally, the COVID-19 pandemic has disrupted traditional in-person data collection, making online data collection ever more prominent [10] .

In the last decade, crowdsourcing platforms such as MTurk have mitigated some recruitment challenges [11] and driven down the overall cost of conducting online research. While online crowdsourcing has its merits [12] , recruiting older adults on crowdsourcing platforms presents many challenges. For example, a research cohort of participants aged 55-and-older is considered a "Premium Qualification" on MTurk and therefore incurs a higher cost than a cohort of younger adults [1, 13] . Also, the availability of older adult crowdworkers may be hindered by factors such as a lack of awareness of the platforms, a lack of access to technology, insufficient technological skills, and a lack of motivation [14] [15] [16] .

The constant churn of the participant pool in crowdsource markets such as MTurk [17] also makes it hard to generalize research outcomes. Although MTurk and similar platforms help overcome the homogeneity of university participant pools, they tend to yield samples significantly younger and less racially diverse than the American population [2] . This limitation is exacerbated by the fact that Amazon does not publish an age-based breakdown of MTurk workers [18] , and that the number of MTurk workers available for a study at any given time is well below the number registered on the platform [19] [20] [21] . The relatively small number of active MTurk workers increases the chance that they are familiar with the methods that researchers typically use to ensure data quality [1] .

An overarching concern with online research is that recruitment on crowdsourcing platforms may be undermined by bots or malicious actors misrepresenting age, location, or other characteristics, in an effort to obtain tasks and payments [18, 22] . In addition to these concerns, survey studies on online crowdsourcing platforms are vulnerable to response satisficing, with participants paying inadequate attention while they take surveys, thus threatening data validity [23] . These challenges, intensified by the COVID-19 pandemic, motivated us to explore other avenues for online data collection that did not limit us to one monolithic platform such as MTurk. Our search led us to Prime Panels, an aggregation of online research panels maintained by CloudResearch 1 that has been found to be better at approximating national probability samples than MTurk as reported in previously published comparisons [1] .

This study was approved by the Institutional Review Board of The University of Texas at Austin. Informed consent was obtained prior to any data collection.

Our survey study was driven by the following overarching questions: What differences exist in the information behavior of older and younger adults with respect to , and what factors could explain those differences? We used Qualtrics to design and implement the survey, with two attention checks: one instructional manipulation check [25, 26] , and one repeated question to check for consistency of responses. At the end of the survey, we included demographic questions to collect participants' age, gender, race, educational attainment, and political beliefs.

We used a 5-point scale for each Likert-type item, with a visible numeric label for each point from 1 to 5 and provided descriptive labels only at endpoints. This strategy offers the following advantages: (i) the odd number of options avoids forcing neutral participants to choose a side [27, 28] ; (ii) the use of 5 points balances the information gained with the cognitive demands on respondents [28] ; and (iii) endpoint-only labeling reduces the cognitive load in comparison with labeling all gradations [29] . This strategy permits the use of statistical methods such as correlations and linear regression [28] . To comply with Web Content Accessibility Guidelines (WCAG 2.0) 2 and thus be inclusive of older adults who rely on assistive technologies (e.g., text-to-speech applications), we implemented all Likert-type items using drop-down menus instead of rows of radio buttons.

To ensure the clarity and readability of the survey instrument, we pilot-tested it over the phone with 3 older adults (2 females, 1 male) 3 recruited from participants of our prior studies. We conducted cognitive interviews, a commonly used method to enhance the quality of data collection [30] [31] [32] , to collect verbal feedback on the design of the instrument. One researcher guided the interview while another took detailed notes. Following each cognitive interview, after participants completed the survey on their computer or tablet, we asked them to re-enter the survey to retrospectively share their thoughts, experiences, and challenges through probing questions and think-aloud prompts. The main interviewer asked pre-scripted questions and, when necessary, unscripted probing questions to gather requisite feedback [31] . These cognitive interviews enabled us to identify and fix accessibility issues related to clicking fatigue and font legibility. Each interview lasted approximately 45 min. Each participant received a $20 Amazon gift card.

We deployed the survey using Prime Panels, which provided a price and feasibility estimate based on the approximate duration of the survey, the desired sample size, and demographic parameters. To reach our target of 500 valid responses split into 2 comparable group sizes for older (65 + years) and younger (18-64 years) adults, we conducted 5 batches of data collection between June 26 and July 20, 2020 punctuated by data validation procedures described below. For each batch we used the Prime Panels interface to exclude participants who had taken the survey in a previous batch.

To ensure that participants accurately reported their age, we requested Prime Panels to provide us the current age of each participant. The age data provided by Prime Panels is based on the participant's year of birth according to self-report at the time of signing up with one of Prime Panels' providers. Since we did not have access to the years of birth of our participants, we allowed for a difference of 1 year in each participant's age as reported in the 2 data sets. We removed duplicates and used the attention checks to discard invalid responses. To filter out careless responses [33] we used two criteria: (i) long-string responses: all items on a page with 5 or more items were marked with the same option; and (ii) response time: where participants took less than an average of 2 s per item on a page [33] [34] [35] .

Across 5 batches of data collection, we received responses from 669 participants: 272 older adults (40.7%) and 397 younger adults (59.3%; in each batch we got a few more responses than requested; and more responses from younger adults than those from older adults were excluded due to data quality concerns, specified below). Older adults took 10.9 min on average (SD = 12 min) to complete the survey, whereas younger adults took 10.2 min (SD = 21.4 min). The data collection cost $1,173. We paid Prime Panels for all responses even though we discarded some responses during data validation.

Of the 669 responses collected, 185 failed one or more of our validation criteria, leaving 484 responses in our final sample. Table 1 breaks down our data collection by batch, including the time elapsed and responses collected. Table 2 summarizes the number of invalid responses per criterion, broken down by age category. 

During the COVID-19 pandemic, the older adult population has been especially vulnerable. Compared with younger people, older adults are more likely to develop serious health conditions if infected by the virus, but they are less likely to obtain digital information and services [36, 37] . When data for research must be obtained online, older adults risk being further underrepresented, hindering subsequent decision making based on the data. In this paper, we have outlined methodological strategies for our recently completed online survey study with a large sample of older and younger adults. Our strategies were aimed to ensure the survey's readability and usability, obtain a large, stratified sample of both older adult and younger adults, and validate the data including participants' ages. Data collection via Prime Panels is far less expensive than the personnel and financial resources required by traditional, in-person studies [6] [7] [8] [9] . Thus, this platform offers a viable alternative not only to MTurk, but also to traditional data collection methods. Our initial request for 250 older adult participants cost $660 ($2.64 per participant). Our validation process required us to request additional participants, but the minimal cost per participant was a clear benefit. Indeed, one impressive finding of our study was the stronger performance among older adults with the validation, as 248 of the 272 responses from older adults satisfied all validation criteria (91.2%), while 236 of the 397 responses from younger adults satisfied all validation criteria (59.4%).

It is also important to note that it took very little time to collect data from a large number of older adults. As shown in Table 1 , across the 5 batches of data collection, it took less than 14 h to obtain valid responses from 250 older adult participants. As such, Prime Panels provides a rapid means to collect data from older adults compared to other methods. For example, in another research project conducted by our team during the summer of 2020, we conducted a telephone survey to gather data from 200 older adults. Data collection took over 3 months, with a cost of $4000 in participant compensation (this manuscript is available from the authors). Overall, our methodological strategies have significant implications for researchers currently working with online crowdsourcing platforms as well as researchers who have had to shift their data collection from in-person to online due to COVID-19 [10] .

One overarching concern with studying older adults online is that about a third of older adults in the U.S. lack internet access, and only two-fifths of older adults own smartphones [38] . Therefore, despite our rapid and cost-efficient data collection, our sample was likely skewed by technology adoption among older American adults, and thus arguably less representative of this population as a whole than for younger adults.

Moreover, crowdwork remains a largely unregulated part of the internet economy and is vulnerable to exploitative business practices often leading to unfair wages and low quality of work [39, 40] . In our prior studies, we have taken care to ensure that we always pay participants at least the U.S. national minimum wage [41] [42] [43] . However, for this study, our ability, as researchers, to control the compensation to our participants to meet our ethical standards was obfuscated by Prime Panels' inability to specify the exact payment received by participants [44] . As a result, it is unclear to us how much compensation our research participants had received, or what form the compensation was in (e.g., monetary compensation or donation to charity, as some crowdsourcing platforms have done [45, 46] ). All we were able to do in this circumstance was to ensure that our estimate of the time required was conservative, to ensure that the compensation would be on the higher end of what Prime Panels allows. However, the lack of knowledge about and control of compensation provided by Prime Panels is a limitation of their current implementation.

Given the circumstances of the COVID-19 pandemic and the subsequent move to online data collection, more work needs to be done to ensure access to populations traditionally hard to reach online, including, but not limited to, older adults. To facilitate this access, crowdsourcing platforms must be transparent about the demographic make-up of their respective worker populations. Also, to ensure the quality of collected data, crowdsource platforms and online panels should offer researchers a communication channel to allow them to ascertain the validity of demographic data while preserving participants' privacy. An underlying concern in recruiting hard-to-reach-online populations is the availability and reach of the internet in the U.S., as well as the computer literacy of the population of interest. It is, therefore, the responsibility of policymakers, crowdsourcing platforms, and researchers to ensure that hard-to-reach populations are represented in research. Such inclusion efforts will ensure that these populations are represented in research findings and, subsequently, policies developed using the research findings. Finally, these stakeholders need to work together to ensure that participants are compensated fairly for their participation in research.

Online panels in social science research: expanding sampling methods beyond Mechanical Turk

The demographic and political composition of mechanical Turk samples

An aging world: 2015. International Population Reports P95/16-1

Recruiting older adults as participants in applied social research: applying and evaluating approaches from clinical studies

Older adults and internet use

Rural dementia caregivers in Southwest Texas: an exploratory study of advance directives and end-of-life proxy decision making

Effects of an e-health literacy intervention for older adults

Improving older adults' e-health literacy through computer training using NIH online resources

Public library computer training for older adults to access high-quality Internet health information

Conducting research during the COVID-19 pandemic: advice from psychological researchers on protecting participants, animals and research plans

The viability of crowdsourcing for survey research

Use of rapid online surveys to assess people's perceptions during infectious disease outbreaks: a cross-sectional survey on COVID-19

Evaluation of pharmaceutical pictograms by older "turkers": a cross-sectional crowdsourced study

Why would anybody do this? Older adults' understanding of and experiences with crowd work

Using Amazon's mechanical Turk: benefits, drawbacks, and suggestions

Older adults and crowdsourcing: Android TV app for evaluating TEDx subtitle quality

Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations

How do older adults recruited using MTurk differ from those in a national probability sample?

Demographics and dynamics of mechanical Turk workers

Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the mechanical Turk participant pool

The average laboratory samples a population of 7,300 Amazon mechanical Turk workers

The shape of and solutions to the MTurk quality crisis

Preventing satisficing in online surveys: a "kaptcha" to ensure higher quality data

TurkPrime.com: a versatile crowdsourcing data acquisition platform for the behavioral sciences

No harm in checking: Using factual manipulation checks to assess attentiveness in experiments

Instructional manipulation checks: detecting satisficing to increase statistical power

Experiments with the middle response alternative in survey questions

The effect of rating scale format on response styles: the number of response categories and response category labels

Formulating questions about knowledge and attitudes

Research synthesis: The practice of cognitive interviewing

Cognitive interviewing: verbal data in the design and pretesting of questionnaires

Cognitive laboratory approach to designing questionnaires for surveys of the elderly

Identifying careless responses in survey data

Detecting and deterring insufficient effort responding to surveys

Methods for the detection of carelessly invalid responses in survey data

A double burden of exclusion? Digital and social exclusion of older adults in times of COVID-19

When going digital becomes a necessity: ensuring older adults' needs for information, services, and social inclusion during COVID-19

Tech adoption climbs among older adults

The internet is enabling a new kind of poorly paid hell. The Atlantic

Responsible research with crowds: pay crowdworkers at least minimum wage

The relationship between human values and attitudes toward the Park51 and nuclear power controversies

Human values and trust in scientific journals, the mainstream media and fake news

Understanding perception of algorithmic decisions: fairness, trust, and emotion in response to algorithmic management

One Survey at a Time. SurveyMonkey

Clear Outcomes. Americans' COVID-19 preventative practices in April and

Acknowledgements. This material is based upon work supported by the National Science Foundation under Grant No. 2027426. We thank John Bellquist, Ph.D., Editor of the Cain Center in the School of Nursing at The University of Texas at Austin, for his professional proofreading of an earlier draft of this manuscript; Le (Betty) Zhou, Ph.D., at Carlson School of Management, the University of Minnesota for helping us with data validation; and the anonymous participants of this study.