key: cord-0253781-lefpbmbo
authors: Ye, S.; Huang, B.; Sun, K.; Phi, H.; Ko, B.; Parsey, C. M.; Hosseini Ghomi, R.
title: BrainCheck: Validation of a Computerized Cognitive Test Battery for Detection of Mild Cognitive Impairment and Dementia
date: 2020-11-13
journal: nan
DOI: 10.1101/2020.11.10.20229369
sha: 9f3ec0d84bb193367d51bb85249346a5c5eff7db
doc_id: 253781
cord_uid: lefpbmbo

INTRODUCTION: Early detection of dementia is critical for intervention and care planning, but remains difficult. This study evaluated a computerized cognitive testing battery (BrainCheck) for its diagnostic accuracy and ability to distinguish the severity of cognitive impairment. METHODS: 99 participants diagnosed with Dementia, Mild Cognitive Impairment (MCI), or Normal Cognition (NC) completed the BrainCheck battery. Statistical analyses compared participants performance on BrainCheck based on their diagnosis group. RESULTS: BrainCheck battery performance showed significant differences between the NC, MCI, and Dementia groups, achieving >88% sensitivity/specificity for separating NC from Dementia, and >77% sensitivity/specificity in separating the MCI group from NC/Dementia groups. Three-group-classification found true positive rates >80% for the NC and Dementia groups and >60% for the MCI group. DISCUSSION: BrainCheck was able to distinguish between diagnoses of Dementia, MCI, and NC, providing a potentially reliable tool for early detection of cognitive impairment.

In proportion with the increasingly aging population, the incidence of dementia is on the rise; projected to affect nearly 14 million people in the United States and upwards of 152 million people globally in the coming decades [1] [2] [3] . Current rates of undetected dementia are reported as high as 61.7% [4] , and available treatments are limited to promoting quality of life rather than reversal or cure of the disease process. The ability to properly identify and treat dementia at this scope requires an active approach focused on early identification and intervention. Early detection of dementia provides access to timely interventions and knowledge to promote patient health and quality of life before symptoms become severe. Known treatment interventions help to address underlying conditions and minimize distressing symptoms, providing reason to implement these earlier on. Early and accurate diagnosis allows for proper preparation of patients, caregivers, and families which results in improved caregiver well-being and delayed institutionalization [5, 6] . Further, it helps to characterize early-stage dementia patients for clinical trials, exploring the latest therapeutics and validating biomarkers indicative of specific pathologies. Despite the benefits, early detection is a current challenge with current clinical protocols, leaving many patients undiagnosed until symptoms are more apparent in later stages of the illness.

Considered an early symptomatic clinical stage of dementia, mild cognitive impairment (MCI) signifies a level of cognitive impairment between normal cognition (NC) and dementia [7] . While not all MCI cases progress, the conversion rate of MCI to dementia has been observed to be approximately 5-10% [8] . This stresses the importance of MCI identification in the early detection and clinical intervention for dementia, included in recommendations from The National Institute on Aging and the Alzheimer's Association [9] . Detection of MCI has been fairly successful using brief cognitive screening assessments. The widely used Montreal Cognitive Assessment (MoCA) has demonstrated 83% sensitivity and 88% specificity in distinguishing MCI from NC its ability to distinguish dementia from MCI has demonstrated 90% sensitivity and 63% specificity [10, 11] . The Saint Louis University Mental Status (SLUMS) exam and the Mini-Mental State Examination (MMSE) also show the ability to distinguish impaired individuals. For example, the diagnostic accuracy of the MMSE has been reported to be 62% sensitivity and 87% specificity for distinguishing MCI from NC and 80% sensitivity and 85% specificity for distinguishing dementia from MCI [12, 13] . Similar performance has been observed with the SLUMS exam, reporting 94% sensitivity and 87% specificity for diagnosing dementia and 82% sensitivity and 86% specificity for diagnosing cognitive impairment [14] . While traditional 'paper and pen' based versions of these assessments have typically been used in standard practice, digitized or computerized versions are now available; however, they have been slow in adoption in clinical practice. Although these screening tests do well in their ability to detect MCI, they often are utilized to prompt an indication for further comprehensive testing to better specify patterns of cognitive impairment seen in neurological disorders [15] .

After a diagnosis of MCI, there is often a referral for neuropsychological testing (NPT); a comprehensive evaluation of cognitive domains (e.g., attention, working memory, language, visuospatial skills, executive functioning, memory), that are used to support clinical diagnoses and further delineate specific neurocognitive disorders. NPT can determine patterns of cognitive functioning that relate to normal aging, MCI, and dementia progression with specificities from 67 -99% [16] . A major strength of NPT is the ability to characterize cognitive impairments, providing clues to underlying pathology, and thereby improving diagnostic accuracy to guide appropriate treatment.

However, NPT comes with downsides including financial cost, long appointment times, and high levels of required training and expertise to conduct and interpret tests. To address these pitfalls and meet future demands, computerized and/or digitized NPT have also been developed to address issues of accessibility and efficiency [17, 18] .

Prior studies have shown that some computerized cognitive tests demonstrate high accuracy in differentiating dementia patients from healthy participants, but do not have adequate psychometrics to distinguish MCI from dementia [19] [20] [21] [22] . Ideally, computerized cognitive tests would aim to be more rapid and maximize accessibility to both patients and providers to address the expected uptick in dementia. This would also yield multiple benefits including maintaining testing standardization, alleviating time pressures of modern clinical practice, and providing a comprehensive assessment of cognitive function to strengthen a clinical diagnosis. Importantly, in the new era of practicing amidst the COVID-19 pandemic [23] , increasing the accessibility of remote cognitive testing for vulnerable/high-risk patients is essential.

This study evaluated BrainCheck, a computerized cognitive test battery previously validated for its diagnostic accuracy for the detection of concussion [24] and dementia-related cognitive decline [25] . BrainCheck is portable, allows for self and remote administration, and can be reimbursed by insurance plans in the US. Furthering its validation for dementia-related cognitive decline, we sought to assess BrainCheck's utility as a diagnostic aid to accurately distinguish the severity of cognitive impairment using existing clinical diagnoses of Normal Cognition (NC), Mild Cognitive Impairment (MCI), and Dementia. We compared BrainCheck scores to comprehensive clinical diagnosis (subjective cognitive impairment or NC, MCI, or Dementia). Our goal was to further demonstrate the utility of BrainCheck for cognitive assessment, specifically as a diagnostic aid in cases where NPT may be unavailable, or when a comprehensive evaluation is not indicated.

This study was approved by the Institutional Review Board of The University of Washington for human subject participation. Participants were recruited using the University of Washington Alzheimer's Disease Research Center's (UW ADRC) research registry [26] , a database of individuals willing to participate in dementia-related research. Those on the registry were contacted by phone or email provided within the registry. While there were no formal exclusionary criteria, those with listed addresses outside a 70-mile radius of Seattle, Washington were not contacted to participate. If the person was unable to physically use an iPad, too cognitively impaired to understand or follow instructions, or if the primary contact (e.g., spouse) indicated that the person was unable to participate, they were not recruited for the study. When study procedures were modified from in-person to remote administration due to the COVID-19 pandemic (approximately March 2020), participants outside of the initial geographical range were . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint

The copyright holder for this this version posted November 13, 2020. ;  contacted. We required that these participants had access to either an iPad with iOS 10 or later, or a touchscreen computer with Wi-Fi connectivity. A small number of participants were recruited via snowball sampling by participants who had completed the study.

Most participants from the research registry were former or current patients of a memory disorders clinic in Seattle, Washington where a recent clinical diagnosis by a physician had been made. Using their primary cognitive diagnosis by the memory clinic, participants were divided into one of three groups: 1) Normal Cognition (NC), indicated by subjective cognitive complaint or no diagnosis of cognitive impairment, 2) Mild Cognitive Impairment (MCI), representing both amnestic (aMCI) and non-amnestic (naMCI) subtypes, or 3) Dementia, which included dementia due to Alzheimer's Disease (AD), Frontotemporal dementia, Vascular dementia, Lewy body dementia, Mixed dementia, or Atypical AD, delineated by the clinical diagnosis. Participants who did not have a formal diagnosis and denied cognitive symptoms or a history of cognitive impairment were placed into the NC group. 

Data collection for on-site administration was collected from October 2019 to February 2020. A session was held either in the participant's home or in a public setting that was well-lit, quiet, and distraction-free. Consent forms were reviewed and signed by the participant, or legally authorized representative, and examiner, with both parties . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint

The copyright holder for this this version posted November 13, 2020. ;

obtaining a copy. The study was designed for participants to complete one session with a moderator using a provided iPad (Model MR7G3LL/A) connected to Wi-Fi to complete the BrainCheck battery. Prior to testing, participants were briefed on BrainCheck, and moderator guidance was limited to questions and assistance requested by the participant during the practice portions. Participants received a gift card for participation at the conclusion of the study session.

Due to the COVID-19 pandemic, study procedures were modified to accommodate stay-at-home orders in Washington state. Data collection resumed from April 2020 to May 2020, with modified procedures using remote administration. These participants provided written and verbal consent and were administered the BrainCheck battery remotely over a video call with the moderator. Participants used their personal iPads or touchscreen computer browser to complete the BrainCheck battery. The same method for on-site administration (described above) was used for remote administration.

A short description of each of the 5 assessments that make-up the BrainCheck battery (V4.0.0) is listed in Table 1 . More detailed descriptions may be found in a previous validation study [24] . After completion of the BrainCheck battery, the score of each assessment is calculated using assessment specific measurements by the BrainCheck software (shown in Table 1 ). The BrainCheck Overall Score is a single, cumulative score for the BrainCheck battery that represents general cognitive functioning. This score is calculated by taking the weighted average of all assessment scores. The normalized assessment scores and overall scores are corrected for participant age and device used (iPad versus computer) using the mean and standard deviation of the corresponding score from a normative database previously collected by BrainCheck.

The score generated follows a standard normal distribution, where a lower score indicates lower assessment performance and cognitive functioning.

. CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint

The copyright holder for this this version posted November 13, 2020. ; is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint

The copyright holder for this this version posted November 13, 2020. ; https://doi.org/10.1101/2020.11.10.20229369 doi: medRxiv preprint

To evaluate BrainCheck performance among participants in different diagnosis groups, calculations were generated to determine diagnostic sensitivity and specificity. In assessing BrainCheck for three group classification, we used an R package developed by Luo et al. [27] , to define optimal cutoffs for the BrainCheck Overall Score and find the which the word and font color are the same (e.g., the word RED presented in red font), and NON-CONGRUENT in which the word indicates a different color than the font (e.g., the word RED presented in green font). A time-out mechanism is triggered if there is not a completion of a trial in the assessment within 30 seconds. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint

The copyright holder for this this version posted November 13, 2020. ; maximum diagnostic accuracy. The analysis was performed using Python version 3.8.5 and R version 3.6.2 programming languages.

We found most participants in the NC group were able to complete the assessments whereas the Dementia group had a higher 'timeout' rate, with MCI falling in between the two (Figure 1) . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this this version posted November 13, 2020. ; https://doi.org/10.1101/2020.11.10.20229369 doi: medRxiv preprint

BrainCheck assessments were compared across the three groups. Individual scores, such as the BrainCheck Overall Score, were normalized for age and device. Overall, participants with greater cognitive impairment showed lower BrainCheck assessment scores. All individual assessments showed significant differences in performance between the NC and Dementia groups. Three of the seven assessments (Digit Symbol, Flanker, and Trails A) showed significant differences in performance between all three groups on post-hoc Dunn's test ( Table 2 ). The Immediate and Delayed Recognition tests showed significant differences ( p <0.05) in performance between the MCI and NC group, but there was not a significant difference between the MCI and Dementia group ( Table 2 ). The Stroop and Trails B tests showed significant differences between the NC and Dementia groups but could not significantly differentiate the MCI group from the other groups (Table 2) .

. CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this this version posted November 13, 2020. ; . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this this version posted November 13, 2020. ; https://doi.org/10.1101/2020.11.10.20229369 doi: medRxiv preprint scored significantly higher than the MCI and Dementia groups. The Dementia group demonstrated significantly lower scores than the other two groups (Figure 2 ). 

Using ROC analysis, Braincheck Overall Scores achieved a sensitivity of 94% and a specificity of 88% for classifying NC and Dementia participants (AUC=0.954); a sensitivity of 83% and a specificity of 86% for classifying between NC and MCI participants (AUC=0.842); and a sensitivity of 77% and a specificity of 83% for classifying between MCI and Dementia participants (AUC=0.785) (Figure 3 ).

. CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint

The copyright holder for this this version posted November 13, 2020. ; Using methods described by Luo and colleagues for three group classification [27] , the optimal lower and upper cutoffs of the normalized BrainCheck Overall Score in maximizing diagnostic accuracy were -3.64 and -0.06, respectively. This achieved true positive rates of 83% for the NC group, 64% for the MCI group, and 83% for the Dementia group (Figure 4) .

. CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint

The copyright holder for this this version posted November 13, 2020. ; https://doi.org/10.1101/2020.11.10.20229369 doi: medRxiv preprint 

Consistent with prior findings in concussion [24] and dementia/cognitive decline [25] samples, this study demonstrated that BrainCheck is consistent in its capability to detect cognitive impairment. Our study demonstrated that BrainCheck can detect MCI, and reliably differentiate between the severity of cognitive impairments in the diagnosis groups (NC, MCI, and Dementia). As expected, participants with more severe cognitive impairment performed worse across the individual assessments and on BrainCheck Overall Scores. The BrainCheck Overall Scores separated participants of different diagnostic groups successfully with high sensitivity and specificity.

. CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint

The copyright holder for this this version posted November 13, 2020. ; https://doi.org/10.1101/2020.11.10.20229369 doi: medRxiv preprint BrainCheck Overall Scores were more robust in distinguishing these groups where participants in the Dementia group had significantly lower scores than ones in the NC group. The BrainCheck battery was able to distinguish between NC and Dementia participants, with 94% sensitivity and 88% specificity. These findings show that the BrainCheck Overall Score demonstrates better accuracy for differentiating NC from Dementia, compared to the MMSE, SLUMS and MoCA screening measures [14, 15, 29 ] .

People with MCI usually experience fewer cognitive deficits and preserved functioning in activities of daily living compared to those with Dementia [ 30 ] , and our findings of sensitivity and specificity with separating MCI from other groups were slightly lower than the NC versus Dementia differentiations ( Figure 3) . Nonetheless, the BrainCheck Overall Score showed sensitivities and specificities >80% in distinguishing MCI from NC and Dementia groups, which is comparable to the MoCA, SLUMS, and MMSE [11] [12] [13] [14] [15] .

Furthermore, a review of validated computerized cognitive tests indicated AUC's ranging from 0.803-0.970 for detecting MCI, and AUC's of 0.98 and 0.99 in detecting dementia due to AD [28] , which were comparable with the results found in this study.

Although not all individual assessments in the BrainCheck battery could differentiate NC, MCI, and Dementia, we observed a general trend for each assessment showing that Dementia participants had the lowest scores whereas the NC participants had the highest scores. The individual assessments that did show significant differences in the scores between NC and MCI groups and between Dementia and MCI groups included the Digit Symbol, Flanker, and Trails A. Notably, in this study the BrainCheck Digit Symbol test showed significant differences in performance between all three diagnosis groups, whereas a previous study found that the BrainCheck Digit Symbol test did not show significant differences between cognitively healthy and cognitively impaired groups (n=18, p =0.29) [25] . This is likely due to the present study having a larger sample size. The individual assessments with no significant differences between the MCI group and the NC/Dementia groups were the BrainCheck Stroop and Trails B tests ( Table 2 ). Both of these tests include timeout mechanisms if participants are unable to complete the test, and time out rates were higher in the more cognitively impaired groups (Figure 1 ).

. CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint

The copyright holder for this this version posted November 13, 2020. ;  In comparison to comprehensive NPT, which can typically last a few hours and sometimes require multiple visits [30] , BrainCheck demonstrated shorter test duration, with average completion times of 18. This would make the distinguishing of NC from cognitive impairment more difficult, yet diagnostic accuracy among the groups still remains high. Furthermore, the mean number of days since the last clinical evaluation for NC participants was as high as 753 days. This could suggest that the NC participants did not feel an inclination to seek out further cognitive evaluation during the extended time period, and not have experienced noticeable cognitive decline. Future validity studies should ensure that a physician . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint

The copyright holder for this this version posted November 13, 2020. ; https://doi.org/10.1101/2020.11.10.20229369 doi: medRxiv preprint evaluation and diagnosis occur closer to the time of BrainCheck testing to address these limitations.

When administration type was considered, scores showed significant differences among all three groups for onsite administration but not for remote administration. While scores showed similar trends, the sample size of NC (n=6) and MCI (n=6) within the remote groups were fairly small. While remote administration was not designed within the original study, stay at home orders due to COVID-19 required modifications, and efforts were made to provide preliminary data for remote use and self-administration abilities.

With preliminary outcomes indicating feasibility for remote administration, a more robust study and increased sample size will be needed to fully validate BrainCheck's cognitive assessment via it's remote and self-administered feature.

The use of computerized cognitive tests is an opportunity to increase test accessibility for an aging population and an increasing presence of cognitive impairment. Remote test options with quick and accurate results could save on time and costs. Technological advances have allowed for these tools to obtain greater sensitivity and specificity as well [28] . Our findings in this study demonstrate that BrainCheck can distinguish between three different levels of cognitive impairment, NC, MCI, and Dementia. This study paves the way for a comprehensive longitudinal study, exploring BrainCheck in early detection of dementia and monitoring of cognitive symptoms over time, including comparison to 'gold standard' neuropsychological assessments. BrainCheck is automated, self-administrable, and quick to administer. These qualities are critical to increasing accessibility, especially in early detection, for a population facing an expected increase of dementia in upcoming years. Increasing access may lead to the growth in the normative data needed to better characterize dementia subtypes and provide earlier detection and diagnostic utility.

. CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint

The copyright holder for this this version posted November 13, 2020. ; https://doi.org/10.1101/2020.11.10.20229369 doi: medRxiv preprint

Alzheimer's disease facts and figures -2020 -Alzheimer's & Dementia -Wiley Online Library n

Racial and ethnic estimates of Alzheimer's disease and related dementias in the United States (2015-2060) in adults aged ≥65 years

Prevalence and determinants of undetected dementia in the community: a systematic literature review and a meta-analysis

The impact of early dementia diagnosis and intervention on informal caregivers

A family intervention to delay nursing home placement of patients with Alzheimer disease. A randomized controlled trial

Mild Cognitive Impairment Represents Early-Stage Alzheimer Disease

Rate of progression of mild cognitive impairment to dementia -meta-analysis of 41 robust inception cohort studies

The diagnosis of mild cognitive impairment due to Alzheimer's disease: Recommendations from the National Institute on Aging-Alzheimer's Association workgroups on diagnostic guidelines for Alzheimer's disease

A re-examination of Montreal Cognitive Assessment (MoCA) cutoff scores -Carson -2018 -International Journal of Geriatric Psychiatry -Wiley Online Library n

Diagnostic accuracy of the Montreal Cognitive Assessment (MoCA) for cognitive screening in old age psychiatry: Determining cutoff scores in clinical practice. Avoiding spectrum bias caused by healthy controls

Cognitive Tests to Detect Dementia: A Systematic Review and Meta-analysis

Screening for Early Alzheimer's Disease: Is There Still a Role for the Mini-Mental State Examination? Prim Care Companion

The usefulness of the SLUMS test for diagnosis of mild cognitive impairment and dementia

Cognitive Screening Tests Versus Comprehensive Neuropsychological Test Batteries: A National Academy of Neuropsychology Education Paper

Neuropsychological testing and assessment for dementia

Status of computerized cognitive testing in aging: A systematic review

Computerized Neurocognitive Test (CNT) in mild cognitive impairment and Alzheimer's disease

Three-Month Stability of the CogState Brief Battery in Healthy Older Adults, Mild Cognitive Impairment, and Alzheimer's Disease: Results from the Australian Imaging, Biomarkers, and Lifestyle-Rate of Change Substudy (AIBL-ROCS)

Validity of a Brief Computerized Cognitive Screening Test in Dementia

Validity of a novel computerized cognitive battery for mild cognitive impairment

Computerized Cognitive Testing for Older Adults: A Review

WHO Declares COVID-19 a Pandemic

Diagnostic accuracy of tablet-based software for the detection of concussion

Mobile Neurocognitive Screening Test to Aid in Identifying Cognitive Impairment and Dementia (BrainCheck): Cohort Study

Using ADRC and Related Resources -Alzheimer's Disease Research Center

DiagTest3Grp: An R Package for Analyzing Diagnostic Tests with Three Ordinal Groups

Brief cognitive screening instruments for early detection of Alzheimer's disease: a systematic review

Mild cognitive impairment

Neuropsychological Testing -an overview | ScienceDirect Topics n

BK report personal fees from BrainCheck, outside the submitted work; BH, RHG reports receiving stock options from BrainCheck

Acknowledgments: Funding was provided by BrainCheck, Inc. Registry was provided by the University of Washington Alzheimer's Disease and Research Center along the Memory and Brain Wellness Center Data Availability -Data may be made available by contacting the corresponding author