key: cord-0073004-ommfiy0w
authors: Monje, Mariana H. G.; Domínguez, Sergio; Vera-Olmos, Javier; Antonini, Angelo; Mestre, Tiago A.; Malpica, Norberto; Sánchez-Ferro, Álvaro
title: Remote Evaluation of Parkinson's Disease Using a Conventional Webcam and Artificial Intelligence
date: 2021-12-23
journal: Front Neurol
DOI: 10.3389/fneur.2021.742654
sha: dd6b53b8ab4a492e368392c9ba83eaf17c5f2221
doc_id: 73004
cord_uid: ommfiy0w

Objective: This study aimed to prove the concept of a new optical video-based system to measure Parkinson's disease (PD) remotely using an accessible standard webcam. Methods: We consecutively enrolled a cohort of 42 patients with PD and healthy subjects (HSs). The participants were recorded performing MDS-UPDRS III bradykinesia upper limb tasks with a computer webcam. The video frames were processed using the artificial intelligence algorithms tracking the movements of the hands. The video extracted features were correlated with clinical rating using the Movement Disorder Society revision of the Unified Parkinson's Disease Rating Scale and inertial measurement units (IMUs). The developed classifiers were validated on an independent dataset. Results: We found significant differences in the motor performance of the patients with PD and HSs in all the bradykinesia upper limb motor tasks. The best performing classifiers were unilateral finger tapping and hand movement speed. The model correlated both with the IMUs for quantitative assessment of motor function and the clinical scales, hence demonstrating concurrent validity with the existing methods. Conclusions: We present here the proof-of-concept of a novel webcam-based technology to remotely detect the parkinsonian features using artificial intelligence. This method has preliminarily achieved a very high diagnostic accuracy and could be easily expanded to other disease manifestations to support PD management.

Bradykinesia, defined as the slowness of movement and decrement in amplitude or speed (or progressive hesitations/halts) in continuous movement, is the most relevant clinical motor feature of Parkinson's disease (PD) (1) . For its evaluation, the clinicians analyze the multiple aspects of movement, such as amplitude, speed, fatigue, and arrests when executing a motor task. Typically, a clinician integrates all these features. The best example is its rating into a single severity score of different bradykinesia tasks part of the Movement Disorders Society-sponsored revision of the Unified Parkinson's Disease Rating Scale motor subscale (MDS-UPDRS part III). This scale is the most used standard evaluation of motor function in PD (2) with a high test-retest reliability and inter-and intra-rater reliability (3, 4) . However, it is an ordinal scale with only five discrete categories, and often its accuracy can be compromised due to the subjectivity of the assessment and the difficulty to detect the subtle changes in the consecutive time points.

To accurately quantify and analyze the motor performance of the patients with PD, the technology-based tools, such as the wearable sensors composed of accelerometers and gyroscopes can be used (5) . These objective measurement tools can overcome the subjective and non-linear measures resulting from the clinical ratings (6) . Additionally, they can be used to analyze the motor status of the patient in the home setting (7) .

The optical motion capture systems based on video processing can also be used to study motor performance (8, 9) . Specifically, some video-based systems are developed for the automated assessment of the upper limb movement in the patients with PD (10) . These systems included cameras combined with the colored and reflective markers, bare hand tracking by the depthsensing devices that traced the upper limb movement while performing the MDS-UPDRS part III bradykinesia tasks (11) (12) (13) . These systems are traditionally used in a lab setting and have not yet been transitioned to the home environment. With the surge of telemedicine and remote consultation, there is a need for the supportive tools that permit an objective evaluation of movement remotely.

In this work, we propose a markerless video-based motion method to prove the concept that video-based objective classification of PD motor function based on bradykinesia is possible using a standard laptop webcam and an artificial intelligence algorithm. This analysis provides an ideal proof-ofconcept for capturing bradykinesia of a patient with PD remotely while using an accessible, standard webcam video-camera.

We recruited a consecutive cohort of 22 patients with PD and 20 healthy subjects (HSs). The eligible patients (i) had a PD diagnosis in the preceding 5 years according to the UK Brain Bank Clinical Criteria (14), further supported with (ii) a PET- 18 FDopa neuroimaging. We excluded the HSs in the presence of personal history, and first-and second-degree family history of any movement disorder (i.e., tremor or parkinsonism), and any known condition that could affect motor performance of the upper limbs. The demographic characteristics were assessed for both the groups, such as handedness (Laterality Preference Index, LPI) (15) . An independent dataset containing N = 12 videos (six PD and six HSs) were also included as validation cohort for the test. The Ethics Committee of HM Hospitales approved the study protocol (protocol number: 18.05.1245-GHM). The participants provided the written informed consent before participating in the study.

The participants were always evaluated after overnight off medication, and clinical evaluation included a motor assessment performed by two trained specialists (MHGM and ASF) using the MDS-UPDRS Part III. To evaluate the concurrent validity of the new method with other objective tests, motor performance was also evaluated with objective measures using the inertial measurement units (IMUs) (Kinesia TM One system; Great Lakes Neurotechnologies Inc., Cleveland, OH, USA) (16) . We quantified the motor function while performing the MDS-UPDRS-III bradykinesia upper limb tasks (finger tapping, hand movements, and pronation and supination movements of the hand). For that, the IMU was placed on the index finger. Output data from Kinesia TM is a continuous score from 0 (less) to 4 (maximum) impairment.

The participants were recorded with a computer webcam (640 × 426 pixels at 30 fps). During the examination, the participants rested their elbow on an armchair, and the camera was adjusted such that the hand and forearm were always present in the field of view. The participants were instructed to perform the MDS-UPDRS III bradykinesia upper limb tasks (finger tapping, hand movements, and pronation and supination movements of the hand) in front of the camera. Each task was performed three times with each hand separately (i.e., single-hand tasks, named unilateral motor tasks), and with both hands simultaneously (i.e., two-hand tasks, named bilateral motor tasks). For the normalization purposes, for each task the subjects were asked to stay in a certain position for a few frames. In the finger tapping and hand movements tasks, the patients were asked to do a maximum aperture and closing of the fingers or hand. In the pronation and supination movement of the hand task, the subjects were asked to extend the arm with the palm down and do the maximum supination movement. Each video sample was restricted to 12 s.

The video frames were processed with a Single Shot MultiBox Detector (SSD) network trained to detect the hands in realtime using the EgoHands dataset (17) . The output of the SSD is a series of bounding boxes each marked with the probability of it containing a hand. The algorithm detects each hand and processes each of the two bounding boxes separately. To refine the detection process, we introduced some post-processing rules depending on the task. For single hand tasks, we selected the highest-ranking bounding box on the side required by the task, and for the two-hand tasks, we selected the highest-ranking bounding box on each side. We also performed a temporal correction. If the probability of a certain bounding box was not higher than its probability in the previous frame, we keep the previous bounding box. This pre-process ensures a correct and efficient detection of hands due to a varied background of the videos. After the bounding boxes for the hands were computed for all the frames, these boxes were cropped and processed by a second CNN model named OpenPose (18) , to detect the joints of the hands (Figure 1) . The specific landmarks extracted by OpenPose for each hand are shown in Supplementary Figure 1 . From these landmarks, we select specific key-points to generate time curves to describe each task. To compensate for the camera distance and the size of the hand in the amplitude measurements, we normalized the measurements to the maximum amplitude. For finger tapping, the Euclidean distance (in pixels) between the thumb and the index finger was computed for every frame. For the hand grasp task, the Euclidean distance in pixels between the wrist and the average of the tips of all fingers, except the thumb. For the pronation-supination task, we computed the vector resulting from subtracting the key points of the pinky finger and the thumb. This vector was then transformed into the polar coordinates to obtain the degrees of the rotation for every frame, with respect to the normalization frames, as explained above.

The time signals were pre-processed with a Butterworth low pass frequency filter. To select the frequency of the filter we computed the Fourier transform of each graph to calculate their frequency components. The frequency selected to perform the low pass filter was the highest frequency of the peaks that have an amplitude at least higher than one-fourth of the amplitude of the highest frequency peak.

In addition, we applied an amplitude correction, to eliminate the peaks due to noise. For the finger tapping task, we used a normalized pixel threshold of 0.1, while for the hand grasp and pronation/supination tasks, a 0.25 threshold was applied. After filtering every signal, we extracted the upper envelopes of the filtered signals, by detecting the peaks and interpolating among them (Figure 2) . We extracted several features from the time curves: mean amplitude and SD of the peaks, speed (number of peaks per second), and fatigue (difference between the highest and the lowest values of the upper envelope of the curve). The three features were computed for both the left-hand and the righthand tasks.

To confirm the accuracy of the time signals generated by our pipeline, the videos were assessed by an external clinician that manually labeled every landmark of the hand in videos of the finger tapping task of nine subjects. The clinician labeled the frames from each video of both hands and the software skipped two frames after each labeled frame. The salient points were automatically extracted for the remaining frames by the algorithm. Visual inspection of the salient points aligned to the original video were used to confirm the accuracy of the algorithm output.

We designed a model to differentiate between the PD and HSs. For this, we extracted several single features that are known to be related with bradykinesia (mean amplitude, SD of the amplitude, speed, and fatigue) (19, 20) from both the hands either in the single-hand tasks and in the twohands tasks. We trained three classifiers: Logistic Regression, Gaussian Naive-Bayes, and Random Forest. Each classifier received two values as input. These were the values of each feature for the left and right hands, respectively. We trained the classifiers using the features from each hand in the single hand tasks and using the features from each hand in the tasks performed with both hands simultaneously. A 4-fold cross-validation per classifier was applied and the receiver operating characteristic (ROC) curve per fold of the classifier was produced. Subsequently, an average ROC curve was calculated FIGURE 2 | Acceleration traces during the single-hand (unilateral) and two-hand (bilateral) motor tasks using the webcam. Representative segment of the kinematic signal reconstructed during unilateral (upper) and bilateral (bottom) motor tasks from finger tapping, hand movement, and pronation/supination movements of the hand in a patient with Parkinson's disease (PD). Note the general worse performance in the dual tasks shown in the lower part of the image when compared with the corresponding task perform with just one hand. and the corresponding area under the curve (AUC) was used to compare the performance of the three candidate classifiers per classification task.

An external validation dataset included 12 videos (six videos from the patients with PD and six from HSs) recorded with the same protocol. We evaluated the extracted features from each hand in the single-hand tasks, and the extracted features from each hand in the two-hand tasks.

We compared the demographic characteristics between the PD and HSs groups using Mann-Whitney's U non-parametric test (continuous) and the chi-square test (categorical). The extracted motor features from the videos of the finger tapping, hand movement, and pronation-supination movements of the hands stratified by side were compared between the PD and HSs groups using the non-parametric tests (Mann-Whitney test). Spearman's r correlations between the bradykinesia MDS-UPDRS-III sub-scores and the quantitative assessment methods were calculated. To rule out the confounding effects due to gender and hand size, we evaluated the performance of the classifiers on a strata containing only the male participants. The significance level was set at a 2-sided Pvalue of 0.05 and RStudio version 1.1.414 was used for the statistical analysis. 

The single features extracted by the classifiers showed differences between the most affected side (MAS) and less affected side (LAS) in the patients with PD and between the dominant side (DS) and non-dominant side (NDS) according to handedness in HSs ( Table 2 ). There were statistically significant differences for the unilateral finger tapping speed, hand movement speed, and pronation-supination movement amplitude (P < 0.05) ( Table 2 ). For the bilateral task, there were statistically significant differences for the finger tapping amplitude, finger tapping speed, and hand movement speed (P < 0.05) ( Table 2) .

In the patients with PD, the video-extracted speed of the MAS showed significant moderate negative correlation with the MDS-UPDRS-III score of the MAS for hand movement (r = −0.50, 

When we applied our predictive model to an external validation dataset containing 12 videos (six PD and six HSs), in the unilateral motor tasks, the combined right-left speed for finger tapping, hand movement, and pronation-supination movement of the hand had the highest values and consistency across the three different classifiers, along with the combined right-left amplitude of the pronation-supination movement of the hand ( Table 3 and Figure 3) . The AUC range for the Logistic Regression model in the unilateral finger tapping, hand movement, and pronation-supinations tasks ranged from 0.47 to 1, 0.28 to 1.00, and 0.40 to 0.94, respectively. For Naïve Bayes model, the AUC results for the unilateral tasks ranged from 0.47 to 0.83 in the finger tapping, 0.22 to 0.97 for hand grasp, and 0.41 to 0.89 in the pronation supination task. Finally, for the Random Forest model, the AUCs ranged from 0.41 to 0.75 in the finger tapping, from 0.49 to 0.78 in the hand grasp, and from 0.61 to 0.89 in the pronationsupination task ( Table 3) . For the bilateral motor tasks, the combined right-left amplitude for finger tapping, pronationsupination movement of the hand, and the combined rightleft amplitude variability for hand movements and speed for pronation-supination movement of the hands showed the highest value and consistency across the three classifiers ( Table 3 and Figure 3 ). 

In this study, we have proven the concept that the motor performance can be assessed objectively using a conventional webcam. We found differences in the motor performances between the different sides of the body and between the patients with PD and HSs in all the upper limb motor tasks used to evaluate bradykinesia. The best performing classifiers were unilateral finger tapping and hand movement speed achieving an almost perfect classification similar to other diagnostic tests in PD (21) . In addition to this high classification performance, the model correlated both with IMUs for quantitative assessment of motor function and the most used standard MDS-UPDRS part III, hence demonstrating concurrent validity with the existing gold standard methods.

Different video-based systems have been described for the automated assessment of upper limb motor performance in the patients with PD (10) . Most of these studies are restricted to the performance of one single motor task of the MDS-UPDRS-III with scarce comparisons of the motor performance between the different body sides (10, 11) . Additionally, most of these systems have been traditionally used in a lab setting and have not yet been transitioned to the home setting (10, 22) . Recently, some video-based technologies have approached the study of motor performance using the low-quality video cameras. Using the smartphones cameras, different studies showed that it is possible to predict the presence of bradykinesia while performing the finger-tapping tasks, with an over 0.70 test accuracy, but with low discriminative capacity between the PD and HSs (23) (24) (25) . Other studies, assessing the finger tapping motor task with conventional webcams also showed similar results to our study (11, 22) . One of the most interesting aspects of this work is that the performance of the model was lower when we evaluated the different motor features extracted individually than when we integrated them. Specially, the inclusion of the side reflecting the asymmetry typical of PD increased the diagnostic performance dramatically and emphasized how critical the integration of information is for the clinical diagnosis.

The integration of information happens during a standard clinical evaluation. To evaluate a patient and establish a diagnosis, the neurologist needs to: (1) have a predefined criteria of average and non-average motor performance for each body side, (2) evaluate the different aspects of the motor performance (e.g., speed, amplitude, and fatigue), (3) compare the motor performance between each body side, (4) evaluate if the observed pattern matches the previous cases with the disease or that of the HSs. In our study, we replicated this behavior and trained a model that can recognize decrements in amplitude and speed of movement between the PD and HSs groups. This information was also combined with the motor performance asymmetry of one side compared with the other one. Thus, the accuracy obtained for some of the extracted motor features (i.e., finger tapping speed or hand grasp speed) could complement the one obtained from an expert neurologist (26) . It is remarkable that the computer needed a 12 s video per task to perform a classification of PD vs. HSs based on the performance of bradykinesia tasks. Therefore, our method has many conceptual resemblances with the routine diagnostic process, making it interpretable and aligned with the standard of care, and supporting the potential of feature integration mimicking the behavior of the human brain. Importantly, the system could also detect the differences in motor performance between the unilateral tasks and bilateral tasks, showing a worse performance when performing bilateral tasks in the group of patients with PD compare with HSs ( Table 2) . This is in line with the described impaired ability to perform the bilateral tasks, either simultaneously or sequentially, that occurs in the patients with PD. Thus, when performing a bilateral task, the motor performance in the patients with PD can show a dramatic reduction in the movement of the most affected side compared to the unilateral tasks (27, 28) . This proof-of-concept study hence demonstrates that a standard webcam coupled with an artificial intelligence method has potential for accurately assessing upper limb bradykinesia in the patients with PD remotely. The camera employed for all the experiments was a webcam of standard laptop produced in 2010 (a 2010 MacBookPro). Any current laptop or mobile device allows video recording with a higher resolution and frame rate. This tool expands the portfolio of technologies available for evaluating the patients with PD, with the advantages of not needing any dedicated equipment outside of a standard laptop and using a video which also permits a simultaneous verification of the motor performance using the traditional clinical methods (i.e., the healthcare professional could review the raw video that generated the score when needed for quality control purposes). The most salient applications of this video-based technology could be the use in remote teleconsultations, which have surged with the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic (29) and the decentralized clinical trials, as a complement for the standard assessments using MDS-UPDRS or other objective evaluation methods (i.e., sensors) (30, 31) . In addition, this system could help minimize variability in clinical assessment in the clinical trials.

In this way, and in keeping with the previous initiatives (11, 22) , we have recently developed a web app for the remote recording of the upper limb bradykinesia motor tasks. The app includes conformance statement signing, video instructions for every task, and the indications to facilitate hand positioning. Furthermore, it requires no software to be installed, thus using any standard laptop and any operating system. The upcoming studies will show the feasibility of its implementation for the assessment of the tasks in the at-home setting.

Our work has several limitations. First, this is a proof-ofconcept study with a reduced sample size, with specific disease characteristics (age and duration of disease), and the comparison was made only with healthy controls. Both the factors limit the generalizability of the results. Additionally, the study cohort is integrated by early patients with PD with predominant unilateral motor symptoms. Yet, this provides a convenient scenario to test the discriminative strength of the method. In the future, our results should be verified in larger cohorts with a representation of a broad spectrum of the patients with PD, such as the groups with different age and disease duration. In addition, the discriminative power of the method when including other parkinsonian syndromes in the mix remains to be established as well. Another limitation is that we restricted the assessment to the upper limb bradykinesia motor tasks of the MDS-UPDRS III. However, the motor performance of upper limbs is a predictive characteristic of onset and PD progression (32, 33) . Therefore, its analysis is of the utmost relevance for the clinical evaluation and outcome of the patients with PD. The assessment of the global motor performance, such as lower limbs, and other disease manifestations, such as tremor, axial signs, and gait represents a future expansion of the current concept of objectively measuring other disease motor features using a standard video. This can improve the model performance and hence needs to be investigated. Finally, we focused our analysis on the evaluation of the binary classification performance of the present method. Future work should evaluate the additional quantitative aspects of the motor performance, increasing the granularity of the information, be able to rate the disease severity, and the detection of subtle changes in the motor status along the time, and after a therapeutic intervention. Those aspects will be critical for the application of this system in telemedicine and potentially clinical trials.

We proved the concept that a novel webcam-based technology can accurately evaluate bradykinesia, the single core feature that allows the diagnosis of parkinsonism, in a remote setting, using artificial intelligence. This method has an accuracy performance that could complement the usual diagnostic process performed by the experienced movement disorders specialists and could be easily expanded to other disease manifestations. Our results need to be confirmed in the larger studies, such as patients with other forms of parkinsonism, age groups, and disease status.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

The studies involving human participants were reviewed and approved by the Ethics Committee of HM Hospitales who approved the study protocol (Protocol Number: 18.05.1245-GHM). The patients/participants provided their written informed consent to participate in this study.

MHGM has contributed to the design of the solution, the recruitment and evaluation of the participants, the analysis of the data, and the manuscript draft. SD, JV-O, and NM have participated in the technology development, the analysis, and the draft of the manuscript. AA and TM have participated in the critical review of the project, evaluation of the results, and the manuscript elaboration. AS-F has contributed to the conceptualization of the solution, its development, the clinical validation, the draft, and the review of the manuscript. All authors contributed to the article and approved the submitted version.

This study was supported by the Joint Program for Neurodegenerative Diseases and the Instituto de Salud Carlos III (Reference Number: HESOCARE-329-073) as part of the iCARE-PD consortium.

MDS clinical diagnostic criteria for Parkinson's disease

Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS): scale presentation and clinimetric testing results

Interrater reliability of the unified Parkinson's disease rating scale motor examination

Factor structure of the Unified Parkinson's Disease Rating Scale: motor examination section

New sensor and wearable technologies to aid in the diagnosis and treatment monitoring of Parkinson's disease

Clinician versus machine: reliability and responsiveness of motor endpoints in Parkinson's disease

Free-living monitoring of Parkinson's disease: lessons from the field

A study of Vicon system positioning performance

Deep neural networks enable quantitative movement analysis using single-camera videos

Video-based analyses of Parkinson's disease severity: a brief review

A computer vision framework for finger-tapping evaluation in Parkinson's disease

Leap motion evaluation for assessment of upper limb motor skills in Parkinson's disease

Optical hand tracking: a novel technique for the assessment of Bradykinesia in Parkinson's disease

The lateral preference inventory for measurement of handedness, footedness, eyedness, and earedness: norms for young adults

Objective measure of upper extremity motor impairment in Parkinson's disease with inertial sensors

Lending A hand: detecting hands and recognizing activities in complex egocentric interactions

OpenPose: realtime multi-person 2D pose estimation using part affinity fields

The modified bradykinesia rating scale for Parkinson's disease: reliability and comparison with kinematic measures

Differential response of speed, amplitude, and rhythm to dopaminergic medications in Parkinson's disease Alberto

Diagnostic performance of clinical motor and non-motor tests of Parkinson disease: a matched case-control study

The PARK framework for automated analysis of Parkinson's disease characteristics

Bradykinesia recognition in Parkinson's disease via single RGB video

Supervised classification of bradykinesia for Parkinson's disease diagnosis from smartphone videos

The discerning eye of computer vision: can it measure Parkinson's finger tap bradykinesia?

Accuracy of clinical diagnosis of Parkinson disease

The execution of bimanual movements in patients with Parkinson's, Huntington's and cerebellar disease

Unilateral versus bilateral tasks in early asymmetric Parkinson's disease: differential effects on bradykinesia

Trends in the use of telehealth during the emergence of the COVID-19 pandemic -United States

Telemedicine in movement disorders: Leçons du COVID-19

Movement Disorder Society-Unified Parkinson's disease rating scale use in the Covid-19 era

The natural history of Parkinson's disease

Motor onset topography and progression in Parkinson's disease: the upper limb is first

The authors are particularly thankful to Francisco Navarro Muñoz (FNM) for external labeling of the videos during the algorithm performance optimization.

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fneur. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.Publisher's Note: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.Copyright © 2021 Monje, Domínguez, Vera-Olmos, Antonini, Mestre, Malpica and Sánchez-Ferro. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.