key: cord-0610043-489mezwy authors: Alam, Mohammad Arif Ul title: Activity-Aware Deep Cognitive Fatigue Assessment using Wearables date: 2021-05-05 journal: nan DOI: nan sha: 24bdb3b6a7a6af45bd40331cb2a11113489462c5 doc_id: 610043 cord_uid: 489mezwy Cognitive fatigue has been a common problem among workers which has become an increasing global problem since the emergence of COVID-19 as a global pandemic. While existing multi-modal wearable sensors-aided automatic cognitive fatigue monitoring tools have focused on physical and physiological sensors (ECG, PPG, Actigraphy) analytic on specific group of people (say gamers, athletes, construction workers), activity-awareness is utmost importance due to its different responses on physiology in different person. In this paper, we propose a novel framework, Activity-Aware Recurrent Neural Network (emph{AcRoNN}), that can generalize individual activity recognition and improve cognitive fatigue estimation significantly. We evaluate and compare our proposed method with state-of-art methods using one real-time collected dataset from 5 individuals and another publicly available dataset from 27 individuals achieving max. 19% improvement. Cognitive fatigue is a syndrome conceptualized as resulting from chronic workplace stress that has not been successfully managed [1] . Although, cognitive fatigue is not a clinical condition which can occur in any workplace or home environment where there is stress, it is recognized by the World Health Organization (WHO) as a syndrome [1] . In short term, cognitive fatigue may cause sleeping disturbances, anxiety, irritability and hormonal disturbances and in long run, this may result more severe impacts on health safety such as cardiovascular, gastrointestinal and neuropsychological disorders [2] . Current frameworks for cognitive fatigue estimation are mostly self-reported questionnaire based [29] , [30] , which is impossible to generate continuous fatigue report by avoiding recall bias [29] . Recent advancement of wearable physical and physiological sensor technologies enable accurate estimation of cognitive fatigue related partial outcomes such as stress, anxiety, sleep quality, mobility etc, which provides ultimate opportunity to researchers to estimate cognitive fatigue continuously [16] , [18] - [20] , [22] - [24] that includes actigraphy [25] , [26] , heart rate (HR) [17] , [25] , Electrocardiography (ECG) [21] , Electroencephalography (EEG) [27] and Electromyography (EMG) [28] sensors along with traditional and deep machine learning techniques. Combining accelerometer with ECG has been a successful attempt as well before [3] which proposed to use deep learning frameworks (LSTM with Consistency Self-Attention, LSTM-CSA) but suffers with the lack of adaptability across diverse population. Due to the dissimilarities among different individual group's responses on cognitive fatigue in terms of physi-Department of Computer Science, University of Massachusetts Lowell mohammadariful alam@uml.edu cal and physiological contexts, current wearable cognitive fatigue estimation research is constrained in group specific cognitive fatigue estimation [cite] . For example, fatigue detection of video game players [31] , athletes [31] , basketball players or heavy exercise performers [32] . As per many clinical psychologists and mental health researchers, cognitive fatigue estimation should be more personalized, rather than generalized on specific group of people to keep mental healthcare systems sustainable for future generations [33] . While, the emergence needs of building personalized cognitive fatigue estimation tool, we have the following key question: can we develop personalized cognitive fatigue assessment tool considering activity as their activity as domain invariant feature and fatigue as personalized response on each activity? As we know, autonomic nervous system (ANS) restrains the body's major physiological activities including the heart rate (HR) and gland secretion or electrodermal activity (EDA) [34] . However, these responses are contaminated with physical activity artifacts significantly [35] . The central hypothesis of the this paper is: each performed activity context generates similar artifacts on same activity over diverse population, thus, we can align similar activities (activity-awareness) as person invariant feature and its physiological responses as personalized fatigue feature. For example, in Fig. 1 , we illustrate the EDA responses on two different activities: (i) steady hand (ii) waving hand over two different physiological states: (i) stress and (ii) no stress. The Fig. 1 clearly shows that same activity (waving hands) has similar EDA response patterns (but different amplitude) due to similar artifacts which signifies our hypothesis. In this paper, we develop a novel Activity-Aware Recurrent Neural Network (AcRoNN) model and utilize it to design a personalized cognitive fatigue assessment framework, that provides the following key contributions • We develop a novel Activity-Aware Recurrent Neural Network (AcRoNN) framework that is able to exploit contextual cues present in any event from actigraphy sensor and then assess cognitive fatigue from physiological (EDA and HR) sensor signal using a deep recurrent neural network. • Apply AcRoNN on two publicly available data and evaluate the capability of AcRoNN framework to improve cognitive fatigue assessment. shows the overall schematic diagram of our contextaware cognitive fatigue assessment framework. In this framework, recognized activity information is intertwined with the cognitive fatigue assessment architecture. In this regard, we take (stage 1) cognitive fatigue assessment scoring in terms of gestural activity, and then postural activity information, and finally we (stage 2) re-evaluate the cognitive fatigue assessment scoring based on activity relationships it has learned from stage 1. This stage involves feature extraction, activity recognition and activity-based score mapping for cognitive fatigue detection. 1) Wearable Sensor Signal Processing: Wearable sensors can be two types: physical and physiological. Physical sensors (accelerometer, gyroscope etc.) signal values change over the movements of the sensor devices. Physiological sensors change over physiological condition of body such as EDA changes over stress and PPG changes over heart rate. However, physical movements also impose noises on physiological sensor signals which is called motion artifacts. Physiological Signal Processing: A continuous and descrete decomposition of EDA, and time and frequency domain analytics of EDA signal have been investigated before to extract relevant physiological features which were contaminated with noises and motion artifacts [38] . [39] denoised and classified EDA from cognitive load and stress with accuracy higher than 80%. Though motion artifacts removal techniques such as exponential smoothing and lowpass filters provide significant improvement in filtering EDA signals, wavelet transforms offer more sophisticated refinement for any kind of physiological sensors such as electroencephalogram, electrocardiogram [37] , and PPG [40] . [41] proposed a stationary wavelet transform (SWT) based motion artifacts removal technique. 'cvxEDA' proposed a convex optimization technique considering EDA as a mixture of white gaussian noise, tonic and phasic components where white gaussian noise includes motion artifacts and external noises [37] . We combine SWT and 'cvxEDA' together to remove noises and motion artifacts from EDA signal. Researchers proposed different methods such as frequency analytics [42] , statistical analytics [43] and digital filter [44] to reduce noises and motion artifacts from PPG. We used Periodic Moving Average Filter (PMAF) in this regard [45] . After the noise reduction, we generated 33 heart rate variability (HRV) features from PPG (as per [3] ) and 12 statistical features from EDA (as per [34] ) signals with a 10-seconds window. Accelerometer Signal Processing: We used Bai et. al. proposed accelerometer signal processing method [3] . We used ActiLife tool [46] , and calculated the Actigraphy counts (from accelerometer) every 5 seconds, and detect the nonwear time (for invalid data removal). Within every 10seconds window, based on Actigraphy counts we further extracted 8 statistical features, i.e.,mean, median, standard deviation, variance, minimum value, maximum value, skewness and kurtosis for further processing. Multimodal Feature Sequence Construction: After preprocessing and feature engineering, the original segment can be transformed into a D-dimensional sequence where T is the sequence length (i.e., the number of windows/epochs within a segment), and x t = f eat acc ∪ f eat eda ∪ f eat hrv where f eat acc , f eat eda , f eat hr represent accelerometer, EDA and HR features extracted above. Since, each of the extracted features were in 10-seconds window, the concatenated input feature x t has a dimension of 53 (33+12+8) per 10-seconds in the time-series window. 2) Activity Recognition Module: We develop a two step multi-label activity recognition framework which consists of two LSTM with Consistency Self-Attention (LSTM-CSA) [36] models, (1) gestural activity recognition and (2) postural activity recognition. Both of the LSTM-CSA models are independent from each other, trained and tested separately using hand gesture and postural activity labels respectively using the input accelerometer features (f eat acc ) and their corresponding labels. For all LSTM-CSA models, we used the following regularisation term where T , Γ(α) tends to penalize heavily with a larger contextual scores (which will be fined later) to maintain its global consistency. We develop a contextual feature mapping for each cognitive fatigue label which can be represented as follows The output of the contextual feature map layer is a (H c × W c × C) tensor, where H c and W c are the segment dimensions, and C is the number of classes. We consider (after trial and error), we have set H c = 23 while we have already defined W c = 53. We designed a scoring function to measure the contextual relevance of cognitive fatigue detection in relation to multiple-label's presence in a window [47] . Scores are computed using the contextual feature maps generated in each stage of our pipeline, and are used as the ranking score in AP calculations to measure whether contextual learner confirms or refutes detections passed to it, based on learned semantic relationships. The scoring process is designed as a new network layer, and appended to the end of each stage in our pipeline. We defined two contextualscoring method as per [47] . • Individual Contextual Scoring: Gestural and posturalbased cognitive fatigue scoring has been estimated using the following equation where F M b represents the activity bounding box (start and end of an activity) related relevance score and b represents each activity type i.e. gestural or postural activity as per [47] . We have two types of individual contextual scoring in our framework, gestural-based cognitive fatigue scoring and postural-based cognitive fatigue scoring (Fig. 3 ). • Cumulative Contextual Scoring: In this scoring method, we add both gestural and postural activity based cognitive fatigue scoring together for producing final Activity-based cognitive fatigue re-scoring which can be defined as follows where CF M b represents the cumulative activity bounding box (start and end of an activity) related relevance score and c represents cumulative activity type i.e., either gestural or postural activity-based cognitive fatigue scoring or re-scoring [47] . The second stage is an LSTM with Consistency Self-Attention (LSTM-CSA) model that is trained to learn semantic relationships using the cumulative contextual score mapping generated by the primary cognitive fatigue detector using the Equation 1. In this section, we aim to evaluate our proposed Activity-Aware Recurrent (AcRoNN) performance towards developing a personalized cognitive fatigue assessment system using wearables without any target labels. We use two datasets to evaluate AcRoNN model performance which are described as follows: • A1: Activity Recognition Dataset: Previously, we collected hand gestural (8-hand gestures) and postural (4 postural) activity dataset to serve the purpose of our previous papers [35] , [48] using Empatica E4 watch. We utilized the same dataset and developed hand gesture and postural activity recognition framework as per our proposed framework in this paper. • D1: Gamer's Fatigue Dataset: We recruited 5 student video games players (age ranges from 19-25) for 7 days who stayed up during a 22 hour shift every alternative day (4 days each) to simulate cognitive fatigue while wearing Empatica E4 watch [4] . Empatica E4 watch consists of accelerometer (ACC), electrodermal activity (EDA), photoplethysmography (PPG) and skin temperature (TEMP) sensors. During the data collection (including non-gaming days), participates were asked to measure their sleepiness based on the 'Stanford Sleepiness Scale' (SSS) [5] , [8] (ranges 1-7 representing active to extremely sleepy) and the 'Sleep-2-peak' score [6] (ranges 1-7 representing active to extremely sleepy) using Sleep2Peak Android App [7], [9] . • D2: Healthy Adults Fatigue Dataset: We have used publicly available health adults fatigue dataset [10] , [11] . Data from 28 healthy individuals (26-55 years of age, average age 42 years, 41/51% female/male), of which 17 enrolled up to 2 days after returning from long-haul flights with 3-7 time zone differences and hence were recovering from jet lag, from 1 to 219 consecutive days (mean 35, median 9, total 973 days) were collected. Objective data were collected using a multisensor wearable device, Everion (Biovotion AG, Switzerland [12]), in conjunction with a mobile app, SymTrack (Gastric GmbH, Switzerland), to deliver a daily fatigue questionnaire. Volunteers were asked to continuously wear the Everion device around their non-dominant arm over a 1-week period. The device combines a 3-axis accelerometer, barometer, galvanic skin response electrode, and temperature and photo sensors. Dataset tracked a total of 12 parameters at 1-Hz temporal resolution on physical activity and physiology. Volunteers were instructed to complete a 4-item daily questionnaire in the evening to capture their subjective assessment of fatigue, adapted from the Fatigue Assessment Scale [13] and Visual Analogue Scale to evaluate fatigue severity [14] : (10 Physical fatigue score (PhF), (2) Mental fatigue score (MF), (3) Visual analogue scale score (VAS), and, (4) Indicator of relative perception (RelP) (see [10] , [11] for more details). For each subject and parameter, we excluded days where more than 80% of the samples were missing to ensure an acceptable performance of downstream analysis. Missing samples were due to subjects not wearing the device (e.g., during charging) or low-quality segments (e.g., loss of skin contact). This filtering step led to a total of 5 subjects and 821 hours of data annotated (Stanford Sleepiness Scale and Sleep-2-peak) labels with continuous Empatica E4 sensor data (excluding 1 hour recharging sessions). Finally, we imputed missing data gaps using the state-of-the-art unidirectional uncorrelated recurrent imputation model from Cao et al [15] . We re-implemented latest cognitive fatigue estimation framework [3] . In Bai et. al. [3] , authors generated highest accuracy of cognitive fatigue assessment using LSTM-CSA model. Although, Bai et. al. provided cognitive fatigue detection method based on PPG (ECG) and Accelerometer sensor signal processing, we implemented the following baselines from Bai et. al. method as follows. • AcRoNN: We combine everything together as proposed in this paper. Table I shows details of our experimental results and comparisons with our different baseline models. We can easily identify that our model AcRoNN outperforms all of the baseline models significantly in both of our collected datasets and already available datasets. Also, we can firmly say that, our AcRoNN model outperforms baseline significantly even though we chose to use baseline proposed sensors (Actigraphy and ECG) related features only. We have collected data from only 5 student volunteers in a limited setting due to the on-going pandemic related lockdown that blocked us from reaching mass population in the campus and dormitory of University of Massachusetts Lowell. However, our unique Activity-Aware attention model has been evaluated on a publicly available data that provides us ample confidence of the efficacy of our developed framework. We also could not validate the activity recognition accuracy in the collected data due to the unavailability of camera data as per the IRB exemption. However, our collected activity data (A1) has been well-validated by our previous researches which were published in top venues [34] , [35] , that provides us validity of the activity recognition dataset as well as the related outputs that have been used in our AcRoNN framework's stage-1. In future, while the lockdown will be ended, we plan to collect more data engaging more students in the campus and out of campus, more likely in real cognitively stressed and fatigue community such as healthcare workers, construction workers and scuba divers. To develop an automated cognitive fatigue assessment system, we introduced a new pipeline from data collection, data preprocessing, feature engineering, attention based LSTM and a novel context-aware LSTM model flow. To our best knowledge, AcRoNN is the best cognitive fatigue detection model in the existing literature which can be extended to any other physiological health assessment with proper study design and data collection. Our efficient two-step feature map scoring method provides a new concept in contextaware activity and health monitoring research area that can be utilized to provide appropriate care to patients with dementia, asthma, post-traumatic stress disorder and so on. We acknowledge Eliza Doering and Alexa Mai for helping us to collect data from students. This project has been funded by University of Massachusetts Lowell Internal Seed Grant to Addressed COVID Related Nursing Community Impact Study. The collected datasets will be made public upon acceptance of the manuscript as per IRB exemption. Burn-out an "occupational phenomenon": International Classification of Diseases Circadian rhythm disruption and mental health Wan-Fai Ng: Fatigue assessment using ECG and actigraphy sensors The development and use of the stanford sleepiness scale (SSS) Validation of sleep-2-Peak: A smartphone application that can detect fatiguerelated changes in reaction times during sleep deprivation Management of Excessive Daytime Sleepiness Reviewed Validation of sleep-2-Peak: A smartphone application that can detect fatigue-related changes in reaction times during sleep deprivation Continuous multi-sensor wearable data and daily subject-reported fatigue of heathy adults Assessment of Fatigue Using Wearable Sensors: A Pilot Study Psychometric qualities of a brief self-rated fatigue measure: The Fatigue Assessment Scale Validity and reliability of a scale to assess fatigue Brits: Bidirectional recurrent imputation for time series Towards Reliable, Automated General Movement Assessment for Perinatal Stroke Screening in Infants Using Wearable Accelerometers Recognizing Digital Biomarkers for Fatigue Assessment in Patients with Multiple Sclerosis Remote Monitoring of Stroke Patients' Rehabilitation Using Wearable Accelerometers PD Disease State Assessment in Naturalistic Environments Using Deep Learning Deep learning-based automated speech detection as a marker of social functioning in late-life depression Detection of mental fatigue state with wearable ECG devices Automatic Assessment of Problem Behavior in Individuals with Developmental Disabilities Selecting Clinically Relevant Gait Characteristics for Classification of Early Parkinson's Disease: A Comprehensive Machine Learning Approach Evaluating upper limb function after stroke using the free-living accelerometer data A data-driven approach to modeling physical fatigue in the workplace using wearable sensors Jerk as an indicator of physical exertion and fatigue EEG-based mental fatigue measurement using multi-class support vector machines with confidence estimate Physical fatigue detection through EMG wearables and subjective user reports: a machine learning approach towards adaptive rehabilitation Information bias in health research: Definition, pitfalls, and adjustment methods Psychometric qualities of a brief self-rated fatigue measure: The Fatigue Assessment Scale EEG correlates of video game experience and user profile in motor-imagerybased brain-computer interaction Towards Detecting Biceps Muscle Fatigue in Gym Activity Using Wearables. Sensors (Basel) Cognitive fatigue effects on physical performance: A systematic review and meta-analysis Automated Functional and Behavioral Health Assessment of Older Adults with Dementia 17th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services Understanding and improving recurrent networks for human activity recognition by continuous attention Combining Electrodermal Activity and Speech Analysis towards a more Accurate Emotion Recognition System Automated Functional and Behavioral Health Assessment of Older Adults with Dementia Discriminating stress from cognitive load using a wearable EDA device Reduction of motion artifacts from photoplethysmographic recordings using a wavelet denoising approach Wavelet-based motion artifact removal for electrodermal activity Adaptive Pulse Oximeter with Dual-wavelength based on Wavelet Transforms Motion Artifact Removal from Photoplethysmographic Signals by Combining Temporally Constrained Independent Component Analysis and Adaptive Filter Improved Elimination of Motion Artifacts from a Photoplethysmographic Signal using a Kalman Smoother with Simultaneous Accelerometry The Periodic Moving Average Filter for Removing Motion Artifacts from PPG Signals A Systematic Analysis of a Context Aware Deep Learning Architecture for Object Detection AI-Fairness Towards Activity Recognition of Older Adults