key: cord-0858462-sjngszs8
authors: Sakalle, Aditi; Tomar, Pradeep; Bhardwaj, Harshit; Alim, Md. Abdul
title: A Modified LSTM Framework for Analyzing COVID-19 Effect on Emotion and Mental Health during Pandemic Using the EEG Signals
date: 2022-03-11
journal: J Healthc Eng
DOI: 10.1155/2022/8412430
sha: 071f8d534fbf3b1dba80994ba6e13f5149bbe16a
doc_id: 858462
cord_uid: sjngszs8

COVID-19, a WHO-declared public health emergency of worldwide concern, is quickly spreading over the world, posing a physical and mental health hazard. The COVID-19 has resulted in one of the world's most significant worldwide lockdowns, affecting human mental health. In this research work, a modified Long Short-Term Memory (MLSTM)-based Deep Learning model framework is proposed for analyzing COVID-19 effect on emotion and mental health during the pandemic using electroencephalogram (EEG) signals. The participants of this study were volunteers that recovered from COVID-19. The EEG dataset of 40 people is collected to predict emotion and mental health. The results of the MLSTM model are also compared with the other literature classifiers. With an accuracy of 91.26%, the MLSTM beats existing classifiers when using the 70–30 partitioning technique.

e COVID-19 epidemic has brought the world to a halt because of its rapid spread throughout the globe. Governments must impose lockdowns to stop the virus from spreading, which results in widespread social isolation, which can have serious mental illness repercussions [1] . e worldwide lockdown has affected people's livelihood and causes huge damage to their mental health. However, still, the resemblance of the effects of a pandemic on mental illness is not widely studied, and only little consequences have been identified. Given the significant links between pressure and the start of troubles among people's children due to emotion, it is critical to look at the impact that instructions to isolate themselves and stay at home are having on their mental illness [2] . During this time, it is critical to look into how individuals are concerned about becoming infected or suffering other COVID-19-related repercussions, and how this risk perception is changing [3, 4] and influencing their emotions, eventually leading to mental health issues [5] .

It has been observed through research that the side effects of a pandemic on mental illness are regulated by different factors, such as age, networks in the society, occupation, responsibilities, financial condition, mental health problems, health issues, and personality behaviors [6, 7] . It should be a priority to investigate these connections since they are crucial in informing rules and healthcare choices, as well as guiding academics and physicians. However, there is a scarcity of information on how pandemics affect mental illness. Previous research on the association between COVID-19 and mental illness has focused on approximately restricted areas of mental health, failing to include the many psychosocioeconomic factors that are expected to alter impact, as well as the pandemic's negative-positive impact [8, 9] .

To overcome this issue, we suggested a modified Long Short-Term Memory (MLSTM)-based Deep Learning model framework using electroencephalogram (EEG) signals for studying COVID-19's effect on emotion and mental health during the pandemic in this research paper. is research study was volunteered by participants who are detected with COVID-19 and recovered from it. In this study, we have tried to analyze the emotions of an individual by studying their brain waves using EEG signals which describe the mental health of individuals during pandemics. Emotion recognition-based behavior analysis benefits society and also it is an evolving research area [10, 11] . Emotion identification refers to a person's ability to recognize successful responses that occur in a variety of daily interactions. Awareness of emotional state can help society in many ways. For example, a person with a negative state of emotion can be given emotional help or medication if required and his or her emotional state can be improved, which further can improve their performance in society and at workplaces [12] . At the same time, the person with a positive emotional state can work for the welfare of society. e deployment of EEG signals to predict mental health has recently gained a lot of attention [13] .

EEG signals have been more popular since they cannot be faked [14] . EEG signals reflect the electrical activity of neurons in the brain, and they have been frequently employed to study the working of the brain [15] [16] [17] [18] .

As it is difficult to counterfeit brain signals, in this paper, we are proposing a method that uses EEG for analyzing mental health during the pandemic. In this study, we have collected the dataset using an EEG device. e data captured with the help of EEG is nonmanipulative which is a major advantage in the research [19] . Explicit surveybased analysis of mental health and emotions during COVID-19 takes a lot of time, so this process needs to be automated so that a person can make the most of his/her potential and assistance can be provided to them to improve their mental health. e recording of brain signals directly cannot be falsified or disturbed. It helps in the detection of mental health to achieve a reasonable degree of precision because EEG is cheap, cost-effective, noninvasive, and speedy, making it a popular tool for testing brain changes to emotions. Using the NeuroSky MindWave Mobile 2 device, we have created the dataset by putting the electrode at the FP1 position because frequencies less than and larger than 0.5 Hz to 50 Hz, respectively, are not caught correctly [20, 21] .

An MLSTM-based framework is developed for the analysis of COVID-19 effect on emotions and mental illness during the pandemic using EEG signals in this study. e 70-30 partitioning strategy is used to determine our model's performance. e results show that our model outperforms all other classifiers. is research includes the early detection of personality disorders, emotional distress, and other mental illnesses [22, 23] . During a pandemic, the psychological consequences are to improve people's quality of life and to maximize their performance [24, 25] .

A short overview of this research work is as follows: (i) For analyzing COVID-19 effect on emotion and mental illness during the COVID-19 pandemic using EEG signals, a novel framework Deep learning-based modified MLSTM approach is proposed. (ii) A new EEG signal data collection (Dataset) is developed with an EEG portable single-channel computer-efficient platform for Indian emotional clusters in the Hindi language.

In this section, the detail about the device used, dataset created, and the proposed algorithm is presented.

e NeuroSky MindWave Mobile 2 is a very cost-effective, portable, and easy to handle device. It captures the brain signals, and the components present in the device are flexible and long-lasting. e dataset required for this work is based on brain waves or EEG signals. ese data are self-acquired using the Neu-roSky MindWave Mobile 2 device, a single-channel EEG adjustable headband. e device outputs 12-bit raw brainwaves (1-100 Hz) with a sampling rate of 512 Hz and outputs EEG power spectra in different frequency and morphology bands. e dataset is recorded using the eegId application, in which the FFT technique is implemented as a feature extraction method. A total of 10 features are extracted, referred as F1, F2. . ..F10.

e EEG signals are generated as a person experiences different emotions or feelings when exposed to situations or scenarios through visual content. We recognize an individual's mental health by analyzing brain waves while watching emotional or situational materials and classifying the emotions into two classes, i.e., positive and negative. e elicitation materials included around 40 videos (Hindi-English languages). e defined process for data acquisition involves approximately 40 participants to watch precisely a set of 8 video clips that characterize real-life emotional experiences to analyze mental health. is recorded data of brain waves is processed, followed by applying deep learning-based and machine learning-based algorithms to study and analyze mental health during the pandemic and precisely predict the state of emotions.

Forty (20 males and 20 females) nonclinical participants were considered for this research. e study was volunteered by participants detected with COVID-19 and recovered from it. e participants also signed an informed consent form. All nonclinical participants are from various cultures and education classes yet Hindi speaking. However, 5 data samples were dropped due to failure in equipment or excessive EEG signal artifacts in the final analysis. erefore 35 samples were left as credible subjects (17 male and 18 female). e age group of participants is divided into three age groups 15-20 years, 21-26 years, and 27-35 years. e participants are divided according to their educational background as undergraduates, postgraduates, and working professionals. In our nonclinical population with healthy eyesight, two were left-handed and the remaining 33 were right-handed. Participants were instructed 24 hours before the experiment not to drink nicotine or caffeine. At the outstep, the context and process for the study were initially told to all participants in a manner consistent with the Helsinki Declaration definition. e experiment was performed twice with the same nonclinical population within one week. Figure 1 explains the experimental protocol followed to create a brain signals dataset. A 10 sec hint was given to the participants to start the experiment. After watching the elicitation clips, the subject has to fill out the 3-point selfassessment form (1 � "agree", 2 � "neutral", and 3 � "disagree") to rate the emotions. It consists of the binary class of emotions. Participants were urged to respond to their true feelings to all the questions.

In this study, the MLSTM model was employed to classify two classes of emotions (positive and negative) to analyze the effect of COVID-19 on mental health using EEG signals. Figure 2 shows the fundamental LSTM design. ere are three gates in the modified LSTM network: a forget, an input, and an output. e MLSTM network uses these gates to decide what information to keep and remove from memory. Except for the hidden state, there is nothing else to recollect. Figure 2 reveals the network's secret. It shows the forget gate, which selects which information to keep and which cell states to discard. e choice is made with the help of the sigmoid layer.

e second gate with a sigmoid layer is the input gate, which determines values to be modified, and the layer tanh, which generates newly updated values.

Long EEG sequences are difficult to learn from a recurrent neural network because they are trained using time backpropagation (BPTT), which causes gradient disappearance. To tackle this problem, the recurrent neural network (RNN) cell is replaced with a port cell called the LSTM cell. Due to its reliance on subjective evaluations and EEG data categorization, MLSTM has no concept of gap length. Learning the long-term time-series dependencies and examining the temporal correlations of the EEG signals helps MLSTM achieve better outcomes when it comes to EEG signal categorization difficulties. MLSTM supremacy is also due to the model's explicit nature to prevent the longterm problem of reliance. e MLSTM 1, MLSTM 2, and MLSTM 3 architectures are shown in Figure 3 . Python 3.8 and TensorFlow 2.2.0 were used to build the MLSTM networks on the back-end. e simplest option is the MLSTM 1 design, which has 164 memory units in a single layer. e blocks of the memory store information, and these blocks are modified by three major processes, known as gates, which control the memory. Using the forget gate, it is possible to do what was left out of the cell state in the suggested way. e incoming EEG signal dataset is purged using the forget gate, which multiplies a filter. e MLSTM network must be optimized by making this change.

In the MLSTM_2 design, there is just one LSTM layer and a maximum of 268 memory units. A total of 732, 564, and 236 memory units are available in the MLSTM_3 architecture's three LSTM levels.

ere are 0.4 and 0.2 dropout layer probabilities. With dropout regularization, the MLSTM model has the advantage of better memorization without losing any of its core features. Overfitting is decreased, the model is trained quicker, and MLSTM model prediction performance is improved by employing the dropouts of 0.4 and 0.2. An activation function, known as " tanh," is used at the model's internal layer. Each of the three MLSTM networks has an output of 32, 56, and 64 units attributed by the " tanh" activation function. e "Softmax" activation function is utilized in the output layer to classify two types of emotions:

In our three MLSTM designs, overfitting was decreased by limiting unit coadaptation in the dropout layer. Categorical cross-entropy is the loss function in the dense layer for various network configurations. e Adaptive Moment Estimation optimizer (Adam) is used with a learning rate of 0.001. Using the MinMaxScaler function, the input attributes of the dataset are normalized. Each feature's contribution is consistent, thanks to this method's normalization. Network parameters vary during training, reducing internal covariate transition and shifting network activation distribution. Internal covariance change is reduced as a result of network normalization. Ping weights were also restricted to prevent them from expanding throughout the whole site due to the optimization procedure. e mechanism becomes more regular due to normalization, which was not the desired outcome. 1000 iterations with 40 batch sizes were used to evaluate the proposed model's output. e gathered EEG dataset was used to evaluate each MLSTM prototype. Based on our trials with nearby architecture, we selected these buildings and qualities (in terms of layers and nodes). Because they outperform their nearby design in terms of accuracy throughout 500 epochs, these three structures were considered in our analysis. According to the correctness of the MLSTM_3 architecture, Table 1 presents the assessment results on several epochs. e parameters for evaluating performance will be explained in the next section. Journal of Healthcare Engineering

precision � TP TP + FP .

It is the ratio of true negative to the true negative plus false positive: [26] test can provide the significant difference of the samples. It calculates the p values of the model and describes its effectiveness.

e result of the proposed architecture is presented in this section. e details of the device used for implementing all the models MLSTM_1, MLSTM_2, MLSTM_3, MLP, KNN, SVM, LibSVM, and CNN are Intel 5, 16 GB RAM, 1 TB hard disk, and the language used for implementing all the models is Python. e parameter values for implementing the MLP, KNN, SVM, LibSVM, and CNN models are taken from [27] [28] [29] [30] [31] , respectively.

e parameter values used for implementing the MLSTM architectures are dropouts of 0.2 and 0.4, Adam optimizer, 0.001 learning rate, 500 epochs (best accuracy achieved but trained till 1000 epochs). e 70-30 partition is used to evaluate the performance. In this, 70% of the sample is kept in training data and 30% is used for testing the architecture. All the performance measures are calculated using this partition scheme, demonstrating the results.

e three deep learning MLSTM architecture has been implemented in the research for two classes of emotion classification to analyze the mental state during the pandemic with EEG signals. To show the dominance of the LSTM [11] architecture, it has been compared with the ML models. It is proved from the literature that due to the presence of computational and classification unit, LSTM architecture showed superiority over ML models that is also confirmed from our results. e ability to handle the sequential and time-series data makes them one of the best choices for emotion recognition. is is much higher than the MLSTM_1 architecture. e MLSTM_3 architecture has the best accuracy among all the architecture. e maximum, average, and minimum accuracy achieved is 91.26%, 88.92%, and 85.18%, respectively. is confirms that MLSTM_3 is the best among the three architectures. Figure 4 represents the accuracy as compared to the training loss. e loss calculation is done by calculating the difference between actual and predicted values. e loss function used here is categorical cross-entropy, as it is one of the most used loss functions. Figure 5 represents the accuracy as compared to the testing loss. It is clear from the graph that the model has an average accuracy of almost 90% with minimal loss, which shows the superiority of our model for emotion recognition. Table 3 describes the performance of the model in terms of accuracy.

e results show that MLSTM_3 architecture outperforms the other two architectures.

A comparison of the proposed architecture with the other machine learning models is made. All the models used in this study are implemented using the same set of features.

e environment for developing the model is also the same that gives a fair comparison of the work. e accuracy of MLP is 75.34%, the accuracy of KNN is 72.42%, the accuracy of SVM is 78.23%, the accuracy of LibSVM is 81.72% the accuracy of CNN is 79.72%, and the accuracy of MLSTM_3 is 91.26%. e accuracy of the proposed architecture is almost 10% higher than the second-best model that showcases the effectiveness of MLSTM_3 architecture. e other performance measures are described in Table 4 . It is clear from the result that MLSTM_3 architecture has very low false-positive and false-negative rates. at is also very significant in terms of measuring the model performance. Bold shows the sensitivity, precision, and specificity values in percentage obtained from the confusion matrix when 70 percent data are used for training, 30 percent data are used for testing and MLSTM_3, and the proposed classifier is applied for data classification. 6 Journal of Healthcare Engineering Table 5 presents the Mann-Whitney test [26] .

e p value describes the significance of the model with other models. If the p value is larger than highly significant, their low p value describes the insignificance of the architecture. Here, all the values are highly significant, which is very important in measuring the model.

In this paper, an MLSTM architecture for two classes of emotion classification to analyze the mental state during the pandemic is proposed. We have collected a dataset of 40 people for doing the emotion classification. Also, the MLSTM architecture was developed to classify positive and negative emotions; the results ensure that it outperforms all other methods for emotion classification. e limitation of this work is that we have used a 1-channel EEG device for data collection. It will be interesting to analyze the result of other EEG devices like 4-channel, 14-channel, and 16channel. In the future, we want to create a more extensive dataset and analyze our result on more participants that may help our classifier to improve.

e data are available on request from the corresponding author.

e authors declare that there are no conflicts of interest.

e psychological impact of quarantine and how to reduce it: Rapid review of the evidence

Adolescent development and risk for the onset of social-emotional disorders: A review and conceptual model

Risk perceptions and risk characteristics

Risk perceptions of covid-19 around the world

Public mental health under the long-term influence of covid-19 in China: Geographical and temporal distribution

Multidisciplinary research priorities for the covid-19 pandemic: A call for action for mental health science

Ethnicity and covid-19: An urgent public health research priority

e complex neurobiology of resilient functioning after childhood maltreatment

Social isolation, loneliness and physical performance in older-adults: Fixed effects analyses of a cohort study

Emotion recognition using fourier transform and genetic programming

A lstm based deep learning network for recognizing emotions using wireless brainwave driven system

A long short term memory deep learning network for the classification of negative emotions using eeg signals

Personality, emotion, and individual differences in physiological responses

Emotion assessment: Arousal evaluation using EEG's and peripheral physiological signals

Classification of electroencephalogram signal for the detection of epilepsy using innovative genetic programming

Classification of extraversion and introversion personality trait using electroencephalogram signals

Twin neural networks for efficient eeg signal classification

Toward standard user-centered eeg bmi performance evaluation

A novel fitness function in genetic programming to handle unbalanced emotion recognition data

e dependence of electrode impedance on the number of performed eeg examinations

Quality assessment of single-channel eeg for wearable devices

Detecting antisocial personality disorder using a novel machine learning algorithm based on electroencephalographic data

Electroencephalographic abnormalities in antisocial personality disorder

Deep in thought while driving: An EEG study on drivers' cognitive distraction

Review on portable eeg technology in educational research

On a test of whether one of two random variables is stochastically larger than the other

Human emotion recognition and analysis in response to audio music using brain signals

Evaluation of feature extraction techniques in emotional state recognition

Single trial classification of eeg and peripheral physiological signals for recognition of emotions induced by music videos

Emotion detection using perceptual based speech features

Mean amplitude spectrum based epileptic state classification for seizure prediction using convolutional neural networks