key: cord-0950242-rspx897u
authors: Malche, Timothy; Tharewal, Sumegh; Tiwari, Pradeep Kumar; Jabarulla, Mohamed Yaseen; Alnuaim, Abeer Ali; Hatamleh, Wesam Atef; Ullah, Mohammad Aman
title: Artificial Intelligence of Things- (AIoT-) Based Patient Activity Tracking System for Remote Patient Monitoring
date: 2022-03-01
journal: J Healthc Eng
DOI: 10.1155/2022/8732213
sha: ed85e745d5b457d19a2d7a2905acb5a184b42394
doc_id: 950242
cord_uid: rspx897u

Telehealth and remote patient monitoring (RPM) have been critical components that have received substantial attention and gained hold since the pandemic's beginning. Telehealth and RPM allow easy access to patient data and help provide high-quality care to patients at a low cost. This article proposes an Intelligent Remote Patient Activity Tracking System system that can monitor patient activities and vitals during those activities based on the attached sensors. An Internet of Things- (IoT-) enabled health monitoring device is designed using machine learning models to track patient's activities such as running, sleeping, walking, and exercising, the vitals during those activities such as body temperature and heart rate, and the patient's breathing pattern during such activities. Machine learning models are used to identify different activities of the patient and analyze the patient's respiratory health during various activities. Currently, the machine learning models are used to detect cough and healthy breathing only. A web application is also designed to track the data uploaded by the proposed devices.

Remote Patient Monitoring (RPM) is an emerging research field that has gained much attention as today's world faces a tremendous need for it in the time of the pandemic. e pandemic has exposed many problems, such as overloading hospitals, the risk of healthcare workers being infected, etc. As a result, there is a need for an RPM system to remotely monitor COVID-negative patients and make the healthcare facility available to COVID-positive patients in the event of an emergency. Even COVID-positive patients can be monitored remotely if they are not in a critical situation. e RPM system assists in remote monitoring in real-time, which enables the detection of specified illnesses rapidly. Using RPM, patients can be monitored from a remote location away from hospital environments using technology. e benefits of using RPM are early detection of patients' health conditions, real-time acquisition of patients' health status, preventing critical health situations, hospitalization, and even deaths.

With the help of RPM, medical care services can be provided to patients in case of an emergency. RPM can be implemented for patients with different target groups, for example, patients having chronic illnesses and disabilities, patients with mobility issues, and elderly patients. e primary purpose of RPM is to deliver good healthcare facilities to patients from the comfort of their homes and give them psychological advantages as they are away from the hospital environment. erefore, the RMP gives patients the freedom and comfort of home and allows them to perform daily activities in their friendly environments while being actively monitored for health conditions. e doctors can also monitor and keep track of patients' health remotely with the help of an automated system capable of generating an alert if something is wrong with the patients' health parameters [1] .

To enable such features in the RMP system, this study aims to build a health monitoring device with machine learning (ML) capabilities, network infrastructure, and software applications that will help doctors monitor the activities of their patients and vitals during such activities. Currently, this study focuses on monitoring the activity parameters of patients, such as whether a patient is walking, sleeping, or exercising, and the vitals during these activities, such as heart rate, body temperature, and oxygen level. e novelty of this research is that it uses an edge intelligent decision-making system that immediately identifies the health conditions, suggests patients, and alerts doctors in real-time. Although the proposed approach can analyze only one aspect of RPM, the other parameter for analysis can be easily added to the system, enhancing its capabilities. erefore, the proposed approach focuses on tracking the patients' activities and different vitals during those activities by implementing the innovative idea to build future intelligent healthcare devices capable of making a decision. e main contributions of this article are as follows:

(i) An Internet of ings-(IoT-) enabled health monitoring device is designed using the machine learning models to track patient's activities such as running, sleeping, walking, and exercising, the vitals during those activities such as body temperature and heart rate, and patient's breathing pattern during such activities. (ii) Machine learning models are used to identify different activities of the patient. (iii) ey also analyze the patient's respiratory health during various activities. (iv) A web application is also designed to track the data uploaded by the proposed devices.

e rest of the article can be organized as follows: Section 2 presents the related work. Section 3 presents the proposed methodology. System architecture is presented in Section 4. Experimental analyses are presented in Section 5. Section 6 concludes the article.

Nguyen and Silva [2] discussed applications for remotely checking patients for neutralizing cardiovascular illnesses. A telemonitoring system that uses hardware and programming contraptions is proposed by Szydloand Konieczny in [3] . e system is designed to screen diverse heart-related diseases. ECG and heartbeat monitoring system is described in [4] by Lanata et al. In [5] , Kozlovszky et al. proposed a method for checking cardiovascular ailments. Blood pressure (BP) and ECG checking systems are discussed in [6] by Ramesh et al. ey have developed an algorithm that uses five states. Another approach by Kumar and Kotnana in [7] monitors pulse, heart rate, blood pH level, ECG, and body temperature with the help of different sensors. In [8] , Ferreira et al. proposed a system for bedridden patients to monitor ECG, body temperature, oxygen levels, galvanic skin response, and airflow in the lungs. Another approach in [9] proposed by Sannino et al. is used for fall detection of patients using an accelerometer. e system can also monitor other vital signs such as body temperature, oxygen level, ECG, and heart rate. Mishra and Agrawal created a system to collect physiological data for ECG and pulmonary artery pressure (PAP) [10] . e system sends the collected data like text messages. In [11] , Pinheiro et al. developed a system for cardiac analysis of a patient in a wheelchair.

Wang et al. in [12] discussed a patient activity monitoring system with fall detection that uses an Android-based Smartphone. Gibson et al. [13] addressed fall detection that uses multiple comparators and a classifier. In [14, 15] , a fall detection system is proposed by Paoli et al. and Naranjo-Hernandez et al., which uses a wireless sensor node equipped with an accelerometer and location device. Yu et al. in [16] and Mastorakis and Makris in [17] developed a contactless system based on a camera module for fall detection and physical activity. Greene et al. in [18] developed a method for fall detection based on the Internet of ings (IoT) that uses an Amazon Echo device, a speaker, and a webcam for the functionality.

Lanata et al. in [4] proposed a system to monitor patients suffering from mental bipolarity. Nadeem et al. [19] developed a system for EEG signal analysis using Bayesian learning. In [20] , Karan et al. proposed a plan to monitor diabetic patients with the help of artificial neural networks. Zhan et al. [21] developed a system called HopkinsPD. is system is designed for patients who are suffering from Parkinson's disease. e system can monitor such patients remotely. Price et al. in [22] developed a system to monitor cognitive fatigue for brain injury people. Prabhakar and Rajaguru proposed the RPM system to classify epilepsy in their study [23] . In [24] , Adams et al. proposed another remote patient monitoring system for psychoanalysis. Lakshminarayanan et al. in [25] developed a smartphonebased eye care system that has shown promising results. Rotariu et al. in [26] developed a wireless sensor networkbased system for composition and respiration analysis. e system can also analyze melanoma and other skin-related diseases. In [27] , Khattak et al. proposed a constrainedoriented application protocol (CoAP) based low-power personal area network system for healthcare. Another application [28] by Gonzalez et al. was developed to assist patients in an accident. Patel et al. in [29] discussed wearable sensors and systems based on microelectromechanical systems (MEMS) and system on a chip (SoC) implementation for rehabilitation applications. Sardini et al. in [30] developed a system using a thin copper wire embedded in a t-shirt with a piezoelectric actuator. e system is used for posture monitoring during rehabilitation. Benelli et al. in [31] proposed a method to measure BP, ECG, body weight, spirometry, and glycemia. A similar approach is proposed in [32] by Sorwar and Hasan to monitor vitals on multiple parameters. E-Ambulance [33] is a system developed by Almadani et al. for remote monitoring of patients. e system is deployed in an ambulance and monitors patients' vital signs while going to the hospital or medical care facility. A Smart Rehabilitation Garment (SRG) for posture correction was developed by Wang et al. [34] . Magno et al. [35] demonstrated a wireless, low-power remote monitoring system with on-body sensors. Serhani et al. [36] created the Smart Mobile End-to-End Monitoring (SME2EM) system for long-term monitoring of disorders that cause illness. e system is deployed as web services and an algorithm to filter and wrapper selection. Al-Naji et al. in [37] developed a camera-based monitoring system to monitor children in a hospital environment. e system monitors respiration rates and detects apnoea using a Kinect camera.

Yew et al. [38] proposed an RPM system based on IoT. e system monitors ECG signals and uses the MQTT protocol for data transmission. Annis et al. [39] , in their research, studied the RPM system to monitor COVID-19 patients. e system tracks the details of patients based on vitals. Taiwo and Ezugwu [40] also proposed a system to monitor COVID-19 patients.

e system is called an intelligent home healthcare support system (ShHeS) and is based on IoT. Iranpak et al. in [41] proposed RMP based on LSTM (long short-term memory) deep neural network algorithm. El-Rashidy et al. in [1] also proposed RPM. e authors have conducted a research survey on principles, trends, and challenges of RPM for chronic disease. ese systems are summarized in Table 1 . In comparison to the systems described in related works, the proposed system has a novel method of analyzing and monitoring the health parameters of patients, and the proposed system is also extensible as more health parameters can be configured easily.

Remote patient monitoring (RPM) systems are used to collect various physiological data from patients, which includes electrocardiogram (ECG), body temperature, heart rate, blood oxygen level, blood pressure, respiration rate, and EEG. Various research has been conducted on RPM systems. e proposed system consists of a sensor node to track patients' vitals during different activities which patients perform. e sensor node runs a machine learning model to analyze the vitals during various activities. e sensor node detects the patient's body temperature, heart rate, and oxygen level when the patient is walking, sleeping, or exercising [42, 43] . Analyzing vitals during various activities is critical because these vitals are different during various activities.

us, by analyzing the vitals during different activities, the doctor can prescribe treatment or give suggestions to patients. For example, if the heart rate is higher during exercise, the doctor may advise the patient or prescribe therapy. Similarly, if the doctor has advised a patient to do some exercise regularly, the doctor can track whether the patient is doing it. erefore, the proposed solution is helpful for doctors and patients. e doctor can track whether the patient is taking proper rest and exercising daily; on the other hand, patients get health information and alerts if there is a critical situation. e proposed sensor node collects patients' data using the sensors attached to the nRF5340 Development Kit (DK). e connected sensors are accelerometer, microphone, pulse oximeter, heart rate sensor, and temperature sensor. e accelerometer enables monitoring different patient physical activities, including walking, sleeping, exercising, and running. All these various activities of a patient are identified using the ML model. ML model runs locally on the device and can classify all these patients' activities accurately. With the help of a microphone, the patient's respiratory health can be analyzed. e classification of the patient's respiratory health is also done with the help of an ML model running on the sensor node.

e study "Artificial intelligence model detects asymptomatic Covid-19 infections through cell phone recorded coughs" by MIT [44] and research paper "COVID-19 Artificial Intelligence Diagnosis Using Only Cough Recordings" [45] show that this technique is beneficial in detecting the infections. e proposed system uses the Mel Frequency Cepstral Coefficients (MFCC) [46, 47] approach to extract features from the patient's speech signal and compare it with the existing COVID-19 and other sound data available in the database. e study uses two ML models for identifying physical activities and respiratory health. e results of these ML models are programmatically aligned with the results of other sensors such as the pulse oximeter, heart rate sensor, and body temperature sensor to give an overall report of patients' health conditions. For example, when a patient is exercising, the temperature, heart rate, and pulse data may be different from the data acquired while a patient is sleeping. e sensor node also continuously captures the patient's respiratory data via a microphone and detects any anomaly in the respiratory health, such as whether the patient is breathing normally, breathing fast, or having a frequent cough. All these data enable a further study on a different aspect of a patient's health. Figures 1 and 2 depict the other components of the sensor node. Table 2 contains the list of essential hardware components used for developing the system prototype.

In the proposed system, the sensor node captures the oxygen level, heart rate, and temperature of patients during different physical activities.

is information helps the doctors to study different health parameters based on the activities performed by patients. Along with the physical activities, the sensor node also monitors the respiratory health of the patient. Respiratory health is analyzed while the patient is breathing or making vocal sounds during different activities.

e microphone attached to the sensor node is used to Journal of Healthcare Engineering 3 capture data for respiratory health analysis. One outcome of the analysis of respiratory health is to detect whether the patient is breathing normally or fast or having a cough. By respiratory analysis using vocal sounds, other conditions such as pneumonia can also be detected. e system works by collecting the data from a sensor node and sending them to the gateway node via Bluetooth or Zigbee. e gateway is present at the patient's sight and displays the result of the analysis by the sensor node. e gateway forwards the data to IoT cloud via Message Queue Telemetry Transport (MQTT) protocol. e data are stored in cloud and can be accessed by the doctors in the dashboard of the web application. Figure 3 illustrates the overall system architecture. Figure 4 depicts the sensor node and gateway node prototype. e gateway node is built using Raspberry Pi 3 (RPi3). e RPi3 has built-in Bluetooth Low Energy (BLE), which is used to communicate with the sensor node. e gateway node displays the health information to the patient locally and transmits the data to the IoT cloud server via the MQTT protocol. e gateway node also stores the most recent data in the local database. Once the new readings have been received, the gateway node deletes the previous data, thus making space for more data. erefore, the gateway node only keeps the current data in a local database, which will be displayed to the patient locally. If patients wish to see historical data, they may use the web application and fetch data from the remote server. Figure 5 shows the application dashboard. 

Application area AIenabled? Nguyen and Silva [2] Cardiovascular illnesses monitoring No Szydlo and Konieczny in [3] Heart-related diseases monitoring No Lanata et al. in [4] , and Kozlovszky et al. [5] Cardiovascular ailments analysis No Ramesh et al. [6] Blood pressure (BP) and [22] Cognitive fatigue for brain injury monitoring No Prabhakar and Rajaguru [23] Classifying epilepsy No Adams et al. [24] Psychoanalysis monitoring No Lakshminarayanan et al. [25] Eyecare Yes

Rotariu et al. [26] Respiration analysis, melanoma, and other skin-related diseases No

Khattak et al. [27] Healthcare system No Gonzalez et al. [28] Assist patients in an accident No Patel et al. [29] Rehabilitation applications No Sardini et al. [30] Posture monitoring during rehabilitation No Benelli et al. [31] BP, ECG, body weight, spirometry, and glycemia No Sorwar and Hasan [32] E-health monitoring No Almadani et al. [33] Ambulance and monitors patients' vital signs No Wang et al. [34] Posture correction No Magno et al. [35] On-body sensors for monitoring No Serhani et al. [36] Monitoring of disorders that cause illness No Al-Naji et al. [37] Camera-based monitoring system to monitor children in a hospital environment No Yew et al. [38] ECG monitoring No Annis et al. [39] and Taiwo and Ezugwu [40] Monitor COVID-19 patients No Iranpak et al. [41] General monitoring Yes Wang et al. [12] Disease-based monitoring Yes It is the brain behind the system. It is an SoC, powered with two-arm ® cortex ® -M33 processors, has all the processing power to execute machine learning model and built-in communication modules such as BLE, Zigbee, read. e kit is used to design sensor nodes.

It has Broadcom BCM2711, quad-core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5 GHz. is kit is used to design gateway nodes for the system.

It is an industrial motion MEMS sensor expansion board. It embeds the following modules. (i) ISM330DHCX 3-axis accelerometer (ii) 3-axis gyroscope (iii) IIS2MDC 3-axis magnetometer (iv) IIS2DLPC 3-axis accelerometer (v) IMP34DT05 digital microphone e accelerometer and microphone are used in the proposed system. SparkFun Digital Temperature Sensor It is used for reading body temperature.

SparkFun Pulse Oximeter and Heart Rate Sensor It is used to monitor patients' heart rate and oxygen saturation.

Journal of Healthcare Engineering 5

e proposed system works on two machine learning models: voice data and physical activity data. e proposed method uses Audio Mel Frequency Cepstral Coefficients (MFCC) to extract features from the audio signals for voice data. Edge Impulse TinyML Tool [48] is used to build the proposed model. Tables 3 and 4 show the parameters and cepstral coefficients used for MFCC.

Currently, the ML model predicts two labels for respiratory health that are negative and positive. Negative here means that the patient has healthy breathing. Positive means that the patient has some sort of respiratory sicknesses, such as a cough. e main aim is to detect COVID-19 and Journal of Healthcare Engineering pneumonia-positive patients, but the model could not be built due to the unavailability of many audio datasets. e future scope will be the process of collecting the dataset and detecting COVID-19 positive and pneumonia patients. Table 5 shows the different settings used for training the model. e neural network architecture for the respiratory health model is given in Figure 6 . It consists of an input layer having 650 features, a hidden layer, and an output layer with 2 features. e hidden layer has reshaped layer (13 columns), 1 D convolutional/pool layer (8 neurons, 3 kernels, and 1 layer), dropout layer (rate 0.25) for more details refer to [49, 50] . Tables 6 and 7 show the training performance of the model. e model is working accurately while being tested on the sample data. As shown in Table 6 , the accuracy of a model during training is 100%. e confusion matrix of the model is as follows. e result of the live classification is represented in Table 8 . In the obtained results, it has been observed that the model can predict cough and healthy breathing with 100% accuracy. Table 8 shows how the model is performing during the live classification for the negative and positive labels at different time stamps. e second model, which was developed for the proposed system, works on physical activity data. e patient's physical activity data are collected using the 3-axis accelerometer sensor. Currently, the model is trained and tested on four physical activities: exercising, sleeping, running, and stationary. e model works using spectral analysis, which is excellent for analyzing motions. e frequency and power characteristics of a signal over time are extracted from the data collected using input axes X, Y, and Z of the accelerometer. en, the learning block for classification using Keras [51, 52] has been applied, which helps to learn patterns from data. Anomaly Detection using K-means [53] [54] [55] is also used to find outliers in new data.

is enables the recognition of unknown states and complements classifiers. Table 9 shows the parameters of the spectral feature used in the model, which is scaling, filter, and spectral power. e frequency domain and spectral power are represented in Figures 7(a) and 7(b) , respectively. e architecture of the neural network (NN) has been illustrated in Figure 8 . NN consists of an input layer with 33 features, hidden layers, and an output layer with four attributes. e hidden layer consists of two dense layers of 20 and 10 neurons, respectively. e training settings are given in Tables 10-12. e training settings to train the model are provided in Table 10 .

e parameters used are some training cycles, learning rate, and minimum confidence rating. e training performance of the ML model for physical activity is given in Table 11 , which indicates that the model is performing accurately. e confusion matrix of the model is given in Table 12 . Currently, the model uses four labels that are exercising, moving, running, and stationary. F1 score for each label is also given. Figure 9 (a) depicts the feature explorer for the model. It is observed that the model can predict each label accurately. Each label in the graphical representation in Figure 9 (a) can be clearly seen.

After the live classification, the obtained results are represented in Figures 9(b) and 9(c). Figure 9 (b) shows the anomaly explorer. Figure 9 (c) shows spectral features from the live classification. e live classification for the "moving" label is shown in Figures 9(b) and 9(c). It is proved that there is no anomaly while classifying data for the moving label. Similarly, from Figure 9 (c), it is proved that the model is classifying the moving label accurately. Table 12 contains a few sample rows of data from the live classification that show that the model is classifying the "moving" label accurately. Table 13 shows that at a different timestamp, the moving label is classified accurately among all four labels during the live classification of data and no anomaly in the data has occurred. Similarly, Figures 9(d) , 10(a), and 10(b) and Tables 14-16 depict the results from the live classification of accelerometer data for the labels "stationary," "exercising," and "running," respectively. ese figures and table data show that like the moving label, the model is also classifying the other three labels "stationary," "exercising," and "running" accurately and no anomaly has occurred while classifying the data. Table 17 and Figure 10 (c) show the overall performance of the model developed for the physical activity analysis of the patient. Figure 11 shows the workflow of the models. As discussed in the previous section, the two models have been developed. For the respiratory health model, the audio data have been fed, which are processed using MFCC. e accelerometer data have been provided for the physical activity model, which are analyzed using spectral analysis.

An Internet of ings-(IoT-) enabled health monitoring device was designed using the machine learning models to track patient's activities such as running, sleeping, walking, and exercising, different vitals during these activities such as body temperature and heart rate, and patient's breathing pattern during such activities. Machine learning models were used to identify different activities of the patient. Currently, the machine learning models are used to detect cough and healthy breathing only. A web application was also designed to track the data uploaded by the proposed devices. e proposed health monitoring device developed in this study did not cause any discomfort to the patient. It can be easily removed and worn again whenever required without any assistance. e obtained results revealed that the proposed system can be very helpful in monitoring the patients remotely. In the near future, we will extend the proposed device for other kinds of diseases. Additionally, in this article, no machine learning model is proposed. erefore, we will try to design our own lightweight machine learning model to obtain better results.

e data used to support the findings of this study are available from the corresponding author upon request.

is article does not contain any studies with human participants performed by any of the authors.

ere are no conflicts of interest between authors.

Mobile health in remote patient monitoring for chronic diseases: principles, trends, and challenges

Use of smartphone technology in cardiology

Mobile devices in the open and universal system for remote patient monitoring

Complexity index from a personalized wearable monitoring system for assessing remission in mental health

Cardiovascular and diabetes focused remote patient monitoring

A mobile software for health professionals to monitor remote patients

Design and Implementation of Portable health monitoring system using PSOC mixed signal Array chip

Vital signs monitoring system using radio frequency communication: a medical care terminal for beddridden people support

A supervised approach to automatically extract a set of rules to support fall detection in an mHealth system

Continuous health condition monitoring by 24× 7 sensing and transmission of physiological data over 5-G cellular channels

Dual architecture platform for unobtrusive wheelchair user monitoring

A remote health care system combining a fall down alarm and biomedical signal monitor system in an android smart-phone

Multiple comparator classifier framework for accelerometer-based fall detection and diagnostic

A system for ubiquitous fall monitoring at home via a wireless sensor network and a wearable mote

Personalization and adaptation to the medium and context in a fall detection system

A posture recognition-based fall detection system for monitoring an elderly person in a smart home environment

Fall detection system using Kinect's infrared sensor

IoT-based fall detection for smart home environments

Application specific study, analysis and classification of body area wireless sensor network applications

Diagnosing diabetes using neural networks on small mobile devices

High Frequency Remote Monitoring of Parkinson's Disease via Smartphone: Platform Overview and Medication Response Detection

Towards a mobile assistive technology for monitoring and assessing cognitive fatigue in individuals with acquired brain injury

Development of patient remote monitoring system for epilepsy classification

Mobile devices for the remote acquisition of physiological and behavioral biomarkers in psychiatric clinical research

Smartphone science "in eye care and medicine" in Eye Care and Medicine

A wireless sensor network for remote monitoring of bioimpedance

CoAP-based healthcare sensor networks: a survey

Survey of WBSNs for pre-hospital assistance: trends to maximize the network lifetime and video transmission techniques

A review of wearable sensors and systems with application in rehabilitation

Wireless wearable T-shirt for posture monitoring during rehabilitation exercises

Health monitoring and wellness for all, a multichannel approach through innovative interfaces and systems

Smart-TV based integrated e-health monitoring system with agent technology

E-AM-BULANCE: real-time integration platform for heterogeneous medical telemetry system

Smart rehabilitation garment for posture monitoring

A versatile biomedical wireless sensor node with novel drysurface sensors and energy efficient power management

SME2EM: smart mobile end-to-end monitoring architecture for lifelong diseases

Real time apnoea monitoring of children using the Microsoft Kinect sensor: a pilot study

Iot based real-time remote patient monitoring system

Rapid implementation of a COVID-19 remote patient monitoring program

Smart healthcare support for remote patient monitoring during covid-19 quarantine

Remote patient monitoring and classifying using the internet of things platform combined with cloud computing

COVID-19 artificial intelligence diagnosis using only cough recordings

Novel deep transfer learning model for COVID-19 patient detection using X-ray chest images

Multiobjective evolutionary optimization techniques based hyperchaotic map and their applications in image encryption

Graphology based handwritten character analysis for human behaviour identification

Deep neural networkbased speaker-aware information logging for augmentative and alternative communication

Edge Impulse

Smartphone app detects asymptomatic COVID-19 BY recording forced coughs analysis

Visibility improvement and mass segmentation of mammogram images using quantile separated histogram equalisation with local contrast enhancement

Metaheuristic-based deep COVID-19 screening model from chest X-ray images

A fully convolutional neural network-based regression approach for effective chemical composition analysis using near-infrared spectroscopy in cloud

Deep learning approach for microarray cancer data classification

Multi-modality medical image fusion technique using multi-objective differential evolution based deep neural networks

Human activity recognition and embedded application based on convolutional neural network

Providing security and privacy to huge and vulnerable songs repository using visual cryptography