key: cord-0783674-k22g5byk
authors: Safari, Aref; Hosseini, Rahil; Mazinani, Mahdi
title: A Novel Deep Interval Type-2 Fuzzy LSTM (DIT2FLSTM) Model Applied to COVID-19 Pandemic Time-Series Prediction
date: 2021-09-30
journal: J Biomed Inform
DOI: 10.1016/j.jbi.2021.103920
sha: bd20737c8e76bb6210acfe93548c0dc4ac8dc22e
doc_id: 783674
cord_uid: k22g5byk

Currently, the novel COVID-19 coronavirus has been widely spread as a global pandemic. The COVID-19 pandemic has a major influence on human life, healthcare systems, and the economy. There are a large number of methods available for predicting the incidence of the virus. A complex and non-stationary problem such as the COVID-19 pandemic is characterized by high levels of uncertainty in its behavior during the pandemic time. The fuzzy logic, especially Type-2 Fuzzy Logic, is a robust and capable model to cope with high-order uncertainties associated with non-stationary time-dependent features. The main objective of the current study is to present a novel Deep Interval Type-2 Fuzzy LSTM (DIT2FLSTM) model for prediction of the COVID-19 incidence, including new cases, recovery cases, and mortality rate in both short and long time series. The proposed model was evaluated on real datasets produced by the world health organization (WHO) on top highly risked countries, including the USA, Brazil, Russia, India, Peru, Spain, Italy, Iran, Germany, and the U.K. The results confirm the superiority of the DIT2FLSTM model with an average area under the ROC curve (AUC) of 96% and a 95% confidence interval of [92-97] % in the short-term and long-term. The DIT2FLSTM was applied to a well-known standard benchmark, the Mackey-Glass time-series, to show the robustness and proficiency of the proposed model in uncertain and chaotic time series problems. The results were evaluated using a 10-fold cross-validation technique and statistically validated through the t-test method. The proposed DIT2FLSTM model is promising for the prediction of complex problems such as the COVID-19 pandemic and making strategic prevention decisions to save more lives.

The COVID-19 pandemic was first reported in Wuhan in South China in early December 2019. The cause of the pandemic was later recognized as a novel coronavirus known as SAR-COV-2. As of November, 2020, there have been 51,100,521 COVID-19 cases, with 1,324,714 deaths and 32,951,517 recovered cases. The U.S. had the highest number of COVID-19 death cases with 247,000 deaths [1] . During the last few months, several types of research have been conducted to realize the incidence of the disease and modelling uncertainties and prediction of the COVID-19 patterns during time series. The WHO is collecting and encouraging researchers to provide new methods and knowledge on COVID-19 and compiling it in a database to obtain a better strategy for handling this global issue. There are nonstationary time series features in Covid-19 pandemic patterns whose statistical attributes such as means and variances change over time. This study aims to model uncertainty sources associated with non-stationary time series features in real-world applications such as the COVID-19 pandemic. 

This section presents an overview of data analytics and computational intelligence models applied to COVID-19

prediction.

In [2] , the author described a bioinformatics inference system for modeling the SARS-CoV-2, which focuses on the design of a synthetic vaccine and a preventative peptidomimetic antagonist against the SARS-CoV-2 with computeraided techniques. In [3] , the authors designed a framework for predicting the incidence of the COVID-19 pandemic based on the data transportation in China and applied machine learning methods for prediction of epidemic spread of Coronavirus driven by Spring Festival Transportation in time-series forecasting. In [3] - [5] , population-based methods were proposed to predict the candidate targets. These methods used immune responses information through the sequence homology of the COVID-19 patterns. Also, the authors deliberated what could still be done to manage the COVID-19 pandemic with the standard measures of separation and contact tracing in risky countries. A computational spatial-temporal approach for modeling the distribution of SARS-CoV-2 in China was proposed in [6]- [7] with a focus on culinary spice bioactive as potential therapeutics against sars-cov-2: computational investigation and data-driven modeling analysis. In [8] , the authors proposed a propagation analysis method for modeling the time series. Their research proposed the transmission characteristics of the epidemic at different stages designed based on a Gaussian distribution theory to construct an innovative method of coronavirus transmission during a period.

On the other hand, in [9] and [10] , the authors describe various data mining and statistical techniques for predicting the COVID-19 pandemic in China with reasonable outcomes. They modeled an estimation of the daily number of new cases. Also, their proposed model emphasizes the control of the infection rates in short time-series. The authors in [11] proposed a deep LSTM approach to predict the SARS-CoV-2 new infection rates. They showed the trends of different countries and compared them to Canadian data to forecast the upcoming infections. They used LSTM to model the ending era of the pandemic in Canada for long time-series, but the challenge of non-optimal parameters in their model affected on the results of their work. Also, a computational intelligence framework for COVID-19

incidence based on a deep neural network technique and a social mimic tuning method associated with fuzzy logic and stacking methods in [12] . They described the dynamics of epidemic incidence of Coronavirus disease during a daily horizon. Also, in [13] , the authors model a real-time prediction framework for the COVID-19 pandemic in a monthly format. These models were based on LSTM, convolutional neural network (CNN), and multi-layer perceptron neural nets using the dataset. Meanwhile, a novel SIR method has been proposed to model the pandemic incidence in Pakistan [14] . They projected the epidemic curve as realistic quarantine, social distancing, and probable medication scenarios in a short time-series.

In [15] - [16] , COVID-19 patients; however, they failed to use reasonable pandemic predictions and described a clinical SIR estimation method for modeling a time-series prediction of the COVID-19 new cases rates in a short time-series based on parameter approximation of time-dependent rates in China and Iran datasets. However, the main issue and limitation of their work is the lack of an appropriate dataset, which affects the reliability of their proposed model. In [17] - [20] , the authors proposed a dynamic model of the incidence of the viruses based on Auto-regressive integrated moving average (ARIMA), Transmission Dynamics Susceptible Latent Mild Critical Removed (SLMCR), Seasonal-ARIMA and the SIR model to predict the incidence of the COVID-19 in short time series.

The work presented in [21] predicted total cases in the most infected African countries with an ARIMA model in the short time-series. However, their proposed model does not consider the associated uncertainty in the procedure of forecasting. In [22] , a comprehensive mathematical hybrid SIR and ARIMA were applied to predict the new cases in a short-term time series of the COVID-19 pandemic but using a limited number of cases in the dataset. Also, an intelligent hybrid ensemble artificial neural network with fuzzy inference aggregation was proposed in [23] for daily prediction of the COVID-19 incidence. Overall, the reviewed literature in this section, such as [15] - [16] and [21] , cannot model the high-ordered uncertainty for the COVID-19 time-series forecasting. On the other hand, most of the related works in the kinds of literature were focused on the short time-series horizon, which is not suitable for strategic predictions. Also, most of the related works used a limited dataset in their model architecture which affects the reliability of their proposed approaches.

During the recent decade, applications of fuzzy logic systems, especially type-2 fuzzy systems in problems with highorder uncertainty, have been emerged, especially for predictions problems with dynamic and non-stationary problems [24] . Various types of research [25] - [28] have been reported to model uncertainty using fuzzy set theory. Also, they have been applied to real-world applications through type-2 fuzzy logic systems, mostly in modeling uncertainty, control, and predictions [29] - [31] . Also, various neural network models such as recurrent neural networks (RNNs) have been applied to model and predict and model the time series data [11] - [12] and [32] - [34] .

It has been proven that the RNNs, especially the LSTM network, have an excellent capability for solving complex problems. The LSTM model can learn the multi-layered inter-relationships between short and long time series [35]-

[37]. The benefit of an LSTM cell compared to a regular recurrent unit is its cell memory [38] . The cell vector of an LSTM can capture the notion of forgetting part of its earlier stored memory and the part of its new information [39] .

These capabilities can be used in the prediction of non-stationary problems [40] . In addition to the general benefits of using RNNs for time series prediction, the LSTM network can also automatically learn the temporal dependence from the data [41] . Hence, by definition, the LSTM has the capabilities of classical non-linear prediction methods to learn an arbitrary complex mapping from inputs to outputs. However, it cannot model the associated uncertainty in nonstationary features.

 This study takes advantage of type-2 fuzzy logic to develop a new architecture for handling high-order uncertainty in a deep time-series learning model.  A novel deep interval type 2 fuzzy LSTM (DIT2FLSTM) model has been proposed to predict non-stationary problems such as the COVOID-19 pandemic.  The proposed DIT2FLSTM model is capable for both short-and long time-series forecasting.

The rest of the article is ordered as follows: a theoretical research background has been presented in Section 2; the detailed structure and the mathematical model of the proposed DIT2FLSTM model have been denoted in Section 3; Section 4 presented the performance evaluation and comparison results of the proposed DIT2FLSTM based on ROC curve analysis and t-test statistical evaluation; Section 5 provides the discussion and the comparative results. The paper is concluded in Section 6.

This section presents a brief review of the LSTM network. Then, it follows by a review of interval type-2 fuzzy sets (IT2FS) concepts and the mathematic definitions.

The LSTMs have been developed to address classic RNNs' limitations by enhancing the gradient vanishing in the network structure. The cell state in LSTM network stores the long-term information as follows [42] :

Input gate:

(1)

Output gate:

Forget Gate:

is the hidden state of the LSTM at time and are the weight matrices, index is the time step, and is the sigmoid activation function. Fig.1 shows the architecture of the LSTM cell. [42] Also, the gates adjust the states and hidden cells of the LSTM using the following equations [42] :

Hidden state:

where is the element-wise product, and is the activation function. ⨀ ℎ

An interval type-2 fuzzy set represented as , is characterized through a type-2 MF where and

, is the domain of fuzzy set and is the domain of the secondary MF at . is as [46] : 0 ≤ ( , ) ≤ 1

where represents union overall admissible and [44] . ∬

where is the main variable, , an interval in [0,1], is the primary MF of x, u is the secondary variable, and is ∫ ∈ the secondary MF at x. of , i.e., , [46] as:

Also, the FOU in the IT2FLS provides more degree of freedom when designing a fuzzy system [49] . An interval type-2 fuzzy system (IT2FS) architecture contains four components. Fig.3 shows the stricture and components of an IT2FLS. The TSK fuzzy rule type was considered in the proposed model, which has more precision than Mamdani rules [48] .

In this work, the singleton fuzzifier was implemented. [51] . For defuzzification in the last step, the Karnik-Mendel (K.M.) algorithm [52] was applied. The following section presents the novel DIT2FLSTM model.

This section presents the detailed architecture of the novel DIT2FLSTM model. In 3.1.2, the components of the DIT2FLSTM model are described. Then, in 3.2, the mathematical model of the DIT2FLSTM is presented. Finally, the cell structure of the proposed DIT2FLSTM has been discussed in 3.2.1.

The structure of the proposed DIT2FLSTM and it's layers have been illustrated in Fig. 4 . As shown in this architecture, the DIT2FLSTM is fuzzified using the IT2FS.

Input Layer: The first layer is the input layer, which handles the original data in the input of the deep model. The output of the input layer can feed the inputs of the next layer, known as the hidden deep layers.

The main idea is mapping the entire input sequence to a vector and then using an encoder to generate the output sequence. In this layer, the encoder represents the entire input sequence in the hidden layer activities. The DIT2FLSTM model presents a novel cell structure in the encoder/decoder layer. The proposed structure can simply reform according to the dimension of the training time-series data set. The details are described in section 3.2.1 and shown in Fig. 5 .

Output Layer: The final layer is the prediction layer which decides and provides the final prediction based on the input features received from the previously hidden layers. 

= (

= (

where is the sigmoid function, signifies the weight matrices, is a matrix of fuzzy weights from the input cell to the output gate, signifies the bias vector, and and are the input gates, the hidden (forget) gate, the output , , , gate, and cell activation function, is the cell output, and is the element-wise product, respectively. However, ⊙ the hidden unit in this architecture is represented in memory blocks. Each block contains one or a large number of memory cells. This procedure allows these cells to preserve information for a particular time in an uncertain time series and decide which information needs to be stored and when to use it.

the input variables of the DIT2FLSTM model can be defined as p, and . Moreover, the Takagi Sugeno Kang 

The inference part of the DIT2FLSTM model can be entirely characterized by fuzzy rules for inference process as:

where is the inputs And the output ∈ . The following equation represents the membership 1 ∈ 1 , …, ∈ function as:

where ∩ signifies the product t-norm operation [43] . The output of each fuzzy rule is , with membership =°f

where ∘ represents the composition operation and ∪ signifies the maximum t-conorm operation [43] , and denotes ( ) the firing interval for the fuzzy rule, where and is as:

The firing output set is produced through a fuzzy inference using ruleset and the aggregation of the consequent of as follows [44] :

where * represents the product t-norm operation. The final output is obtained through the integration of all rule firing sets on the output:

where ∨ represents the maximum operation. Then the type reduced set is obtained through computing the ( ) centroid of :

where the two points and are computed through the K.M. algorithm in [44] . ( ) ( )

In this research, the sigmoid squashing function was used on all the gates for the DIT2FLSTM cell structure as described follows:

The input of the cell uses the following equation:

The input to the cell passes a non-linear function , as follows:

In the following equation, the main MFs for each antecedent are defined through Gaussian distribution with uncertain income as follows:

where is the uncertain mean, k =1,2 denotes the number of the antecedents, M denotes the number ∈ ⌊ 1 . 2 ⌋ of the fuzzy rules, and is the standard deviation. C in represents the number of memory cells in the training phase as follows: Also, the input enters the cell whenever the input gate is activated. The cell state is defined as follows:

Then in the output gate of the cell is as follows:

The activation function at the output gate is given by:

where the cell output is ( ) at the next step for prediction at time t as follows:

In this section, the DIT2FLSTM model was applied to an official dataset collected from the WHO official website.

First, an ROC curve analysis of applying the DIT2FLSTM model to the official dataset is presented. Then a statistical evaluation of the DIT2FLSTM has been conducted to represent the model's capability for both short and long timeseries predictions.

The dataset contains the latest public data on the COVID-19 pandemic, including daily updates. This dataset includes detailed information about the number of confirmed cases, confirmed death, and confirmed recovery cases during the COVID-19 pandemic (February 2019 to November 2020).

In this section, an ROC curve analysis was conducted to have a reliable estimate of the performance of the DIT2FLSTM model, and the results were statistically verified as follows: where is the means of the accuracy of the ROC curve for the 10-fold cross-validation. A 10-fold cross-validation technique was applied, randomly partitions the original sample into k equal sized subsamples. A single subsample is retained as the validation data for testing the model from the ten sub-samples, and the remaining nine are used as training data. This research was repeated ten times, while each subsample was used exactly once as validation test data. The obtained results of 10-fold cross-validation are reported in Table 2 . Table 3 *The obtained accuracy is an average percentage of three cases, including new cases, recovery, and deaths in its fold number. 

In this section, the proposed DIT2FLSTM model was applied to both Mackey-Glass as a global benchmark and COVID-19 time series to show the proposed model's capability for predicting short and long time series. On the other hand, to perform a statistical evaluation of the performance of the proposed method, a t-test was conducted to validate the obtained results.

This section presents the experimental results of applying the proposed DIT2FLSTM prediction method to the Mackey-Glass time series. The proposed DIT2FLSTM model was evaluated on a high-dimensional chaotic time-series system generated by the Mackey-Glass delay differential equations as follows:

where is the delay of the time series. Furthermore, a statistical test was conducted on applying the DIT2FLSTM to ∆ the Mackey-Glass time series through t-test. First, we calculate the mean of observation as follows [53] :

where, x̄ is the sample mean, is the time delay mean, s is the sample size, and S is the standard deviation as follows Δ [53] :

Also, the RMSE was computed to assess the error of the DIT2FLSTM model in the Mackey-Glass time-series. The detailed calculation of RMSE is given by:

where denotes the forecasted value, and is the actual value. Fig.7 shows the statistical analysis of the Also, the obtained results showed that the minimum error of the DIT2FLSTM model was 0.00251, and the average error was 0.00301. Table 4 shows the quantitative comparison of DIT2FLSTM uncertainty in Mackey-Glass and COVID-19 datasets. 

In this section, a two-sample t-test (left tailed) has been conducted. The null hypothesis was defined as [55] : H 0 = > and, H 1 : , where and are the means of the area under the ROC curve (AUC) of DIT2FLSTM and < IT2FLS for ten different iterations of 10-fold cross-validation assessment. The obtained results of the t-test analysis are shown in Table 5 , which demonstrates the advantage of the proposed DIT2FLSTM method for COVID-19 timeseries prediction compared to the IT2FLS. The obtained results in Table 5 shows that the t-test failed to reject the null hypothesis. 

In this section, Table 6 presents a comparison of conventional methods and the proposed DIT2FLSTM model.

According to the obtained results, the proposed DIT2FLSTM model has reported the highest performance compared to the counterpart models, including the LSTM, IT1FLS, IT2FLS, T1FLSTM, and T2ANFIS for the COVID-19 timeseries prediction. Meanwhile, the time-series complexity was determined by the number of computational steps needed to run the algorithm of the DIT2FLSTM as a function for several data samples as the input size. Time measurement was computed by order of the models and consumption, including the hour, minute, and seconds (00:00:00) as shown in Table 7 , where N is the number of instructions and M is the input size. The prediction results of the 10-top countries based on the proposed DIT2FLSTM model are shown in Fig. 9 . Also, in the final part of this section, the DIT2FLSTM model has been compared to related works in Table 8 . However, the capability of the proposed model is not limited to this specific application. Furthermore, the proposed DIT2FLSTM model has the potential to be applied to other biomedical applications, including biopharmaceutical and clinical studies in horizon of time series.

In this study, a novel DIT2FLSTM model was proposed to predict the incidence of new cases, recovery cases, and 

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this article.

[35] C. Li [37] C. Li, C. Zhou, W. Peng, Y. Lv, and X. Luo, Accurate prediction of short-term photovoltaic power generation via a novel double input-rule-modules stacked deep fuzzy method, Energy, 11(7) (2020) 118700.

Understanding epidemic data and statistics: A case study of COVID-19

Preliminary bioinformatics studies on the design of a synthetic vaccine and a preventative peptidomimetic antagonist against the SARSCoV-2 (2019-nCoV, COVID-19) coronavirus

Novel Coronavirus Driven by Spring Festival Transportation in China: A Population-Based Study

Rigidity of the Outer Shell Predicted by a Protein Intrinsic Disorder Model Sheds Light on the COVID-19 (Wuhan-2019-nCoV) Infectivity

A, A Sequence Homology and Bioinformatic Approach Can Predict Candidate Targets for Immune Responses to SARS-CoV-2

Syed Musthapa Meeran, Culinary spice bioactives as potential therapeutics against SARS-CoV-2: Computational investigation

Spatial-temporal distribution of COVID-19 in China and its prediction: A data-driven modeling analysis

Propagation analysis and prediction of the COVID-19

Trend and forecasting of the COVID-19 outbreak in China

Understanding Unreported Cases in the COVID-19 Epidemic Outbreak in Wuhan, China, and the Importance of Major Public Health Interventions

Time series forecasting of COVID-19 transmission in Canada using LSTM networks

COVID-19 detection using deep learning models to exploit Social Mimic Optimization and structured chest X-ray images using fuzzy color and stacking approaches

Deep learning model for forecasting COVID-19 outbreak in Egypt

Analysis and Prediction of COVID-19 Pandemic in Pakistan using Timedependent SIR Model

A conceptual model for the coronavirus disease 2019 (COVID-19) outbreak in Wuhan, China with individual reaction and governmental action

A Time-dependent SIR model for COVID-19 with Undetectable Infected Persons

A Model for COVID-19 Prediction in Iran Based on China Parameters

Modelling the Transmission Dynamics of COVID-19 in Six High-Burden Countries

Modeling and forecasting of COVID-19 using a hybrid dynamic model based on SEIRD with ARIMA corrections

Optimization Method for Forecasting Confirmed Cases of COVID-19 in China, Special Issue Real Time Clinical and Epidemiological Investigations on Novel Coronavirus

Forecasting of COVID-19 Confirmed Cases in Different Countries with ARIMA Models, COVID-19 research community

Modeling the transmission dynamics of COVID-19 epidemic: a systematic review

Neural Network Models with Fuzzy Response Aggregation for Predicting COVID-19 Time Series: The Case of Mexico

The concept of a linguistic variable and its application to approximate reasoning-I

Type-2 Fuzzy Logic Controllers: A Way Forward for Fuzzy Systems in Real World Environments

Computational Intelligence: Research Frontiers. WCCI

Joint propagation of ontological and epistemic uncertainty across risk assessment and fuzzy time series models

Probabilistic Forecasting with Fuzzy Time Series

Ensemble Neural Network with Type-2 Fuzzy Weights Using Response Integration for Time Series Prediction

Automated Ontology Extraction from Unstructured Texts using Deep Learning, Intuitionistic and Type-2 Fuzzy Logic Enhancements in Neural and Optimization Algorithms: Theory and Applications

A Novel Type-2 Adaptive Neuro Fuzzy Inference System Classifier for Modelling Uncertainty in Prediction of Air Pollution Disaster (RESEARCH NOTE)

A survey on LSTM memristive neural network architectures and applications

multi-step least squares support vector machine modeling approach for forecasting shortterm electricity demand with application

Prediction of photovoltaic power output based on similar day analysis, genetic algorithm and extreme learning machine

Photovoltaic power forecasting based lstm-convolutional network

A Mutually Recurrent Interval Type-2 Neural Fuzzy System (MRIT2NFS) With Self-Evolving Structure and Parameters

Long short-term memory networks in memristor crossbar arrays

Memristor-based LSTM network with in situ training and its applications

A memristor-based long short-term memory circuit

A review on the long short-term memory model

On the Fundamental Differences Between Interval Type-2 and Type-1 Fuzzy Logic Controllers

Interval type-2 fuzzy logic systems made simple

Recommendations on designing practical interval type-2 fuzzy systems

A Vector Similarity Measure for Interval Type-2 Fuzzy Sets

on km algorithms for solving type-2 fuzzy set problems

Approaches for reducing the computational cost of interval type-2 fuzzy logic systems Overview and comparisons

Group decision making under generalized fuzzy soft sets and limited cognition of decision makers

Interval Type-2 Mutual Subsethood Fuzzy Neural Inference System (IT2MSFuNIS)

Double-Input Rule Modules Stacked Deep Interval Type-2 Fuzzy Model with Application to Time Series Forecasting

Study on Centroid Type-Reduction of Interval Type-2 Fuzzy Logic Systems Based on Noniterative Algorithms

Fuzzy Rules Interpolation for Sparse Fuzzy Rule-Based Systems Based on

Gaussian Fuzzy Sets and Genetic Algorithms

Corrections to An Automatic Approach for Learning and Tuning Gaussian Interval Type-2 Fuzzy Membership Functions Applied to Lung CAD Classification System

A Type-2 Fuzzy Time Series Model for Pattern Similarity Analysis: A Case Study on Air Quality Forecasting

Corresponding Author, Supervision, Methodology, Validation, Writing preparation, Reviewing, Editing

Consultant, Reviewing

The notations, petameter, and the variables of the proposed DIT2FLSTM model are summarized and described in Table 9 . 

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent was obtained from all participants of the study.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.