key: cord-0970014-c6xqwimz
authors: Suri, Jasjit S.; Bhagawati, Mrinalini; Paul, Sudip; Protogerou, Athanasios D.; Sfikakis, Petros P.; Kitas, George D.; Khanna, Narendra N.; Ruzsa, Zoltan; Sharma, Aditya M.; Saxena, Sanjay; Faa, Gavino; Laird, John R.; Johri, Amer M.; Kalra, Manudeep K.; Paraskevas, Kosmas I.; Saba, Luca
title: A Powerful Paradigm for Cardiovascular Risk Stratification Using Multiclass, Multi-Label, and Ensemble-Based Machine Learning Paradigms: A Narrative Review
date: 2022-03-16
journal: Diagnostics (Basel)
DOI: 10.3390/diagnostics12030722
sha: db9f491673326917e12a932c4efa10e6ea6f181d
doc_id: 970014
cord_uid: c6xqwimz

Background and Motivation: Cardiovascular disease (CVD) causes the highest mortality globally. With escalating healthcare costs, early non-invasive CVD risk assessment is vital. Conventional methods have shown poor performance compared to more recent and fast-evolving Artificial Intelligence (AI) methods. The proposed study reviews the three most recent paradigms for CVD risk assessment, namely multiclass, multi-label, and ensemble-based methods in (i) office-based and (ii) stress-test laboratories. Methods: A total of 265 CVD-based studies were selected using the preferred reporting items for systematic reviews and meta-analyses (PRISMA) model. Due to its popularity and recent development, the study analyzed the above three paradigms using machine learning (ML) frameworks. We review comprehensively these three methods using attributes, such as architecture, applications, pro-and-cons, scientific validation, clinical evaluation, and AI risk-of-bias (RoB) in the CVD framework. These ML techniques were then extended under mobile and cloud-based infrastructure. Findings: Most popular biomarkers used were office-based, laboratory-based, image-based phenotypes, and medication usage. Surrogate carotid scanning for coronary artery risk prediction had shown promising results. Ground truth (GT) selection for AI-based training along with scientific and clinical validation is very important for CVD stratification to avoid RoB. It was observed that the most popular classification paradigm is multiclass followed by the ensemble, and multi-label. The use of deep learning techniques in CVD risk stratification is in a very early stage of development. Mobile and cloud-based AI technologies are more likely to be the future. Conclusions: AI-based methods for CVD risk assessment are most promising and successful. Choice of GT is most vital in AI-based models to prevent the RoB. The amalgamation of image-based strategies with conventional risk factors provides the highest stability when using the three CVD paradigms in non-cloud and cloud-based frameworks.

The PRISMA model was used for searching and selecting the final studies for the review. The search was done using Science Direct, Google Scholar, IEEE Xplore, and Pub-Med by adapting the following keywords "multiclass classification for CVD risk", "multilabel classification for CVD risk", "ensemble classification for CVD risk", "CVD risk using Machine Learning/Artificial Intelligence for multiclass", CVD risk using Machine Learning/Artificial Intelligence for multi-label, "CVD risk using Machine Learning/Artificial Intelligence for ensemble", "CVD risk assessment in ML/AI framework", and "Bias in ML/AI". The total number of ML/AI-based CVD studies is shown in Figure 1 . An exhaustive search resulted in a total of 19,454 studies. The three criteria used for exclusion were (a) non-relevant studies (b) articles removed after search and screening of the studies (c) records rejected due to insufficient data. The implementation of exclusion criteria provides 19,084, 88, and 17 studies for exclusion showed by E1, E2, and E3 ( Figure 1 ). The important scientific knowledge from these final studies was gained and the statistical classification was drawn. Further, a comprehensive analysis of the studies was done between the three techniques with the determination of AI bias. 

The statistical distributions derived from the selected studies are shown in Figure 2 .

The following attributes were used for the statistical distribution (a) types of CVD paradigms, (b) types of risk classes in multiclass CVD (c) ML-based CVD systems without/with feature extraction, (d) # GTs in multi-label-based CVD, (e) feature selection techniques, and (f) ML-based CVD publications. 

The statistical distributions derived from the selected studies are shown in Figure 2 . The following attributes were used for the statistical distribution (a) types of CVD paradigms, (b) types of risk classes in multiclass CVD (c) ML-based CVD systems without/with feature extraction, (d) # GTs in multi-label-based CVD, (e) feature selection techniques, and (f) ML-based CVD publications. The percentage of studies for each of the three kinds of CVD risk prediction had the following distributions: multiclass (26%) [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] , multi-label (15%) [83] [84] [85] [86] [87] [88] [89] [90] , and ensemble (59%) [80, (Figure 2a) . Several different kinds of risk classes were identified in multiclass CVD framework, namely binary (65%), tertiary (22%), quaternary (6%), and greater than quaternary (7%) (Figure 2b ). The distribution of the ML-based CVD studies with and without feature selection are shown in Figure 2c . It was found that almost 82% of MLbased CVD studies performed feature selection for risk prediction whereas only 18% [69, 70, 73, 75, 83, 94, 96, 110, 120] did not perform it. For the ML-based multi-label CVD (Figure 2d ), the total number of GT's used for each study were as follows and given in the [89] , and Zeng et al. (4) [90] . There were eight sectors in the pie chart and each sector represents a study (publication) in the area of multi-label-based ML system. Below the study shows the number of gold standards used for the design of the multi-label ML system paradigm. For example, Ventakesh et al. had 6 types of gold standards ((death, stroke, coronary heart disease (CHD), CVD, heart failure (HF), atrial fibrillation (AF)) during the design of their multi-label-based ML system. Similarly, Jamthikar et al. had three types of gold standard (coronary artery disease (CAD), acute coronary syndrome (ACS), composite cardiovascular event (CVE)) during the design of the multi-label ML system. Since the The percentage of studies for each of the three kinds of CVD risk prediction had the following distributions: multiclass (26%) [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] , multi-label (15%) [83] [84] [85] [86] [87] [88] [89] [90] , and ensemble (59%) [80, (Figure 2a) . Several different kinds of risk classes were identified in multiclass CVD framework, namely binary (65%), tertiary (22%), quaternary (6%), and greater than quaternary (7%) (Figure 2b ). The distribution of the ML-based CVD studies with and without feature selection are shown in Figure 2c . It was found that almost 82% of ML-based CVD studies performed feature selection for risk prediction whereas only 18% [69, 70, 73, 75, 83, 94, 96, 110, 120] did not perform it. For the ML-based multi-label CVD (Figure 2d ), the total number of GT's used for each study were as follows and given in the ground braces: Venkatesh et al. (6) [83] , Jamthikar et al. (3) [84] , Kumar et al. (3) [85] , Mehrang et al. (3) [86] , Mohamend et al. (8) [87] , Priyanka et al. (10) [88] , Zamzmi et al. (4) [89] , and Zeng et al. (4) [90] . There were eight sectors in the pie chart and each sector represents a study (publication) in the area of multi-label-based ML system. Below the study shows the number of gold standards used for the design of the multi-label ML system paradigm. For example, Ventakesh et al. had 6 types of gold standards ((death, stroke, coronary heart disease (CHD), CVD, heart failure (HF), atrial fibrillation (AF)) during the design of their multi-label-based ML system. Similarly, Jamthikar et al. had three types of gold standard (coronary artery disease (CAD), acute coronary syndrome (ACS), composite cardiovascular event (CVE)) during the design of the multi-label ML system. Since the number of gold standards are important during the multi-label paradigm, the pie-chart shows the statistical distribution of the different studies using the number of gold standards. The number of studies (given in curly braces) that used the following feature selection techniques were 2D convolutional neural network (CNN) (6) [71, 79, 81, 89, 101, 111] , continuous wavelet transform (1) [72] , principal component analysis (PCA) (9) [76, 79, 84, 98, 102, 112, 114, 119, 121] , Mel frequency cepstral coefficient (1) [77] , amplitude magnitude (1) [78] , gain ratio (1) [80] , Matlab (1) [86] , association technique (2) [87] , SHAP (1) [90] , extreme gradient boost (XG-Boost) (1), genetic algorithm (5) [91, 103, 104, 122, 123] , Tunicate Swarm (1) [116] , chi-Square (2) [117] , least absolute shrinkage and selection operation (LASSO) (1) [99] (Figure 2e ). The increasing trend of CVD publications from the year 2009 to 2021 is shown in Figure 2f .

The fundamental cause of CVD is the disease of atherosclerosis [124] . The process of plaque formation is known as atherogenesis as shown in Figure 3a (A-I) [125] . It is a process when the plaques develop in the arteries where there is low endothelial shear stress [126] . The shear stress depends on the flow velocity characteristics like type of flow, direction, and velocity. Leukocytes attack the epithelium in this region (Figure 3 (bA)) [126] . Mainly there is the migration of monocytes into the sub-epithelial layer where it is oxidized by the low amount of low-density lipoprotein (LDL) cholesterol and turns into macr0ophage ( Figure 3 (bB)) [127, 128] . Eventually, these macrophages become large foam cells with oxidized LDL cholesterol leading to the formation of necrotic core (Figure 3 (bC)). Microscopic calcium granules expand in the necrotic cells and forms lumps of calcium deposits. This necrotic core is separated from the blood vessel by a fibrous cap [129] . The blood remains uninterrupted when the plaque is small as the arteries do remodeling by themselves [130] . However, when the plaques increase, the lipid-core volume decreases leading to structural stabilization of plaque ( Figure 3a ) [131] . number of gold standards are important during the multi-label paradigm, the pie-chart shows the statistical distribution of the different studies using the number of gold standards. The number of studies (given in curly braces) that used the following feature selection techniques were 2D convolutional neural network (CNN) (6) [71, 79, 81, 89, 101, 111] , continuous wavelet transform (1) [72] , principal component analysis (PCA) (9) [76, 79, 84, 98, 102, 112, 114, 119, 121] , Mel frequency cepstral coefficient (1) [77] , amplitude magnitude (1) [78] , gain ratio (1) [80] , Matlab (1) [86] , association technique (2) [87] , SHAP (1) [90] , extreme gradient boost (XGBoost) (1), genetic algorithm (5) [91, 103, 104, 122, 123] , Tunicate Swarm (1) [116] , chi-Square (2) [117] , least absolute shrinkage and selection operation (LASSO) (1) [99] (Figure 2e ). The increasing trend of CVD publications from the year 2009 to 2021 is shown in Figure 2f .

The fundamental cause of CVD is the disease of atherosclerosis [124] . The process of plaque formation is known as atherogenesis as shown in Figure 3a (A-I) [125] . It is a process when the plaques develop in the arteries where there is low endothelial shear stress [126] . The shear stress depends on the flow velocity characteristics like type of flow, direction, and velocity. Leukocytes attack the epithelium in this region (Figure 3 (bA)) [126] . Mainly there is the migration of monocytes into the sub-epithelial layer where it is oxidized by the low amount of low-density lipoprotein (LDL) cholesterol and turns into macr0ophage (Figure 3 (bB)) [127, 128] . Eventually, these macrophages become large foam cells with oxidized LDL cholesterol leading to the formation of necrotic core ( Figure  3 (bC)). Microscopic calcium granules expand in the necrotic cells and forms lumps of calcium deposits. This necrotic core is separated from the blood vessel by a fibrous cap [129] . The blood remains uninterrupted when the plaque is small as the arteries do remodeling by themselves [130] . However, when the plaques increase, the lipid-core volume decreases leading to structural stabilization of plaque ( Figure 3a ) [131] . Progressive deposition of lipids results in the thinning of the fibrous cap leading to rupturing the plaque [132] . The ruptures of the cup result in healing by the platelets in the bloodstream, which leads to the formation of the clot of blood or thrombus which yields blocking of artery than atrial stiffness [133] . Due to this, the tissues become deprived of blood supply, causing cell death. If the coronary artery gets blocked, causing a myocardial infarction or CVD (Figure 3 (bD)) [3, 7] .

The core aim of this review is to understand the three kinds of paradigms for CVD risk stratification. This allows understanding the (a) types of gold standards used for different kinds of applications, (b) types of fundamental architectures used, and (c) finally the comparison between the three different types of paradigms.

The most fundamental type of CVD risk stratification is the multiclass framework [134] . There are three main characteristics in multiclass framework, namely (i) it divides the outcome into two or more granular risk classes, (ii) the drug prescription is better controlled for CVD treatments based on which class the disease stage or risk lies, and (iii) the risk of CVD is better understood when divided into several stages such as low, mild, low-ofmoderate, high-of-moderate, low-of-high, and high-of-high.

For any CVD system, there are two most important attributes: (a) the types of the covariates used and (b) the gold standard adopted. Accordingly, in the multiclass framework, there are 14 published studies (see Table 1 ). It shows the three attributes represented in three columns: covariates, gold standard, and the AI category, namely ML or DL. The types of covariates considered for the multiclass systems were OBBM [71, 76, 80, 82] , LBBM [71, 76, 80, 82] , CUSIP [76, 80] for office-based setups (Table 1 : row 1-5), and Electrocardiogram (ECG) [79, 81, 82] , PCG [77] , Acceleration Plethysmogram (APG) [78] signals for cardiac stress test laboratories (Table 1 : row 6-9), and coronary artery calcium (CAC) for CTbased CVD models [135] . The ground truths considered for CVD risk assessment (Table 1) were death [80] , coronary heart disease (CHD) [82] , chronic heart conditions (CHC) [79] , cardiovascular event (CVE) [76] , sudden cardiac death (SCD) [72] , heart failure (HF), myocardial infarction (MI) [75] , coronary artery calcification (CAC) score [69] , fatal/non-fatal CVD [73] , joint CVD and diabetes [70] . Note that these gold standard choices along with AI attributes, scientific and clinical validations are key to preventing bias in AI.

The comparison between CVD and non-CVD applications [136] is shown in Table 2 . Seven attributes were used for this comparison. The image modalities used in the CVDbased system were US, CT, MRI, and ECG (Table 2 : row 4, CVD column). The architectures applied were ML and DL. DL provided better results due to its unique automated feature selection process (Table 2 : row 6, CVD column). The defined number of classes was in the range of 3-9 (Table 2 : row 5, CVD column) [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] . The multiclass approach for classification has been applied to non-CVD applications such as Alzheimer's prediction or different cancer types. The interpretation of multiclass in the non-CVD system can be thought of as different stages of the diseases, for example, in the case of Alzheimer's disease (AD), it can be categorized as different stages of memory loss with age. Similarly, in the case of cancer, it can be different stages or grades of cancer. Our observations show that the gold standard types in the non-CVD system are very different from the CVD systems. For example, for the early detection of AD/Mild Cognitive Impairment (MCI), the classification is done between (1) AD vs. normal control (NC), (2) MCI vs. NC, (3) AD vs. MCI, and (4) progressive MCI (PMCI) vs. Significant Memory Concern (SMCI) for Alzheimer's. In the case of breast cancer, GTs can be proliferation and non-proliferation cancer types.

Note that the number of classes considered for multiclass differs from disease-todisease. The different architecture followed for CVD are mainly ML and DL, whereas for non-CVD it ranges from deep learning retinal CAC score (RetiCAC) [137] , pooled cohort equation (PCE) [138, 139] , support vector machine (SVM) [70, [75] [76] [77] 140] , convolutional neural networks (CNN), decision tree (DT) [71, 79] , random forest (RF), logistic regression (LR), naive Bayesian (NB), K-nearest neighbor (KNN), and ensemble. The different types of covariates for no-CVD-based systems were breast histopathology images (BHI), OBBM, and LBBM (Table 2 : row 2, column non-CVD). Modalities for the non-CVD-based system were EEG, MRI, CT [137, 139] (Table 2 : row 4, non-CVD column), and the number of risk classes varied from 5-14 [137] [138] [139] 141, 142] (Table 2 : row 5, non-CVD column).

The architectures opted for multiclass prediction of CVD risk has very basic components (a) data collection (b) training system, and (c) testing system. The training system is basically used for training the ML system based on different covariates (or risk factors) [143, 144] , with the support of different ground truths while using the training-based classifiers. The system can be trained to identify the granular risk classes from no, low, and medium, to high class. Feature selection is also performed during the training of the system [145, 146] . For prediction, the training model is applied to transform the testing features either in Seen AI framework or the Unseen AI framework [147] . Two types of architectures were described in this section in terms of the above-mentioned factors. A typical online system for multiclass CVD risk stratification is shown in Appendix A, Appendix A.1. BHI [139] , OBBM [137, 138, 141, 142] , LBBM [137, 138, 141, 142] 3

Disease Type CVD [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] Diabetes [142] , Cancer (Breast, Lung, Brain) [138, 139] , Alzheimer's [138, 141] A generalized ML system is applied to office-based CVD or stress-test-based CVD systems as shown in Figure 4 . Considering the office-based CVD system, the covariates were collected from OBBM, LBBM, CUSIP, and MedUSE [76] , while for the CVD-based stress-test system, EEG was the input. The rest of the configuration remains the same which consists of four parts: Part A is the preprocessing of the input data (covariates) and augmentation for balancing the classes. Part B consists of a training system, Part C consists of a prediction system, and Part D consists of a performance evaluation system (Appendix E). In Part A, the objective is to balance the classes if there is a multiclass scenario, Part B consists of two subparts: (i) selection of the best feature given the set of covariates and (ii) model generation using (a) classifier, (b) selected features, and the (c) gold standard. Part C consists of the application of the trained model on the selected set of best features from the test data set by transforming the test features to compute the predicted label. Part D is used for performance evaluation of the ML system where the predicted labels are compared against the gold standard labels. Note that during the training system, the two ingredients are the classifier bank and the gold standard used. The classifier bank, for example, can be classifiers like SVM, XGBoost, KNN, NB, etc., while the gold standard is the coronary artery disease syndrome, such as coronary artery disease stages that include the four types of risk stages. Note that since the system is a K-fold (either of the K types such as K2, K3, K4, K5, and K10 can be used), every patient gets to be in the test pool, and then at the end of all the folds, the complete set can be used for performance evaluation. Further to note a classifier bank can be used during the design of the training model, that uses the gold standard (such as coronary risk scores derived from coronary angiography) and training covariates. The CVD example in Figure 4 uses four sets of covariates, which can be flipped to ECG signals [148] [149] [150] when using the stress test-based system for CVD risk assessment. The longitudinal ultrasound model is used typically for the collection of the CUSIP risk factors such as cIMT (max., min., and ave.), intima-media thickness variability (cIMTV), maximum plaque height (MPH), and total plaque area (TPA). Diagnostics 2022, 12, x FOR PEER REVIEW 10 of 48 

Another set of architecture for multiclass CVD risk prediction was used by Hussein et al. [75] ( Figure 5 ). The ECG signals [151] [152] [153] are obtained from the stress test laboratory for the analysis of CVD risk. The model uses the multiclass SVM classifier that takes the ECG signals as risk factors or covariates. And the ground truth used for the training system is myocardial infarction (MI). The multiclass outcomes that were identified were normal, low MI, and high MI. The feature of ST (it is the interval between ventricular depolarization and repolarization, and PR (the flat line that runs from the end of the P-wave 

Another set of architecture for multiclass CVD risk prediction was used by Hussein et al. [75] ( Figure 5 ). The ECG signals [151] [152] [153] are obtained from the stress test laboratory for the analysis of CVD risk. The model uses the multiclass SVM classifier that takes the ECG signals as risk factors or covariates. And the ground truth used for the training system is myocardial infarction (MI). The multiclass outcomes that were identified were normal, low MI, and high MI. The feature of ST (it is the interval between ventricular depolarization and repolarization, and PR (the flat line that runs from the end of the P-wave till the start of the QRS complex) were extracted from the time-frequency (TF) power spectrum. The created training model was the input to the prediction systems along with the test data and the final classifications were made into the normal, low MI, and high MI. till the start of the QRS complex) were extracted from the time-frequency (TF) power spectrum. The created training model was the input to the prediction systems along with the test data and the final classifications were made into the normal, low MI, and high MI. The general algorithm for multiclass CVD risk stratification is explained in form of pseudo-code. A detailed explanation is provided in Appendix A, Appendix A.2.

The second technique used for CVD risk stratification is multi-label-based [154] [155] [156] . The ground truth is very important for the proper classification of CVD risk [157] [158] [159] . CVD risk prediction systems were said to be multi-label-based depending on the number of ground truth (GT) used in the system [160] [161] [162] . The paradigm was considered as a multi-label-based classification if more than one number of GT is used for CVD risk detection [90, [163] [164] [165] [166] [167] . The GTs, risk factors, and the architecture used were discussed in the next sub-sections. The pseudo-code that represents a multi-label-based risk stratification process can be referred to in Appendix B.

Eight multi-label-based studies for CVD risk prediction were considered in this review [83] [84] [85] [86] [87] [88] [89] [90] . Different types of ground truths used in these studies were death, stroke, CHD, CVD, HF, atrial fibrillation (AF) [83] , CAD, ACS, composite CVE [84] , large vessel disease (LVD), small vessel disease (SVD) [168] , intracerebral hemorrhage (ICH) [85] , non-AFib-non-ADHF, AFib-non-ADHF, AFib-ADHF [86] , systolic heart failure (acute, chronic type), diastolic heart failure (acute and chronic type) [87] , congestive heart failure, hypertension, AF, acute kidney failure, diabetes type II, acute respiratory failure, hyperlipidemia, coronary atherosclerosis, urinary tract infection, esophageal reflux [88] , CAD, dilated cardiomyopathy (DCM), MI [89] , lung complication, cardiac, infectious and rhythmic complication [90] .

The risk factors used were OBBM, LBBM, CUSIP, MRI, and CT image phenotypes (input covariates column, Table 3 ). The algorithms used for the multi-label classifications were namely binary recursive (BR), label powerset (LP), multi-label adaptive resonance associative map (MLARAM), random k-labelset (RakEL), classifier chain (CC), multi-label k-nearest neighbor (MLkNN), seismocardiography (SCG-Z), gyrocardiography (GCG-Z), principal component analysis (PCA), DCT, consensus-based risk model. Other characteristics of this classification technique were described in Table 3 . The general algorithm for multiclass CVD risk stratification is explained in form of pseudo-code. A detailed explanation is provided in Appendix A, Appendix A.2.

The second technique used for CVD risk stratification is multi-label-based [154] [155] [156] . The ground truth is very important for the proper classification of CVD risk [157] [158] [159] . CVD risk prediction systems were said to be multi-label-based depending on the number of ground truth (GT) used in the system [160] [161] [162] . The paradigm was considered as a multi-label-based classification if more than one number of GT is used for CVD risk detection [90, [163] [164] [165] [166] [167] . The GTs, risk factors, and the architecture used were discussed in the next sub-sections. The pseudo-code that represents a multi-label-based risk stratification process can be referred to in Appendix B.

Eight multi-label-based studies for CVD risk prediction were considered in this review [83] [84] [85] [86] [87] [88] [89] [90] . Different types of ground truths used in these studies were death, stroke, CHD, CVD, HF, atrial fibrillation (AF) [83] , CAD, ACS, composite CVE [84] , large vessel disease (LVD), small vessel disease (SVD) [168] , intracerebral hemorrhage (ICH) [85] , non-AFib-non-ADHF, AFib-non-ADHF, AFib-ADHF [86] , systolic heart failure (acute, chronic type), diastolic heart failure (acute and chronic type) [87] , congestive heart failure, hypertension, AF, acute kidney failure, diabetes type II, acute respiratory failure, hyperlipidemia, coronary atherosclerosis, urinary tract infection, esophageal reflux [88] , CAD, dilated cardiomyopathy (DCM), MI [89] , lung complication, cardiac, infectious and rhythmic complication [90] .

The risk factors used were OBBM, LBBM, CUSIP, MRI, and CT image phenotypes (input covariates column, Table 3 ). The algorithms used for the multi-label classifications were namely binary recursive (BR), label powerset (LP), multi-label adaptive resonance associative map (MLARAM), random k-labelset (RakEL), classifier chain (CC), multi-label k-nearest neighbor (MLkNN), seismocardiography (SCG-Z), gyrocardiography (GCG-Z), principal component analysis (PCA), DCT, consensus-based risk model. Other characteristics of this classification technique were described in Table 3 . 

The architecture design for the multi-label plays an important in the outcome results of the system. The basic component of the architecture for the CVD prediction system is training and testing. The proper choice of GT leads to non-biased results in the risk prediction of CVD. The architecture system used by Jamthikar et al. [84] is shown in Figure 6 below. The total number of ground truths considered for this system were three, namely (a) coronary artery disease, (b) acute coronary syndrome, and (c) a composite CVE, and the covariates used were OBBM, LBBM, and the CUSIP phenotype. Six types of classification techniques used include (i) four problem transformation methods (PTM) and (ii) two algorithm adaptation methods (AAM) are used for multi-label CVE prediction. The four PTM techniques were binary relevance (BR), label powerset (LP), classifier chain (CC), and random k-labelset (RAkEL). Under AAM-based, two techniques, namely multi-label k-nearest neighbor (MLkNN), and multi-label adaptive resonance associative map (MLARAM) were used. The details can be seen in Appendix B. Evaluation was performed by calculating the accuracy, sensitivity, specificity, F1-score, and AUC for all the classification techniques. The BR classification was found to be the best performer with the values for accuracy, sensitivity, specificity, F1-score, and AUC as 81.2%, 76.5%, 83.8%, 75.37, and 0.89 (p < 0.0001), respectively.

Another architecture [86] used for multi-label CVD classification is described in Figure 7 . The mechanocardiography (MCG) data were used by the system. Four kinds of ground truth were used, namely AFib, non-AFib, ADHF, and non-ADHF. The covariates were gender, age, height, weight, BMI, given for the training and testing system. The ML classification algorithm used were random forest (RF), Xtreem Gradient Boost (XGB), and logistic regression (LR). RF gave the best performance among all the three ML classifiers. The system was validated by nested cross-validation. In this system, feature extraction was also performed using a feature vector. The hierarchal classification was also adapted in this system. Another paradigm that can use multiple classifiers at the same time is under the ensemble framework as presented in the next section. Diagnostics 2022, 12, x FOR PEER REVIEW 13 of 48 

The ensemble-based technique was the third type of technique considered for CVD risk classification [169] [170] [171] . This classification was characterized by the fusion of different types of ML or DL classifiers (Table 4 ). It can be used with multiclass and multi-label classification [172] [173] [174] . Figure 8 shows the concept of the ensemble paradigm. There are two sets of strategies, namely homogeneous ensemble and heterogeneous ensemble (see the separation shown by dotted line). In homogenous ensemble, the conventional classifier techniques are combined using homogeneous ensemble algorithm to yield homogeneous ensemble classifier, which when trained using classifier A while using the gold standard. This homogeneous system yields the trained model A. The same protocol can be adapted for the heterogeneous ensemble paradigm yielding the trained model B. These trained models can be used by the prediction system on the test feature to produce prediction labels. Finally, the performance can be evaluated by comparing predicted labels to goldstandard labels yielding performance parameters. The key benefit of using an ensemble classifier is its superior performance compared to either multiclass or multi-label strategies. The pseudo-code that represents the ensemble-based risk stratification process can be seen in Appendix C. The ensemble technique can be applied to the CVD field, as well as to other fields, such as education, Alzheimer's, etc. 

The ensemble-based technique was the third type of technique considered for CVD risk classification [169] [170] [171] . This classification was characterized by the fusion of different types of ML or DL classifiers (Table 4 ). It can be used with multiclass and multi-label classification [172] [173] [174] . Figure 8 shows the concept of the ensemble paradigm. There are two sets of strategies, namely homogeneous ensemble and heterogeneous ensemble (see the separation shown by dotted line). In homogenous ensemble, the conventional classifier techniques are combined using homogeneous ensemble algorithm to yield homogeneous ensemble classifier, which when trained using classifier A while using the gold standard. This homogeneous system yields the trained model A. The same protocol can be adapted for the heterogeneous ensemble paradigm yielding the trained model B. These trained models can be used by the prediction system on the test feature to produce prediction labels. Finally, the performance can be evaluated by comparing predicted labels to goldstandard labels yielding performance parameters. The key benefit of using an ensemble classifier is its superior performance compared to either multiclass or multi-label strategies. The pseudo-code that represents the ensemble-based risk stratification process can be seen in Appendix C. The ensemble technique can be applied to the CVD field, as well as to other fields, such as education, Alzheimer's, etc. 

The ensemble-based technique was the third type of technique considered for CVD risk classification [169] [170] [171] . This classification was characterized by the fusion of different types of ML or DL classifiers (Table 4 ). It can be used with multiclass and multi-label classification [172] [173] [174] . Figure 8 shows the concept of the ensemble paradigm. There are two sets of strategies, namely homogeneous ensemble and heterogeneous ensemble (see the separation shown by dotted line). In homogenous ensemble, the conventional classifier techniques are combined using homogeneous ensemble algorithm to yield homogeneous ensemble classifier, which when trained using classifier A while using the gold standard. This homogeneous system yields the trained model A. The same protocol can be adapted for the heterogeneous ensemble paradigm yielding the trained model B. These trained models can be used by the prediction system on the test feature to produce prediction labels. Finally, the performance can be evaluated by comparing predicted labels to gold-standard labels yielding performance parameters. The key benefit of using an ensemble classifier is its superior performance compared to either multiclass or multi-label strategies. The pseudo-code that represents the ensemble-based risk stratification process can be seen in Appendix C. The ensemble technique can be applied to the CVD field, as well as to other fields, such as education, Alzheimer's, etc. 

The different classifiers used in ensemble techniques were kNN, Reglog, GaussNB (GNB), linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), random forest (RF) [91, [95] [96] [97] [98] , multilayer perceptron (MLP), SVM [91, 94, 95, 97, 101, 103, 104] , CNN, long short term memory network (LSTM), gated recurrent unit (GRU), bidirectional LSTM, bidirectional GRU [92] , bagging, XGBoost, Adaboost [93, 99] , DNN [94] , generalized additive models (GAMs), elastic net, penalized logistic regression (PLR), gradient boosted machines (GBMs), Bayesian logistic regression [96] , K-NN [98, 99, 102, 104, 121] , NB [101, 104] , light GBM, GBDT, LR, BPNN, DT [98, 99, 104, 109] , GB [99] , Adaboost ensemble [100] , ANN [101, 104] , GNB, LDA, LR, QDA, AdaBoost [105, 113, 118] , XGBoost [102, 118] , ensemble SVM [104] , CART [106] , bagging, VS, LASSO, boosting, Bassian, MARS, logistic [107] , ensemble boosting [80] , ensemble learning, deep learning [108] , ET, sequential minimal optimization (SMO), IBk, AdaBoostM1 with decision stump (DS), AdaBoostM1 with LR, REPTree, [109] , neural network (NN), GB [110, 114] , linear Cox model [110] , ensemble gradient boosting [111] , ET [112] , NB, multi-layer defense system (MLDS) [114] , averagevoting (AVEn), majority-voting (MVEn), weighted-average voting (WAVEn) [115] , HTSA, ensemble deep learning [116] , XGBoost Meta [117, 119] , SOM [120] , extreme learning machine (ELM) [121] .

All the architecture can be combined to achieve the functionality of all the three models, namely multiclass, multi-label [13] , and ensemble. Both multiclass, multi-label modalities can be combined with the ensemble to acquire a better accuracy in the prediction of CVD risk. The comparison between the three has been shown in Appendix D, Table A1 . 

The different classifiers used in ensemble techniques were kNN, Reglog, GaussNB (GNB), linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), random forest (RF) [91, [95] [96] [97] [98] , multilayer perceptron (MLP), SVM [91, 94, 95, 97, 101, 103, 104] , CNN, long short term memory network (LSTM), gated recurrent unit (GRU), bidirectional LSTM, bidirectional GRU [92] , bagging, XGBoost, Adaboost [93, 99] , DNN [94] , generalized additive models (GAMs), elastic net, penalized logistic regression (PLR), gradient boosted machines (GBMs), Bayesian logistic regression [96] , K-NN [98, 99, 102, 104, 121] , NB [101, 104] , light GBM, GBDT, LR, BPNN, DT [98, 99, 104, 109] , GB [99] , Adaboost ensemble [100] , ANN [101, 104] , GNB, LDA, LR, QDA, AdaBoost [105, 113, 118] , XGBoost [102, 118] , ensemble SVM [104] , CART [106] , bagging, VS, LASSO, boosting, Bassian, MARS, logistic [107] , ensemble boosting [80] , ensemble learning, deep learning [108] , ET, sequential minimal optimization (SMO), IBk, AdaBoostM1 with decision stump (DS), AdaBoostM1 with LR, REPTree, [109] , neural network (NN), GB [110, 114] , linear Cox model [110] , ensemble gradient boosting [111] , ET [112] , NB, multi-layer defense system (MLDS) [114] , average-voting (AVEn), majority-voting (MVEn), weighted-average voting (WAVEn) [115] , HTSA, ensemble deep learning [116] , XGBoost Meta [117, 119] , SOM [120] , extreme learning machine (ELM) [121] .

All the architecture can be combined to achieve the functionality of all the three models, namely multiclass, multi-label [13] , and ensemble. Both multiclass, multi-label modalities can be combined with the ensemble to acquire a better accuracy in the prediction of CVD risk. The comparison between the three has been shown in Appendix D, Table A1 . The data size varies from 212-66,363 (for multiclass) [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] , 300-46,520 (for multi-label) [83] [84] [85] [86] [87] [88] [89] [90] , 459-823,627 (for ensemble) [80, . The number of risk factors for multiclass is low, multi-label is more, and for the ensemble is moderate. The risk factors considered for multiclass are family history and BMI. For multi-label-based studies and ensemble-based studies, the risk factors considered were BMI, ethnicity, hypertension, and smoking. The image modalities used for multiclass and multi-label were MRI [175, 176] , ECG [177] [178] [179] , and CUSIP whereas ECG is not used in ensemble-based studies. The range of performance evaluation parameters used for the multiclass, multi-label, and ensemble was 1-5, 1-8, and 1-8, respectively. The different types of classifiers used for these three techniques were SVM [91, 94, 95, 97, 101, 103, 104] , RF [91, [95] [96] [97] [98] , CNN, DT, k-NN, Agatston classifier, Elastic Net, NN, NB, XGBoost, SVM, ELM, one against one (OAO), one against all (OAA), decision direct acyclic graph (DDAG), exhaustive output error correction code (ECOC) [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] . The power analysis is also done on more multi-label and ensemble-based techniques. The detailed description can be seen in Appendix F. The general presentation of the NN algorithm was made in Appendix H.1 Right. The ML-based systems also lead to bias as it lacks clinical evaluation which is discussed in the next section. 

Performance evaluation (PE) strategies are very vital for understanding the reliability of the ML-based CVD risk stratification systems. The main metrics used by the PE systems are sensitivity, specificity, accuracy, precision, F1-score, positive predictive value (PPV), negative predictive value (NPV), false-positive rate (FPR), false-negative rate (FNR), p-value, hamming loss, C-index in multiclass, multi-label, and ensemble-based CVD risk assessment systems. The formulae used for determining these parameters are described in Appendix E. These different PE strategies were analyzed in different techniques. It was found that PE for multi-label-based CVD is different as compared to multiclass and ensemble. There are two types of PE techniques for multi-label, namely label-based and instance-based PE. The label-based is done using micro and macro-averaging techniques. Details of these techniques can be seen in Appendix E. Figure 9 (top) shows the label-based and instance-based performance evaluation. The number of studies that used this PE parameter is the accuracy (46) followed by sensitivity (32) , precision (27) , F1-score (27) , specificity (26) , p-value (10), PPV (8), NPV (6), FPR (6), FNR (5), c-index (4), Hamming Loss (1). Hamming Loss has opted only for the ensemble-based CVD risk stratification [181] [182] [183] [184] . The PE metrics used in the stress test-based (ECG) [185] [186] [187] techniques are area-under-the-curve (AUC), sensitivity, specificity, PPV, and NPV [188] [189] [190] [191] [192] .

As seen from the above discussion, the most important characteristic of the multiclass paradigm is the selection of gold standards having greater than two classes. The highest flexibility in the multiclass framework is the amalgamation of different sources of covariates, namely OBBM, LBBM, CUSIP, and MedUSE. We could take characteristics of plaque in the carotid ultrasound such as information about plaque symptomatology. The same principle holds in the stress test-based CVD paradigm or non-CVD framework. The ML systems sometimes overestimate the accuracies in prediction and underestimate the scientific validation, which results in bias in the prediction systems that we discuss in Section 5.

stance-based PE. The label-based is done using micro and macro-averaging techniques. Details of these techniques can be seen in Appendix E. Figure 9 (top) shows the labelbased and instance-based performance evaluation. The number of studies that used this PE parameter is the accuracy (46) followed by sensitivity (32) , precision (27) , F1-score (27) , specificity (26) , p-value (10), PPV (8), NPV (6), FPR (6), FNR (5), c-index (4), Hamming Loss (1). Hamming Loss has opted only for the ensemble-based CVD risk stratification [181] [182] [183] [184] . The PE metrics used in the stress test-based (ECG) [185] [186] [187] techniques are area-under-the-curve (AUC), sensitivity, specificity, PPV, and NPV [188] [189] [190] [191] [192] . As seen from the above discussion, the most important characteristic of the multiclass paradigm is the selection of gold standards having greater than two classes. The highest flexibility in the multiclass framework is the amalgamation of different sources of covariates, namely OBBM, LBBM, CUSIP, and MedUSE. We could take characteristics of plaque in the carotid ultrasound such as information about plaque symptomatology. The same principle holds in the stress test-based CVD paradigm or non-CVD framework. The ML systems sometimes overestimate the accuracies in prediction and underestimate the scientific validation, which results in bias in the prediction systems that we discuss in Section 5.

The ML-based systems for CVD risk classification generate a bias due to various reasons [193] [194] [195] . Thus, it is important to understand the risk of bias (RoB) in these MLbased systems. As the ML systems were clustered in three different clusters, namely multiclass, multi-label, and ensemble, the bias nature was compared in three independent categories, and finally by considering all the three mixed together. For the RoB in the ML- 

The ML-based systems for CVD risk classification generate a bias due to various reasons [193] [194] [195] . Thus, it is important to understand the risk of bias (RoB) in these ML-based systems. As the ML systems were clustered in three different clusters, namely multiclass, multi-label, and ensemble, the bias nature was compared in three independent categories, and finally by considering all the three mixed together. For the RoB in the ML-based systems, the ML systems were ranked on the basis of the average mean score along with cumulative mean values ( Table 5 ). The mean and the cumulative score were generated by scoring the ML attributes for each study. There were 52 ML studies (14 in multiclass, 8 in multi-label, 30 in ensemble cluster) with 41 attributes each. The score was given to each AI attribute using a grading scheme [196] . In this grading scheme, a high-score was assigned to the AI attribute, if the AI attribute was adopted (used) in a particular study (publication). The score is between 0 and 5. For example, a high-score was given if the attribute "data size" had a value higher than 1000 patients, else a low-score was assigned. Similarly, as another example, a high-score of 5 was given to the attribute "feature extraction", if it was implemented in a study, else a score of 0 was assigned, if not implemented. Later the ML-based studies were clustered into low-bias, moderate-bias, and high-bias groups. The distributions were done on the basis of the two cut-offs values. The low-moderate (LM) and moderate-high (MH) cutoff values for each cluster of ML studies were determined based on the mean values along with the cumulative-mean values.

The cutoffs values obtained for the multiclass cluster are 1.8 and 1.35 for LM and MH respectively (Figure 10a ). The studies belonging in the low-bias, the moderate-bias, and the high-bias bins are 4, 5, and 5, respectively. Similarly, the cutoffs for the multi-label cluster are LM: 1.9 and MH: 1.4 (Figure 10b ). Multi-label-based CVD ML studies in low-bias group are 3, moderate-bias group are 3 and high-bias group is 2. The values of LM cutoff for the ensemble cluster are 1.8 and HM cutoff value is 1.6. The studies in low-bias bin are 8, in moderate-bias are 16 and high-bias bin is 6 respectively for ensemble-based ML studies ( Figure 10c) . Alternatively, as all the studies are based on CVD risk prediction, the LM and MH cutoffs were determined by combining all the 52 studies. The LM, HM cutoff for the combined approach is 1.9 and 1.7 respectively (Figure 10d) . Thus, we see that the ensemblebased ML CVD risk estimation systems are low-biased among all the selected studies followed by multiclass-based (moderate-biased) while the multi-label-based was found to be low-biased. The AI-based CVD risk stratification systems can be further improved by incorporating the mobile, cloud, and e-health infrastructure as discussed in the next Section 6. 

The CVD risk prediction was taken to next level by integration of mobile, cloud, and telemedicine technologies. The mobile-based CVD systems follow both ML and non-ML approaches [197] [198] [199] [200] [201] [202] [203] [204] [205] [206] [207] . The classifier techniques used for the mobile-based ML systems were k-NN [208] , SVM [201, 209] , CNN [201, 202] , NB [204] , DT [207] , and RF [207] . The number of outcomes for the mobile-based CVD systems [197] varies between 1 and 2, basically CVD and diabetes. The cloud-based CVD systems also used both ML and non-ML approaches for CVD risk prediction [197] [198] [199] [200] [201] [202] [203] [204] [205] [206] [207] [208] [209] . The types of classifiers used for the cloudbased ML CVD risk prediction systems were quite similar to the mobile-based systems, namely SVM [201, 209] , k-NN [208] , CNN [201, 202] , RF [207] , Bayesian [204] , and DT [207] . The number of outcomes changes to 1 in the cloud-based CVD systems [197] [198] [199] [200] [201] [202] [203] [204] [205] [206] [207] [208] [209] . All the mobile and cloud-based studies have performed the feature extraction along with the analysis for the CVD risk prediction. Cross-validation was also done by using the K-fold CV protocol (Column C17) for the mobile, as well as cloud-based systems [197] [198] [199] [200] [201] [202] [203] [204] [205] [206] [207] [208] [209] . For performance evaluation of the mobile and cloud-based CVD, systems were analyzed by the use of different parameters such as sensitivity [207, 209] , specificity [207, 209] , accuracy [207, 209] , precision, F1-Score, P-value, Silberg score [199] , and receiver operating characteristic (ROC) [200] (Column C22-C29). However, the number of performance parameters used by each study ranges from 0 to 3 as described in Table A4 .

Scientific validation (Column C12) was also performed for a high number of mobile and cloud-based CVD studies. Only one cloud-based CVD risk prediction system has been FDA approved (Column C6) [208] . All the characteristics are described in detail in Table A4 . It can be noticed that the AI-based systems have gained the advantage of more accuracy, reliability with the addition of mobile and cloud-based infrastructure. It is also helpful in remote prediction, which is very much important in the COVID-19 framework. 

The CVD risk prediction was taken to next level by integration of mobile, cloud, and telemedicine technologies. The mobile-based CVD systems follow both ML and non-ML approaches [197] [198] [199] [200] [201] [202] [203] [204] [205] [206] [207] . The classifier techniques used for the mobile-based ML systems were k-NN [208] , SVM [201, 209] , CNN [201, 202] , NB [204] , DT [207] , and RF [207] . The number of outcomes for the mobile-based CVD systems [197] varies between 1 and 2, basically CVD and diabetes. The cloud-based CVD systems also used both ML and non-ML approaches for CVD risk prediction [197] [198] [199] [200] [201] [202] [203] [204] [205] [206] [207] [208] [209] . The types of classifiers used for the cloudbased ML CVD risk prediction systems were quite similar to the mobile-based systems, namely SVM [201, 209] , k-NN [208] , CNN [201, 202] , RF [207] , Bayesian [204] , and DT [207] . The number of outcomes changes to 1 in the cloud-based CVD systems [197] [198] [199] [200] [201] [202] [203] [204] [205] [206] [207] [208] [209] . All the mobile and cloud-based studies have performed the feature extraction along with the analysis for the CVD risk prediction. Cross-validation was also done by using the K-fold CV protocol (Column C17) for the mobile, as well as cloud-based systems [197] [198] [199] [200] [201] [202] [203] [204] [205] [206] [207] [208] [209] . For performance evaluation of the mobile and cloud-based CVD, systems were analyzed by the use of different parameters such as sensitivity [207, 209] , specificity [207, 209] , accuracy [207, 209] , precision, F1-Score, p-value, Silberg score [199] , and receiver operating characteristic (ROC) [200] (Column C22-C29). However, the number of performance parameters used by each study ranges from 0 to 3 as described in Table A4 .

Scientific validation (Column C12) was also performed for a high number of mobile and cloud-based CVD studies. Only one cloud-based CVD risk prediction system has been FDA approved (Column C6) [208] . All the characteristics are described in detail in Table A4 . It can be noticed that the AI-based systems have gained the advantage of more accuracy, reliability with the addition of mobile and cloud-based infrastructure. It is also helpful in remote prediction, which is very much important in the COVID-19 framework. As the CVD prediction systems have evolved in the COVID-19 times, we, therefore, discuss this in the upcoming section.

The main scope of this review was to compare comprehensively the three kinds of machine learning (ML) techniques mainly multiclass, multi-label, and ensemble in office-based settings. Further, the scope of the study had a limited discussion on (a) CVD risk prediction using ECG signals-based settings and (b) deep learning (DL) techniques for CVD risk prediction. Therefore, the main or principal findings from this review were (i) three types of CVD risk stratification techniques, namely (a) multiclass (b) multi-label, and (c) ensemble; (ii) types of covariates used where OBBM, LBBM, MedUSE, and CUSIP. The OBBM, LBBM, MedUSE were used widely when compared to image-based phenotypes (CUSIP), which is now evolving more rapidly since is a surrogate marker for coronary artery disease; (iii) ground truth is a very vital factor so as to avoid the risk of bias (RoB) during the ML-based CVD risk prediction; (iv) popularity of the classification techniques used in the field of CVD were in the order as multiclass-based, ensemble-based, multi-label-based;

(v) clinical and scientific validation is another set of AI attributes that must be accompanied in any ML-based CVD risk prediction systems to prevent the AI bias from in such systems; (vi) the performance evaluation metrics used for the three techniques were analyzed. It was found that the most commonly used PE parameter was accuracy. The cloud-based AI techniques comprising all the three classifications techniques are more likely to be the future for CVD risk prediction. In the future, advanced computer-aided diagnosis techniques can be applied based on image processing [210] . Edge devices with mobile and cloud-based AI infrastructure are now highly emerging in the medical industry as it provides remote facility and is a much faster, the most necessary feature in the COVID-19 era. Table   Table 6 shows the benchmarking table with a comparison between eighteen review studies that focused on multiclass, multi-label, and ensemble techniques for CVD risk prediction. This table shows thirteen attributes (column C1 to column C13) for each of the eighteen studies [35, [211] [212] [213] [214] [215] [216] [217] [218] [219] [220] [221] [222] [223] [224] [225] [226] corresponding to the rows R1 to R18. These thirteen attributes presented were the Author (C1), year of the study (C2), name of the journal (C3), data size (C4), the study belongs to CVD or not (C5), the domain of the study (C6), machine learning (C7), classifier type (C8), cross-validation protocol (C9), the studies are multiclass (C10), multi-label study (C11), ensemble study (C12), and finally the summary of the study (C13). The data size for each study is shown in column C4, which is ranging from 8 to 86,155, whereas our study (row R18) has used 94 studies. Column C5 describes whether the study is of CVD type or not. Studies (rows R2, R3, R5, R9, R10, R11, R12 , R16, and R17) along with our study (row R18) are in the field of CVD while the rest are not. Column C6 describes the different domains for the studies (rows R1, R4, R6, R7, R8, R13, R14, and R15) which does not belong to CVD. The domains are EEG, blood pressure, education, statistics, software, chronic fatigue, and sickle cells. The technical approach of the studies is shown in column C7, i.e., whether machine learning (ML) or not. Most of the studies including our proposed study are ML (rows R1, R3, R4, R6, R7, R8, R9, R10,  R11, R12, R13, R14, R15, R16, R17, and R18) . Column (rows R1, R3, R6, R7, R9, R11, R12, R17) shown in column C10 along with our study (row R18). Column C11 shows multi-label studies (rows R8, R13, R14, R15, and R18) likewise column C12 shows the ensemble studies (rows R4, R6, R10, and R18). The last column C13 describes the keyword objectives of each study. The studies' objectives were classification and CVD risk prediction or stratifications. 

The conventional classification CVD risk assessment systems assume the linear relationship between the covariates and the gold standard. The linear systems typically use the covariates like OBBM and LBBM or ECG signals [228] [229] [230] . With the additions of CUSIP and MedUSE, the requirement becomes more stringent on CVD calculators. In today's times, it was observed that COVID-19 can play the role of a new covariate or risk factor due to its relationship with CVD [231, 232] . The risk of CVD gets accelerated in the individual with COVID-19 [233, 234] . This inclusion can result in a more non-linear classification paradigm for CVD risk prediction [235] . This can improve the reliability and the accuracy of the prediction results [236] . The AI/ML approaches help in understanding the non-linear relationship between the covariates and the ground truth. Hence there is a need for the development of non-linear classifiers in the ML/DL domain. It includes non-linear SVM classifiers [237] , PCA, XGBoost [235] , RF [233] , generalized discriminant analysis (GDA), ELM, LDA [238] . Different non-linear methods which are applied in the CVD field are Poincare plot (PP), approximate entropy (ApEn) [235] , quasi period density-prototype distance (QPD-PD) [239] , fuzzy entropy [238] , recurrence period density prototype distance (RPD-PD) [237] , non-linear ensemble classifiers [233] . These are all out of the scope of the current study. The other application of non-linear classifiers are in the field of stroke [240] and sleep apnea [241] . The non-linearity can also be handled by using the DL approaches along with multiclass, multi-label, and ensemble-based techniques for CVD risk prediction in the future.

This is one of the greatest assets of the machine learning system. The most important ingredient for accomplishing this solution is to ensure that we have a follow-up gold standard for the clinical data. This means one must have the gold standard (events) for the times such as 1st-year, 3rd-year, 5th-year, and 10th-year. Further, the risk factors (so-called covariates or variables) must be available for the development of the training model. Given the two pairs (covariates and the gold standard-even for that time), one can develop the machine learning model for that time-zone (1st-year, 3rd-year, 5th-year, and 10th-year). Should you intended to predict for 1st-year, 3rd-year, 5th-year, and 10th-years, it requires four kinds of machine learning models. Each time-event has to have its own machine learning model. The atherosclerosis disease which has transformed over different years and leads to the event needs to be used for the development of the training model. The only challenge with this setup is the length of time it takes to collect the event data. It is both expensive and tedious since we have to follow the patients over the 10-year period. Recently, Kakadiaris et al. [62] perused this strategy using the machine learning paradigm. The ML paradigm has the same fundamental concept of training and testing as shown in Figure 4 . The left half is the training model where the gold standard will change as per the time-zone (1st-year, 3rd-year, 5th-year, and 10th-years), while the prediction will be applied for the patient for the corresponding time-zones (1st-year, 3rd-year, 5th-year, and 10th-years). It is painful to wait to accomplish this validation, since it is costly, and a large cohort is needed.

To overcome such a scenario, another way to predict the CVD risk is using the surrogate marker of carotid artery disease. Since the formation of the atherosclerotic disease in coronary artery has the same genetic make-up as the carotid artery disease, the surrogate artery can be used for the prediction of CVD or the coronary artery disease risk. Further, note that over time (1st-year, 3rd-year, 5th-year, and 10th-years), the plaque formation changes and so does the image phenotypes such as intima-media thickness, plaque burden, or plaque area/volume. Thus, one can compute the time-dependent image phenotypes which uses the ingredients which make the atherosclerotic disease. This includes rate of change of cIMT over time (age), obesity index over time (age), cholesterol change over (age), one can use this paradigm to predict the plaque burden in carotid artery-based age. This is sometimes called as vascular age of the patient. This has been shown by Khanna et al. [34] . Later, this was commercialized as AtheroEdge™ 2.0 (AtheroPoint™, Roseville, CA, USA) [36] . The CVD risk can be computed based on the intensity of the risk factors. This is called a non-ML method (also known as the statistical solution for the prediction of the 10th-year CVD risk.

Machine learning paradigm for CVD risk prediction has provided us with a way to obtain more accurate, early, and fast results. The ML systems offer following advantages against the previously published studies: (i) it handles the non-linear nature between the covariates and ground truths (GT) [31] ; (ii) ability to predict the CVD risk in granular classes, such as six different risk classes (no-risk, low-risk, mild-risk, moderate-risk, high-risk, and very-high-risk) [34, 35] ; (iii) ability to augment the training data using popular augmentation paradigms such as adaptive synthetic (ADASYN) and synthetic minority over-sampling technique SMOTE [227] ; (iv) incorporate the cohort's knowledge during training and predicting the CVD risk; (v) flexibility of amalgamating of different types of covariates such as OBBM, LBBM, CUSIP, and MedUSE during the design of the model training; (vi) ability to interface with different types of classification techniques like multiclass, multi-label and ensemble for improving the overall performance of the system; and (vii) ability to enhance the risk factor (or covariates) such as genetic and comorbidities such as cancer. Thus, all the above-mentioned factors puts ML-based system a very strong paradigm for CVD risk stratification, unlike the conventional statistical models.

The Deep learning (DL) paradigm has started to emerge in the field of CVD risk prediction. The DL approach can be applied for both (a) the office-based [242, 243] and (b) stress-based test settings [244] [245] [246] [247] [248] . DL approaches have been applied for CVD risk stratification using multiclass [249] , multi-label [250] , and ensemble-based paradigms [116] . Even though there are evolving CVD risk stratification techniques in the DL framework, this review does not venture deep since it is not the main focus of this review. As a result, we have not analyzed publications related to the DL paradigm. Note that, the main advantage of DL techniques is (i) automated feature selection process from the input covariates (such as OBBM, LBBM, CUSIP, and EGC signals phenotype) and (ii) prediction of more accurate and reliable results due to a large number of layers in DL network. Advanced stochastic imaging methods can be applied [251] to improve the loss function during the training paradigm. This evolving DL paradigm will flourish more in the very near future in office-based imaging and stress-based test settings.

The CVD risk estimation at an early stage is very much important to reduce the mortality rate due to CVD [252, 253] . As it was observed that not only ML but extreme machine learning (ELM) can also be applied and further developed for CVD risk stratification [254] . Moreover, COVID-19 accelerates the atherosclerosis condition due to which fast detection of CVD in COVID-19 patients is needed [255, 256] . The above circumstances are leading to an evaluation in the CVD risk stratification techniques. In the near future, cloud-based AI modalities will be very much in use for CVD risk detection. It also promotes the remote and fast prediction of the risk of CVD. It also helps in reducing prediction errors. Other non-invasive imaging techniques like carotid, femoral, arterial imaging can be used as an indirect measure of plaque build-up in these arteries. Deep learning technologies will evolve in the field of CVD risk estimation [257] . This will also include pruning of weights using evolutionary techniques such as genetic algorithms in the Deep Learning framework [147] . Devices equipped with cutting edge technologies like mobile-based AI, cloud-based AI, multiclass, multi-label, and ensemble-based systems for CVD risk prediction will be emerging in the medical imaging industry market.

This was the first review study of its kind that presented three different kinds of AI-based CVD risk stratification, namely multiclass, multi-label, and ensemble, where multiclass was most popular and multi-label was least, which happened to be our first key contribution. The second contribution was exhaustive analysis by selecting the best 265 studies using the PRISMA model for understanding the three kinds of machine learningbased systems for prediction of the CVD risk. This was based on our hypothesis that there exists a biological link between atherosclerotic disease formation and the CVD risk. The third contribution was the identification of the top four covariates, namely OBBM, LBBM, CUSIP, and MedUSE for designing the training model using a machine learning framework. The fourth contribution was on the choice of the gold standard for an unbiased AI system design for CVD risk prediction, which leads to a robust and reliable CVD prediction system. The fifth finding and contribution required that the ML system undergo clinical and scientific validation for reliability, stability, and robustness of the system design. Lastly, we observed that with the advancement of telecommunication systems, mobile and cloud-based strategies are speedily penetrating the CVD risk stratification system designs. Low-powered edge devices like Rasberry Pi and Jetsen Nano are like to be adopted in the future. 

The authors declare no conflict of interest.

Appendix A. 1 

This system shows the amalgamation of online covariates, which are then transformed by the ML-based training model using multiclass-based models. The output yields the multiclass risk marked in color (low, mild, moderate, and high risk). 

The pseudo-code describes the process used by the multiclass algorithm for CVD risk stratification into granular risk classes. It uses the "for" loop for training and prediction of each fold of data, which were divided into K folds. The training model is applied to the test data and the PE was predicted and stored in form of accuracy (ACC), ROC, sensitivity (Sen), specificity (Spec), F1-score, the area-under-the-curve (AUC), and precision. 

The pseudo-code describes the process used by the multiclass algorithm for CVD risk stratification into granular risk classes. It uses the "for" loop for training and prediction of each fold of data, which were divided into K folds. The training model is applied to the test data and the PE was predicted and stored in form of accuracy (ACC), ROC, sensitivity (Sen), specificity (Spec), F1-score, the area-under-the-curve (AUC), and precision. Figure A1 . Typical online system for multiclass CVD risk stratification.

The pseudo-code describes the process used by the multiclass algorithm for CVD risk stratification into granular risk classes. It uses the "for" loop for training and prediction of each fold of data, which were divided into K folds. The training model is applied to the test data and the PE was predicted and stored in form of accuracy (ACC), ROC, sensitivity (Sen), specificity (Spec), F1-score, the area-under-the-curve (AUC), and precision. Figure A2 . Pseudo-code for multiclass technique.

The problem transformation method (PTM) makes the multi-label classification problem to one or more single label classification tasks. Basically, four PTM, namely BR, CC, LP, and RakEL were used as discussed below:

Binary Relevance: In the BR technique, the problems get divided into one or more single-label classification problems. The single-label classification resembles the binary prediction. An example can be described as, say M is a set of "q" labels with M = {m1, m2, …, Mq}, the BR technique makes "q" single-label binary classifiers for each label. The multilabel training sets get converted to binary datasets ("q"), and Elj = 1…..q, where Elj has all Figure A2 . Pseudo-code for multiclass technique.

The problem transformation method (PTM) makes the multi-label classification problem to one or more single label classification tasks. Basically, four PTM, namely BR, CC, LP, and RakEL were used as discussed below:

Binary Relevance: In the BR technique, the problems get divided into one or more single-label classification problems. The single-label classification resembles the binary prediction. An example can be described as, say M is a set of "q" labels with M = {m1, m2, . . . , Mq}, the BR technique makes "q" single-label binary classifiers for each label. The multi-label training sets get converted to binary datasets ("q"), and Elj = 1 . . . q, where Elj has all samples of the original dataset but with single positive or negative values. The dataset gets divided into "q" single label datasets with classifier C and the next classifier set is obtained as Cj (E), j = 1 . . . q by the training set Elj. The label dependency was not considered by the BR classification algorithm. Thus, it shows less complexity in the computation as compared with other multi-label techniques. The process is shown in the following Figure A3 [258] . As shown in Figure A3 four examples were considered as multi-label dataset and label set M with four labels (m1, m2, m3, and m4) which is split as four single labels that are independent.

Classifier Chain: This algorithm also works in single-label classification. This technique takes a class of classifiers where the very initial classifier is trained with the dataset, which acts as the input, following that each classifier gets trained with the whole feature space. The feature set has an original dataset with the label set used in the earlier base classifier that is in the chain. Each base classifier uses the earlier label information for training and testing models. Thus, a correlation exists in the CC algorithm. Figure A4 describes the functioning of CC [259] .

Classifier Chain: This algorithm also works in single-label classification. This technique takes a class of classifiers where the very initial classifier is trained with the dataset, which acts as the input, following that each classifier gets trained with the whole feature space. The feature set has an original dataset with the label set used in the earlier base classifier that is in the chain. Each base classifier uses the earlier label information for training and testing models. Thus, a correlation exists in the CC algorithm. Figure A4 describes the functioning of CC [259] . Label Powerset: It also converts the prediction situation to a single-label multiclass prediction technique. In this technique, all possible individual group of labels is given special or unique class. Such as if three types of labels are there, then eight different types of combinations can come into the picture. LP technique has eight types of labels that get nique takes a class of classifiers where the very initial classifier is trained with the dataset, which acts as the input, following that each classifier gets trained with the whole feature space. The feature set has an original dataset with the label set used in the earlier base classifier that is in the chain. Each base classifier uses the earlier label information for training and testing models. Thus, a correlation exists in the CC algorithm. Figure A4 describes the functioning of CC [259] . Label Powerset: It also converts the prediction situation to a single-label multiclass prediction technique. In this technique, all possible individual group of labels is given special or unique class. Such as if three types of labels are there, then eight different types of combinations can come into the picture. LP technique has eight types of labels that get Label Powerset: It also converts the prediction situation to a single-label multiclass prediction technique. In this technique, all possible individual group of labels is given special or unique class. Such as if three types of labels are there, then eight different types of combinations can come into the picture. LP technique has eight types of labels that get trained for prediction. This technique deals with a large number of classes that are related to small instances, and also consideration of correlation is done. The transformation was shown in Figure A5 [260] . In Figure A5 trained for prediction. This technique deals with a large number of classes that are related to small instances, and also consideration of correlation is done. The transformation was shown in Figure A5 [260] . In Figure A5 the 1st table shows the original datasets, and the 2nd table is showings the transformed datasets. Random k-label set: It is a type of combination technique used for multi-label prediction. Every combination method gets trained on a small size of the randomly selected subset of labels by a single-label-based classifier. This process is described as if L labels in the dataset (E), the RAkEL classifier turns this data to all the possible k-label sets (Lk). Each label set is then trained for prediction. Finally, the prediction is made into positive (1) and negative (0) values in accordance with the threshold (0.5). The further implementation can Random k-label set: It is a type of combination technique used for multi-label prediction. Every combination method gets trained on a small size of the randomly selected subset of labels by a single-label-based classifier. This process is described as if L labels in the dataset (E), the RAkEL classifier turns this data to all the possible k-label sets (L k ). Each label set is then trained for prediction. Finally, the prediction is made into positive (1) and negative (0) values in accordance with the threshold (0.5). The further implementation can be seen in [261] .

Multi-label KNN: This algorithm is basically an implementation of the KNN algorithm in multi-label datasets. The neighbors are selected from unseen training sets. Next, the labelset are found for the instance which are unseen in nature by utilizing the maximum of posteriori (MAP) principle. The full algorithm can be seen in [262] .

Multi-label ARAM: It is associated with the neural network model based on resonance theory. The advantage of this algorithm is its fast learning ability. The detailed algorithm can be seen in [263] .

Multi-label pseudo-code describes the multi-label algorithm where more than one multi-label endpoint was considered. For each multi-label endpoint, the risk class was defined. In this pseudo-code, two "for" loops are used one for multi-label and the next for multiclass prediction. Finally, the PE was determined as accuracy, sensitivity (Sen), specificity (Spec), area-under-the-curve (AUC), sample-based, and label-based metrics. 

Ensemble-based-CVD risk prediction uses combinations of multiple classifiers. The pseudo-code shows that the data are divided into testing and training with K folds. The prediction was done using each type of classifier for multiclass and multi-label prediction. Then each type of classifier is combined into an ensemble classifier and the final prediction was made. 

Ensemble-based-CVD risk prediction uses combinations of multiple classifiers. The pseudo-code shows that the data are divided into testing and training with K folds. The prediction was done using each type of classifier for multiclass and multi-label prediction. Then each type of classifier is combined into an ensemble classifier and the final prediction was made. Diagnostics 2022, 12, x FOR PEER REVIEW 31 of 48 Figure A7 . Pseudo-code for ensemble-based technique. 

Comparison of ML-Based Multiclass, Multi-Label, and Ensemble CVD Classification 

The PE for the multiclass and ensemble basically have accuracy (ACC), sensitivity (Sen), specificity (Spec), AUC, F1-Score which were calculated using values of true positives (TPs), false positives (FPs), false negatives (FNs), and true negatives (TNs). The formulae can be referred from Table A2 . The performance evaluation for multi-label-based CVD is different as compared to multiclass and ensemble. They are label-based, instance-based performance evaluations.

In the label-based techniques, the PE parameters are checked for each label by the values of TPs, FPs, FNs, and TNs. All the labels have their own values. S, these are calculated by averaging methods (i) macro-averaging and (ii) micro-averaging [181] . The performance metrics say β is calculated by the values of TPs, FPs, FNs, and TNs, the macroaveraging techniques, macro-averaging (β macro ) for all labels (L) is given by averaging β for each label "p", as shown in Equation (A1).

In the same manner, for the micro-averaging techniques, the PE metrics are computed for each individual label and finally obtaining the micro-average (β micro ) by using the Equation (A2).

For instance-based performance evaluation, the parameters are calculated for individual instances, then the average value is computed and final the performance metric is performed. The final metric has a hamming loss, precision, recall, F1-score, Jaccard similarity coefficient score, and accuracy.

The multi-label dataset is supposed to be |E| with multi-label examples (pi, Qi), i = 1 . . . |E|, and Qi ⊆ L, L is a set of all multiple labels. C is a multi-label classifier and Mi = C (pi) be the set of labels predicted by C. |E| indicates the features of the set E, while |Qi∩ Mi| indicates the feature of the intersection of true labels and the predicted labels. |Mi| indicates the features of predicted labels, and |Qi| indicates the features of the true labels.

Hamming loss shows the number of times when the label pair is misclassified. The lower value of Humming loss presents the better performance of the multi-label classifier. Jaccard score presents the ratio of the size of the intersection between predicted and the ground truth labels. Precision is the proportion of correct predictions out of all predictions. Likewise, recall is the ratio of correct predicted labels to the actual labels. F1-score is the combination of precision and recall Table A3 . Table A2 . Performance evaluation metrics used in CVD risk assessment. 

Subset Accuracy, Acc Subset Acc Subset = 1

Power analysis can be done for multi-label and ensemble-based CVD systems. Its objective was to state the smallest data or sample size (s) needed to perform the multi-label, ensemble-based CVD risk classification. The parameters which are required for calculating power analysis are confidence interval, a margin error (e) as ±5%, and a sample proportion (q), the z-score (z * ) (taken standard z-table). Therefore, the formula used is shown in Equation (A3) [264, 265] . 

World Health Organization cardiovascular disease risk charts: Revised models to estimate risk in 21 global regions

Projected costs of informal caregiving for cardiovascular disease: 2015 to 2035: A policy statement from the

Calcium detection, its quantification, and grayscale morphology-based risk stratification using machine learning in multimodality big data coronary and carotid scans: A review

Low-cost preventive screening using carotid ultrasound in patients with diabetes

Cardiovascular disease and stroke risk assessment in patients with chronic kidney disease using integration of estimated glomerular filtration rate, ultrasonic image phenotypes, and artificial intelligence: A narrative review

Integration of estimated glomerular filtration rate biomarker in image-based cardiovascular disease/stroke risk calculator: A south Asian-Indian diabetes cohort with moderate chronic kidney disease

Cardiovascular risk assessment in patients with rheumatoid arthritis using carotid ultrasound B-mode imaging

Cardiovascular disease detection using machine learning and carotid/femoral arterial imaging frameworks in rheumatoid arthritis patients

Carotid plaque imaging profiling in subjects with risk factors (diabetes and hypertension)

Carotid intraplaque-hemorrhage volume and its association with cerebrovascular events

Automated diagnosis of epileptic EEG using entropies

Use of principal component analysis for automatic classification of epileptic EEG activities in wavelet framework

Multi-Label Active Learning-Based Machine Learning Model for Heart Disease Prediction

A wearable smartphone-based platform for real-time cardiovascular disease detection via electrocardiogram processing

ECG criteria for the detection of high-risk cardiovascular conditions in master athletes

Machine learning improves risk stratification after acute coronary syndrome

Ranking of stroke and cardiovascular risk factors for an optimal risk calculator design: Logistic regression approach

Advances in Cardiac Signal Processing

Automated diagnosis of coronary artery disease affected patients using LDA, PCA, ICA and discrete wavelet transform

Heart rate variability: A review

Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: Prospective cohort study

General cardiovascular risk profile for use in primary care: The Framingham Heart Study

Estimation of ten-year risk of fatal cardiovascular disease in Europe: The SCORE project

Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: The Reynolds Risk Score

ACC/AHA guideline on the assessment of cardiovascular risk: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines

ESC guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation: Comments from the Dutch ACS working group

ESC guidelines on the management of stable coronary artery disease: The Task Force on the management of stable coronary artery disease of the European Society of Cardiology

ESC Guidelines for the diagnosis and management of chronic coronary syndromes: The Task Force for the diagnosis and management of chronic coronary syndromes of the European Society of Cardiology (ESC)

Canadian Cardiovascular Society guidelines for the management of dyslipidemia for the prevention of cardiovascular disease in the adult

update of the Canadian Cardiovascular Society guidelines for the diagnosis and treatment of dyslipidemia for the prevention of cardiovascular disease in the adult

Moving beyond regression techniques in cardiovascular risk prediction: Applying machine learning to address analytic challenges

Contrast-enhanced ultrasound imaging of carotid plaque neovascularization is useful for identifying high-risk patients with coronary artery disease

Combined femoral and carotid plaque burden identifies obstructive coronary artery disease in women

Nonlinear model for the carotid artery disease 10-year risk prediction by fusing conventional cardiovascular factors to carotid ultrasound image phenotypes: A Japanese diabetes cohort study

Artificial intelligence framework for predictive cardiovascular and stroke risk assessment models: A narrative review of integrated approaches using carotid ultrasound

Low-cost office-based cardiovascular risk stratification using machine learning and focused carotid ultrasound in an Asian-Indian cohort

Review of Imaging biomarkers for the vulnerable carotid plaque

Multi-Detector CT Imaging: Principles, Head, Neck, and Vascular Systems

Superharmonic imaging for medical ultrasound: A review

Speckle reduction in medical ultrasound images using an unbiased non-local means method

State-of-the-art medical image registration methodologies: A survey. In Multi Modality State-of-the-Art Medical Image Segmentation and Registration Methodologies

Prognostic Factors of Fatal and Nonfatal Cardiovascular Events in Patients with Type 2 Diabetes: The Role of Renal Function Biomarkers

Relation between age and carotid artery intima-medial thickness: A systematic review

Current status of carotid ultrasound in atherosclerosis

Mannheim carotid intima-media thickness and plaque consensus

Use of carotid ultrasound to identify subclinical vascular disease and evaluate cardiovascular disease risk: Summary and discussion of the American Society of Echocardiography consensus statement

Ankle-brachial index and its link to automated carotid ultrasound measurement of intima-media thickness variability in 500 Japanese coronary artery disease patients

Carotid intima-media thickness and plaque in cardiovascular risk assessment

Association of carotid plaques and common carotid intima-media thickness with modifiable cardiovascular risk factors

Association between carotid intima media thickness, age, and cardiovascular risk factors in children and adolescents

Role of artificial intelligence in cardiovascular risk prediction and outcomes: Comparison of machine-learning and conventional statistical approaches for the analysis of carotid ultrasound features and intra-plaque neovascularization

Carotid ultrasound maximum plaque height-A sensitive imaging biomarker for the assessment of significant coronary artery disease

Handheld versus conventional vascular ultrasound for assessing carotid artery plaque

Association of automated carotid IMT measurement and HbA1c in Japanese patients with coronary artery disease

Automated deep learning-based paradigm for high-risk plaque detection in B-mode common carotid ultrasound scans: An asymptomatic Japanese cohort study

Ultrasound carotid plaque features, cardiovascular disease risk factors and events: The Multi-Ethnic Study of Atherosclerosis

State-of-the-art review on deep learning in medical imaging

Plaque tissue morphology-based stroke risk stratification using carotid ultrasound: A polling-based PCA learning paradigm

Machine learning for prediction of all-cause mortality in patients with suspected coronary artery disease: A 5-year multicentre prospective registry analysis

Van der Schaar, M. Cardiovascular disease risk prediction using automated machine learning: A prospective study of 423,604 UK Biobank participants

Machine learning outperforms ACC/AHA CVD risk calculator in MESA

On over-fitting in model selection and subsequent selection bias in performance evaluation

Detection of Cardiovascular Disease using Machine Learning Classification Models

Cardiovascular Disease Prediction using Classification Algorithms of Machine Learning

Using recurrent neural network models for early detection of heart failure onset

Cardiovascular Disease Prediction from Electrocardiogram by using Machine Learning Method: A Snapshot from the Subjects of the Malaysian Cohort

A hybrid information mining approach for knowledge discovery in cardiovascular disease (CVD). Information

Machine learning improves cardiovascular risk definition for young, asymptomatic individuals

SVM multi-classification of T2D/CVD patients using biomarker features

Deep learning predicts cardiovascular disease risks from lung cancer screening low dose computed tomography

A novel multi-class approach for early-stage prediction of sudden cardiac death

Bragatston study protocol: A multicentre cohort study on automated quantification of cardiovascular calcifications on radiotherapy planning CT scans for cardiovascular risk prediction in patients with breast cancer

Identification of novel pheno-groups in heart failure with preserved ejection fraction using machine learning

An Automated High-Accuracy Detection Scheme for Myocardial Ischemia Based on Multi-Lead Long-Interval ECG and Choi-Williams Time-Frequency Analysis Incorporating a Multi-Class SVM Classifier

Multiclass machine learning vs. conventional calculators for stroke/CVD risk assessment using carotid plaque predictors with coronary angiography scores as gold standard: A 500 participants study

Classification of Multi-Class Cardiovascular Disorders using Ensemble Classifier and Impulsive Domain Analysis

Multiclass Classification of APG Signals using ELM for CVD Risk Identification: A Real-Time Application

Multiclass classification of myocardial infarction with convolutional and recurrent neural networks for portable ECG devices

Machine learning adds to clinical and CAC assessments in predicting 10-year CHD and CVD deaths

Multi-class Cardiovascular Disease Detection and Classification from 12-Lead ECG Signals Using an Inception Residual Network

Performance analysis of multiclass support vector machine classification for diagnosis of coronary heart diseases

Cardiovascular event prediction by machine learning: The multi-ethnic study of atherosclerosis

A machine learning framework for risk prediction of multi-label cardiovascular events based on focused carotid plaque B-Mode ultrasound: A Canadian study

CIMT as a risk factor for stroke subtype: A systematic review

Classification of Atrial Fibrillation and Acute Decompensated Heart Failure Using Smartphone Mechanocardiography: A Multilabel Learning Approach

Multi-label classification and evidential approach in diseases diagnoses using physiological signals

Applying Deep Learning to ICD-9 Multi-Label Classification from Medical Records

Harnessing machine intelligence in automatic echocardiogram analysis: Current status, limitations, and future directions

Explainable machine-learning predictions for complications after pediatric congenital heart surgery

A new machine learning technique for an accurate diagnosis of coronary artery disease

Ensemble deep learning models for heart disease classification: A case study from Mexico

Roles of anxiety and depression in predicting cardiovascular disease among patients with type 2 diabetes mellitus: A machine learning approach

Using ensemble of ensemble machine learning methods to predict outcomes of cardiac resynchronization

Cardiovascular disease detection using a new ensemble classifier

Machine learning versus traditional risk stratification methods in acute coronary syndrome: A pooled randomized clinical trial analysis

Improving the Accuracy for Analyzing Heart Diseases Prediction Based on the Ensemble Method

Disease prediction via Bayesian hyperparameter optimization and ensemble learning

Efficient Prediction of Cardiovascular Disease Using Machine Learning Algorithms with Relief and LASSO Feature Selection Techniques

Abnane, I. A systematic mapping study for ensemble classification methods in cardiovascular disease

Ensemble approach for developing a smart heart disease prediction system using classification algorithms

Ensemble Machine Learning and its Validation for Prediction of Coronary Artery Disease and Acute Coronary Syndrome using Focused Carotid Ultrasound

Enhanced Evolutionary Feature Selection and Ensemble Method for Cardiovascular Disease Prediction

A novel Ensemble Learning Paradigm for Medical Diagnosis with Imbalanced Data

Diagnosing coronary heart disease using ensemble machine learning

An improved ensemble learning approach for the prediction of heart disease risk

Prediction Model Using Machine Learning for Mortality in Patients with Heart Failure

Novel deep genetic ensemble of classifiers for arrhythmia detection using ECG signals

Heart Disease Risk Prediction Using Machine Learning Classifiers with Attribute Evaluators

Can machine learning bring cardiovascular risk assessment to the next level? A methodological study using FOURIER trial data

An Ensemble Basedheart Disease Predictionusing Gradient Boosting Decision Tree. Turk

A soft voting ensemble classifier for early prediction and diagnosis of occurrences of major adverse cardiovascular events for STEMI and NSTEMI during 2-year follow-up in patients with acute coronary syndrome

The prediction of cardiovascular disease based on trace element contents in hair and a classifier of boosting decision stumps

An Ensemble Method Based Multilayer Dynamic System to Predict Cardiovascular Disease Using Machine Learning Approach

Ensemble of heterogeneous classifiers for diagnosis and prediction of coronary artery disease with reduced feature subset

Effective prediction of heart disease using hybrid ensemble deep learning and tunicate swarm algorithm

Analysis of Heart Disease Using Parallel and Sequential ensemble Methods with Feature Selection Techniques: Heart Disease Prediction

Prediction of incident hypertension within the next year: Prospective study using statewide electronic health records and machine learning

Prediction of heart disease using ensemble learning and Particle Swarm Optimization

Comparison of machine learning approaches toward assessing the risk of developing cardiovascular disease as a long-term diabetes complication

A hybrid cost-sensitive ensemble for heart disease prediction

XGBoost: A scalable tree boosting system

Xgboost: Extreme gradient boosting. R Package Version 0

Inflammation and plaque vulnerability

Inflammatory biomarkers of coronary heart disease

Inflammation in atherosclerosis: From pathophysiology to practice

Cell biology of atherosclerosis

Subendothelial lipoprotein retention as the initiating process in atherosclerosis: Update and therapeutic implications

Pathology of the thin-cap fibroatheroma: A type of vulnerable plaque

Morphological predictors of arterial remodeling in coronary atherosclerosis

A review on atherosclerotic biology, wall stiffness, physics of elasticity, and its ultrasound-based measurement

Mechanisms of plaque rupture: Mechanical and biologic interactions

Material properties of components in human carotid atherosclerotic plaques: A uniaxial extension study

Classification of atherosclerotic and non-atherosclerotic individuals using multiclass support vector machine. Technol. Health Care

Presence of calcium-like tissue composition in carotid plaque is indicative of significant coronary artery disease in high-risk patients

Comparison between statistical models and machine learning methods on classification for highly imbalanced multiclass kidney data

Deep-learning-based cardiovascular risk stratification using coronary artery calcium scores predicted from retinal photographs

Multiclass magnetic resonance imaging brain tumor classification using artificial intelligence paradigm

Multi-instance multi-label learning for multi-class classification of whole slide breast histopathology images

Development of health parameter model for risk prediction of CVD using SVM

Robust multi-label transfer feature learning for early diagnosis of Alzheimer's disease

Prediction of heart disease using machine learning algorithms

Association of psychosocial risk factors with risk of acute myocardial infarction in 11,119 cases and 13,648 controls from 52 countries (the INTERHEART study): Case-control study

Effect of potentially modifiable risk factors associated with myocardial infarction in 52 countries (the INTERHEART study): Case-control study

Reliable and accurate psoriasis disease classification in dermatology images using comprehensive feature space in machine learning paradigm

PCA-based polling strategy in machine learning framework for coronary artery disease risk assessment in intravascular ultrasound: A link between carotid and coronary grayscale plaque morphology

Evolutionary algorithm-based classifier parameter tuning for automatic ovarian cancer tissue characterization and classification

How machine learning is impacting research in atrial fibrillation: Implications for risk prediction and future management

Detection of abnormal left ventricular geometry in patients without cardiovascular disease through machine learning: An ECG-based approach

Risk prediction for cardiovascular disease using ECG data in the China Kadoorie Biobank

Machine learning in electrocardiography and echocardiography: Technological advances in clinical cardiology

Use of artificial intelligence and deep neural networks in evaluation of patients with electrocardiographically concealed long QT syndrome from the surface 12-lead electrocardiogram

An hybrid ECG-based deep network for the early identification of high-risk to major cardiovascular events for hypertension patients

Multiscale Residual Network Based on Channel Spatial Attention Mechanism for Multilabel ECG Classification

Classifying syndromes in Chinese medicine using multi-label learning algorithm with relevant features for each label

A deep learning approach to predict diabetes' cardiovascular complications from administrative claims

Weighing features of lung and heart regions for thoracic disease classification

Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks

Predict multicategory causes of death in lung cancer patients using clinicopathologic factors

Nuclear imaging of vulnerable plaque: Contrast improvements through multi-labeling of nanoparticles

Image classification algorithm based on LTS-HD multi instance multi label RBF

A multi-label tree ensemble method for multi-label classification

Patient classification of hypertension in Traditional Chinese Medicine using multi-label learning techniques

Multi-label ECG signal classification based on ensemble classifier

An efficient multi-layer extreme learning machine using combined kernels for multi-label classification

Regularizing extreme learning machine by dual locally linear embedding manifold learning for training multi-label neural network classifiers

Automatic diagnosis of multiple cardiac diseases from PCG signals using convolutional neural network

Cerebral small vessel disease: A review focusing on pathophysiology, biomarkers, and machine learning strategies

Application of ensemble machine learning algorithms on lifestyle factors and wearables for cardiovascular risk prediction

Available online: https://machinelearningmastery. com/nested-cross-validation-for-machine-learning-with-python

Special issue on ensemble learning and applications

Generalized k-labelsets ensemble for multi-label and cost-sensitive classification

A biomedical hybrid MUlti-LABel classifier based on multiple linear regression

Multi-label classification methods for green computing and application for mobile medical recommendations

A deep CNN based multi-class classification of Alzheimer's disease using MRI

Automatic right ventricle segmentation using multi-label fusion in cardiac MRI

Deep learning analysis of resting electrocardiograms for the detection of myocardial dysfunction, hypertrophy, and ischaemia: A systematic review

Spatial QRS-T angle and cognitive decline in older subjects

New concepts in sudden cardiac arrest to address an intractable epidemic: JACC state-of-the-art review

Multi-label classifier performance evaluation with confusion matrix

Multi-label classification: An overview

A comparison of multi-label feature selection methods using the problem transformation approach

Coronary Artery Calcium and Cognitive Function in Dutch Adults: Cross-Sectional Results of the Population-Based ImaLife Study

Tele-electrocardiography and bigdata: The CODE (Clinical Outcomes in Digital Electrocardiography) study

Predictive performance and impact of algorithms in remote monitoring of chronic conditions: A systematic review and meta-analysis

Epidemiology of sudden cardiac death: Clinical and research implications

Risk stratification of sudden cardiac death in patients with heart failure: An update

Early prognostic value of an Algorithm based on spectral Variables of Ventricular fibrillAtion from the EKG of patients with suddEn cardiac death: A multicentre observational study (AWAKE)

Spectral analysis-based risk score enables early prediction of mortality and cerebral performance in patients undergoing therapeutic hypothermia for ventricular fibrillation and comatose status

Performance evaluation of time-frequency distributions for ECG signal analysis

A survey on bias and fairness in machine learning

Visual analytics for discovering intersectional bias in machine learning

Taming technical bias in machine learning pipelines

Systematic review of artificial intelligence in acute respiratory distress syndrome for COVID-19 lung patients: A biomedical imaging perspective

Primary Prevention of Cardiovascular Disease and Type 2 Diabetes Mellitus Using Mobile Health Technology: Systematic Review of the Literature

Evaluation of mHealth Applications Related to Cardiovascular Diseases: A Systematic Review

e-Health. Current status of cardiovascular disease-related smartphone apps downloadable in China

Intra-and inter-operator reproducibility of automated cloud-based carotid lumen diameter ultrasound measurement

A Low-cost, Low-energy Wearable ECG System with Cloud-Based Arrhythmia Detection

An Artificial Intelligence based mobile solution for early detection of valvular heart diseases

Congestive heart failure risk assessment monitoring through internet of things and mobile personal health systems

The lookAfterRisk Project: Dynamic Cardiovascular Risk Assessment based on Remote Monitoring Solutions

Rationale and design of a digital trial using smartphones to detect subclinical atrial fibrillation in a population at risk: The eHealth-based bavarian alternative detection of Atrial Fibrillation (eBRAVE-AF) trial

A smartphone-based survey in mHealth to investigate the introduction of the artificial intelligence into cardiology

Pit-a-Pat: A smart electrocardiogram system for detecting arrhythmia

Symmetrical compression distance for arrhythmia discrimination in cloud-based big-data services

A clinical decision support platform for the risk stratification, diagnosis, and prediction of coronary artery disease evolution

Breast Imaging, Mammography, and Computer-Aided Diagnosis of Breast Cancer

Multiclass classification of brain-computer interface motor imagery system: A systematic literature review

Global cardiovascular risk assessment in the primary prevention of cardiovascular disease in adults: Systematic review of systematic reviews

Comparative Study on Heart Disease Prediction Using Feature Selection Techniques on Classification Algorithms

A review on ensembles for the class imbalance problem: Bagging-, boosting-, and hybrid-based approaches

Primary prevention of cardiovascular disease: A review of contemporary guidance and literature

A Review on Handling Multiclass Imbalanced Data Classification In Education Domain

Implementation of Multiclass Algorithm for Sickle Cell Identification and Categorization-A Review

Reliable multi-label learning via conformal predictor and random forest for syndrome differentiation of chronic fatigue in traditional Chinese medicine

Intelligence system for diagnosis level of coronary heart disease with K-star algorithm

A survey on coronary atherosclerotic plaque tissue characterization in intravascular optical coherence tomography

Assessment of cardiovascular risk markers from ultrasound images: System reproducibility

A Survey of Statistical Topic Model for Multi-Label Classification

A comprehensive and didactic review on multilabel learning software tools

A review on dimensionality reduction for multi-label classification

Healthcare Management of Major Cardiovascular Disease-A review

Machine learning algorithm to predict coronary artery calcification in asymptomatic healthy population

Synthetic minority over-sampling technique

Heart rate variability

A deep learning approach for annotating the pathogenicity of genetic variants

Non-linear effects of the built environment on automobile-involved pedestrian crash frequency: A machine learning approach

Securing your relationship: Quality of intimate relationships during the COVID-19 pandemic can be predicted by attachment style

as a major predictor of COVID-19 basic reproduction number in the USA

Rapid triage for COVID-19 using routine clinical data for patients attending hospital: Development and prospective validation of an artificial intelligence screening test

Application of non-linear and wavelet based features for the automated identification of epileptic EEG signals

Automated diagnosis of coronary artery diseased patients by heart rate variability analysis using linear and non-linear methods

Application of machine learning in liver transplantation: A review

Non-linear dynamical signal characterization for prediction of defibrillation success through machine learning

Detection of coronary artery disease by reduced features and extreme learning machine

Integration of Attributes from Non-Linear Characterization of Cardiovascular Time-Series for Prediction of Defibrillation Outcomes

PCA-based selection of distinctive stability criteria and classification of post-stroke pathological postural behaviour

A hybrid feature selection and extraction methods for sleep apnea detection using bio-signals

Detection and classification of arrhythmia using an explainable deep learning model

Deep CNN with LM learning based myocardial ischemia detection in cardiac magnetic resonance images

Deep learning algorithm classifies heartbeat events based on electrocardiogram signals

DeepMI: Deep multi-lead ECG fusion for identifying myocardial infarction and its occurrence-time

Cardiovascular disease diagnosis using cross-domain transfer learning

One dimensional deep capsule networks for coronary artery disease detection using ECG signals

Automatic ECG classification using continuous wavelet transform and convolutional neural network

Deep multi-scale fusion neural network for multi-class arrhythmia detection

Automatic Multi-Label ECG Classification with Category Imbalance and Cost-Sensitive Thresholding

Stochastic Modeling for Medical Image Analysis

Cardiovascular risk prediction: Basic concepts, current status, and future directions

Cardiovascular disease risk prediction models: Challenges and perspectives

Extreme learning machine framework for risk stratification of fatty liver disease using ultrasound tissue characterization

Excess deaths in people with cardiovascular diseases during the COVID-19 pandemic

Cardiovascular manifestations of COVID-19 infection

Unseen Artificial Intelligence-Deep Learning Paradigm for Segmentation of Low Atherosclerotic Plaque in Carotid Ultrasound: A Multicenter Cardiovascular Study

Multi-label problem transformation methods: A case study

Classifier chains for multi-label classification

Learning multi-label scene classification

Random k-labelsets: An ensemble method for multilabel classification

A lazy learning approach to multi-label learning

Haram: A hierarchical aram neural network for large-scale text classification

Determination of sample size

Determining Sample Size: How to Ensure You Get the Correct Sample Size

Appendix H. 1 

Appendix H. 1