key: cord-323581-qtasvgtd
authors: Zhang, Yu-Dong; Dong, Zhengchao; Wang, Shui-Hua; Yu, Xiang; Yao, Xujing; Zhou, Qinghua; Hu, Hua; Li, Min; Jiménez-Mesa, Carmen; Ramirez, Javier; Martinez, Francisco J.; Gorriz, Juan Manuel
title: Advances in Multimodal Data Fusion in Neuroimaging: Overview, Challenges, and Novel Orientation
date: 2020-07-17
journal: Inf Fusion
DOI: 10.1016/j.inffus.2020.07.006
sha: 
doc_id: 323581
cord_uid: qtasvgtd

Multimodal fusion in neuroimaging combines data from multiple imaging modalities to overcome the fundamental limitations of individual modalities. Neuroimaging fusion can achieve higher temporal and spatial resolution, enhance contrast, correct imaging distortions, and bridge physiological and cognitive information. In this study, we analyzed over 450 references from PubMed, Google Scholar, IEEE, ScienceDirect, Web of Science, and various sources published from 1978 to 2020. We provide a review that encompasses (1) an overview of current challenges in multimodal fusion (2) the current medical applications of fusion for specific neurological diseases, (3) strengths and limitations of available imaging modalities, (4) fundamental fusion rules, (5) fusion quality assessment methods, and (6) the applications of fusion for atlas-based segmentation and quantification. Overall, multimodal fusion shows significant benefits in clinical diagnosis and neuroscience research. Widespread education and further research amongst engineers, researchers and clinicians will benefit the field of multimodal neuroimaging.

Neuroimaging has been playing pivotal roles in clinical diagnosis and basic biomedical research in the past decades. As described in the following section, the most widely used imaging modalities are magnetic resonance imaging (MRI), computerized tomography (CT), positron emission tomography (PET), and single-photon emission computed tomography (SPECT). Among them, MRI itself is a non-radioactive, non-invasive, and versatile technique that has derived many unique imaging modalities, such as diffusion-weighted imaging, diffusion tensor imaging, susceptibility-weighted imaging, and spectroscopic imaging. PET is also versatile, as it may use different radiotracers to target different molecules or to trace different biologic pathways of the receptors in the body. Therefore, these individual imaging modalities (the use of one imaging modality), with their characteristics in signal sources, energy levels, spatial resolutions, and temporal resolutions, provide complementary information on anatomical structure, pathophysiology, metabolism, structural connectivity, functional connectivity, etc. Over the past decades, everlasting efforts have been made in developing individual modalities and improving their technical performance. Directions of improvements include data acquisition and data processing aspects to increase spatial and/or temporal resolutions, improve signal-to-noise ratio and contrast to noise ratio, and reduce scan time. On application aspects, individual modalities have been widely used to meet clinical and scientific challenges. At the same time, technical developments and biomedical applications of the concert, integrated use of multiple neuroimaging modalities are trending up in both research and clinical institutions. The driving force of this trend is twofold. First, all individual modalities have their limitations. For example, some lesions in MS can appear normal in T1-weighted or T2-weighted MR images but show pathological changes in DWI or SWI images [1] . Second, a disease, disorder, or lesion may manifest itself in different forms, symptoms, or etiology; or on the other hand, different diseases may share some common symptoms or appearances [2, 3] . Therefore, an individual image modality may not be able to reveal a complete picture of the disease; and multimodal imaging modality (the use of multiple imaging modalities) may lead to a more comprehensive understanding, identify factors, and develop biomarkers of the disease.

In the narrow sense, a multimodal imaging study would mean the use of multiple imaging devices such as PET and MRI scanners, different imaging modes such as structural MRI, diffusion-weighted imaging, and magnetic resonance spectroscopy, or even different contract mechanism such as with or without contract agents in a single examination or experiment of a subject. This practice has been widely used in clinical diagnosis and medical research. For example, a routine protocol of MRI examination of a stroke patient may include T1-weighted, T1-weighted high-resolution structural MRI scans, diffusion-weighted imaging, SWI, etc [4, 5] . A protocol of an MRI study of a psychiatric disorder may contain a combination of structural MRI, functional MRI, MR spectroscopic imaging, etc [6, 7] .

In the broad sense, a multimodal imaging study may mean the use of multimodal imaging data obtained separately, from different subjects, and/or from different clinical or research sites. This practice offers the advantages of large and diverse datasets. However, it also comes with challenges of sophisticated models, complicated data normalization (that includes correction of errors and variations imbedded in data from different institutions), data fusion, and data integration [8, 9] .

In recent years, the quantity of peer-reviewed journal articles on neuroimaging has been increasing steadily. A database (PubMed) query using the keywords in titles of "neuroimaging" OR "brain imaging" returned more than 39 ,000 articles from 2010 to the present time when this paper was drafted in Feb 2020 ( Figure 1 ). These publications include not only applications of multimodal neuroimaging in clinical examinations and biomedical research but also methodological studies in imaging processing and fusion of multimodal neuroimaging. Therefore, this present paper will focus on the following two main aspects: (1) we will review some of the recent, typical papers that exhibit the strength and limitations of the neuroimaging modalities and the corresponding analysis methods, and in particular, the needs for improved image fusion methods and (2) we will review recent methodological development in data preprocessing and data fusion in multimodal neuroimaging. We note that although we tried to cover all neuroimaging modalities, we inevitably paid more attention to MRI modalities. This is not only due to the most practical application and versatility of the MRI but also due to the limitations of our expertise. Figure 2 shows the taxonomy of this review.

The main contents of the paper are organized as follows. Chapter 2 will give a brief introduction to neuroimaging, and challenges of multimodal imaging; Chapter 3 introduces the commonly used neuroimaging modalities, which include computerized tomography, positron emission tomography, single-proton emission computed tomography, and magnetic resonance imaging, which has many modalities in its own right. For each modality, we will concisely describe its signal source, energy level, spatial resolution, temporal resolution, and major applications; Chapter 4 describe applications of neuroimaging in three major areas: the developing brains, the degenerative brains, and mental disorders. In each part, we will first briefly describe what the clinical and/or biomedical problems are, we then review recent papers on how neuroimaging has been used to address these problems, and we point out what the unmet needs and challenges; Chapters 5 to 9 are devoted to the multimodal neuroimaging fusion, covering some important procedures in data fusion. The topics are not necessarily complete and their order of presentation is not necessarily coherent with the pipeline of fusion processing. Chapter 5 reviews the fundamental methods, which covers types, rules, atlas-based segmentation, decomposition, reconstruction, and quantification; Chapter 6 reviews subjective and objective assessment of data fusion in multimodal neuroimaging; Chapter 7 reviews the advantages of data fusion in improving the spatial/temporal resolution, distortion correction, and contrast; it also reviews the benefits of these advantages in fusing structural and functional images; Chapter 8 reviews atlas-based segmentations in multimodal imaging fusion; Chapter 9 reviews the quantification in multimodal neuroimaging fusion. While the focus of this part is given to PET and SPECT, some of the approaches and principles discussed here, such as partial volume correction and attenuation (relaxation), can be applied to quantitative MRI modalities, such as DTI, ASL, quantitative susceptibility mapping (QSM), etc. Chapter 10 concludes the paper. Figure 1 Numbers of peer-reviewed papers with the keywords of "neuroimaging" or "brain imaging" in titles (the numbers and the bar graph were generated by PubMed in Feb 2020). 

In this part, we will review the current challenges of neuroimaging, including limited spatial/temporal resolution, lack of quantification, and imaging distortions. These challenges often create fundamental limitations on individual modalities of neuroimaging, while some challenges also exist in current multi-modal neuroimaging. This part will mainly cover the challenges of individual neuroimaging modalities that led to the development and ongoing research of multimodal neuroimaging methods.

Neuroimaging can be divided into structural imaging and functional imaging according to the imaging mode.

Structural imaging is used to show the structure of the brain to aid the diagnosis of some brain diseases, such as brain tumors or brain trauma. Functional imaging is used to show how the brain metabolizes while carrying out certain tasks, including sensory, motor, and cognitive functions. Functional imaging is mainly used in neuroscience and psychological research, but it is gradually becoming a new way of clinical-neurological diagnosis [10] .

The amount of information obtainable through single-mode imaging is limited and often cannot reflect the complex specificity of organisms. For instance, although CT imaging is effective in identifying normal structures and abnormal diseased tissues according to their density and thus can provide clear anatomical structure information, it cannot image soft tissue well. Generally speaking, MRI imaging has good soft-tissue contrast resolution for most sequences, but its display of bone structure is relatively poor. PET imaging and SPECT imaging are not limited by the detection depth, with high imaging sensitivity, and are easy to quantify, but their spatial resolutions are low [11] .

Optical imaging refers to the detection of fluorescence or bioluminescent dyes using the principle of light emission.

This technique has high sensitivity, no radioactivity, good specificity, and low cost. Optical imaging allows dynamical monitoring of the replication process of virus bacteria in organisms. However, it has low spatial resolution and limited imaging depth [12, 13] . Therefore, it can be observed that various imaging technologies all have both benefits and drawbacks, as shown in Figure 3 , and it is difficult to provide comprehensive and accurate information through utilizing individual modality imaging. The nowadays most commonly used noninvasive functional imaging methods and their spatial and temporal range are illustrated in Figure 4 . It can be distinctly observed that among these most advanced methods, functional MRI (fMRI) reaches the highest range of spatial resolution. fMRI can assess the whole brain and image the Microscopic Level hemodynamic processes at the layered and columnar levels of the human cortex, under the condition of a high-intensity magnetic field (i.e., submillimeter level) [14] . However, it has a relatively lower temporal resolution in terms of imaging the neuronal population dynamics. Electroencephalogram (EEG) and Magnetoencephalography (MEG) can both measure electromagnetic changes in the scale of milliseconds. However, their spatial resolution/uncertainty is more than several millimeters [15, 16] . The microscopic level of neuroscience is often beyond the reach of noninvasive imaging techniques due to the requirement of high spatial or temporal resolution.

The majority of imaging modalities are non-quantitative and have to gain complement information from other data. This additional information allows the normalization of signals, acquirement of absolute units, and inter-subject comparison. As an example, the fMRI signal is a measure of neuronal activity incited hemodynamic changes caused by a combination of complex physical and physiological processes. In different subjects or brain areas, the same level of neuronal activity can evoke different corresponding fMRI signals. As a consequence, fMRI signals can only be considered as roughly proportional to the activity of neurons. Four years later, in 2008, the studies from Ances have found that the cerebral blood flow (CBF) is relevant to fMRI signal variations in individual's brain regions, patients' age groups, and health conditions. This broad relevance brings fMRI signal high sensitivity [17] . Various approaches have been proposed to explain the ensuing sensitivity differences. Amongst these approaches, the so-called calibrated BOLD approach proposed and improved by Blockley, Chiarelli and Hoge over the years has been the most widely used [18] [19] [20] . However, the more reliable absolute quantitative results of CBF with improved spatial and temporal resolutions are provided by the Arterial spin labeling (ASL) technique by the UMICH fMRI lab. [21] 

Some neuroimaging modalities are prone to geometric distortions. Echo-planar imaging (EPI) is a fast imaging approach that could obtain the completespace data set just in a single acquisition. Due to its unmatched acquisition speed, it has revolutionized the field of neuroimaging and has served as the standard readout module for most fMRI and dMRI acquisition. Nevertheless, EPI suffers from distortion and intensity loss mainly caused by field inhomogeneities, leading to relatively poor image quality [22] . In general, the imperfection of equipment may result in information loss, noise amplification and artifacts, resulting in the distortion of images.

To conclude, in practical application, utilizing individual modality imaging often has limitations, such as low sensitivity and specificity, low spatial/temporal/contrast resolution, distortion and so on. Because of these deficiencies, we need to introduce the usage of multimodal neuroimaging to eliminate those shortcomings in some degree.

Positron Emission Tomography, shortened as PET, is a combination of nuclear medicine and biochemical analysis that is mostly used for the diagnosis of brain or heart conditions and cancer. Instead of detecting the amount of a radioactive substance existing in body tissues of a specific location to check the tissue's function, PET detects the biochemical changes within body tissues. The biochemical changes can reveal the onset of a disease process before other imaging processes can visualize the anatomical changes related to the disease. During PET studies, only a tiny amount of radioactive substance is needed for the examination of targeted tissues. PET scans not only can be used to detect the presence of disease or other conditions of organs or tissues but can also be used to evaluate the function of organs, like the heart or the brain. The most common application of PET scan is cancer detection and treatment. Figure 6 (a) shows a PET image of the brain, and Figure 6 (b) shows a PET scan of the kidney [27] .

(a) brain scan via PET scan (b) Kidney scan via PET scan (c) brain scan via SPECT (d) brain scan via MRI Figure 6 Common scan modalities

Single-photon emission computed tomography, commonly known as SPECT, uses gamma rays as the tracer to detect blood flow in organs or tissue. Therefore, a gamma-emitting radioisotope, such as isotope gallium, should be injected into the bloodstream of the patient for SPECT. The computer collects the gamma rays from the tracer and shows it on the CT cross-section. It bears similarity with the traditional nuclear medicine planar imaging but provides 3D information as multiple images of cross-sectional slices through the patient. The information can be manipulated or reformatted freely according to diagnostic or research requirements. Besides detecting blood flow, SPECT scanning is also applied for the presurgical evaluation of medically controlled seizures. Figure 6 (c) [27] shows a SPECT scan of the brain

Magnetic resonance imaging (MRI) utilizes strong magnetic field, magnetic field gradients, and radio waves to generate pictures of the anatomy and the physiological processes of the body [28] . Different from PET or CT, MRI does not need the injection of ionizing radioisotopes or involve X-rays. As all radiation instances can cause ionization that leads to cancer, MRI, without exposing the body to radiation, becomes a better choice than CT and one of the safest medical procedures. MRI is widely used in hospitals and clinics for the medical diagnosis of different body regions, including the brain, spinal cord, bones and joints, breasts, heart and blood vessels, and other internal organs, such as the liver, womb or prostate gland [29] [30] [31] . Besides, MRI can also be used for non-living objects [32] . Figure 6 (d) shows an MRI brain image [33] .

T1, T2 and Proton density (PD) are three basic types of MRI imaging. T1, T2 and PD, which all vary with sequence parameters, can simultaneously determine the contrast of the MR images [34] . Via selecting different pulse sequences with different timings, we can decide the contrast in the region being imaged. There are also other types of sequences, such as fluid attenuates inversion recovery (FLAIR) and short tau inversion recovery (STIR). In this section, we will only mention the three main types. Figure 7 shows the relationship between and , of which refers to plane [35] . Different from T1 and T2, which mainly focuses on the magnetic characteristics of the hydrogen nuclei, PD is more related to the number of nuclei in the region being imaged. PD weighted images are obtained by a short echo time and a long repetition time, which can provide a more apparent distinction between the gray matter and white matter. PD weighted imaging is specifically useful for detecting joint disease and injury. Figure 8 (a) and (b) show a T1 weighted brain image and T2 weighted brain image, respectively [27] .

(a) T1 weighted brain image (b) T2 weighted brain image Figure 8 Two types of weighted images of the same brain

Functional magnetic resonance imaging or functional MRI (fMRI) measures brain activity by detecting changes associated with blood flow. As it has been proved that when an area of the brain is in use, the blood flow to that area increases, which means that the neuronal activation and cerebral blood flow are matched. fMRI is a particular type of imaging technology used to map the neuron activities in the spinal cord and brain of humans or animals by visualizing the change in blood flow, which is related to the energy use by brain cells. fMRI also includes resting-state fMRI [36] or task-less fMRI, which can provide subjects' baseline BOLD service [37] .

Diffusion Weighted/Tensor Imaging (DWI) generates image contrast from the differences in the magnitude of diffusion of water molecules within the brain. Diffusion in biology is defined as the passive movement of molecules from a higher concentration region to a lower concentration region, which is also known as Brownian motion [38] .

Diffusion within the brain is affected by many factors, such as temperature, type of molecule under investigation, and the microenvironmental architecture in which the diffusion takes place. Based on the MRI sequences of which diffusion is sensitive to, the image contrast can be generated according to the difference in diffusion rates. DWI is highly effective for the early diagnosis of ischemic tissue injury, even before the pathology can be shown by the traditional MR sequence. Therefore, DWI provides the time window for tissue salvaging interventions.

Perfusion weight imaging (PWI) is defined as a variety of MRI techniques that are able to provide insights into the perfusion of tissues by blood [39] . PWI can be used for the evaluation of ischaemic conditions, neoplasms, and neurodegenerative diseases. Perfusion MRI mainly has three main techniques: Dynamic susceptibility contrast (DSC), Dynamic contrast-enhanced (DCE), and Arterial spin labeling (ASL).

Susceptibility weighted imaging (SWI), previously known as BOLD venographic imaging, is a type of MRI sequence that is extremely sensitive to venous blood, hemorrhage, and iron storage. As an fMRI technique, SWI can explore the susceptibility differences between tissues and detect differences based on the phase image. An enhanced contrast magnitude image can be obtained by combing the magnitude and phase data. SWI is commonly used in traumatic brain injuries (TBI) and high-resolution brain venographies as it is sensitive to venous blood.

Magnetic resonance fingerprinting (MRF) [40] is new MRI technique that integrates MR physics theory and computer pattern recognition technology and realizes fast and multi-parameter parallel quantization imaging. The technique consists of three modules. First, the fingerprinting signals are excited and acquired from the subject in the MR scanner by the pseudorandom temporal varied pulse sequence to reflect the physiological property of tissue.

Second, the evolution of fingerprinting signals with different physiological parameter combinations are predicted by the computer simulation using the Bloch equation; and a fingerprint dictionary indexed by the quantized parameters is constructed. Finally, the pattern recognition technology is applied to find the matched fingerprinting entries for the measured fingerprinting signals, so as to obtain the corresponding quantization parameters and realize quantization MR imaging. Different from most of the conventional MRI modalities, which provide qualitative contrast-based images that are determined not only by the tissue properties but also by experimental conditions, MRF provides quantitative images of tissue properties that reflects pathological conditions of the subject. Figure 9 shows Currently, the applications of MRF have been limited to biomedical research and the fusion of MRF with other neuroimaging modalities has not been reported. Given its parametric and quantitative features, the MRF technique will play an important role not only in neuroimaging but also in fusion of multimodal neuroimaging. Figure 9 Digital phantom experiments of conventional T1 weighted and T2 weighted MRIs, and MRF Table 1 lists the main advantages, disadvantages and applications of each neuroimaging technology. 

In this section, we listed some public databases, as shown in 

In this part, we will review recent advancements in the application of multimodal neuroimaging in some clinical and research areas such as early brain development, neurodegenerative diseases, psychiatric disorders, and neurological diseases. It is not our intention to cover all aspects or provide a complete review of these areas. Instead, we focus on the aspects related to the development and applications of the multimodal neuroimaging techniques that meet the expectations and challenges of biomedicine. As such, each of the areas will begin with a brief description of background information such as clinical features, pathology, diagnosis, treatment of the diseases; then a general introduction of the roles, applications, and current status of the medical imaging techniques to the disease; the major part will be a review of recent papers that used one or more imaging modalities and used image fusion in multiple imaging modalities.

Recent studies show that the human brain experiences a rapid development in the first eight years and continues to develop and change into adulthood. During this long period, the brain develops in size, neuroanatomy, and functions. This period is significant for a person's physical and mental health, intellectual and emotional development, and learning, working, and life success [41] [42] [43] .

Many factors have influences on the brain development of young children, which will have an impact on cognitive abilities and mental health in later life. These influencing factors include genes, maternal stress, and drug abuse, exposure to toxic environments, infectious diseases, socioeconomic status of the family, etc. Approximately one-third of genes in the human genome are expressed primarily in the brain and will affect brain development.

Many psychiatric and mental disorders, such as autism, ADHD, bipolar, and schizophrenia, are highly heritable or have genetic risk factors. Maternal stress and drug abuse are associated with preterm birth and low birth weight and increased risk of neurodevelopmental disorders and mental disorders in children [44] . The nutritional status of a child, which is affected by the socioeconomic status of the family, has a significant impact on neurocognitive development [45] .

Neuroimaging techniques have been used to study normal and/or abnormal development of the brain, enhancing our understanding of neuroanatomy, connectivity, and functionality of the brain. These techniques also reveal the etiological associations of abnormal brain development with risk factors and contribute to the development of intervention procedures for diseased children [42] . Young children are more sensitive to radiation than adults are, so the use of PET and CT is limited. Thanks to the in vivo nature and versatility of MRI, not only young children but also newborn babies can be imaged, offering the opportunity to study white matter development and cognition in babies [46] [47] [48] . MRI has become the most important pediatric neuroimaging modality and has been widely used to study normal and abnormal brain development, allowing repeated longitudinal observation of the changes of brains of the same individuals before and after birth [49] . In the following, our review will focus on the major MRI modalities in pediatric imaging, which include structural, functional, and diffusion tensor imaging.

Early pediatric brain MRI studies focused on the anatomical aspects using T1-weighted and T2-weighted

images. Qualitative studies provided information about changing patterns of gray matter and white matter differentiation and myelination in the first months of birth [50] and early childhood [51] . Quantitative studies also revealed the changes in water contents, T1, and T2 relaxation times in both gray matter and white matter; age-related changes in gray matter, white matter, and CSF volumes. All these reflect ongoing maturation and remodeling of the central nervous system [52, 53] .

Compared with adult cohorts, brain MR imaging of young children is challenging because of several factors.

Young children are less cooperative than adults with scanning procedures, which can be long, noisy, and uncomfortable when lying still for long; the images are often plagued with motion artifacts. The brain changes rapidly with age in early life after birth; the brain is not well myelinated; the contrast between gray matter and white matter is low. These pose difficulties to the optimized parameters for data acquisition protocol and also the standard parameters or criteria for the postprocessing procedures, such as the segmentation of the brain to determine cortical thickness [54] . As a result, the physical properties, such as relaxation times, water content, diffusion coefficients, of the developing brain are not very well characterized. Other technical challenges exist to scan young children [55] .

Knowledge of the variations of biophysical properties, such as T1 and T2 relaxation times, water contents in GM and WM during the early life of children, is of critical importance to the understanding of neurodevelopment of young children and also to the development of diagnostic protocols of abnormal brain development. The measurement of these biophysical properties is challenging due to the prolonged scan time. Recent technical development of magnetic resonance fingerprinting (MRF) allows rapid and quantitative analysis of multiple tissue properties [40] . For example, MRF can provide T1, T2, and proton density maps of the brain in contrast to the conventional T1-weighted, T2-weighted, or proton density-weighted images. A recent paper reported an application of the MRF to study the T1, T2, and MWF of children aged from 0 to 5 years old [56] . This study was able to record different patterns of variations of tissue biophysical parameters over different age stages. MRF techniques were also used to parametrically characterize brain tumors in children and young adults [57] . In a broad sense, the parametric information in MRF opened doors to studies of the correlations between brain tissue properties and brain development, impairment, and physiopathology. Techniques of image fusion can play an essential role in the processing, interpretation, and application of MRF data.

Degenerative brain diseases are caused by the decline of neuronal function and the reduction of numbers of neurons in the central nervous system (CNS). Known degenerative brain diseases include mild cognition impairment, Alzheimer's disease (AD), Parkinson's disease, etc. The patients of these diseases suffer from losses of functions in memory, speech, movement, etc. Most of these diseases (except for some mild cognitive impairment subtypes) are progressive, i.e. the symptoms deteriorate as the brains age. As the population is rapidly aging, degenerative brain diseases post enormous impacts on individuals, families, and society. The etiology of these diseases is still unknown, and there is currently no cure. In the following sessions, we will review advances of neuroimaging on MCI, ADs, and PDs.

Mild cognitive impairment (MCI) is a clinical transition between normal aging and dementia or Alzheimer's disease (AD), in which individuals have memory or other cognitive impairments beyond their age, but not to the extent of dementia. Patients with MCI often only have minor difficulties in functional ability.

In studies based on people older than 65 years of age, the incidence of MCI is estimated to be at 10-20% [58] , and the Mayo clinical study on aging shows an 11.1% incidence of amnestic MCI (aMCI) and 4.9% incidence of non-amnestic MCI (naMCI) in undiagnosed patients aged 70-89 years [59] . Several longitudinal studies have shown that most MCI patients have a significantly higher risk of developing to dementia compared to the general U.S.

population (1-2%/ year) [60] , the community population (5-15%/ year), and the clinical patients (10-15%/ year) [61] [62] [63] . The latter data suggest that cognitive impairment tends to develop more rapidly for the patients that display serious symptoms. Although some studies have shown that the incidence of MCI reversals to normal cognitive function is as high as 25-30%, recent studies suggest that the incidence may be lower. In addition, cognitive reversals over a short period of follow-up study showed that they did not prevent subsequent disease progression.

Magnetic resonance imaging Magnetic resonance imaging (MRI) techniques have been used in the clinical identification of MCI and various types of dementia to predict the progression of MCI to dementia. For MRI measurements of brain structure, linear, area, or volume measurements can be used. The results showed that the area of MCI brain atrophy was consistent with AD, but to a lesser extent, between the normal elderly (control group) and

AD patients [64] [65] [66] [67] . Similar results were found using voxel-based measurement and analysis, with abnormal changes in not only gray matter but also white matter [68, 69] . The previous diagnosis of AD by structural MRI was mainly based on the degree of brain atrophy, especially in the medial temporal lobe. The structural MRI studies showed the atrophy along the hippocampal pathway (entorhinal cortex, hippocampus and posterior cingulate cortex), which was consistent with the loss of early memory. As the disease progresses, the temporal, frontal, and apical lobes shrink with neuronal loss, causing abnormalities in language, practice, vision, and behavior [70, 71] . However, no definitive biomarkers have been identified by structural MRI alone to distinguish MCI and AD, to stage MCI, and to predict MCI conversion to AD or not [72, 73] .

MRI-based functional imaging has been applied to the understanding of and to the discrimination between AD and MCI. These techniques include perfusion-weighted imaging (PWI), diffusion-weighted imaging (DWI), diffusion tensor imaging (DTI), and blood oxygen-dependent fMRI (including task execution and resting state) [72] .

Functional MRI allows the delineation of microstructural brain changes, which is complementary to structural MRI that can depict the global changes of the brain in MCI. An MRI-based functional imaging study that employed PWI, DTI and proton MRS showed significant abnormalities in parameters derived from the three imaging modalities for AD patients. PWI and DTI parameters showed a significant, but a lower degree of abnormalities in some areas for MCI patients. fMRI has also been used to distinguish AD and MCI and to predict the transition from cognitive normal to MCI and from MCI to AD. Recent studies show that BOLD-fMRI can detect changes in brain function before MCI progresses to AD, making it an important technique to study the neural mechanism of MCI [74, 75] .

Proton magnetic resonance spectroscopy ( 1 HMRS) is a noninvasive imaging method that can detect biochemical and metabolic changes in brain tissue in vivo and conduct quantitative analysis. Early MRS studies show abnormal concentrations of N-acetylaspartate (NAA), creatine, and choline are associated with the status of memory and cognition impairment and have a promise for assessing cognitive status, evaluating response to medicine, and monitoring progression during treatment [76] [77] [78] . In recent years, with advances in the technical development of MR hardware and pulse sequences, the roles of glutamate, the excitatory neurotransmitter, and GABA, the inhibitory neurotransmitter, in MCI patients became the main focus [79] [80] [81] . For example, with ultra-high field 7 Tesla MR scanner, abnormal concentrations of GABA, glutamate, NAA, glutathione, and myo-inositol (mI) in different brain regions were detected [82] . The manifestations of 1 HMRS in MCI patients were mainly shown in decreased NAA/Cr ratio and increased mI/Cr ratio. The pathological results showed neuronal deletion and glial proliferation, and the changes in metabolite concentration were consistent with the pathological results [82, 83] .

Multimodal imaging PET and SPECT provide insight into blood perfusion and metabolism in tissues and organs, as well as explore changes in function. The nuclear medical images of aMCI patients showed decreased perfusion and metabolism in the hippocampus, temporoparietal lobe, and posterior cingulate gyrus. Studies using PET, SPECT, and MRI have shown that glucose metabolism in the hippocampus, glucose metabolism rate in the bilateral temporal-parietal lobe, and blood perfusion in patients with aMCI are lower than those in normal elderly.

These studies have also shown that low glucose metabolism in the temporal-parietal lobe is a reliable indicator of conversion to AD [84] [85] [86] . Excessive deposition of β-amyloid peptide in the brain and the cascade reaction caused by it are the early onset of AD. Therefore, early detection of β-amyloid peptide in the brain can help identify patients with aMCI, and monitor the progression of the disease and treatment effect. It was found that the 11 C-PiB-PET could attach to Aβ in the brain. PET imaging showed the amount and location of Aβ deposition in the brain, which was expected to be an early diagnostic method for AD [87] [88] [89] .

Multimodal imaging techniques involving MRI-based imaging and PET-based imaging have been frequently used for prediction, characterization, and classification of MCI [90, 91] . In facilitating these complex tasks, imaging fusion methods based on artificial intelligence, neural network, deep learning and graph theory have been used [92] [93] [94] . Brain network studies based on multimodal MRI and graph theory analysis have found that the topological properties of AD and aMCI affected brain networks have undergone abnormal changes, which mainly manifested as the imbalance between functional differentiation and integration. This approach provided a new way to reveal topological mechanisms and pathophysiological mechanisms of brain networks [93, 95] . In addition, the combination of graph theory analysis and classification analysis suggests that the brain network topology attribute can be used as an imaging marker of AD and has a good clinical application prospect.

Alzheimer's disease (AD) is a neurodegenerative disorder and the most common cause of dementia. AD is characterized by progressive memory loss, aphasia, loss of use, loss of recognition, impairment of visual-spatial skills, executive dysfunction, and personality and behavior changes [96, 97] . It has become one of the major diseases that seriously threaten the health and quality of life of the elderly [98] . The onset of AD is slow or insidious, with patients and their families often unable to tell when it starts. It is more common in the elderly over the age of 70 (the average male is 73, and the average female is 75 years old), with more females than males (female to male ratio of 3:1) [99] .

There is currently no cure for AD, but large numbers of novel compounds are currently under development that have the potential to modify the course of the disease and to assess the efficacy of these proposed treatments. There is a pressing need for imaging biomarkers to improve understanding of the disease and to assess the efficacy of these proposed treatments.

Magnetic resonance imaging Structural MRI (sMRI) is the most widely used imaging modality for the study of AD. The techniques for analyzing sMRI are classified into volume-based and surface-based methods [100] .

Previous studies have shown that hippocampal volume atrophy and whole-brain atrophy independently predicted the progression of AD [101] . Hippocampal damage or atrophy occurs in the early stage of AD, which is an important structural basis for the clinical manifestations of AD. Although global hippocampal atrophy in AD was well accepted, the differences were often detected large sample-size studies [102] .

De Winter et al. studied 48 elderly AD patients with depression and 52 healthy control elderly people and examined all the subjects with sMRI and neuropsychology [103] . They found that there was no significant difference in the positive rate of Aβ between the depression group and the healthy control group. However, the hippocampal volume in the depression group was significantly smaller than that in the healthy control group. There is significant hippocampal atrophy in elderly depression patients, and hippocampal atrophy has nothing to do with Aβ, which challenges the reliability of hippocampal atrophy in the clinical diagnosis of AD. It is suggested that hippocampal atrophy not only occurs in AD but also in senile depression. The study of sMRI indicates that the brain atrophy shown by brain morphology and structure has reference value for the diagnosis of AD. However, the diagnosis of AD still needs to be confirmed by combining clinical manifestations, neuropsychological assessments, and other examination methods. It also indicates that follow-up is needed for suspected depression in patients with AD. The above studies showed the limitations of structural MRI and the necessity of the multimodal approach in the study of AD [104] .

Other MRI modalities, including functional MRI, DWI, PWI, have also been widely used in the study of neurodegenerative diseases. We will review recent advances of the resting-state functional magnetic resonance imaging (rs-fMRI) as an example. As opposed to the conventional task-based fMRI, rs-fMRI does not require the subject to perform any task or be subjected to any external excitation. The rs-fMRI captures the low-frequency oscillations signals that are related to the spontaneous neural activity of the brain by analyzing the brain blood oxygen level dependent (BOLD) signal. Sophisticated methods of analysis of the rs-fMRI data depict the functional connectivity of the brain. The rs-fMRI has been used to reveal how the networks of the functional connectivity are correlated to the brain functions of individuals with cognitive impairment. Zamboni et al. found that the recognition task of AD patients was related to the increased activation of the lateral prefrontal area, which also overlapped with the functional connection enhancement area indicated by the rs-fMRI [105] . Zhou et al. predicted the pathological changes of AD by using the calculation model of resting brain function network and studied five different brain regions vulnerable to neurodegenerative diseases through the use of task state fMRI [106] . They found that the brain network of AD patients may have the phenomenon of weak functional connectivity and their ability to transmit information of functional brain network decline. Wang et al. found that the functional brain network of MCI patients had different degrees of functional connectivity disorder. The evaluation of overall functional brain connectivity of patients plays an important role in the early diagnosis and treatment of AD [107] . Abnormal brain connectivity can be a biomarker of the disease.

Many neuropsychiatric diseases and dementia can change the default mode network (DMN) of the brain.

Identification of the change in the connectivity of DMN is constructive for the early recognition of AD. Jin et al.

collected 8 patients with aMCI and 8 healthy people to analyze rs-fMRI data by independent component analysis (ICA) [108] . They found that the functional activities of the lateral prefrontal cortex, left medial temporal lobe, left middle temporal gyrus and right angular gyrus in aMCI patients decreased, while the activity of the middle and medial prefrontal cortex and the left parietal cortex increased. Further studies found that the functional activities of the left lateral prefrontal cortex, left middle temporal gyrus and right angular gyrus were positively correlated with memory, especially delayed memory [109] . Although there was no significant difference between the two groups in the degree of medial temporal lobe atrophy, the functional activities of the left medial temporal lobe decreased. This decrease suggests that the functional changes of DMN may occur in the early stage of AD, i.e. aMCI, and the functional changes may occur before the obvious change of brain structure.

Multimodal imaging Due to severe overlap in symptoms and findings of individual imaging modalities of the neurodegenerative diseases, it is difficult to identify the biomarkers that could be used to differentiate the types of these diseases and/or to stage the progress of a disease. Therefore, multimodal neuroimaging techniques are used to overcome the challenges [110] . As pointed out in [111] , individual modalities of MRI and EEG lack precision in AD diagnosis and staging. By employing both imaging modalities, with the MRI measuring the cortical thickness and the EEG measuring the rhythmic activities, the authors found joint markers that identified the subjects of Alzheimer's disease with an accuracy of 84.7%, a significant increase from those of individual modalities. While some studies of multimodal imaging confirmed correlations of findings among individual modalities as in a study of sMRI and fMRI [112] , multimodal imaging studies can also be used to dissociate the tau deposition and brain atrophy in early ADs using PET and MRI. The study found that the tau load had little effect on the gray matter atrophy, and this might imply that tau protein deposit precedes and predicts brain atrophy. The multimodal imaging studies require statistical and analytical models, advanced computing algorithms, and especially, novel data fusion methods [113] [114] [115] [116] , which will be reviewed in detail in the following sections.

Parkinson's disease (PD) is a chronic progressive degenerative disease of the central nervous system, which is commonly seen in elderly patients. Typical clinical manifestations of PD include static tremor, myotonia, bradykinesia, and abnormal posture and pace [117] . With the continuous increase of the aging population, the incidence and disability rates of the population are also increasing year by year. The results of epidemiological surveys indicate that the prevalence of PD in people over 65 is about 1.7%, and the prevalence of PD in people over 80 is as high as 4% [118] [119] [120] . PD is more and more harmful to the health of the middle-aged and elderly, especially involving the central nervous system.

Due to the lack of objective basis and diagnostic criteria for the diagnosis of PD, the previous clinical diagnosis of PD was mainly based on the clinical symptoms, resulting in a low coincidence rate between the clinical diagnosis and pathology of PD and in a significant lag behind the pathological changes of brain microstructure. With the increasingly standardized diagnosis and treatment of PD, neuroimaging examination has become an indispensable part of the diagnosis. This differential diagnosis of PD can help identify different movement disorders, locate anatomical dysfunction sites, and determine the causes of the lesion, which will improve clinical evaluation and prognosis [121, 122] .

Magnetic resonance imaging Structural cranial MRI can distinguish white matter from gray matter by setting different imaging parameters while avoiding radiation. It is better than cranial CT in revealing white matter lesions, small infarcts, subacute intracerebral hemorrhage, and lesions in the brain stem, subcortical regions, and posterior fossa. On structural MRI such as T1, T2-weighted, and fluid-attenuated inversion recovery (FLAIR) images, PD patients usually exhibit broadening of the ventricles (caused by extrapyramidal atrophy) and widened sulci (diffuse brain cortical atrophy) [123] . The quantitative measurements of cortical atrophy can be measured based on voxel morphometric assessment. When the compact belt of the substantia nigra shrinks and the short T2 signal of the substantia nigra disappears, the width of the dense belt of the substantia nigra, the ratio of the width of the dense belt of the substantia nigra to the diameter of the midbrain, the caudex nucleus, the putamen nucleus, the thalamus and other areas of interest are measured. In evaluating the extent of atrophy, physiological changes such as age increase

and relevant clinical supporting evidence should be taken into account [124] .

Neuromelanin-sensitive MRI is used to detect neuromelanin, a surrogate biomarker for the PD. Neuromelanin is a dark pigment found in neurons in the substantia nigra pars compacta. The concentration of neuromelanin increases with age but is found to be around 50% higher in PD patients compared with age-matched non-PD subjects, due to the death of cells in the substantia nigra. Neuromelanin-sensitive MRI allows the visualization of the neuromelanin-containing neurons in the substantia nigra, pars compacta. With the use of morphological analysis and signal intensity (contrast to noise ratio), the width and CNR of the lateral and central substantia nigra were found to be significantly lower in the PD subjects than in the control group and untreated essential tremor (ET) group [125, 126] . Therefore, this imaging technique can be potentially used as a biomarker to differentiate ET from the de novo tremor-dominant PD subtype. The neuromelanin levels were quantitatively assessed using neuromelanin-sensitive MRI and quantitative susceptibility mapping (QSM) [127] , an MRI modality for measuring the absolute concentrations of iron, calcium, and other substances in tissues based on changes of local susceptibility [128] . While the neuromelanin imaging found significantly lower neuromelanin levels in the PD group than the health controls (HC), which is in agreement with the neuromelanin MRI only study, the QSM values were significantly higher in the PS group than in the HC group. This result suggested the usefulness of QSM in detecting PD [127] .

Resting functional magnetic resonance imaging (fMRI) is a technique that collects the blood oxygen level dependent signal changes of patients in the awake and resting states to obtain the functional activity level of the brain in the baseline state. In recent years, fMRI has been widely used in clinical studies of all motor disorders or neurodegenerative diseases, including PD [129] [130] [131] . Resting-state functional MRI (rs-fMRI) can calculate a variety of brain activity attributes, such as local consistency, range of low-frequency fluctuation, and amplitude of low-frequency fluctuation etc. By observing the correlation between time-dependent signals of blood oxygen levels in different voxels or areas of interest, we can further evaluate the synchronization of functional activity in different brain areas, i.e. functional connectivity [129, 132, 133] . In recent years, calculation methods based on independent component analysis, Granger causality analysis and Graph Theory can help to find complex pattern changes in the brain network of PD patients.

Other imaging modalities and multimodal imaging Other imaging modalities, including PET, SPECT, EEG, CT, have been applied to study the functional and structural abnormalities and changes of PD patients, and they provide much complementary information to MR-based imaging modalities mentioned earlier. PET studies investigated cerebral glucose metabolism with or without medications, with or without brain stimulations [134] [135] [136] . Metabolic and brain chemical changes related to dopamine neurons in PD patients were also studied using SPECT, which cannot be assessed by other MRI modalities including proton magnetic resonance spectroscopy (MRS) [137] . By jointly applying SPECT and DTI, this study identified regions and connections of the brain that differentiate PD patients and healthy controls. Different from the imaging modalities in that study, a recent study employed PET scans with two different tracers and rs-fMRI to investigate variations of metabolism and functional connectivity of the PD patients [138] . It identified correlations between motor impairments with hypometabolism and hypoconnectivity in multiple brain regions. With the use of different modalities under similar aims, results from these studies can provide complementary information for the impaired regions. The data from them can be integrated and analyzed using data fusion like the work in [139] , in which data of anatomical MRI, rs-fMRI, and DTI were analyzed for more accurate and reliable biomarkers of PD.

Mental disorders are conditions that affect a person's thinking, mode, behavior, relationship with others, and 

Depression is a common mood disorder, which can be caused by a variety of reasons. The main clinical feature is marked with persistent depression of mood, which is incompatible with the situation. In severe cases, suicidal thoughts and behaviors may occur. Most cases tend to show recurrence; and most can be relieved each time, while some may have residual symptoms or progress to chronic depression. At least 10% of clinical depression patients also show manic episodes and should be diagnosed as bipolar disorder [150, 151] . What we commonly call depression is clinical or major depression, which affects 16% of the population at some point in their lives [152] . In addition to the severe emotional and social costs of depression, the economic costs are also enormous. According to the World Health Organization, depression has become the fourth most serious disease in the world and is expected to become the second most serious disease after coronary heart disease by 2020 [153] .

So far, the etiology and pathogenesis of depression are not clear, and there are no obvious signs or laboratory indicators of abnormality. Although there have been many basic and clinical studies on depression, no critical breakthrough has been made in the three most important clinical problems: pathogenesis, objective diagnosis, and efficient treatment. A key breakthrough in these issues is to find and establish a stable biological marker from gene to clinical phenotype and then further study its pathogenesis, establish objective diagnostic methods and develop efficient clinical therapy.

Magnetic resonance imaging Up to now, in the clinical research field of mental illness, especially depression, the most sought after biological markers may potentially be provided by the study of neuroimaging, especially brain MRI. Brain MRI examination have characteristics of good clinical applicability, non-invasive, simple operation, universal, relatively stable results and easy to repeat, but its sensitivity and specificity need to be improved. Brain MRI research has become an intermediate mechanism from molecular research to clinical phenotype. Through this mechanism research, we can not only explore how genes, molecules, and proteins affect the brain structure and function of patients with depression but also use MRI as an objective diagnostic tool for the most urgent clinical needs. Over the past 20 years, the application of multi-mode MRI technology to study the brain structural and functional characteristics of depression, especially to establish clear biological marker targets around the characteristics of emotional circuits, has become one of the major scientific frontiers in the basic and clinical research of neuroscience.

Many studies have found that patients with depression have abnormalities in brain structure and function of emotional circuits, as well as in neurotransmitters associated with these circuits [149, 154] . MRI studies in recent years found that the depressive mood is associated with three brain regions, namely in the amygdala and the ventral striatum as the primary mood areas, the orbital gyrus, medial prefrontal cortex and cingulate gyrus as the emotional auto-regulation areas, and the dorsolateral and ventrolateral prefrontal cortex as the center of the active emotional regulation area [154] [155] [156] .

Multimodal MRI techniques used in mental disorders seek to find correlated, complementary, and/or converging image features from multiple image modalities and applied sophisticated analytical methods to identify robust biomarkers for the types of depression.

A study employed DTI, magnetic resonance spectroscopy (MRS), rs-fMRI, and magnetoencephalography (MEG) and revealed patterns of abnormalities of patients with major depressive disorders. These patterns included factors in the neurotransmitters (glutamate concentration), white matter fibers (fractional anisotropy), and functional excitations (fMRI) [157] . A multimodal MRI study involves structural MRI and ASL to assess grey matter volume and regional cerebral blood flow in MDD patients. This multimodal study revealed negative correlations between the extent of depressive symptoms and CBF in the bilateral para-hippocampus and between depressive symptoms and CBF in the right middle frontal cortex [158] . In addition to confirming the correlations among findings of individual modalities, some multimodal MRI studies, however, found disrelations among individual findings in the MDD group [159] . Further multimodal data analysis involving MRI imaging data and clinical, neurobiological metrics of the patients may resolve the disparities.

Among the methods of the multimodal MRI image fusion in depressive disorders, support vector machine (SVM) [160] and linked independent component analysis [161] was recently used, respectively, to identify biomarkers for the classification and prediction of symptom loads of heterogeneous MDD cohorts. Imaging data and neurobiological data were included in the data fusion. In both studies, the results did not show strong support for the hypothesis and did not provide sufficient evidence for the sought biomarkers [160, 161] .

Obsessive-compulsive disorder (OCD) is a group of neuropsychiatric disorders with obsessive thinking and compulsive behavior as the main clinical manifestations. It is characterized by the co-existence of conscious compulsion and anti-compulsion, and the repeated intrusion of thoughts or impulses into the daily life of patients that are often meaningless and involuntary. Although patient perceives that these thoughts or impulses are their own and resist them to the utmost degree, he or she is still unable to control them. The intense conflict between the two causes the patient great anxiety and pain, which affects his other study, work, interpersonal communication, and even daily life.

Magnetic resonance imaging Voxel-based morphometry was widely used in the sMRI studies of OCD. These studies measure the structures and volumes of regions of interest in various OCD groups and healthy control groups.

OCD patients were found to have lower grey matter volumes in specific regions of the brain. For children with OCD, these regions include the bilateral frontal lobe, cingulate cortex, and temporal-parietal junction [162] . For adults with OCD, these regions are the left and right orbitofrontal cortex [67] . Lower volumes were also seen in white matter in the cingulate and occipital cortex, right frontal and parietal and left temporal regions [162] and in a small area of the parietal cortex for patients with OCD [163] . fMRI studies can provide information about the pathophysiology of OCD [164, 165] . However, whether this information from single fMRI modality alone could be of clinical value in the diagnosis of individual patients is not clear [166] . The pathophysiological feature of OCD, as revealed by fMRI studies, suggest that abnormal brain metabolites may be implied in OCDs. Abnormalities in brain metabolite concentrations in patients with OCD were investigated using proton MRS [167] [168] [169] [170] . Among about ten detectable metabolites, glutamate, glutamine, and GABA are of particular interest, as they are involved in neurotransmission. Recent studies show that OCD patients had an elevated GABA level and a higher GABA/glutamate ratio in the anterior cingulate cortex [171] , and they had lower GABA concentration in the prefrontal lobe, as compared to healthy control groups [172] . The roles of glutamate and glutamine in OCD are in the focus of research interest, but the findings lacked reasonable consistency [173] [174] [175] . The heterogeneity of structural neuroimaging findings of OCD may reflect the heterogeneity of the disease itself.

Multimodal MRI Multimodal MRI studies in OCD provide complementary, correlated and/or integrated information of findings from individual modalities [176, 177] . Early structural MRI study suggested that the volume reduction of superior temporal gyrus (STG) is associated with the pathophysiology of OCD [178] . A functional MRI study found increased low-frequency fluctuations in neural activities in STG [179] . A correlation between these findings was found in a combined structural MRI and fMRI study, which shows that the volume of the superior temporal sulcus is strongly correlated with functional connectivity between several brain regions that may form a neuro-network [177, 180] . The simultaneous 1H-MRS and DTI study, to investigate metabolic and white matter integrity alterations in OCD, found that the level of Glx to Cr ratio in the anterior cingulate cortex was higher in the OCD group than the healthy control group [181] . The study also found from DTI analysis that the FA values in the left cingulate bundle of the OCD group were significantly higher than the healthy controls. A limitation of this study is that the Glx level, which is a combination of glutamate and glutamine, was measured instead of measuring glutamate and glutamine individually. It has been recognized that it is difficult to distinguish these two structurally similar metabolites using 3 Tesla scanners and ultra-high magnetic field (e.g. 7T) scanners are required.

Schizophrenia is a group of serious psychosis with unknown etiology, which usually starts slowly or sub-acute in the young and middle-aged individuals [182] . Clinically, it often manifests as a syndrome with different symptoms, including abnormalities in sensory perception, thinking, emotion, and behavior, as well as uncoordinated mental activities [183] . Schizophrenia is a multifactor disease [184] . deterioration. Some patients eventually show recession and mental disability, but some patients can maintain recovery or basic recovery after treatment [185] .

Magnetic resonance imaging Structural MRI was widely used to study the morphology and volumetry of the brains of schizophrenia patients. Studies found that the average brain volume of schizophrenia patients was smaller than that of healthy people [186, 187] . The abnormal volume and structure of white matter usually appear before the onset of the disease, and these abnormalities tend to be stable during the development of the disease [188] ; the change of gray matter volume is more evident after the onset of the disease and decreases progressively over time [189] . According to a longitudinal study, gray matter deficiency in schizophrenia mainly occurs in the first five years [190] .

Quiet complement to the structural MRI, DTI reveals the abnormalities of white matter microstructure of schizophrenia patients [191] . Decreased fractional anisotropy in white matter tracts, different cortical regions, and subcortical regions was found in schizophrenia patients in some studies. However, controversial findings were also reported [192, 193] . These inconsistencies might be attributed, in part, to small sample sizes. A large-scale DTI study involving more 4,000 subjects found widespread white matter microstructural differences between schizophrenia patients and healthy controls [194] . Significantly reduced fractional anisotropy values were found in 20 of the 25 investigated regions within the white matter. Furthermore, significantly higher mean diffusivity and radial diffusivity were also observed in schizophrenia patients than in healthy controls.

Functional MRI techniques are used to detect the deficits in neural networks of patients with schizophrenia [195] . Brain network studies show that the functional connectivity of the default mode network (DMN) in schizophrenic patients has changed. Although the research structures are inconsistent, most studies show that DMN functional connectivity is enhanced in schizophrenia, and functional connectivity in the prefrontal cortex is weakened (especially in the prefrontal cortex) [196] . In addition, the functional connections of auditory/linguistic networks and basal nuclei are related to auditory hallucinations and delusional symptoms. The study of brain structure networks found that the number of frontal and temporal core nodes decreased and the average shortest path increased, indicating decrease in global efficiency. Wang et al constructed a network of DTI images of 79 schizophrenia patients and 96 age-matched normal subjects [197] . They found that: compared with the normal subjects, the global efficiency of the schizophrenic group decreased; the local efficiency of the core nodes distributed in the frontal cortex, the paralimbic system, the limbic system and the left putamen decreased; and the global efficiency of the network was negatively correlated with the PANSS score. Research shows that the change in brain structure network started at the beginning of the disease. More severe symptoms indicate lower the global or local efficiency of the network, and the slower the speed of information integration.

Magnetic resonance spectroscopy studies found that the brain metabolism of schizophrenia patients was abnormal [198] [199] [200] . The levels of N-acetyl-aspartic acid (NAA) in the hippocampus, frontal lobe, temporal lobe, and thalamus of schizophrenic patients decreased; the levels of NAA in the thalamus of high-risk groups also decreased, while the level of NAA in temporal lobe decreased. It was also found that the increase of glutamate level in the hippocampus and medial temporal lobe was related to the decrease of executive function.

Multimodal imaging The heterogeneities of findings of individual imaging modalities have been driving the multimodal neuroimaging approach to the search of the more consistent and precise biomarkers for the deficits, abnormalities in functions of schizophrenia patients. A concerted use of three MRI modalities, namely, resting-state of fMRI, structural MRI, and diffusion MRI, was able to simultaneously reveal abnormalities from these three kinds of MRI images and, thereby, identify the cortico-striato-thalamic circuits that might be related the cognitive impairments in schizophrenia [201] . In addition to the multimodal imaging investigation of structural and functional brain abnormalities in schizophrenia, proton MRS has also been used in combination with fMRI to investigate cognitive impairment in schizophrenia at both neurometabolic and functional levels [202] . The combined proton MRS and fMRI study are particularly useful for short-term longitudinal studies on the effects of medication, which is invariant to brain structural change. In this study [202] , the relationship between Glx/Cr levels and BOLD response significantly changed after six weeks of medication for schizophrenia patients, although factors that confound interpretations of the results remain. More examples of multimodal imaging studies on schizophrenia are given in recent review articles [203, 204] .

Multimodal fusion has gradually entered the center of research interest as an approach to tackle the challenges of neuroimaging. The first main reason is that there exists a great complementarity between different imaging modes.

For instance, the images obtained by positron emission tomography imaging (PET) or single-photon emission tomography imaging (SPECT) do not contain high resolution, three-dimensional anatomical information. On the other hand, high-resolution structural images can be obtained via the use of CT and/or MRI. These images complement each other to provide a complete picture of the targeted organs' anatomy, physiology, and pathology.

The fusion of these images is of great significance for the relevant clinical and pre-clinical studies.

Another outstanding merit of utilizing multimodal fusion is that it efficiently enhances the spatial and temporal resolution in the characterization of brain processes. In other words, multimodal imaging may allow the combination of the hyper-temporal resolution of one imaging mode with the hyper-spatial resolution of another, taking advantage of the spatial-temporal complementarity. Take a study in 2014 as an example, Ke Zhang et al have successfully measured the cerebral blood flow with a combination of Arterial Spin Labeling (ASL), MRI and PET [205] . Apart from this, utilizing EEG together with fMRI to improve spatial and temporal resolution has also been studied by many scientists in neuroscience [206] [207] [208] [209] It is worth mentioning that, in both narrow and wide senses, multimodal data fusion has a high capacity of generalization. A typical instance is the alignment of functional MRI, EEG, and fNIRS images to an anatomical coordinate system. The coordinate system can act as a template to standardize reported results. The alignment of the images to a single coordinate system not only allows comparison with other studies but also allows the combination of functional and structural information [210] .

PET/CT combination is a multimodal absolute quantification approach where CT provides structural data information on bones, which also serves as the main absorber for the γ-rays in PET. This combination allows decay correction, in which the accumulation of radioactive isotopes in human tissues becomes apparent, and the amount of radioactive decay can be absolutely quantified [211, 212] .

Multimodal imaging also has the benefit of utilizing data from one modality to improve the data quality of another modality, such as correcting the geometric distortions of EPI images by acquiring a B0 field map or obtaining EPI with different parameters [213, 214] . Another classic case is in a combined MR-PET study, where the motion information provided by high-temporal resolution MRI data was used to help the reconstruction of the PET data [10] .

The long history of neuroimaging has led to the development of an assortment of imaging technologies and modalities, as seen in Section 3. The existing research and apparatus provided a solid foundation for multimodal fusion, leading to the rapid development of many fusion techniques. In this part, we classify reviewed methodologies into four primary forms: multimodal, multi-focus, multi-temporal, and multi-view.

Modern medical imaging methods aim at revealing possible dysfunctions in patients. For example, neuroimaging methods are often used to image the structure of nervous system on a macroscopic level, which in turn helps explore the neuroglial basis of behavior and cognition of patients. However, how to combine different medical imaging methods to provide images with better quality or clearer structures remains to be the heated research interest in the field. Therefore, multimodal image fusion is proposed to minimize the gap.

In a narrow sense, multimodal image fusion, a technique to improve the interpretation of the structure and functions of target organ or region, generally combines two or even more images collected from different imaging instruments. In order to achieve the goal of simultaneous acquisition, researchers have developed particular instrumentations to allow data of one modality to be acquired with low or neglectable inference from another modality. For instance, the EEG-fMRI combination fuses data acquired from EEG instruments like amplifiers and MRI scanners. Also, the novel instrumentations range from simple arrangements to relatively complex technological innovations. However, in some combinations, simultaneous acquisition of data was impossible due to the physical interactions of the imaging devices.

Multimodal image fusion, in a broader sense, also combines data but collected with the same instrument. In this case, MRI is widely used due to its versatility in the generation of different tissue contrasts with the well-studied phenomenon of magnetic resonance. There is also research that combines two and more contrasts in the same acquisition, which has been routinely used for many years. Multiple contrasts can also be acquired by PET (Positron emission tomography) when radioactive compounds injected are different. A region-based method, which incorporated segmentation into the fusion process, was proposed by S. Li in and local focus measure comparison. It was pointed out that local features such as dense SIFT can also be used to match pixels that were misregistered between multiple source images.

In methods based on frequency, images are transformed into the frequency-domain for fusion. Wavelet-based methods, including Discrete Wavelet Transform (DWT) [221] , Haar wavelet [222] and pyramid-based methods such as the Laplacian pyramid [223] , fall under the category of frequency-domain methods. A DWT based method was proposed in [224] . Principal component analysis (PCA) was used for approximating coefficients of input images.

The principal components were evaluated to obtain multiscale coefficients. The weights for the fusion rule were then acquired by averaging those components. Besides the promising performance, the proposed method was widely applied in medical image fusion for CT and MRI images. In [225] , the authors proposed the Daubechies complex wavelet transform (DCxWT) to fuse multimodal medical image. By using the maximum selection rule, the complex wavelet coefficients of source images are fused. Source images at different levels are decomposed by DCxWT, followed by Inverse DCxWT to form the fused image. Compared to the other five methods, including Dual tree complex wavelet transform (DTCWT) based fusion and PCA based fusion, the proposed method achieved the best performance in terms of five measurements, including standard deviation, entropy, edge strength, fusion symmetry, and fusion factor. It was proved that the proposed method was robust to noises, including speckle, salt and pepper, while maintaining the property of shift-invariance.

Multi-temporal fusion is to fuse images taken at different times but of the same modality. Multi-temporal fusion enables easy detection of changes in images by subtracting one or multiple images from another. Data acquisition consists of separate recording and simultaneous recording. The choice between separate or simultaneous acquisition should be cautiously considered. Compared to separate recordings, where an image from each modality is acquired individually, simultaneous acquisitions have relatively lower data quality and more artifacts. For example, EEG-fMRI simultaneously acquires cardio-ballistic artifacts and MRI gradients. Data acquisition consists of separate recording and simultaneous recording [226] . In the MR-PET scenario, components of the MRI scanner would trigger degradation in PET scanning results. Therefore, the costs of simultaneous acquisition are higher than separate acquisition due to subject discomfort and long set-up time. However, there are also many cases in which costs are of less concern, given the benefits of simultaneous acquisitions.

In multi-view fusion, images of the same modality are taken under different conditions at the same time. This fusion technique is applied to increase the amount of information for the fused images, while source images are taken under different conditions.

The choice between asymmetric and symmetric data fusion makes a significant difference in the integration and joint analysis of multimodal data. For integration methods falling under the asymmetric category, information from different modalities is assigned with different weights so that information from one modality could be treated as a constraint on the other modality. For instance, fMRI contrast maps limit the source localization of EEG/MEG.

Modalities in symmetric data fusion, however, are treated equally in terms of spatial and temporal resolution as well as the uncertainty in the possible indirect relation to neural activity. Hypothesis-driven approaches and data-driven approaches are the two main categories of symmetric fusion approaches. Hypothesis-driven approaches, also called model-driven approaches, are usually in model-based setting, while data-driven approaches belong to blind source separation methods [227] . Examples of different fusion methods are shown in Table 3 . 

Image fusion rules are applied to highlight the features of interest in images while suppressing the unimportant features. Generally, fusion rules are mainly comprised of four components: activity-level measurement, coefficient grouping, coefficient combination, and consistency verification [230] .

The activity-level measurement rule, as can be subdivided into window-based activity (WBA), coefficient-based activity (CBA) and region-based activity (RBA), characterizes coefficients at different scales. In the coefficient grouping component, there are mainly three typical groupings, including single-scale grouping (SG), multi-scale grouping (MG), and no-grouping (NG). SG indicates that the same strategy is applied to fuse different coefficients between sub-images on the same scale.

The coefficient combination component mainly comprises of maximum rules (MR), weighted rules (WAR), and average rules (AR). MR can be given as:

where C F indicates the combined coefficient, 1 and 2 are the coefficients of two input images at the level of .

For WAR, 1 and 2 are combined by multiplying different weights 1 and 2 , that is

AR, which has the coefficient of input images averaged, is a special case of WAR.

The consistency verification component ensures that the same rules are applied to fuse the coefficients in the neighborhood.

There are three different levels of image fusion: pixel level, feature level, and decision level. This categorization can be seen in Table 4 and Figure 10 . Pixel level rules directly deal with the information acquired from each pixel of source images and then generates pixel values for the fused image correspondingly. Feature level rules focus on regional information and features such as texture and salient features. The fused image in the decision level is acquired through rules of fuzzy logic and statistics. Before rules in feature level and decision level apply to the source images, segmentation of the source images is needed. Compared to the pixel level fusion, feature and decision level fusion shows more advantages are less affected by noises and misregistration. Feature and decision level fusion also shows better contrast and lower complexity [231] . In the following sections, we will introduce fusion rules for multi-model image fusion, followed by validation metrics. Figure 10 pixel-based and window-based fusion

Fuzzy logic-based rules belong to decision level fusion. These rules are usually used to solve challenges in blurry fused images. Mamdani and T-S models are two fuzzy logic models. The difference between them lies in the consequence parts. T-S models linearly mapped input variables into functions to form the consequence parts, while

Mamdani models used fuzzy sets. The T-S model is more advantageous than the Mamdani model in regard to number of rules and accuracy.

In general, feature extraction through fuzzy logic algorithms is performed prior to fusion, which generates pixel-wise features 1 and 2 from two input images. The fusion procedure can be divided into three steps. In the first step, the fuzzy logic, which is usually comprised of four conditional rules, is used to label the individual pixels as following.

= { , both of 1 and 2 are high (Rule1) , 1 is low but 2 is high (Rule2) OR 1 is high but 2 is low (Rule3) , both of 1 and 2 are low (Rule4)

Then the new pixel-wise feature values can be calculated through:

where 1, 2, 3 corresponds to low, medium and high components, respectively. and are the mean and variance of each component. By incorporating the center average defuzzifier to process fuzzy outputs, the weight of fuzzy logic can be obtained.

The essence in statistics-based methods lies in the data-driven technique and high order statistics that can reveal the underlying pattern across multiple modes of data. Principal component analysis (PCA) [232] [233] [234] [235] together with Hidden Markow Tree (HMT) [236] [237] [238] are two typical examples of statistic methods in the field of multi-modal medical image fusion.

PCA is an orthogonal linear transformation that reveals the most valuable components of the input images. Let 1 and 2 be the two coefficients of the input images that can be denoted as:

where 1 and 2 (1 ≤ ≤ ) are column vectors of two coefficients 1 and 2 . The importance of components is related to the eigenvalues in the covariance matrix between 1 and 2 . The covariance matrix can be computed through:

where is the expectation of vectors, 1 ̅̅̅ and 2 ̅̅̅ corresponds to the average of 1 and 2 respectively, that is

Given the eigenvalues obtained from the covariance matrix as , the normalized weights 1 and 2 for 1 and 2 can be denoted as: 

Therefore, the fused coefficient can be the combination of two input images:

The two-state HMT method, unlike PCA methods, can be deployed to model the coefficients. Two mixed Gaussian random distributions, as well as the hidden states, depict Intra-coefficients. The hidden states here refer to the one parent and four children coefficients. Given each coefficient denoted as , the coefficient is obtained by probability density function:

when the coefficient in the state n (n=0, 1), ( | = ) is the probability density function correspondingly.

The fused coefficient is then given by:

Methods based on the Human visual system (HVS) aim at solving the fusion problem in the way of image recognition and comprehension. The system includes components such as visibility, smallest univalve segment assimilating nucleus (SUSAN) [239] , and retina-inspired model (RIM) [240, 241] .

The sharpness of an image can be quantified by visibility. Therefore, the images with higher visibility show lower blurriness. Given an image of size × , then the visibility of the can be mathematically expressed as :

where µ is acquired by calculating the mean grey value of the image and , the visual constant varying from 0.6 to 0.7.

SUSAN, proposed in [242] , is a feature extraction algorithm inspired by HVS. SUSAN computes the feature of a pixel by considering a circular mask around the pixel. In the mask, the area consisting of pixels that have similar brightness to the nucleus, or the central pixel, is selected and is called Univalue Segment Assimilating Nucleus (USAN). Let the input image to be and the circular mask with the radius , then simplest USAN can be given as :

where 0 is the central pixel, is a nominal depicting surrounding pixels. ( )is the pixel value at and is the brightness difference threshold. The value of , which specifies the range of pixel values to be considered, must be carefully chosen because extracted features are sensitive to it. In order to lower the sensitiveness, the distance between pixels is also taken into consideration, and the extended USAN function is: 

where is the distance scaling factor.

For components in intensity-hue saturation (IHS) decomposition methods, the RIM model can be used as the fusion rules. There are five layers in RIM. The first cone layer outputs an array of high-resolution cone photoreceptors with high resolution. The second layer is the extractor for spatial feature, while the third layers are horizontal cells. The fourth and fifth layers, which combine features, are bipolar and ganglion cells The RIM based image fusion rule can be demonstrated as:

where 1 and 2 stand for intensity components of two source images. 1 and 2 are the filters of feature extractors. The filter 1 , a high-scale spatial feature extractor, calculates the spatial difference between high-resolution and low-resolution. Filter 2 combines the output of horizontal cells.

Objective evaluation metrics are used to evaluate the efficacy of image fusion rules on improving the quality of the fused image. Widely used metrics includes spatial frequency (SF) [243] , the ratio of spatial frequency error (rSFe) [244] , wavelet entropy (WE) [245] , Signal noise ratio (SNR) [246] , mutual information (MI) [247] and directive contrast (DC) [248] .

SF metric, which measures images' activity, is generally used for PCA integrated HIS methods in multimodal image fusion. SF can be defined as

where and stand for column frequency and row frequency, respectively.

Based on SF, rSFe compares the SF F ′ of the fused image with 1 ′ and 2 ′ of the input images where SF' can be extended as:

where and are the main diagonal SF and the secondary diagonal SF. rSFe can then be formulated as:

Calculated by multi-scale entropy, WE can be given by:

where i is the resolution level, is the density distribution derived from the energy of the detail signal and the total energy =

In [246] , the authors proposed SNR-based image fusion rules. The proposed fusion method can be formulated in the following form.

for a specific region . ( ) is the activity level of that, while is the total number of pixels. The probability of pixel activity can be calculated through:

where is the weight of SNR from the image, is the number of decomposition levels and ( 1 , 2 ) is the detailed wavelet coefficient.

MI is deployed as the fusion rule for WT-based multi-modal medical image fusion. MI between the input images , and the fused image , is maximized to give , which is acquired by:

DC measures the difference between the pixel and its neighbors. The ratio between the high-frequency intensity and the low-frequency intensity is the intensity contrast DC. 

For the multi-modal image fusion of neuroimaging, the selection of various decomposition and reconstruction methods influence the fusion procedure and outcomes. In this survey, we shall discuss seven popular methods: (i) RGB-IHS; (ii) Pyramid representation; (iii) wavelet-based approach and its variants; (iv) multi-resolution analysis;

(v) sparse representation; and (vi) salient features.

The intensity-hue-saturation (IHS) model [249] helps transform the original image in RGB color space to hue, saturation, and intensity channels. This RGB to IHS procedure is calculated by simple equations. Besides, the reconstruction is carried out by inversed transformation (IHS to RGB).

For the RGB->IHS procedure, the intensity of each input image was estimated by the following equations:

Then, we consider three conditions: C 1 : B<R,G; C 2 : R<B,G; C 3 : G<R,B.

The hue value is obtained by

The saturation value was yielded from the equation

On the other hand, after the revision was performed on the I, H, and S components, we can convert from IHS color space to original RGB space based on three various conditions:

For C 1 , we have

For C 2 , we have

For C 3 , we have

Pyramid representation (PR) is commonly employed in image fusion. PR is a typical multi-scale signal representation approach that can be used for 1D signals, 2D images, etc. Figure 11 shows a toy example using the cameraman picture. There are two pyramids commonly seen in practice: lowpass pyramid and bandpass pyramid.

The former smooths the image and then under-samples the smoothed image. The latter generates the difference between images at adjacent levels and carry out image interpolation between neighboring levels of resolution. 

Suppose F k is the filter and down-sampling, PR can be obtained by

where r k is the residual image of I at i-th level, and k is in the range of ∈ ,1, … , -

where K is the maximum iteration number.

After modification on the PR of input images, the fused image I F is obtained by inverse pyramid transformation (IPT) as

The process of PR based fusion is shown in Figure 12 . Figure 12 Diagram of two-level pyramid representation based fusion

Wavelet transform (WT) based fusion is one of the multi-scale analysis methods. The idea is simple, as shown in Figure 13 . WT will decompose the input images into low-frequency (LF) and high-frequency (HF) subbands. The corresponding image fusion rules will be applied to fuse LF and HF subbands. Finally, the fused image is yielded by the WT reconstruction technique [250] .

where Here, the wavelet Φ , ( ) is calculated from the mother wavelet Φ( ) by translation and dilation. The dilation factor c and translation factor b are all positive numbers.

On the other hand, the discrete wavelet transform (DWT) captures both spatial information along x and y axes, by discretizing above two equations

Then, suppose a low-pass filter (LPF) and a high-pass filter (HPF) are created, we have

where n is the discrete version of time t, e represents the coefficients, a and d correspond to approximation and detail, respectively. The symbol ↓ represents the downsampling operation.

The above decomposition process can be iterated with successive approximations being decomposed in turn, so that one signal is broken down into various levels of resolution.

When I(n) is extended to be a 2D brain image I(m, n), the 1D-DWT is applied to row and column directions separately. The approximation (a) and detail (d) subbands now expand to four subbands: approximation subband (a), horizontal subband (h), vertical subband (v), and diagonal subband (d), as shown in Figure 14 . Figure 14 Diagram of 2D-DWT 2D-DWT can even be generalized to 3D-DWT. A straightforward implementation is to apply 1D-DWT to row, column, and slice directions, respectively. Figure 15 shows an example of carrying out 3D-DWT to a cubic image. 

The DWT can provide better performances than traditional signal processing techniques, but it lacks translation-invariance and directional selectivity. Hence, new variants of wavelet-based techniques have been applied to multi-modality image fusion.

The stationary wavelet transform (SWT) can solve the shift-variance problem by getting rid of the downsampling operation from ordinary DWT. SWT can provide more details and texture information than DWT.

(2019) [253] used SWT and convolutional sparse representation for multi-modal image fusion. The DWT calculates each decomposition level by passing only the previous approximation coefficients to quadrature mirror filters (QMF). Nevertheless, the discrete wavelet packet transform (DWPT) [254, 255] passes all coefficients (both approximation and detail) through QMF to create a full binary tree. Sreekala and Kuncheria (2016) [256] used WPT to implement misaligned image fusion. Shah, Merchant (2010) [257] combined curvelet wavelet and the WPT method to carry out image fusion.

To increase directional selectivity, the dual-tree complex wavelet transform (DTCWT) used two separate two-channel filter banks. The scaling and wavelet filters in the dual-tree cannot be selected arbitrarily. In one tree, the wavelet and scaling filters produce a wavelet and scaling function, which are approximate Hilbert transforms of those generated by another tree. At each level of 2D DTCWT, it produces a total of six directionally selective subbands (±15°, ±45°, ±75°). Figure 16 Lift scheme (LS) is a solution to reduce computation time and accelerate the design and application of DWT [258] . The idea of LS is to factorize DWT with finite filters, into a sequence of ordinary convolution operators, which are called "lifting steps" [259] . This procedure can reduce the arithmetic operations by almost two.

Mathematically, the analysis filters ( , ) can be formulated in the form of polyphase matrix

This polyphase matrix P(z) is a 2 × 2 matrix, which contains the analysis LPF and HPF, each split up into their even and odd polynomial coefficients and normalized. [261] used the level-set method and LS-based CDF wavelet to compress magnetic resonance images.

Apart Wavelets also fail to capture the geometric regularity along the singularities of surfaces, because of their isotropic support. Shearlet transform (ST) is one of the best sparse directional image representation methods. Figure   18 showed the ST coefficients, where the input image is Figure 16 

Different from standard multi-scale analytic methods, the sparse representation (SR) assumes that both high-frequency and low-frequency components share the same set of sparse coefficients [269] . SR based fusion method root from compressed sensing. Nowadays, there are some famous variants of SR, such as group sparse representation and joint sparse representation (JSP) [270] , the diagram of which is shown in Figure 19 .

From Figure 19 we can observe the detailed steps of JSP [271] . First, both input images I 1 and I 2 are transformed into vectors via sliding window

where SW represents the sliding window method. An over-complete dictionary sparsely represents the vectors of both images where C represents the intersection of vectors of images I 1 and I 2 , and U represents their differences. The two variances C and U can be obtained by

where E stands for the over-complete dictionary, and T C and T U denote the sparse coefficients (SC) of C and U .

Afterward, SCs from both input images are combined using fusion rules as

where T F stands for the fused SC. The over-complete dictionary can generate the fused vectors

where F stands for the fused vector.

Finally, we can transform F to image space and get the fused image I F .

Salient feature based fusion approaches are a type of novel methods, with the benefits of shift-invariance, low-cost computation, and saliency-feature preservation. The edge-preserving filter (EPF) is an important research method among all salient feature fusion approaches. Scholars have proposed many EPF approaches, such as local extrema scheme [272] , multi-scale edge-preserving decomposition [273] , edge-preserving smoothing pyramid [274] . The BL of each input image at different scales is obtained by 

Assessment is an essential component of multimodal data fusion that provides perspective into the quality of fusion results. There are two main forms of qualitative assessment: subjective quality assessment and objective quality assessment. In this part, we will review the conventional techniques and metrics applied in both forms.

Subjective assessments are established as a reliable form of quality evaluation in image fusion [275] .

Professional subjective assessments based on medical and radiology expertise are widely applied in neuroimaging studies. Conventional subjective quality assessments are in the form of surveys with a set predesignated of questions and solutions, which represents a non-linear mapping to quantitative quality metrics [276] . Typical questions in neuroimaging fusion are the number of artifacts or distortion, while typical solutions can be in the form of basic descriptors (e.g. none, minimal, some, substantial) associated with continuous scores. Subjects are required to answer questions with respect to a set of images. It is ideal for image sets between subjects to overlap in order to ensure multiple surveys for the same image. With a set of high-quality fused images as references, the survey scores are then converted to difference scores with respect to scores of reference images. The difference scores are then converted to Z-scores and rescaled to the required range for analysis [276] . The double stimulus method is commonly applied to account for the potential misalignment in quality scales between different image sets or different assessment sessions. In the double stimulus method, both surveys and references are randomly included in the test set [277] . Conventional quality metrics obtained using these methods are the mean opinion score (MOS) and difference mean opinion score (DMOS). Data noise elimination and outlier removal for images & subjects are then performed after obtaining the scores. Subjective quality assessments are vulnerable to both intra-expert and inter-expert variability, while the requirement for expertise significantly increases the cost of assessments. For extensive studies, considerable efforts are required to standardize and maintain assessment protocols for optimal consistency [278] .

In contrast to subjective Quality Assessment (QA), which involves a complex organization of observers and strict tests, objective QA only requires the computation of a single numerical score. Objective QA should be consistent with subjective QA, and often relies on statistical properties of the images, or even stochastic modeling of the Human Visual System (HVS) [279] .

Objective QA can be classified according to the reference (distortion-free) image used in the score computation. The best-case scenario is the full-reference QA [280] , in which , with the same resolution as the fused image , is known. This approach is, by far, the most widely extended. However, this full-reference is rarely available in practical applications. For example, in multi-spectral satellite imaging or medical imaging, some image modalities are acquired at a lower resolution and then fused with higher-resolution gray-level images. This is known as the reduced-reference case, where is sometimes available at a coarser resolution and sometimes just a set of extracted features, and the QA is performed in this feature space. Finally, a third case is when no reference is available at all [281] . In this case, is considered by itself, and measures like entropy and contrast are computed using only the information contained within.

Depending on the type of quantification used, the metrics can be classified into "signal distortion" and "salient feature" categories. The former uses strict mathematical theory to assess quality, e.g. with entropy, standard deviation, etc. The later is grounded on a modeling of the HVS to assess salient feature transferred from to .

The first signal distortion metric is a commonly used statistic, the standard deviation (STD) between the input image and . Let us consider the images as a function ( , ) and ( , ) defined in the range ∈ ,0, …and ∈ ,0, … -, with the number of rows and the number of columns in the image. The STD is computed as:

being the mean intensity of the fused image.

Similar is the Root Mean Squared Error (RMSE), also widely used in many applications, included objective QA:

An additional measure based only on pixel intensities is the Sharpness (SP), that reflects the level of detail transferred to :

And finally, the Peak Signal-to-Noise Ratio (PSNR), which is perhaps the most widely used objective quality assessment:

where is the number of intensity levels, typically 256 for 8-bit images.

Now there are many other measures based on information theory and entropy [282] . To perform this computation, let us note ( ∨ ) as the ratio of pixels with gray value equal to over the total number of pixels × of image . We define the Entropy (EN) as:

The difference of entropy (DEN) [282] quantifies the differences between and . The smaller, the better:

Another possibility when a reference image is available is to compute the Cross-Entropy (CE) between the reference and the fused image :

From CE we can derive the Overall Cross-Entropy (OCE), which measures the entropy between several input images and the fused image. The formula below refers to images, but the most common case is the one that uses just two images, 1 and 2 :

In Sheikh and Bovik (2006) [276] , 51 The Mutual Information (MI) is then obtained in [284] as:

with = 1 , 2 to obtain them with no reference images. However, the ( , 1 ) and ( , 2 ) are not guaranteed to be on the same scale. The solution is the Normalized Mutual Information, devised in [284] and re-formulated in [283] for the fusion of two source images 1 , 2 as:

( 1 , 2 , ) = 2 0 ( 1 , )

The Spatial Frequency (SF) [285] is often used to measure the overall clarity of the fused images. It is obtained from the row frequency ( ) and column frequency ( ) of the image as follows: 77) and the SP is obtained by the harmonic mean of these two measures.

( ) = √ ( ) 2 + ( ) 2 (78)

Salient feature metrics assess whether the salient features of the source images have passed to the fused image, following the hypothesis that the HVS is highly adapted for the perception of structural information. The most widely used measure in this category is the Structural Similarity (SSIM) index [281] , which is based on the degradation of structural information.

The SSIM compares local patterns of pixel intensities, based on luminance , contrast and structure :

( , ) = , ( , )-, ( , )-, ( , )-

weighted by some exponential variables that were set to = = = 1 in the original paper [281] . Several expressions are provided for ( , ), ( , ) and ( , ), with certain constraints, and after substituting on the previous equation, it becomes:

( , ) = 

where 1 and 2 are two constraints of the model, expressed by 1 = ( 1 ) 2 , 2 = ( 2 ) 2 and 1 ≪ 1, 2 ≪ 1. The notation and corresponds to the mean intensity and unbiased standard deviation of the intensities of , as in previous equations, and is the covariance matrix of the intensities of and . 

with and the mean vector and covariance matrix of the MGM of .

Multimodal neuroimaging and the fusion of multimodal data tackle the challenges of neuroimaging and the fundamental limitations of individual modalities, and therefore provide significant benefits to the overarching aim to achieve higher image quality and reveal brain physiology. This part will review some of the main benefits with specific fusion examples.

MR-PET refers to a functional metabolic and molecular multimodal imaging method integrated through a combination of MRI and PET. It has the potential to achieve the maximum complementary advantages with the examination function of both PET and MRI [287] .

MRI can not only display structural details through multi-parameter sequences but also perform a variety of functional imaging. In other words, it can be seen as a means of anatomical imaging. However, compared with PET, MRI still has certain limitations in metabolite imaging. PET imaging can show trace amounts of radiolabeled molecules, but its image resolution is poor and the anatomical structure is not clear. It is the complementary characteristics of MRI and PET that led to the birth of MR-PET imaging, which not only has high soft-tissue contrast and resolution but also can provide valuable functional information [288] .

At the radiological society of North America meeting in November 2006, Siemens presented the first MR-PET images of the brain from their diagnostic machine. During the imaging process of MR-PET, PET and MR imaging can be performed simultaneously with minimal interference; the PET scan detects the accumulation of fluorodeoxyglucose, while multiple sequences of MR images were obtained [289] .

In neuroimaging studies of MRI, the common approach tends to acquire T1-and T2-weighted anatomical, T2*-weighted functional data within the same session and then make a combination of these different contrasts for further exploration and research. Here, T1 refers to the recovery time of longitudinal magnetization, and T2 refers to the decay time of transverse magnetization. Both of them are special values associated with the spin of the nucleus in the tissues.

In humans, T1 and T2 values of diseased tissue and normal tissue are different, so diseases can be diagnosed by nuclear magnetic resonance imaging [290] .

Magnetic resonance images are presented in different shades of gray, reflecting differences in the intensity of magnetic resonance signals, or in the length of relaxation times T1 and T2.

Pure T2 dephasing is intrinsic to the sample. While T2* dephasing is relevant with true T2, field inhomogeneity (T2M) and tissue susceptibility (T2MS) as shown in Equation (82). 

According to the common neuroimaging protocols, T1-weighted scans have good resolution and gray-white matter contrast, so it is better for observation of anatomical structure. T2-weighted scans perform well in showing histologic lesions, so it is often used for checking permanent brain injury. T2* is mostly used in scanning brain

activity. An increase in T2* weighted signal between baseline and an active condition is associated with brain activation in studies using blood oxygenation-level dependent (BOLD) fMRI. For instance, as displayed in Figure   21 , brain regions become oxygen-rich after activity, which leads to a decrease of the Hbr/HbrO2 ratio and the increase of fMRI signal. Consequently, for T1, T2 and T2*, each method provides a physiologically and physically filtered view on one or more brain processes of interest [291] . Figure 21 Comparison of Hbr/HbrO2 ratio in brain capillaries before and after activity Thus, combining different contrasts has the general merit of getting a more comprehensive physiological view on brain processes than utilizing just one imaging method alone.

In Figure 4 , the spatial and temporal ranges for the most widely used non-invasive functional imaging methods are presented. Functional MRI, as the non-invasive functional imaging method, has the highest spatial resolution though the temporal resolution is relatively low compared to neuronal population dynamics. MEG and EEG can measure magnetic and electrical changes on millisecond time scales while the spatial resolution is beyond seven millimeters [15] . Ultra high-field MRI has been widely used for the mesoscopic level of neuroscience in humans [292] [293] [294] . As a consequence, the Spatio-temporal resolution can be improved by combining different imaging methods, especially through the combination of one modality that has a higher temporal resolution with another modality of superior spatial resolution. The fused images are called validation when Spatio-temporal resolutions are similar.

When considering improving spatio-temporal resolution, the resolution of each modality is not of the biggest concern. Each modality's effective resolution and the additional effect after the combination of data from different modalities should be considered instead. While the nominal resolution is generated according to fixed physical parameters of instruments, the effective resolution depends on the information of the data instead. In fMRI, field-of-view and the k-space acquisition matrix determine the nominal spatial resolution, while the effective resolution is largely affected by fMRI sequences and spread of the hemodynamic response [295] . Because of the slow development of hemodynamic response, the effective resolution is lower than nominal temporal resolution in fMRI, while the latter resolution is determined by the repetition timeTR.

The nominal temporal resolution of electrophysiological methods is usually in the millisecond range. The effective resolution, however, can vary from tens of milliseconds to hundreds of milliseconds due to the slow evolution of neuronal field potentials and statistical detectability. Also, many repetitions have to be implemented before a detectable signal comes up for the evoked potentials in EEG, while a detectable hemodynamic response can be acquired through a single stimulus in fMRI.

EEG-fMRI, as the combination of EEG and fMRI, turns out to be the typical example of improving spatio-temporal resolution through multimodal neuroimaging. Epilepsy research facilitated the development of simultaneous EEG-fMRI recording [296] . In the case of localizing epileptic foci, simultaneously recorded fMRI help by connecting epileptic foci with interictal epileptiform discharges (IED) acquired by EEG. Simultaneous EEG-fMRI can also be utilized as neuronal oscillations in neuroscience [209, 297] . In these studies, the process of analyzing the recorded fMRI data takes power in the frequency domain of the EEG signal as a variable.

An MRI machine uses a static, homogenous, external magnetic field B0, a radio-frequency magnetic field B1, and three orthogonal gradient fields. However, magnetic field variation may result from imperfections of the main magnetic field coil, the exciting gradient coils, and susceptibility of the tissues [298] . Field inhomogeneity is one of the causes of blurring or spatial shifts in MRI images. A typical region of distortion is the borderline between tissue and air, where a difference in magnetic susceptibilities exists. A standard process in neuroimage processing and fusion is the registration of multimodal images to standardized templates. Hellier and Barillot (2000) [299] introduced 3D non-rigid registration of multimodal images by mapping the deformation through cost optimization.

The registration process provides the foundation for higher quality fusion with less distortion.

The fusion of the main imaging sequence with supplementary sequences is also widely employed for distortion correction. As mentioned previously, a well-studied example is EPI, a high temporal resolution MRI technique that can collect multiple images per second. The high imaging frequency of EPI acquisition results in geometric distortions to its images. Acquisition of the static Bo fieldmap before or after EPI can provide additional information on local static field inhomogeneity and magnetic susceptibility. Holland, Kuperman (2010) [300] developed an efficient non-linear registration method by acquiring EPI of opposite phase encoding polarities, while Oh, Chung (2012) [213] applied PSF modeling to map the distortion field.

Another example is the fusion of MRI and PET images in MR-PET systems. In these systems, apart from the combination of structural and metabolic information, the fusion of MRI and PET also allows enhanced distortion correction. In simultaneous MP-PET systems, motion distortions of PET can be corrected with the volumetric information provided by continuous EPI acquisitions or navigator sequences [301, 302] . Joint reconstruction of MRI and PET utilizes dependence between the two modalities for image reconstruction of one mode based on the other.

Reconstructed images show sharper edges and less distortion [303] . This entire process can also be formulated as a single optimization problem [301] .

Segmentation has been one of the main challenges faced by neuroimaging experts during the last decades. It refers to the process of tagging image pixels or voxels with biologically meaningful labels, such as anatomical structures and tissue types [304] . The segmentation task and data analysis are very often guided in most of the neuroimaging experiments by atlases that provide standardized, or stereotaxic, 3D coordinate systems for statistical data analysis and report of findings [305] . One of the earliest works in anatomical brain standardization was by Talairach, Tournoux (1957) [306] , who developed a 3D coordinate space for the brain to assist surgical techniques.

This work was later updated by a full printed three-dimensional atlas of the human brain, which was especially beneficial for clinical studies, electroencephalographic investigations, and statistical computations [307] .

Tzourio-Mazoyer, Landeau (2002) [308] developed an anatomical parcellation approach with the spatially normalized single-subject high-resolution T1 volume provided by the Montreal Neurological Institute (MNI) [309] .

These developments have contributed to better define the relationships between brain structures and their functions in modern human neuroscience research, as well as to reduce the variability found between different subjects.

The traditional approach for brain image segmentation is the manual annotation or delimitation of the regions of interest (ROIs) by a trained expert [304] . This process is subjective and strongly influenced by expert performance. It is then of limited applicability since it is time-consuming, prone to error and difficult to reproduce.

Automated segmentation methods have been developed during the past decades. These methods can be classified into basic tissue classification and anatomical segmentation procedures [310] . Tissue classification methods segment a 3D image of the brain into different tissue types (Grey Matter, White Matter, CSF, etc.), while also correcting for spatial intensity variations (also known as bias field or RF inhomogeneities). These methods normally incorporate probabilistic tissue atlases to improve their performance and have been successfully automated for fMRI studies as well as integrated into several neuroimaging toolkits such as FSL FMRIB's Automated Segmentation Tool (FAST) [311] and Statistical Parametric Mapping (SPM) [312] . However, anatomical segmentation procedures are much more difficult to automate, since different anatomical structures that consist of different tissue types may exhibit similar signal properties. This observation is the reason why automated anatomical segmentation of the brain into non-homogeneous regions needs to be guided by atlases.

For atlas-based segmentation, an atlas is defined as the combination of a brain volume (atlas template) and its corresponding coregistered segmented volume (atlas labels). The atlas-based segmentation method registers the atlas template to the target image, and then the atlas labels are propagated to the target image using an effective image warping method [313] .

Let ( ) be the volume to be segmented, with ∈ ℝ 3 representing a 3D voxel coordinate vector. For the purpose of single-subject atlas-based segmentation, a grey level atlas volume ( ), a labeled volume ( ) ∈ = ,1,2, . . . , -

where is the number of anatomical regions or labels defined in the atlas and, optionally, a probabilistic atlas P(x), all within the same spatial coordinates (the atlas space), are required. The segmentation process consists of a registration step and a label propagation step.

In the registration step, the atlas volume ( ) is registered to the input image ( ) by means of a spatial transformation ( ), which optimizes a given cost function. In the label propagation step, labels defined in the tagged volume ( ) are assigned to each of the voxels of the input image ( ) by applying the normalization transformation ( ) to ( ), thus spatially aligning the atlas to the input image ( ). The atlas probability maps ( ) are normally used for MRI since they enable to improve the segmentation accuracy of T1-weighted MRI into grey matter, white matter and cerebrospinal fluid tissues with higher accuracy when the regional tissue densities need to be quantified and compared for between different groups of subjects [314] .

Medical image registration is routinely used in neuroimaging. It aims at determining the spatial alignment between images of the same or different subjects by means of optimizing a cost function, which measures the similarity between the transformed input image and the reference image (template) [315] . The registration process involves a spatial transformation being global rigid and affine transformations usually enough for intra-subject image registration [313] . However, the atlas-based segmentation task requires intersubject matching or registration of an input image to an atlas image and, as a consequence of the variability of the anatomical structure across different subjects, non-rigid registration algorithms are widely used.

Single-subject atlas-based segmentation approaches suffer from a reduced ability to capture the variability of the spatial distribution of anatomical structures across different subjects. Multi-atlas segmentation was introduced in some pioneering works [310, 316, 317] to address this problem and to offer superior segmentation accuracy. 

For the purposes of reviewing the existing methods for atlas propagation and atlas fusion in multi-atlas based segmentation, a similar notation to the adopted ones in [318, 319] will be used. These works have addressed the atlas-based segmentation problem as a classification task where the atlas is the training set, and the training is associated with the process of computing the registration between the image and the atlas.

Let ( ) be a 3D image to be segmented into different classes belonging to the atlas label set = ,1,2, . . . , -. Multi-atlas segmentation methods use 3D atlas images ( ) and their corresponding atlas labels

The coordinate transformation ∶ ℝ 3 → ℝ 3 defines a mapping from the coordinates of the atlas A k (x) to the target image I(x). Once the registration transformation is obtained, the input image ( ) is automatically segmented for each one of the atlases available by applying a label propagation technique, thus obtaining K different segmentations (x) ∈ Λ, = 1,2, . . . ,

These candidate segmentations must be combined later into a final estimated segmentation (x) ∈ Λ by applying a label fusion technique, as shown in Figure 22 .

Once each of the K atlas images has been coregistered to the input image, the labels are then propagated to the space of the input image in a process called label propagation. Label propagation applies the registration transformations to map each of the available atlas volumes ( ) to the input image, and an image warping technique preserves the discrete nature of the labels. Note that the transformations T k : applied to the atlas label volumes L k (x) are 3D continuous transformations that do not match grid points in the atlas space to grid points in the target input image. The purpose of label propagation is to perform the interpolation of the transformed atlas label volume for grid points in the target image space [320] .

Among the different label propagation techniques, the most widely used are nearest-neighbor interpolation [310] and linear interpolation [316, 320] . Nearest-neighbor provides the unique label of the nearest atlas grid point. Partial volume interpolation, a technique that was first introduced in [321] for image registration, is a more sophisticated method for label propagation that takes into account the contribution of adjacent grip points to the final propagated label. During the last decade, several improvements have been proposed and reviewed for label propagation in multi-atlas segmentation [304] . Most of them incorporate additional information to improve the performance by augmenting the information with a tissue consistency map in nearest-neighbor interpolation [322] or introducing signed distance maps of the original atlas labels [323, 324] .

The final step in atlas-based segmentation is atlas fusion. After registration and label propagation for each of the K available atlases, a final unique segmentation (x) ∈ Λ is obtained by merging the information provided by each of the individual atlas-based segmentations into a single segmented image. This approach has been shown to be more accurate than an individual atlas segmentation [316] in the same manner as a combination of classifiers is generally more accurate than a single classifier in many pattern recognition scenarios [325] [326] [327] [328] [329] [330] .

The conventional method for combining individual segmentations is an equally weighted majority voting framework [319] . A more sophisticated approach estimates the performances of the individual atlas segmentations and combines them by weighting them according to their estimated performance. Both approaches are described and discussed in this subsection.

Majority voting atlas fusion. The combined multi-atlas segmentation output (x) ∈ Λ for a voxel sample given the set of K single atlas segmentations (x) ∈ Λ, with = 1,2, . . . , , is obtained through the vote rule decision function

where the function is defined to be

Atlas label fusion based on majority voting considers all the segmentations equally accurate, and no prior knowledge of segmentation performance is required.

Performance weighting atlas fusion. Majority voting equally weighs the individual atlas segmentation without using a segmentation performance model. In order to improve the performance of the multi-atlas segmentation, the combined segmentation output (x) ∈ Λ should be the class maximizing the probability, given all the individual segmentations (x) ∈ Λ, with = 1,2, . . . , and some available segmentation performance model [319] :

Using the Bayes rule, and assuming independence of the individual classifiers, simplifies the problem and enables us to find an optimal segmentation based on a model of segmentation performance. Rohlfing, Russakoff (2004) [319] proposed different classifier performance models and used an expectation-maximization (EM) algorithm that simultaneously estimated the performance parameters of the segmentations and provided an estimation of the unknown ground truth. This novel strategy enabled us to learn classifier performance parameters and to adopt weighted atlas combination for atlas label fusion in multi-atlas segmentation following a novel supervised training stage [331, 332] .

The application and improvement of the quantification in medical image allows to increase characteristics such as sensitivity and specificity, among others, obtaining more accurate patient's diagnoses. To perform this analysis, the emission computed tomography (ECT) is used, since it is the most important medical imaging modality in nuclear medicine. More specifically, the two image modalities analyzed are positron emission tomography (PET) and single-photon emission computed tomography (SPECT), which differ in the radiotracer used and the nature of the emission measurement. It should be noted that the factors associated with quantification that affect PET and SPECT are also extrapolated to multimodal images. Furthermore, multimodal images can be of great interest to improve quantification. Both, PET and SPECT, have proven to be effective imaging techniques for the diagnosis and monitoring of treatments in different medical applications, offering information about biological processes, which is very important for the study of brain activity [333] . The level of intensity in these types of images is bound to physical parameters, such as cerebral blood flow, glucose or receptor binding among others [334, 335] .

Since these modalities were conceived in the 50's and 60's Jones and Townsend (2017) [336] , several improvements have been developed in terms of quantification. One of the most relevant improvements in this field is due to the development of dual-modality imaging in the 90's [337] . This type of imaging allows the combination of complementing functional information (PET and SPECT) with structural information (computed tomography (CT) and magnetic resonance imaging (MRI)), leading to increases in sensitivity and specificity. The combination of different characteristics that can be observed from each type of image allows us to obtain a better understanding of the structure and function of the human body. For example, PET/MRI combination offers remarkable advantages compared to the use of PET/CT since CT radiation dose is avoided and soft tissue images from MR could be acquired at the same time as the PET ones [338] .

PET offers higher resolution and better quality than SPECT, particularly in PET/CT. Furthermore, it allows for easier quantitative measurements [335] . For example, sensitivity could be up to two orders of magnitude greater with comparable axial fields of view [339] . Nevertheless, the spatial resolution that can be obtained in both techniques tends to be low, due to its dependence on the radiation dose administrated to the patients. Also, ECT images are usually degraded by several factors, such as photon attenuation, partial volume effect or scatter radiation [337] . Moreover, spatial resolutions associated with structural images continue to be higher (≈ 1 mm) than in functional images (≈ 4-6 mm) [340] .

Regarding quantification, an increasing number of relevant research aims to propose new quantification techniques or improve previously existing ones is observed in recent years. The possibility of quantification in nuclear medicine imaging is one of its successes. Due to the increasing use of nuclear images in therapies, the way of measuring functional images is changing. While historically, the use of relative and semiquantitative measures was typical, the recent application of absolute quantification is gaining support. One of the first steps was considered activity concentration and normalized uptake using the standard uptake value (SUV) [341] . Nevertheless, to reach quantitative imaging as a useful and potential tool, several factors must be addressed. As Zaidi and Hasegawa (2006) [339] mentioned among these factors are the system sensitivity and spatial resolution, dead-time and pulse pile-up effects, the linear and angular sampling intervals of the projections, the size of the object, photon attenuation, scatter, partial volume effects, patient motion, kinetic effects, and filling of excretory routes (e.g., the bladder) by the radiopharmaceutical.

In neuroimaging, PET and SPECT are a popular option to detect biomarkers associated with degenerative diseases. For Alzheimer's disease (AD), the accumulation of Amyloid-β plaques and tau aggregates can be detected [342, 343] , while an increased diffusivity in the striatum and thalamus can be observed in Parkinson's disease (PD) patients. A well-defined quantification of these biomarkers would allow more precise and reliable diagnoses.

This section is organized as follows. Firstly, quantification both in PET and SPECT is analyzed considering the associated effects and their correction, especially those related to photon attenuation, scatter, and partial volume effects. Then, novel developments focused on cerebral medical imaging are highlighted.

Even though PET scans could be evaluated qualitatively through the visual examination of the tracer uptake in cortical regions by a trained radiologist [344] , the best way to analyze these images is quantitative. For that, automated or semi-automated localization methods can be used to evaluate regional levels of tracer uptake [345, 346] . Within automated studies, an essential step is to use spatial normalization to register subjects' brain to a standardized template space, so that all subjects in the study can be compared [347] . Another procedure of great importance is intensity normalization, which is the object of study in this analysis.

As Foster, Bagci (2014) [348] mentioned, several semi-quantitative and quantitative parameters that can be considered for intensity normalization exist, such as standardized uptake value (SUV), fractional uptake rate (FUR), tumor-to-background ratio (TBR), nonlinear regression techniques, total lesion evaluation (TLE) or the Patlak-derived methods. Nevertheless, the most widely used is SUV, which can also be used in SPECT [348] . The use of body weight (BW) in SUV has been discussed in the literature, and the general advice is to use more reliable measures such as body surface area (BSA) or lean body mass (LBM) [349, 350] . Moreover, a decay factor that depends on the particular radiotracer used may also be considered [351] .

Several physiological and physical factors can influence the standardized uptake value obtained [348, 352] .

Regarding physiological factors, some of them are weight and fat of the subject or blood glucose concentration, etc.

Physical factors include partial volume effect (PVE), image manipulation (reconstruction, smoothing), and artifacts related to involuntary movements of the patient. The relevance of these factors lies in the variability of SUV. Studies carried out in the literature suggest that these factors may alter SUV in the range of 10%-30% [353, 354] .

Nevertheless, the importance of correcting these effects depends on the clinical trial performed, since it can be a complex task and is not always worthwhile [355] .

Moreover, these factors affect not only semi-quantitative measurements but also affect absolute quantification.

There are several methods to consider within the absolute quantification, such as image reconstruction, effects correction or calibration to obtain the measured activity distribution [356, 357] .

Among the mentioned factors associated with imaging quantification, we analyzed three effects and their respective corrections, since they are popular bias sources for PET and SPECT quantification. These effects are PVE, scatter, and photon attenuation effects.

Regarding the quantitative accuracy of PET image, partial volume effect (PVE) is the most relevant effect [358] .

PVE is related to the finite spatial resolution of PET scanners and its discrete nature. That discrete nature causes a voxel to could be composed of more than one tissue, which leads to a final signal that is an averaged mix of signals, which is also called tissue fraction effect. Moreover, the larger the voxel size, the more tissues can coexist in it. This effect, along with the point-spread effect (or spillover), is the cause of image blur [354, 359] . Also, it is an effect that often causes confusion among clinicians and researchers when analyzing the image, since it is necessary to distinguish between loss of radioactivity due to PVE and the true loss of tissue.

Strategies to address this effect are called partial volume correction (PVC) methods. Distribution of both signal and noise are the main factors to consider for the selection of algorithms for PVC to apply [359] . Several PVC techniques have been proposed in the literature for improving image quality and quantitative accuracy in PET [360] [361] [362] [363] . Historically, these techniques have been classified in several forms. In this work, these methods are grouped as region-based (RB) and voxel-based (VB) techniques, based the most commonly used classifications [340, 355] . The most relevant differences between them are that VB methods produce PVE-recovered images while RB methods do not, and RB methods are associated with regions of interest (ROIs) while VB methods are applied at a voxel level considering the recovery of the spatial resolution of the system. Usually, RB techniques tend to be used more frequently than VB ones since they include regional homogeneity assumptions, simplifying the problem [359] . [364] in 2D and later extended to 3D by Meltzer, Leal (1990) [365] was a segmentation in anatomy into two classes: brain and non-brain. Mullergartner, Links (1992) [366] proposed an improved version of this technique (known as GM algorithm) to use three instead of two classes: grey matter (GM), white matter (WM) and cerebrospinal fluid (CSF). Meltzer, Zubieta (1996) [367] added a fourth class to the previous algorithms in order to compensate for the real heterogeneity in GM tissue. The importance of this method is that PVE can be corrected between high and low intensity brain structures. Regarding the iterative reconstruction algorithms associated to VB methods, as V. Bettinardi, Castiglioni (2014) [340] noted, some of the best known and used is the Van Cittert (VC) algorithm, the re-blurred Van Cittert (R-VC) and the Richardson-Lucy (RL) algorithm. For broader coverage, readers are referred to reviews related to PVE [340] and iterative reconstruction methods [368] . Four classes: [367] Finally, data extracted from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu) is used in order to show some PVE correction methods. This data consists of a patient diagnosed with Alzheimer's disease from which FDG-PET and T1 weighted MR images were taken, as shown in Figure 23 .

Both modalities are registered to respective templates of T1 and PET, and PET image was not normalized in intensity due to use only one subject in this manuscript. PVC techniques are applied by using PVElab [384, 385] , a platform developed by the EU sponsored PVEOut project. PVElab can facilitate PVE correction through a graphical interface with several steps, including registration, segmentation, reslicing, optional application of an atlas (none in our case), and PVE correction. Three methods discussed above are those analyzed: 3D Meltzer technique [365] and GM algorithm [366] regarding VB methods, and the RB method proposed by Rousset, Ma (1998) [374] .

The large difference between methods is shown in Figure 24a . 

Another relevant effect to analyze is photon attenuation. This phenomenon occurs due to the interaction between the photon associated to the radiotracer and elements in the body such as tissue. Normally, this interaction leads to a scatter in the photon radiation. The probability that a photon experiences an interaction is represented by the linear attenuation coefficient [386] . The situation of this phenomenon differs between PET and SPECT imaging.

In PET, two antiparallel photons are detected in the collimator, so the total tissue thickness crossed by them is the same body thickness that intersected the straight line between the two detections (line of response). Nevertheless, in SPECT imaging, the attenuation depends on the total tissue thickness crossed and its type (e.g., soft tissue, bone), which varies depending on where the points of emission and detection are [386] . Traditionally, this correction was much more commonly performed in PET imaging than in SPECT. Nowadays, it is done in both to be able to achieve much more accurate quantitative results, although it is more difficult to apply in SPECT.

Attenuation correction is necessary in order to obtain accurate quantitative results and is widely implemented.

For this, an attenuation map is needed, which Zaidi and Hasegawa (2006) [386] defined as a representation of the spatial distribution of linear attenuation coefficients and delineates the body structures located in the image. Once the attenuation map is done, the reconstruction algorithm can be implemented with more information. Attenuation correction techniques can be classified into two major groups: transmission-less approaches and the ones based on transmission scanning. While the first group considers the measured emission data to develop the attenuation map or assumes a uniform distribution for the coefficients, the second one uses transmission data of external sources such as CT and MRI, i.e., the use of anatomical data. The second group offers a more accurate solution. Nevertheless, there are situations in which the correction made by the first group is sufficient, without the need to complicate the applied technique. A brief introduction to the various existing methods for performing the attenuation map is shown in Table 6 . For a more extensive reading of this topic, readers are referred to reviews about attenuation correction [386] [387] [388] . Transmission-less based methods are easily applied in brain imaging because the brain is considered a practically homogeneous region composed mainly of soft-tissue. Moreover, these methods allow results regarding the skull [390] and optical tracking systems to be used to obtain head contours [391] . Despite being a typology that has not been used frequently, research on these methods has increased over the last years due to shift away from anatomical information. One of the first methods was proposed by Nuyts, Dupont (1999) [392] , which consists of a maximum likelihood approach known as MLAA (maximum likelihood reconstruction of attenuation and activity). It is commonly used in non-time-of-flight (TOF) PET images, and several studies have tried to improve or test it [393, 395, 396] . Another popular algorithm is MLACF (maximum likelihood attenuation correction factors) [394] , mostly used in TOF PET, which jointly estimates the attenuation sinogram and the activity image. Less popular than the previous one, this method has also been tested in other studies [410] . Moreover, machine learning techniques for attenuation correction are emerging, such as deep convolutional encoder-decoders [334] or deep convolutional neural networks [411] have also been applied in attenuation correction. For this last type of technique, there is considerable interest in this field of research, and new algorithms are constantly presented.

Regarding transmission-based techniques, the added difficulty for MRI systems to obtain attenuation maps compared to CT systems should be highlighted. The reason for this is the connection between the attenuation coefficients and the electron density of the tissue. In CT, the data is associated with electron density and photon attenuation properties of the tissues. In contrast, MRI data is correlated to proton density and magnetic relaxation properties of the tissues [387] . As a challenge related to PET/MRI (or SPECT/MRI) systems, several methods have been proposed in literature regarding attenuation estimation. The MRI-based methods can be classified into two groups: segmentation-based and atlas-based methods.

Of the two groups, the segmentation-based one tends to be used more on the literature. technique, which depends on the density and composition of five tissues: air, skull, brain tissue, and nasal sinuses.

Similar to the latter case, Wagenknecht, Kops (2009) [408] presented a method where tissue classification is done with neural networks. First, a voxel classification with five tissues (GM, WM, CFS, adipose tissue (AT) and background (BG)) is made, then a second classification is done depending on the previous classes to detect extracerebral tissue and finally, segmentation is obtained. The main drawback of this method is the possibility of mis-segmentation or over-segmentation of bones, especially in the presence of abnormal anatomy or pathology.

In order to improve bone detection, ultrashort echo time (UTE) MRI began to be investigated. Firstly, dual-echo ultra-short echo time and three tissue classification (bone, soft tissue and air) were used [370, 413] . In parallel, methods related to the Dixon technique were raised. Dixon-Water-Fat-segmentation (DWFS) allows the separation of soft and adipose tissues. The method was applied DWFS for the whole body in [414] . Considering both techniques (UTE and DWFS), Berker, Franke (2012) [415] proposed a method trying to reduce the drawbacks of these two techniques: time-consuming and complex image registration. This new technique applies UTE sampling for bone detection and uses gradient echoes for water-fat separation, obtaining a four-class PET attenuation map. An example of DWFS application in neuroimaging is the study conducted by Andersen, Ladefoged (2014) [416] , where this technique is used to generate the attenuation maps. Taking into account the time-consuming problem with UTE, short echo-time (STE) has been tested as a method to obtain attenuation maps of three (cortical bone, air and soft tissue) [417] and four (cortical bone, air , soft tissue and fat tissue) tissues [418] , both in combination with fuzzy C-means (FCM) clustering method. Besides, the second study also used two-point Dixon sequences in image acquisition. The latest research in this field has been the use of zero echo time (ZTE), initially proposed by Wiesinger, Sacolick (2016) [419] as part of a segmentation method for cranial bone structures. All studies found in the literature, specially the ones related to ZTE, indicate an improvement in results compared to the use of atlas-based methods [420] [421] [422] [423] .

The basis of atlas-based approaches is the use of an MR template instead of multistep segmentation procedures, so atlas-based methods use a general map while segmentation-based methods generate a map for each individual.

This template can be obtained from two sources: an image labeled as if it were a segmentation of the different tissues or a coregistered attenuation map from a PET or CT scan with continuous attenuation values [354] . As usual, it is obtained from an average of co-registered normal patients, and it has to be warped to the target patient image volume. In any case, most of the studies related to this approach combine atlas-based and segmentation-based methods since the use of an atlas can be an important source of information and can reduce computational cost [424] [425] [426] . Despite this, other studies have proposed methods purely based on atlas images [405, 409] , such as the one developed by Johanson, Garpebring (2013) [427] where linear attenuation coefficients are predicted from a Gaussian mixture regression algorithm.

Once the attenuation map is generated, attenuation correction is applied in the process. For that, two techniques are mainly used [386] . The first one multiplies the attenuation coefficients obtained by PET image data in the sinogram (or projection) space [428] . The other procedure is performed when PET image reconstruction is performed with an iterative algorithm, using attenuation coefficients as data weighting. While PET images can apply both methods, only iterative methods are used for SPECT [429] [430] [431] .

Scatter in PET and SPECT is another relevant effect for absolute quantification, especially in SPECT, and which is usually associated with Compton scattering. It consists of an energy loss and a change of direction of a photon after an interaction with surrounding atoms. Nevertheless, this effect is not highly relevant in the clinical environment, according to the literature. The reason for this is that the techniques implemented have little impact on the final result, and since the photon change of direction is practically zero, the energy loss is minimal. However, several researchers are convinced that in order to obtain a high accuracy quantitative image, this artifact should be removed [432, 433] .

Several methods have been proposed for scatter correction, but most of them were developed many years ago and are inefficient. However, a few recent methods are getting remarkable results. Regarding the categorization of these techniques, a similar classification could be made for both PET and SPECT, considering that PET techniques began to be developed much earlier. Following the review of Zaidi and Montandon (2007) [432] , the classification would consist of five groups: hardware approaches using coarse septa or beam stoppers, multiple-energy-window approaches, convolution/deconvolution-based approaches, approaches based on direct estimation of scatter distribution and approaches based on statistical reconstruction. Due to the low clinical implementation and the diversity of existing methods, this work will only highlight the most innovative and relevant methods. Therefore, readers interested in this effect are advised to read the following reviews [389, 433, 434] . Table 7 summarizes the different existing categories indicating some examples.

The first method proposed to narrow the photopeak energy window to avoid the acceptance of scattered photons. However, this technique has significant drawbacks, such as the elimination of unscattered photons in the process, and therefore, a loss of intensity appears in the image [434, 435] . Thus, multiple-energy-window became much more popular, and it is one of the simplest and most used approaches. The techniques have been developed from two [436, 437] three [438, 439] or even multiple energy windows [440, 441] . Decades have passed since it is known that Monte Carlo techniques are ideal for scatter correction [433] [434] [435] . Nevertheless, it has recently become a viable solution in the clinical environment, as the computational cost is reducing and techniques are faster. Moreover, the Monte Carlo technique can be applied to several methods, either iterative reconstruction based or direct calculation methods.

Finally, another factor to consider in the quantification of functional images is the voluntary and involuntary movement of the patient (e.g. breathing). However, since it does not have high relevance in neuroimaging, its analysis has been dismissed in this study. 

Functional imaging is very useful for the diagnosis of neurodegenerative diseases, such as Alzheimer's disease or Parkinson's disease due to differences in brain activity which can be observed in temporal and parietal lobes (AD) or the striatum (PD) with respect to healthy subjects. Therefore, the systems must allow a correct quantification in order to offer an accurate diagnosis of the patient's condition.

As already mentioned, the correction of the aforementioned effects improves the quantification of the images.

For example, some studies associated with attenuation correction demonstrate this improvement. Delso, Kemp tissue segmentation, and these results are also very similar to those that would be obtained with a PET/CT system. In the study developed by Sousa, Appel (2018) [422] , a comparison between using an attenuation correction method based on ZTE and atlas-based correction shows that the former produces less variability than the latter. However, bias correction is similar in analyzed brain regions. For example, the correlation coefficient associated with anterior cortical regions is 0.99 for ZTE-based correction and 0.92 for atlas-based correction. Similar results are obtained in the study by Sgard, Khalife (2020) [423] . Other interesting results are produced by those that apply machine learning methods. For example, the deep convolutional neural network used by Yang, Park (2019) [411] for both attenuation and scatter correction designed for situations where it is difficult to use a combined CT or transmission source. The results were similar to the ones obtain with an CT-based scatter and attenuation correction.

PVC methods are also used in brain images. For example, regarding PD studies, Du, Tsui (2005) [376] proved than using a modified GTM method in brain SPECT images, the underestimation of striatal activities could be reduced to 1.2% from an initial value of 30%.

Finally, a correct normalization of the images can also highly increase the accuracy of the study.

Salas-Gonzalez, Gorriz (2013) [456] proposed a method for intensity normalization of FP-CIT SPECT brain image based on α-stable distribution. This method was tested by Castillo-Barnes, Arenas (2017) [457] , showing significant differences between the images before and after normalizing. For more information on intensity normalization, especially associated with PD, we recommend reading [458] .

In conclusion, the possibility of improving the techniques exposed in this manuscript to reduce unwanted effects on images is highlighted. Despite being applied in clinical systems, they are not considered of high relevance due to the limited improvement they provide or, sometimes, the increase in noise they produce. Therefore, the area of absolute quantification must continue to be investigated for more accurate and faster solutions.

This review presents an overview of multimodal data fusion in the field of neuroimaging, including current developments and challenges. We first outlined the fundamental limitations of individual modalities, which can include distortion, non-quantitative nature, and limited temporal/spatial resolutions. These limitations are the general motivators for the development of multimodal neuroimaging and fusion. Multimodal neuroimaging provides more comprehensive information on pathology.

We have summarised the individual benefits and limitations of the current imaging technologies and modalities, including CT, PET, SPECT, MRI, fMRI, DWI, PWI, and MRF. Building upon the available individual techniques, we summarised current development and application of multimodal neuroimaging and fusion in terms of neurological disorders and brain diseases, with a focus in three areas: developing brains, degenerative brains, and psychiatric disorders. The utilization of multiple modalities helps in clinical diagnosis, prevention of misdiagnosis, progression analysis, and research-oriented studies that allow us to gain a more in-depth understanding of human brain pathologies. Nowadays the effects of COVID in the human body are not well-known and maybe we could find in the future some people affected in the brain structure (micro ictus or ischemia). Nevertheless, the fusion techniques and strategies discussed in this survey may be transferred to COVID-19 multimodality image analysis [459] .

The forms of multimodality fusion include multi-modal, multi-focus, multi-temporal, and multi-view. These forms combine images from different instruments/acquisitions, acquisition focal lengths, time of acquisitions and conditions, respectively. Fusion rules were specified with respect to its components and three levels of fusion and theoretical foundation, i.e. rules derived from fuzzy logic, statistic models, and the human vision system. Then we summarised conventional and novel image decomposition and reconstruction methods for the fusion process, including methods based on RGB-IHS, pyramid representations, multi-resolution analysis, sparse representation, and salient features. In addition, we summarised both subjective and objective methods for fusion quality assessment.

The major benefits of multimodal data fusion in neuroimaging include distortion correction, higher temporal/spatial resolution, the combination of structural and functional information. We summarised these benefits with the current applications of multimodal fusion, e.g. MR-PET, EEG-fMRI, and EPI correction. This review also adds a particular focus on the application of multimodal image fusion in standardization, via atlas-based anatomical brain segmentation and the use of multi-atlas fusion. In addition, we summarised the effect of multimodal data fusion on the shift of neuroimaging diagnosis from qualitative analysis to quantitative evaluation, with an example of the effect of multimodal data in photon attenuation, scatter, and partial volume effects of PET and SPECT quantification.

Modern neuroimaging has seen significant improvements in acquisition quality and a constant increase in the abundance of imaging modalities. The fusion of modalities combines complementary information, expands resolution limits, provides standardization, and improves data quality.

It is expected that the effect of multimodal imaging fusion could also effectively scale the amount and quality of information accessible to radiologists for more both precise diagnosis and higher quality research in multiple aspects.

Imaging in multiple sclerosis

Emotional and behavioral symptoms in neurodegenerative disease: a model for studying the neural bases of psychopathology

The genetic epidemiology of neurodegenerative disease

MR image features predicting hemorrhagic transformation in acute cerebral infarction: a multimodal study

Magnetic resonance imaging to visualize stroke and characterize stroke recovery: a review

A multimodal MRI approach to identify and characterize microstructural brain changes in neuropsychiatric systemic lupus erythematosus

Functional magnetic resonance imaging outcomes from a comprehensive magnetic resonance study of children with fetal alcohol spectrum disorders

Multimodal fusion of brain imaging data: A key to finding the missing link(s) in complex mental illness

Multimodal Neuroimaging: Basic Concepts and Classification of Neuropsychiatric Diseases

General overview on the merits of multimodal neuroimaging data fusion

PET/SPECT/MRI Multimodal Nanoparticles, in Design and applications of nanoparticles in biomedical imaging

From PET/CT to PET/MRI: advances in instrumentation and clinical applications

Optical imaging: current applications and future directions

Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI)

Magnetoencephalography: from SQUIDs to neuroscience: neuroimage 20th anniversary special edition

Towards the utilization of EEG as a brain imaging tool

Regional differences in the coupling of cerebral blood flow and oxygen metabolism changes in response to activation: implications for BOLD-fMRI

A review of calibrated blood oxygenation level-dependent (BOLD) methods for the measurement of task-induced changes in brain oxygen metabolism

A calibration method for quantitative BOLD fMRI based on hyperoxia

Calibrated fMRI

Arterial spin labeling (ASL) fMRI: advantages, theoretical constrains and experimental challenges in neurosciences

Image distortion correction in EPI: comparison of field mapping with point spread function mapping

Conventional vs invert-grayscale X-ray for diagnosis of pneumothorax in the emergency setting

Application of high-resolution CT images information in complicated infection of lung tumors

Application of CT combined with electrocardiographic gating in hypertensive patients with brain and nerve diseases

Novel 3D printing technology for CT phantom coronary arteries with high geometrical accuracy for biomedical imaging applications

MYO-MRI diagnostic protocols in genetic myopathies

Comparison of the clinical application value of mo-targeted X-ray, color doppler ultrasound and MRI in preoperative comprehensive evaluation of breast cancer

Magnetic Fe3O4@PVP nanotubes with high heating efficiency for MRI-guided magnetic hyperthermia applications

Reconfigurable MRI technology for low-SAR imaging of deep brain stimulation at 3T: Application in bilateral leads, fully-implanted systems, and surgically modified lead trajectories

MRI of plants and foods

Unilateral sensorineural hearing loss identification based on double-density dual-tree complex wavelet transform and multinomial logistic regression

Mechanisms of contrast enhancement in magnetic resonance imaging

Magnetic properties and magnetization reversal process in (Pt/CoFe/MgO)10 multilayers at low temperature

Single-subject manual independent component analysis and resting state fMRI connectivity outcomes in patients with juvenile absence epilepsy. Magnetic Resonance Imaging

The role of ongoing neuronal activity for baseline and stimulus-induced BOLD signals in the rat hippocampus

Lamperti transformation of scaled Brownian motion and related Langevin equations

Denoising arterial spin labeling perfusion MRI with deep machine learning

Magnetic resonance fingerprinting

Functional brain development in humans

Dynamic mapping of human cortical development during childhood through early adulthood

The developing mind : how relationships and the brain interact to shape who we are

Maternal antenatal anxiety and behavioural/emotional problems in children: a test of a programming hypothesis

Diet in the early years of life influences cognitive outcomes at 10 years: a prospective cohort study

White matter development and early cognition in babies and toddlers. Hum Brain Mapp

Modeling healthy male white matter and myelin development: 3 through 60months of age

Multi-parametric evaluation of the white matter maturation

Central nervous system abnormalities assessed with prenatal magnetic resonance imaging

Techniques and methods in pediatric magnetic resonance imaging

MRI of normal brain maturation

Brain development, gender and IQ in children. A volumetric imaging study

Late childhood changes in brain morphology observable with MRI

Processing of structural neuroimaging data in young children: Bridging the gap between current practice and state-of-the-art methods

Factors Associated With Successful MRI Scanning in Unsedated Young Children

MR fingerprinting enables quantitative measures of brain tissue relaxation times and myelin water fraction in the first five years of life

Magnetic Resonance Fingerprinting to Characterize Childhood and Young Adult Brain Tumors

The diagnosis and management of mild cognitive impairment: a clinical review

Prevalence of mild cognitive impairment is higher in men. The Mayo Clinic Study of Aging

Alzheimer disease: new concepts on its neurobiology and the clinical role imaging will play

Increased risk of falling in older community-dwelling women with mild cognitive impairment

Hippocampal metabolic abnormalities in mild cognitive impairment and Alzheimer's disease

Current concepts in mild cognitive impairment

Use of magnetic resonance imaging to identify mild cognitive impairment: who should be imaged?

Brain volumes and their ratios in Alzheimer s disease on magnetic resonance imaging segmented using Freesurfer 6.0

Hippocampal volume and hippocampal angle (a more practical marker) in mild cognitive impairment: A case-control magnetic resonance imaging study

Volumetric MRI study of orbito-frontal cortex and thalamus in obsessive-compulsive personality disorder

Correlating Cognitive Decline with White Matter Lesion and Brain Atrophy Magnetic Resonance Imaging Measurements in Alzheimer's Disease

Inhibition of eukaryotic initiation factor 3B suppresses proliferation and promotes apoptosis of chronic myeloid leukemia cells

Linear measures of temporal lobe atrophy on brain magnetic resonance imaging (MRI) but not visual rating of white matter changes can help discrimination of mild cognitive impairment (MCI) and Alzheimer's disease (AD)

Volumetric and visual rating of magnetic resonance imaging scans in the diagnosis of amnestic mild cognitive impairment and Alzheimer's disease

Magnetic resonance imaging in Alzheimer's disease and mild cognitive impairment

Prediction of Progressive Mild Cognitive Impairment by Multi-Modal Neuroimaging Biomarkers

Functional magnetic resonance imaging with encoding task in patients with mild cognitive impairment and different severity of leukoaraiosis

A survey on applications and analysis methods of functional magnetic resonance imaging for Alzheimer's disease

Brain proton magnetic resonance spectroscopy in Alzheimer disease: changes after treatment with xanomeline

Conversion from mild cognitive impairment to probable Alzheimer's disease predicted by brain magnetic resonance spectroscopy

Alzheimer's disease and mild cognitive impairment in a community-based survey

Glutamine and Glutamate Complex, as Measured by Functional Magnetic Resonance Spectroscopy, Alters During Face-Name Association Task in Patients with Mild Cognitive Impairment and Alzheimer's Disease

Glutamatergic response to a low load working memory paradigm in the left dorsolateral prefrontal cortex in patients with mild cognitive impairment: a functional magnetic resonance spectroscopy study

Reduced Hippocampal Glutamate and Posterior Cingulate N-Acetyl Aspartate in Mild Cognitive Impairment and Alzheimer's Disease Is Associated with Episodic Memory Performance and White Matter Integrity in the Cingulum: A Pilot Study

Neurometabolites and associations with cognitive deficits in mild cognitive impairment: a magnetic resonance spectroscopy study at 7 Tesla

Proton MRS in mild cognitive impairment

Analysis of the posterior cingulate cortex with [18F]FDG-PET and Naa/mI in mild cognitive impairment and Alzheimer's disease: Correlations and differences between the two methods

Anosognosia for memory deficits in mild cognitive impairment: Insight into the neural mechanism using functional and molecular imaging

Precuneus and Cingulate Cortex Atrophy and Hypometabolism in Patients with Alzheimer's Disease and Mild Cognitive Impairment: MRI and (18)F-FDG PET Quantitative Analysis Using FreeSurfer

Brain PET in the diagnosis of Alzheimer's disease

Early detection of Alzheimer's disease using PiB and FDG PET

Alzheimer's disease and the amyloid-beta peptide

Prediction and characterization of protein-protein interaction networks in swine

Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer's disease

The impact of marine shipping and its DECA control on air quality in the Pearl River Delta

A Graph-Based Integration of Multimodal Brain Imaging Data for the Detection of Early Mild Cognitive Impairment (E-MCI)

Accurate multimodal probabilistic prediction of conversion to Alzheimer's disease in patients with mild cognitive impairment

Multiparametric graph theoretical analysis reveals altered structural and functional network topology in Alzheimer's disease

Episodic memory in normal aging and Alzheimer disease: Insights from imaging and behavioral studies

Are faces special in Alzheimer's disease? Cognitive conceptualisation, neural correlates, and diagnostic relevance of impaired memory for faces and names

Alzheimer's Disease and Exercise: A Literature Review

Alzheimer's disease facts and figures

An empirical comparison of surface-based and volume-based group studies in neuroimaging

Hippocampal atrophy rates in Alzheimer disease: added value over whole brain volume measures

Hippocampal CA1 apical neuropil atrophy in mild Alzheimer disease visualized with 7-T MRI

No Association of Lower Hippocampal Volume With Alzheimer's Disease Pathology in Late-Life Depression

Can multi-modal neuroimaging evidence from hippocampus provide biomarkers for the progression of amnestic mild cognitive impairment? Neurosci Bull

Resting functional connectivity reveals residual functional activity in Alzheimer's disease

Predicting regional neurodegeneration from the healthy brain functional connectome

Disrupted functional brain connectome in individuals at risk for Alzheimer's disease

Aberrant default mode network in subjects with amnestic mild cognitive impairment using resting-state functional MRI

Current understanding of magnetic resonance imaging biomarkers and memory in Alzheimer's disease

Multimodality Imaging Approach in Alzheimer disease. Part I: Structural MRI, Functional MRI, Diffusion Tensor Imaging and Magnetization Transfer Imaging. Dement Neuropsychol

Neuroimaging markers of global cognition in early Alzheimer's disease: A magnetic resonance imaging-electroencephalography study

Technical note: can resting state functional MRI assist in routine clinical diagnosis? BJR Case Rep

Convolution neural network-based Alzheimer's disease classification using hybrid enhanced independent component analysis based segmented gray matter of T2 weighted magnetic resonance imaging with clinical valuation

Multimodal Magnetic Resonance Imaging Data Fusion Reveals Distinct Patterns of Abnormal Brain Structure and Function in Catatonia

Multiphysics and multiscale modelling, data-model fusion and integration of organ physiology in the clinic: ventricular cardiac mechanics. Interface Focus

Multi-modal data fusion using source separation: Two effective models based on ICA and IVA and their properties

Parkinson's disease: A Review from the Pathophysiology to Diagnosis, New Perspectives for Pharmacological Treatment

Incidence and remaining lifetime risk of Parkinson disease in advanced age

The incidence rates and risk factors of Parkinson disease in patients with psoriasis: A nationwide population-based cohort study

Incidence of Parkinson's disease: variation by age, gender, and race/ethnicity

Neuroimaging advances in Parkinson's disease with freezing of gait: A systematic review

The role of magnetic resonance imaging in the diagnosis of Parkinson's disease: a review

MRI atrophy parameters related to cognitive and motor impairment in Parkinson's disease. Neurologia

Three-dimensional neuromelanin-sensitive magnetic resonance imaging of the substantia nigra in Parkinson's disease

Neuromelanin-sensitive MRI of the substantia nigra: An imaging biomarker to differentiate essential tremor from tremor-dominant Parkinson's disease

Combined Visualization of Nigrosome-1 and Neuromelanin in the Substantia Nigra Using 3T MRI for the Differential Diagnosis of Essential Tremor and de novo Parkinson's Disease. Front Neurol

Comprehensive MRI quantification of the substantia nigra pars compacta in Parkinson's disease

Quantitative susceptibility mapping (QSM): Decoding MRI data for a tissue magnetic biomarker

Imaging of Motor Cortex Physiology in Parkinson's Disease

Functional neuroimaging in Parkinson's disease. Cold Spring Harb Perspect Med

L-Dopa Modulation of Brain Connectivity in Parkinson's Disease Patients: A Pilot EEG-fMRI Study

Sensorimotor connectivity in Parkinson's disease: the role of functional neuroimaging. Front Neurol

Resting-state functional connectivity associated with mild cognitive impairment in Parkinson's disease

Glucose metabolism in small subcortical structures in Parkinson's disease

Functional imaging of deep brain stimulation in idiopathic Parkinson's disease

Resting regional cerebral glucose metabolism in advanced Parkinson's disease studied in the off and on conditions with [(18)F]FDG-PET

Imaging analysis of Parkinson's disease patients using SPECT and tractography

Network degeneration in Parkinson's disease: multimodal imaging of nigro-striato-cortical dysfunction

Multimodal Imaging Signatures of Parkinson's Disease

Mental disorders affect one in four people

Global Burden of Disease and the Impact of Mental and Addictive Disorders

The burden of mental disorders

Mental Illness Will Cost the World $16 USD Trillion by 2030

Neuroimaging in Psychiatry: A Quarter Century of Progress

Neuroimaging in psychiatry

Understanding MRI in clinical psychiatry: perspectives from neuroimaging psychiatry registrars

The Value of Neuroimaging Techniques in the Translation and Trans-Diagnostic Validation of Psychiatric Diagnoses -Selective Review

Promising Neuroimaging Biomarkers in Depression

Rate and predictors of conversion from unipolar to bipolar disorder: A systematic review and meta-analysis

Early Intervention in Bipolar Disorder

Epidemiology of women and depression

Why does the burden of disease persist? Relating the burden of anxiety and depression to effectiveness of treatment

Brain Structural Signatures of Adolescent Depressive Symptom Trajectories: A Longitudinal Magnetic Resonance Imaging Study. J Am Acad Child Adolesc Psychiatry

A magnetic resonance imaging study of hippocampal, amygdala and subgenual prefrontal cortex volumes in major depression subtypes: melancholic versus psychotic depression

Myelination of the brain in Major Depressive Disorder: An in vivo quantitative magnetic resonance imaging study

Multimodal imaging reveals a complex pattern of dysfunction in corticolimbic pathways in major depressive disorder. Hum Brain Mapp

Baseline brain perfusion and brain structure in patients with major depression: a multimodal magnetic resonance imaging study

Altered hippocampal function in major depression despite intact structure and resting perfusion

Development and evaluation of a multimodal marker of major depressive disorder

Multimodal fusion of structural and functional brain imaging in depression using linked independent component analysis. Hum Brain Mapp

Widespread decreased grey and white matter in paediatric obsessive-compulsive disorder (OCD): a voxel-based morphometric MRI study

Brain changes in children and adolescents with obsessive-compulsive disorder before and after treatment: a voxel-based morphometric MRI study

Abnormal regional spontaneous neuronal activity associated with symptom severity in treatment-naive patients with obsessive-compulsive disorder revealed by resting-state functional MRI

Cerebral activation in children and adolescents with obsessive-compulsive disorder before and after treatment: a functional MRI study

Investigating the predictive value of different resting-state functional MRI parameters in obsessive-compulsive disorder

Metabolic abnormality in the right dorsolateral prefrontal cortex in patients with obsessive-compulsive disorder: proton magnetic resonance spectroscopy

Abnormalities in metabolite concentrations in tourette's disorder and obsessive-compulsive disorder-A proton magnetic resonance spectroscopy study

Proton magnetic resonance spectroscopy in obsessive-compulsive disorder: evidence for reduced neuronal integrity in the anterior cingulate

A critical review of magnetic resonance spectroscopy studies of obsessive-compulsive disorder

Investigation of anterior cingulate cortex gamma-aminobutyric acid and glutamate-glutamine levels in obsessive-compulsive disorder using magnetic resonance spectroscopy

Brain Gamma-Aminobutyric Acid (GABA) Concentration of the Prefrontal Lobe in Unmedicated Patients with Obsessive-Compulsive Disorder: A Research of Magnetic Resonance Spectroscopy. Shanghai Arch Psychiatry

Reduced anterior cingulate glutamate in pediatric major depression: A magnetic resonance spectroscopy study

Proton magnetic resonance spectroscopy in pediatric obsessive-compulsive disorder: Longitudinal study before and after treatment

The effect of behavior therapy on caudate N-acetyl-l-aspartic acid in adults with obsessive-compulsive disorder

Modality-specific overlaps in brain structure and function in obsessive-compulsive disorder: Multimodal meta-analysis of case-control MRI studies

The neural correlates of obsessive-compulsive disorder: a multimodal perspective

Morphometric alterations of anterior superior temporal cortex in obsessive-compulsive disorder

Spontaneous neural activity in the right superior temporal gyrus and left middle temporal gyrus is associated with insight level in obsessive-compulsive disorder

Diagnostic neuroimaging markers of obsessive-compulsive disorder: Initial evidence from structural and functional MRI studies

An MRI Study of the Metabolic and Structural Abnormalities in Obsessive-Compulsive Disorder

Schizophrenia-An Overview

A meta-analytic review of the relationship between neurocognition, metacognition and functional outcome in schizophrenia

Schizophrenia in a genomic era: a review from the pathogenesis, genetic and environmental etiology to diagnosis and treatment insights

Just the Facts": what we know in 2008 part 1: overview

Subcortical brain volume abnormalities in 2028 individuals with schizophrenia and 2540 healthy controls via the ENIGMA consortium

Brain volume changes in first-episode schizophrenia: a 1-year follow-up study

Brain structural abnormalities at the onset of schizophrenia and bipolar disorder: a meta-analysis of controlled magnetic resonance imaging studies

Progressive loss of cortical gray matter in schizophrenia: a meta-analysis and meta-regression of longitudinal MRI studies

Mapping adolescent brain change reveals dynamic wave of accelerated gray matter loss in very early-onset schizophrenia

Diffusion Imaging of White Matter In Schizophrenia: Progress and Future Directions

White matter fibertracking in first-episode schizophrenia, schizoaffective patients and subjects at ultra-high risk of psychosis

The corpus callosum in first episode schizophrenia: a diffusion tensor imaging study

Widespread white matter microstructural differences in schizophrenia across 4322 individuals: results from the ENIGMA Schizophrenia DTI Working Group

Brain structure, function, and neurochemistry in schizophrenia and bipolar disorder-a systematic review of the magnetic resonance neuroimaging literature

Default mode functional connectivity is associated with social functioning in schizophrenia

Anatomical insights into disrupted small-world networks in schizophrenia

Levels of glutamatergic neurometabolites in patients with severe treatment-resistant schizophrenia: a proton magnetic resonance spectroscopy study

Glutamatergic Neurometabolite Levels in Patients With Ultra-Treatment-Resistant Schizophrenia: A Cross-Sectional 3T Proton Magnetic Resonance Spectroscopy Study

Proton magnetic resonance spectroscopy and illness stage in schizophrenia--a systematic review and meta-analysis

In search of multimodal neuroimaging biomarkers of cognitive deficits in schizophrenia

A Longitudinal Multimodal Neuroimaging Study to Examine Relationships Between Resting State Glutamate and Task Related BOLD Response in Schizophrenia

Multimodal neuroimaging as a window into the pathological physiology of schizophrenia: Current trends and issues

Multimodal Neuroimaging in Schizophrenia: Description and Dissemination

Comparison of cerebral blood flow acquired by simultaneous [15O] water positron emission tomography and arterial spin labeling magnetic resonance imaging

Present and future of simultaneous EEG-fMRI. Magnetic Resonance Materials in Physics

Simultaneous EEG and fMRI of the alpha rhythm

EEG-correlated fMRI of human alpha activity

simultaneous EEG-fMRI

Assignment of functional activations to probabilistic cytoarchitectonic areas revisited. Neuroimage

Accuracy of in vivo neuroreceptor quantification by PET and review of steady-state, transient, double injection, and equilibrium models

Distortion correction in EPI at ultra-high-field MRI using PSF mapping with optimal combination of shift detection dimension. Magnetic resonance in medicine

How to correct susceptibility distortions in spin-echo echo-planar images: application to diffusion tensor imaging

A new intensity-hue-saturation fusion approach to image fusion with a tradeoff parameter

Survey on multifocus image fusion techniques

Discrete Cosine Transform based fusion of multi-focus images for visual sensor networks

Multi-focus image fusion based on spatial frequency in discrete cosine transform domain. IEEE signal processing letters

Multifocus image fusion using region segmentation and spatial frequency. Image and vision computing

Multi-focus image fusion with dense SIFT. Information Fusion

Contrast-based image fusion using the discrete wavelet transform

MHWT-A Modified Haar Wavelet Transformation for Image Fusion

The Laplacian pyramid as a compact image code

Discrete wavelet transform based principal component averaging fusion for medical images

Fusion of multimodal medical images using Daubechies complex wavelet transform-A multiresolution approach. Information Fusion

Simultaneous intracranial EEG and fMRI of interictal epileptic discharges in humans

A review of multivariate methods for multimodal fusion of brain imaging data

Brain image

Cross-scale coefficient selection for volumetric medical image fusion

Pixel-and region-based image fusion with complex wavelets. Information fusion

Principal component analysis in medical image processing: a study

Iterative block level principal component averaging medical image fusion

Multi-mode medical image fusion algorithm based on principal component analysis

Medical Image Fusion Using Combination of PCA and Wavelet Analysis

EGGDD: An explicit dependency model for multi-modal medical image fusion in shift-invariant shearlet transform domain. Information Fusion

Image fusion using the expectation-maximization algorithm and a hidden Markov model

Contourlet hidden Markov Tree and clarity-saliency driven PCNN based remote sensing images fusion

Human visual system inspired multi-modal medical image fusion framework. Expert Systems with Applications

MRI and PET image fusion by combining IHS and retina-inspired models. Information Fusion

Contrast-enhanced fusion of multisensor images using subband-decomposed multiscale retinex

SUSAN-a new approach to low level image processing

Multimodal medical image fusion based on IHS and PCA. Procedia Engineering

A new metric based on extended spatial frequency and its application to DWT based fusion algorithms. Information Fusion

Medical image fusion method based on wavelet multi-resolution and entropy

Multilevel medical image fusion using segmented image by level set evolution with region competition

Medical image fusion by multi-resolution analysis of wavelets transform

Directive contrast based multimodal medical image fusion in NSCT domain

Investigation of Remote Sensing Image Fusion Strategy Applying PCA to Wavelet Packet Analysis Based on IHS Transform

An intensity factorized thresholding based segmentation technique with gradient discrete wavelet fusion for diagnosing stroke and tumor in brain MRI. Multidimensional Systems and Signal Processing

Detection of Alzheimer's disease and mild cognitive impairment based on structural volumetric MR images using 3D-DWT and WTA-KSVM trained by PSOTVAC

CT and MR Images Fusion Based on Stationary Wavelet Transform by Modulus Maxima, in Computational Vision and Robotics

Multi-focal Image Fusion with Convolutional Sparse Representation and Stationary Wavelet Transform. in Computing, Communication and Signal Processing (ICCASP)

Detection of Dendritic Spines using Wavelet Packet Entropy and Fuzzy Support Vector Machine

Preclinical diagnosis of magnetic resonance (MR) brain images via discrete wavelet packet transform with Tsallis entropy and generalized eigenvalue proximal support vector machine (GEPSVM)

Wavelet Packet Transform Based Fusion of Misaligned Images. International Conference on Circuit, Power and Computing Technologies

Novel Data-Access Scheme and Efficient Parallel Architecture for Multi-level Lifting 2-D DWT. Circuits Systems and Signal Processing

Wavelet-based lifting scheme for the numerical solution of some class of nonlinear partial differential equations

Multiscale fusion of multimodal medical images using lifting scheme based biorthogonal wavelet transform

MRI image compression using level set method and biorthogonal CDF wavelet based on lifting scheme

Nonsubsampled contourlet transform and k-means clustering for degraded document image binarization

An improved multimodal medical image fusion scheme based on hybrid combination of nonsubsampled contourlet transform and stationary wavelet transform

A Practical Medical Image Enhancement Algorithm Based on Nonsubsampled Contourlet Transform

Multi-Modality Anatomical and Functional Medical Image Fusion Based on Simplified-Spatial Frequency-Pulse Coupled Neural Networks and Region Energy-Weighted Average Strategy in Non-Sub Sampled Contourlet Transform Domain

A Novel Medical Image Fusion Approach Based on Nonsubsampled Shearlet Transform

Image Fusion Using Adjustable Non-subsampled Shearlet Transform

Medical image fusion based on nonsubsampled shearlet transform and principal component averaging

Pixel-level image fusion with simultaneous orthogonal matching pursuit. Information Fusion

Multimodal image fusion with joint sparsity model

Image Features Extraction and Fusion Based on Joint Sparse Representation

Medical image fusion using multi-level local extrema. Information Fusion

Infrared and Visible Image Fusion Based on Contrast Enhancement and Multi-scale Edge-preserving Decomposition

Edge-preserving smoothing pyramid based multi-scale exposure fusion

Subjective tests for image fusion evaluation and objective metric validation. Information Fusion

Image information and visual quality

User models of subjective image quality assessment on virtual viewpoint in free-viewpoint video system

An overview of multi-modal medical image fusion

An overview of multi-modal medical image fusion

Contourlet-based image quality assessment for synthesised virtual image

Image quality assessment: From error visibility to structural similarity

A novel algorithm of image fusion using shearlets

Comments on 'Information measure for performance of image fusion

Entropy and correlation

Image quality measures and their performance

Making a "Completely Blind

The current state, challenges and perspectives of MR-PET

Simultaneous MR/PET imaging of the human brain: feasibility study. Radiology

APD-based PET for combined MR-PET imaging

Comparative analysis of ferumoxytol and gadoteridol enhancement using T1-and T2-weighted MRI in neuroimaging

Tissue characterization with T1, T2, and proton density values: results in 160 patients with brain tumors

Alcoholism Identification Based on an AlexNet Transfer Learning Model

Novel imaging techniques to study postmortem human fetal anatomy: a systematic review on microfocus-CT and ultra-high-field MRI

Ultra-High Field MRI in Alzheimer's Disease: Effective Transverse Relaxation Rate and Quantitative Susceptibility Mapping of Human Brain In Vivo and Ex Vivo compared to Histology

Introduction to functional magnetic resonance imaging: principles and techniques

Present and future of simultaneous EEG-fMRI. Magnetic Resonance Materials in Physics

A personalized history of EEG-fMRI integration

Off-resonance frequency filtered magnetic resonance imaging. Magnetic resonance imaging

Multimodal non-rigid warping for correction of distortions in functional MRI

Efficient correction of inhomogeneous static magnetic field-induced distortion in Echo Planar Imaging

From simultaneous to synergistic MR-PET brain imaging: A review of hybrid MR-PET imaging methodologies

MR-based PET motion correction procedure for simultaneous MR-PET neuroimaging of human brain

Joint reconstruction of PET-MRI by exploiting structural similarity. Inverse Problems

Multi-atlas segmentation of biomedical images: A survey

Brain templates and atlases

Atlas d'anatomie stereotaxique du telencephale: etudes anatomo-radiologiques

Tournoux (1988) Co-planar stereotaxic atlas of the human brain. 3-dimensional proportional system: an approach to cerebral imaging

Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain

Design and construction of a realistic digital brain phantom

Automatic anatomical brain MRI segmentation combining label propagation and decision fusion

Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm

Unified segmentation

A review of atlas-based segmentation for magnetic resonance brain images

Voxel-based morphometry -The methods

Medical image registration

Evaluation of atlas selection strategies for atlas-based image segmentation with application to confocal microscopy images of bee brains

Mindboggle: automated brain labeling with multiple atlases

Combination Strategies in Multi-Atlas Image Segmentation: Application to Brain MR Data

Performance-based classifier combination in atlas-based image segmentation using expectation-maximization parameter estimation

Multi-classifier framework for atlas-based image segmentation

Multimodality image registration by maximization of mutual information

Combining atlas based segmentation and intensity classification with nearest neighbor transform and accuracy weighted vote

Multi-atlas multi-shape segmentation of fetal brain MRI for volumetric and morphometric analysis of ventriculomegaly

Weighted Shape-Based Averaging With Neighborhood Prior Model for Multiple Atlas Fusion-Based Medical Image Segmentation

An empirical study of binary classifier fusion methods for multiclass classification. Information Fusion

Data fusion and multiple classifier systems for human activity detection and health monitoring: Review and open research directions. Information Fusion

Score level fusion of classifiers in off-line signature verification. Information Fusion

Fusion of multiple approximate nearest neighbor classifiers for fast and efficient classification. Information Fusion

Robust Ensemble Classification Methodology for I123-Ioflupane SPECT Images and Multiple Heterogeneous Biomarkers in the Diagnosis of Parkinson's Disease

Ensemble of random forests One vs. Rest classifiers for MCI and AD prediction using ANOVA cortical and subcortical feature selection and partial least squares

Combination of multiple classifiers using local accuracy estimates

IMAGING OF NEUROTRANSMITTER RECEPTORS IN THE LIVING HUMAN-BRAIN. Archives of General Psychiatry

Direct attenuation correction of brain PET images using only emission data via a deep convolutional encoder-decoder (Deep-DAC)

PET studies in epilepsy

History and future technical innovation in positron emission tomography

Dual-Modality Imaging: More Than the Sum of its Components

PET/MRI attenuation estimation in the lung: A review of past, present, and potential techniques

Overview of Nuclear Medical Imaging: Physics and Instrumentation

PET quantification: strategies for partial volume correction

Quantitative SPECT: the time is now. Ejnmmi Physics

The use of PET in Alzheimer disease

Brain imaging: Applications of tau PET imaging

Impact of Training Method on the Robustness of the Visual Assessment of 18F-Florbetaben PET Scans: Results from a Phase-3 Study

A Semiautomated Method for Quantification of F 18 Florbetapir PET Images

An Automated Pipeline for the Analysis of PET Data on the Cortical Surface

A Fully Automatic Technique for Precise Localization and Quantification of Amyloid-beta PET Scans

A review on segmentation of positron emission tomography images

Standardized uptake values of normal tissues at PET with 2-[fluorine-18]-fluoro-2-deoxy-D-glucose: variations with body weight and a method for correction

Standardized uptake values of FDG: body surface area correction is preferable to body weight correction

Quantitative Techniques in PET-CT Imaging

Variability in PET quantitation within a multicenter consortium

Improved correction for the tissue fraction effect in lung PET/CT imaging

Resolution modeling in PET imaging: theory, practice, benefits, and pitfalls

Quantitative SPECT/CT: SPECT joins PET as a quantitative imaging modality

Absolute quantification in SPECT

Partial volume correction for PET quantification and its impact on brain network in Alzheimer's disease. Scientific Reports

Positron emission tomography partial volume correction: Estimation and algorithms

Partial-volume effect in PET tumor imaging

Correction for Partial Volume Effects in Emission Tomography

Partial Volume Correction Strategies in PET

A review of partial volume correction techniques for emission tomography and their applications in neurology, cardiology and oncology

Regional correction of positron emission tomography data for the effects of cerebral atrophy

Correction of PET data for partial volume effects in human cerebral-cortex by MR imaging

Measurement of radiotracer concentration in brain gray-matter using positron emission tomography -MRI-based correction for partial volume effects

MR-Based correction of brain PET measurements for heterogeneous gray matter radioactivity distribution

Iterative Reconstruction Methods, in Quantitative Analysis in Nuclear Medicine Imaging

A recovery coefficient method for partial volume correction of PET images

PET/MRI for Neurologic Applications

A new fast algorithm for the evaluation of regions of interest and statistical uncertainty in computed-tomography

A method to correct for scatter, spillover, and partial volume effects in region of interest analysis in PET

A maximum likelihood method for region-of-interest evaluation in emission tomography

Correction for partial volume effects in PET: Principle and validation

Correction of partial-volume effect for PET striatal imaging: Fast implementation and study of robustness

Partial volume effect compensation for quantitative brain SPECT imaging

Symmetric geometric transfer matrix partial volume correction for PET imaging: principle, validation and robustness

Impact of image-space resolution modeling for studies with the high-resolution research tomograph

Improvement in PET/CT Image Quality with a Combination of Point-Spread Function and Time-of-Flight in Relation to Reconstruction Parameters

Clinical evaluation of PET image reconstruction using a spatial resolution model

Influence of the partial volume correction method on F-18-fluorodeoxyglucose brain kinetic modelling from dynamic PET images reconstructed with resolution model based OSEM

Deconvolution Microscopy, in Microscopy Techniques

A multiresolution image based approach for correction of partial volume effects in emission tomography

Integrated software for the analysis of brain PET/SPECT studies with partial-volume-effect correction

MR-based automatic delineation of volumes of interest in human brain PET images using probability maps

Attenuation Correction Strategies in Emission Tomography

Vision 20/20: Magnetic resonance imaging-guided attenuation correction in PET/MRI: Challenges, solutions, and opportunities

Towards quantitative PET/MRI: a review of MR-based attenuation correction techniques

Attenuation compensation in cerebral 3D PET: effect of the attenuation map on absolute and relative quantitation

Automated PET attenuation correction model for functional brain imaging

Acquisition of attenuation map for brain PET study using optical tracking system

Simultaneous maximum a posteriori reconstruction of attenuation and activity distributions from emission sinograms

Completion of a Truncated Attenuation Image From the Attenuated PET Emission Data

ML-Reconstruction for TOF-PET With Simultaneous Estimation of the Attenuation Factors

Optimized MLAA for quantitative non-TOF PET/MR of the brain

A multi-centre evaluation of eleven clinically feasible brain PET/MRI attenuation correction techniques using a large cohort of patients

Transmission scanning in emission tomography

Evaluation of SPET quantification of simultaneous emission and transmission imaging of the brain using a multidetector SPET system with the TEW scatter compensation method and fan-beam collimation

Nonuniform transmission in brain SPECT using 201Tl, 153Gd, and 99mTc static line sources: anthropomorphic dosimetry studies and influence on brain quantification

Investigation of the relationship between linear attenuation coefficients and CT Hounsfield units using radionuclides for SPECT. Applied Radiation and Isotopes

Image fusion using an integrated, dual-head coincidence camera with x-ray tube-based attenuation maps

Impact of metallic dental implants on CT-based attenuation correction in a combined PET/CT scanner

X-ray-based attenuation correction for positron emission tomography/computed tomography scanners

Method for transforming CT images for attenuation correction in PET/CT imaging

Evaluation of an Atlas-Based PET Head Attenuation Correction Using PET/CT & MR Patient Data

Scatter and attenuation correction for brain SPECT using attenuation distributions inferred from a head atlas

Magnetic resonance imaging-guided attenuation and scatter corrections in three-dimensional brain positron emission tomography

Knowledge-based Segmentation of Attenuation-relevant Regions of the Head in T1-weighted MR Images for Attenuation Correction in MR/PET Systems

Quantitative Evaluation of Atlas-based Attenuation Correction for Brain PET in an Integrated Time-of-Flight PET/MR Imaging System

Evaluation of MLACF based calculated attenuation brain PET imaging for FDG patient studies

Joint correction of attenuation and scatter in image space using deep convolutional neural networks for dedicated brain (18)F-FDG PET

Segmented MR images for brain attenuation correction in PET

MRI-Based Attenuation Correction for PET/MRI Using Ultrashort Echo Time Sequences

Tissue Classification as a Potential Approach for Attenuation Correction in Whole-Body PET/MRI: Evaluation with PET/CT Data

MRI-Based Attenuation Correction for Hybrid PET/MRI Systems: A 4-Class Tissue Segmentation Technique Using a Combined Ultrashort-Echo-Time/Dixon MRI Sequence

Combined PET/MR imaging in neurology: MR-based attenuation correction implies a strong spatial bias when ignoring bone

Generation of MR-Based Attenuation Correction Map of PET Images in the Brain Employing Joint Segmentation of Skull and Soft-Tissue from Single Short-TE MR Imaging Modality

Generation of a Four-Class Attenuation Map for MRI-Based Attenuation Correction of PET Data in the Head Area Using a Novel Combination of STE/Dixon-MRI and FCM Clustering

Zero TE MR Bone Imaging in the Head

Evaluation of Sinus/Edge-Corrected Zero-Echo-Time-Based Attenuation Correction in Brain PET/MRI

Improving PET/MR brain quantitation with template-enhanced ZTE

Evaluation of zero-echo-time attenuation correction for integrated PET/MR brain imaging-comparison to head atlas and (68)Ge-transmission-based attenuation correction

ZTE MR-based attenuation correction in brain FDG-PET/MR: performance in patients with cognitive impairment

PET Attenuation Correction Using Synthetic CT from Ultrashort Echo-Time MR Imaging

Probabilistic atlas-based segmentation of combined T1-weighted and DUTE MRI for calculation of head attenuation maps in integrated PET/MRI scanners

Cluster-based segmentation of dual-echo ultra-short echo time images for PET/MR bone localization

Improved quality of computed tomography substitute derived from magnetic resonance (MR) data by incorporation of spatial information -potential application for MR-only radiotherapy and attenuation correction in positron emission tomography

A Method for Attenuation Correction in Radionuclide Computed Tomography

Maximum Likelihood Reconstruction for Emission Tomography

EM reconstruction algorithms for emission and transmission tomography

An attenuated projector-backprojector for iterative SPECT reconstruction

Scatter Compensation Techniques in PET

Review and current status of SPECT scatter correction

Scatter Correction Strategies in Emission Tomography

Absolute quantification in SPECT -a phantom study

Improved SPECT quantification using compensation for scattered photons

Correction for scatter in 3D brain PET using a dual energy window method

Compton scatter compensation using the triple-energy window method for single-and dual-isotope SPECT

Triple energy window scatter correction technique in PET

SPECT Compton-scattering correction by analysis of energy spectra

Energy dependence of scatter components in multispectral PET imaging

A Monte Carlo simulation study on coarse septa for scatter correction in 3-D PET

Novel scatter correction for three-dimensional positron emission tomography by use of a beam stopper device

A fast, energy-dependent scatter reduction method for 3D PET imaging

PET energy-based scatter estimation and image reconstruction with energy-dependent corrections

A convolution-subtraction scatter correction method for 3d PET

A transmission-dependent method for scatter correction in SPECT

Non-stationary convolution subtraction scatter correction with a dual-exponential scatter kernel for the Hamamatsu SHR-7700 animal PET scanner

A technique for the correction of scattered radiation in a PET system using time-of-flight information

A monte-carlo correction for the effect of compton-scattering in 3-d PET brain imaging

New, faster, image-based scatter correction for 3D PET

Optimization of a fully 3D single scatter simulation algorithm for 3D PET

Efficient fully 3-D iterative SPECT reconstruction with Monte Carlo-based scatter compensation

Absolute quantification in dopaminergic neurotransmission SPECT using a Monte Carlo-based scatter correction and fully 3-dimensional reconstruction

Fully 3D Monte Carlo reconstruction in SPECT: a feasibility study

Linear intensity normalization of FP-CIT SPECT brain images using the alpha-stable distribution

On a Heavy-Tailed Intensity Normalization of the Parkinson's Progression Markers Initiative Brain Database

Comparison between Different Intensity Normalization Methods in 123I-Ioflupane Imaging for the Automatic Detection of Parkinsonism

The role of multimodality imaging in COVID-19 patients: from diagnosis to clinical monitoring and prognosis

This study was partly supported by Royal Society International Exchanges Cost Share Award, UK Guangxi Key Laboratory of Trusted Software (kx201901).

YZ and JMG conceptualized the survey paper. All authors organized the references. SW and JMG planned the methodology. YZ and JMG administrated the review project. YZ and JMG provided resources. YZ, ZC, and JMG supervised the paper writing work. All authors validated the whole work. All authors wrote the draft. QZ revised English. All authors gave critical comments and helped improved the quality.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.