key: cord-0120188-s8ul1opk authors: Yang, Tianqi; Karakucs, Oktay; Anantrasirichai, Nantheera; Achim, Alin title: Current Advances in Computational Lung Ultrasound Imaging: A Review date: 2021-03-21 journal: nan DOI: nan sha: 18521affdcbda502c8994ab5401d550241a61bcc doc_id: 120188 cord_uid: s8ul1opk In the field of biomedical imaging, ultrasonography has become increasingly widespread, and an important auxiliary diagnostic tool with unique advantages, such as being non-ionising and often portable. This article reviews the state-of-the-art in medical ultrasound image computing and in particular its application in the examination of the lungs. First, we review the current developments in medical ultrasound technology. We then focus on the characteristics of lung ultrasonography and on its ability to diagnose a variety of diseases through the identification of various artefacts. We review medical ultrasound image processing methods by splitting them into two categories: (1) traditional model-based methods, and (2) data driven methods. For the former, we consider inverse problem based methods by focusing in particular on ultrasound image despeckling, deconvolution, and line artefacts detection. Among the data-driven approaches, we discuss various works based on deep/machine learning, which include various effective network architectures implementing supervised, weakly supervised and unsupervised learning. the patient's bedside to monitor the progress of treatment; (5) operating in real-time enabling a usage for dynamic imaging and hence boost doctor's diagnostic efficiency; (6) hygienic as the US equipment is easy to clean and disinfect, particularly in the case of hand-held devices [3] . Despite all the aforementioned advantages, US remains an auxiliary tool after all since it has limitations that cannot be easily overcome: (1) It is an operator-dependent device, and the US image quality may vary according to clinicians' skills and the precision of the device. Moreover, the interpretation of US imagery is subjective, in that the individual observation, the movement of the probe, and the parameter settings of the device affect image quality. ( 2) It can be hard to conduct US examination in restricted conditions. For example, critically ill patients need a variety of equipment (such as ventilators, blood purification tubes, chest drainage tubes, etc.) for life maintenance, which negatively affects the application of Medical Coupling Gel, and thus affects the accuracy of US examination. (3) Specific patient conditions also influence the US examination efficacy, in particular obesity, thoracic deformity, skin lesions etc. For example, in obese patients, thick subcutaneous fat may lead to serious distortion, resulting in errors in US examination. All these might influence US imagery characteristics, and affect the results of clinical examination [4] . Among the various clinical applications of US, Lung US (LUS) has become more and more important in recent years due to advances in US technology. Indeed, respiratory disease is extremely common, amounting to more than 414.6 million cases around the world [5] . Severe lung problems cause the death of more than 100 thousand people in the UK every year [6] . According to the British Lung Foundation, somebody dies due to lung disease in the UK every five minutes [6] . Lung disease is in fact the third common cause of death in the UK after heart disease and cancer [6] . Following the start of the COVID-19 pandemic in 2020, analysis and diagnosis of lung disease has become even more crucial. Rising demand has been spurred for timely and accurate diagnosis, as well as patient monitoring [7] . LUS has been increasingly used as a support tool in lung disease diagnosis. This is not only because of the advantages mentioned above, but LUS also helps in assessing the fluid status of patients in intensive care as well as in deciding management strategies for a range of conditions. Well known image quality issues affecting ultrasonography (speckle noise, low resolution), together with the peculiarities and added difficulties of LUS investigations (small acoustic window, energy dissipation in air) make the tools of computational imaging ever more important in reconstructing information of the highest quality in order to rip the benefits of LUS in a variety of clinical applications. This paper aims to provide a comprehensive review of the stateof-the-art computational LUS imaging approaches. On the one hand, we will review model-based methods, which are reasonably well studied in the relevant literature. Typically, they involve solving an inverse problem whereby a set of unknown deterministic parameters which are observed through a linear transformation and corrupted by noise are estimated [8] . Despite their ability to produce results by just utilising a single corrupted image, the main disadvantages of such approaches are that the assumed prior distributions or regularization functions may not faithfully represent image features, and the generalization ability of the models is limited. On the other hand, especially with the current advances in machine learning (ML) and artificial intelligence (AI), approaches that learn representative models and features from a (usually high ) number of training data have gained increased attention. Deep learning (DL) in particular, has brought remarkable improvements in LUS image analysis, as well as to LUS image quality. Since the current state of LUS development has not yet reached maturity, this review aims to provide inspiration and references for the development of yet more accurate and efficient computational LUS imaging approaches. The remainder of this contribution is organised as follows: We describe the basic features and clinical applications of LUS imagery in Section II. Section III presents processing methods for the reconstruction of high quality ultrasound images within the two sub-classes of model-based and datadriven methods. Furthermore, Section IV reviews model-based as well as data-driven approaches specifically developed for LUS image processing. The discussions in Section V serve the purpose of revealing future research directions by analysing current challenges, and Section VI gives a short conclusion of the reviewed work. This section is intended to provide a brief overview of medical ultrasound imaging technology, with a specific focus on LUS. We specify the devices that clinicians normally use, the standards that should be followed, and the lung artefacts that diagnosis is based on. Depending on clinical applications, working principles, tasks and operating systems, medical US images can be acquired in different modes [10] , described in Table I , where M-mode and B-mode are the two most commonly used ones in clinical assessment of lungs. Probes are the actual ultrasonic transducers, which employ the piezoelectric effect to perform the bidirectional conversion of mechanical waves into electrical signals and vice versa. A non-exhaustive list of most commonly used clinical probes is given in Table II . For the observation of in-depth lesions, lowfrequency convex array probes or low-frequency phased array probes are usually utilised, whereas high-frequency linear array probes are used for superficial pleural or sub-pleural detection. Image resolution increases with frequency, at the cost of penetration depth [12] . A number of vendors have developed LUS scanners, including the Canon Xario Series 1 , Aplio Series (by GE) 2 and Sonosite 3 . In order to satisfy eventual point-of-care (PoC) needs, many of these vendors have been developing portable devices. One such additional example is the Butterfly iQ+, which is able to perform whole body imaging with one small probe as well as provide real-time in-app monitoring. In-vitro [13] [14] [15] and clinical studies [16] have shown that the visualization of artefact patterns (mainly B-lines) is strongly dependent on imaging frequency, and the pathological sub-pleural process. Specifically, when the center frequency of the ultrasound waves matches with the geometry of the acoustic channels filled with media (tissue, water, or blood), M-mode extends the A-mode signal on the time axis to display the trajectory of the motion. This is especially suitable for the examination of the heart and other dynamic organs. B-mode B-mode uses the amplitude of the echo pulse to modulate the brightness of the display. The abscissa and ordinate of the display correspond to the position of the sound velocity scan, thus forming an ultrasonic cross-sectional image modulated by the brightness. Different types of B-mode scanners exist, including i. manual scanners, ii. real-time scanners, iii. mechanical scanners, iv. electronic scanners, with both linear-array scanners and phased-array scanners [11] . D-mode D-mode uses the Doppler effect to detect the motion information of the human tissue. The probe movement in C-mode and the synchronized scanning form a "Z" shape, and the displayed sound image is perpendicular to the direction of the sound beam. F-mode F-mode imaging is a type of C-mode imaging, which are able to reconstruct three-dimensional images. Probe Applications Electronic convex array probes These are commonly used for abdominal examinations, obstetrics and gynecology examinations. Their typical working frequencies are 3˜5MHz different ranges of frequencies are recommended to children [9] . Electronic linear array probes Their working frequencies are over 7.5MHz, being often used for shallow investigations such as peripheral blood vessels and superficial organs. Electronic phased array probes They can be used for adult and pediatric heart examinations under working frequencies of 2˜5MHz. Including transesophageal, transcatheter, and endoscopic probes, etc. They work at high frequency and can be used directly on the surface of organs or close to organs, displaying the tissue structure and avoiding the interference due to depth or other organs. B-lines appear with higher intensity in the reconstructed US images. The imaging devices and the management of the patients varies in different countries. Hence, to promote an effective global medical exchange, a standardization for the international protocol of LUS is needed. Soldati et. al. [17] had developed a standardized approach for the use of equipment and the acquisition protocol, specifically for COVID-19 patients. A scoring system for severity classification was also proposed. An artificial intelligence system was developed to support the generation of standardized datasets [7] . Another standardisation effort was made by Allinovi and Hayes [18] who demonstrated that a simplified 8-site B-lines score holds an almost identical predictive power to a 28-site score in quantifying the extent of extravascular lung water in children. The use of 8-site lung ultrasound assessments for children on dialysis was therefore proposed, to accurately quantify fluid overload in children. The lungs are the main air-containing organs of the human body. In a pathological condition, the air-liquid ratio in the lungs changes, and abnormal artifacts may appear during a US examination (see examples in Figures 1). The analysis of LUS images is based on these signs. The common features in all clinical conditions, both local to the lungs (e.g. pneumonia, chronic obstructive pulmonary disease (COPD)) and those manifesting themselves in the lungs (e.g. kidney disease, are the presence in LUS of two types of artefacts, known as A-and B-lines. These may carry important information about the severity of diseases. Hence, the majority of research conducted in this area starts from detecting and quantifying these linear features in LUS images. The A-lines (horizontal linear structures) are a kind of artifact caused by multiple reflection of sound waves due to the difference of acoustic impedance between the pleura and the lung. Their presence are indicative of a healthy lung, whereas the B-lines, appearing as a vertical comet-tail artifact arising from the pleural line and extending indefinitely, can indicate many diseases, and characteristics such as their quantity or thickness are directly related to the severity of the disease [19] , [20] . Therefore, the quantification of B-lines can help with assessing the particular disorder [21] . The position of an artifact relative to the pleural line (which is the horizontal echo reflection formed by the surface of visceral and parietal pleura) is also important in determining whether the detected artifact is a B-line or merely air or another organ [22] . Therefore, in some studies, the pleural line is used as reference for the positioning of other line artefacts [23] , [24] . Indirect signs in LUS can reveal the severity of lung disease. In Table IV , we present a non-exhaustive list of lung diseases, observable with LUS, along with their peculiarities. It is also worth noting that LUS signs have also been shown to become a good indicator of extravascular lung water [25] , and fluid overload in infants and children on dialysis when compared to other measures [26] , [27] . This is because when the center frequency of the ultrasound waves matches with the geometry of the acoustic channels filled with media (tissue, water, or blood), B-lines appear with higher intensity in the reconstructed US images [16] , so the B-lines reflects the air/fluid ration of the patient. In order to accurately identify the various signs of disease in LUS images, the reconstruction of high quality images through observations and the extraction of information therein are necessary. This is commonly posed as an inverse problem. The solution however may not be unique, or it may be highly sensitive to changes in the data, which illustrates the illposedness of the problem. In the following, we discuss the Pleural line and pleural sliding The horizontal linear high echo below the ribs constitutes the pleural line. The relative motion of the parietal pleura and the visceral pleura forms a reciprocating movement with breathing. In B-mode, A-lines are a series of hyperechoic lines that are equally spaced and parallel to the pleural line and occur below the pleural line. Quadrilateral sign and sine wave sign Below the pleural line, a line regular and roughly parallel to the pleural line (the lung line), together with the pleural line and the shadow of the ribs, display a quad sign. In M-mode, the movement of the lung line is sinusoidal [1] . B-line A vertical artifact with clear boundaries and moving synchronously with lung sliding. Lung consolidation and atelectasis The echogenic US appearance of the lungs is similar to that of the liver or spleen. Bronchial inflation sign A dividing point seen in real-time ultrasound where the lung sliding exists and disappears alternately with breathing movement. Lung points A dividing point seen in real-time ultrasound where the lung sliding exists and disappears alternately with breathing movement. As air-filled lung moves up and down with the breathing motion and other abdominal organs are covered periodically. The disappearance of the pulmonary sliding sign in the initial stage and the M-mode LUS shows a superposition of parallel lines that lack motion characteristics [1] . Varying degrees of lung consolidation and abnormal pleural lines, some with fusion of B-lines and pleural effusion. Early stage of COVID-19 pneumonia shows single and/or confluent vertical artifacts [7] , [28] , [29] . Further evolution of COVID-19 pneumonia shows evident consolidations and widespread patchy artifactual changes [30] . Chronic obstructive pulmonary disease(COPD) Pulmonary A-line, pulmonary sliding sign and no right ventricular overload [9] . Lung consolidation with bronchial inflation sign, abnormal pleural line, diffuse pulmonary edema, and disappearance of A-lines. extra-vascular lung water (EVLW) is another indicator to classify the severity of ARDS [31] . Lung cancer US can clearly captures the chest wall, pleura and peripheral lung lesions, and can show the morphology, boundary and blood flow of the lesions and the anatomical relationship between fine structures and the surrounding tissue, which provides basic information for clinical diagnosis [9] . Through their US morphologies, one can distinguish benign and malignant lung tumors. Congenital diaphragmatic hernia (CDH) i. partial absence of the hyperechoic line representing the normal diaphragmatic profile, ii. partial absence of the pleural line in the affected hemithorax, iii. absence of A lines in the affected area, iv. presence of multi-layered area with hyperechoic contents in motion (normal gut), and v. possible presence of parenchymatous organs inside the thorax (i.e., liver or spleen). ICU uses of LUS LUS can effectively detect lung consolidations with a thickness greater than 20 mm in ICU patients [1] , evaluate the respiratory system and circulatory system in real time. Suitable for detecting neonatal respiratory distress syndrome (NRDS), meconium aspiration syndrome, acute pneumothorax, occult atelectasis. most widely studied inverse problems in conjunction with medical US image analysis, which are also relevant to LUS. Approaches peculiar to LUS specifically are discussed in Section IV. A. Model-Based Methods 1) Despeckling: Generally, the final envelope detected US image is composed of two elements, the useful signal component (corresponding to structure inside the human body) and the noise component (comprising multiplicative speckle and additive measurement noise). where g(x, y) and f (x, y) represent the speckle free and the observed signals, respectively. n(x, y) and w(x, y) represent the multiplicative and additive noise components, respectively. (x, y) are the two-dimensional spatial coordinates. Since the influence of the additive noise is far less obvious than that of the multiplicative noise [32] , the image formation model (1) can usually be approximated by The term n(x, y) corresponds to speckle noise, which is an inherent phenomenon in US images. Speckle noise has been shown to be correlated with the tissue structure [32] , so its statistical description generally depends on the type of tissue and the imaging system. It exhibits granular patterns, which obscure fine anatomical details, and thereby reduce the diagnostic accuracy. Speckle noise is hence normally regarded as an undesirable phenomenon in most clinical applications. However, speckle noise may constitute useful information, such as when used for speckle tracking (i.e. motion estimation) and tissue characterization [33] . To mitigate speckle noise, a logarithmic transformation is usually employed to convert the multiplicative characteristics into an additive model. The early work described speckle noise characteristics in the wavelet domain with a heavy-tailed α-stable distributions [32] . This captures the significant non-Gaussian behaviour of US signals. Based on this statistical characterisation, they developed a blind speckle-suppression processor, with nonlinear characteristics related to the degree of non-Gaussianity of the data. Recently, Choi and Jeong [34] have improved descpeckling performance by combining various approaches, including speckle reduction anisotropic diffusion (SRAD) filter [35] , discrete wavelet transform (DWT) using symmetric characteristics, gradient domain guided image filtering (GDGIF) and weighted guided image filtering (WGIF). Firstly, the multiplicative noise in the SRAD composite image wss converted into additive noise through a homomorphic transformation. Following the application of two-level DWT decomposition, in order to suppress the residual noise of an SRAD filtered image, GDGIF and WGIF were exploited to reduce noise from seven high-frequency sub-band images and one lowfrequency sub-band image, respectively. Finally, a speckle-free image was obtained through inverse DWT and an exponential transformation. Compared with previous denoising methods, their algorithm showed superior despeckling performance, better feature information conservation, and lower computational cost. Another stat-of-the-art technique proposed by Chen et. al. [36] introduced the alternating direction multiplier (ADMM) algorithm [37] to optimize the denoising effect of the new speckle noise recovery model based on adaptive variational method. This computational framework for solving optimization problems is shown to be suitable for solving distributed convex optimization problems. It effectively reduced the speckle and achieves considerable performance in terms of visual evaluation and quantitative measurements. 2) Deconvolution: Another important post-acquisition operation often needed in ultrasonography is image deconvolution, whereby US images are modelled as a convolution between a blurring kernel or point spread function (PSF), and the tissue reflectivity function [38] , [39] . The linear image formation model can be rewritten as where y(r) is the image pixel observed at position r, x(r) is the TRF to be estimated, h(r) is the system PSF, n(r) is the additive measurement noise. R refers to the image domain [33] . Deconvolution in medical imaging is commonly employed to improve visual quality and achieve better contrast. This translates in easier interpretation for the physicians. The most common strategies for medical US image restoration is represented by MAP-based deconvolution. It tackles the problem using a two-step scheme: the PSF is estimated first and subsequently image restoration is performed. The main advantages of these strategies lie in (i) two dimensional or even three dimensional PSFs can be accounted for, (ii) no assumption is made on the PSF, neither on the number of zeros or poles, and nor on the position of the zeros in the complex plane, and (iii) more advanced models than white Gaussian can be assumed for the tissue reflectivity [40] . Although commonly employed, MAP deconvolution brings an increased computational cost, even if simple schemes are employed such as based on Wiener filtering or 1 -norm optimization. Another problem is that the PSF estimation is still tedious. In particular this is due to the need for phase unwrapping procedures where non-minimum phase PSFs are considered. Due to the inherent bandwidth limitations of US scanners and the adverse effects of measurement noise, image deconvolution is very sensitive to errors occurring in the PSF estimation. Even slight errors in the PSF estimates can lead to obvious artefacts that render the reconstructed images worthless. To address this, Michailovich et. al. [41] proposed a "hybrid" deconvolution technique. Their method was based only on partial information about the PSF, especially its power spectrum to estimate tissue reflectivity. While directly estimating the reflectance of the tissue from the relevant radiofrequency (RF) data, the proposed approach simultaneously eliminated errors caused by inaccuracies in PSF estimation. Pham et. al. [42] proposed a novel algorithm for the retrieval of blood flow from a sequence of ultrafast US images, based on the combination of two different techniques: deconvolution robust principal component analysis (DRPCA) and blind deconvolution (BD). The proposed method overcame the main limitation of the former related to the requirement for PSF estimation, whilst providing similar performances. The PSF was however assumed to be spatial-temporally invariant, and the algorithm appeared to be computational complex. B. Data-Driven Approaches 1) Despeckling: Deep learning methods are able to play a compelling role in improving the quality of US images, in terms of speckle mitigation in particular. Such data-driven systems can indeed be leveraged across the US imaging domain [43] . The work in [44] proved the applicability of CNNs as a method that can quickly and accurately perform image restoration. A multi-resolution fully convolutional neural network (FCN) was used to approximate an ultrasound image of "CT quality". This end-to-end ultrasonic image de-speckling CNN structure effectively improved the image resolution and contrast while preserving all the relevant anatomical and pathological information. It offered low complexity, making it applicable in real-time settings. However, in practice, CT-US paired data is difficult to obtain, so the US data used for training were simulated. In real cases, the shortage of the data pair reduces the applicability of the method. The method proposed by Feng et. al. [45] was shown to retain all relevant anatomical and pathological information in the restored images. They introduced a new CNN architecture, called US-Net, and proposed a new hybrid loss function that contains two parts. One was designed for speckle noise removal, the other consisted of error summation of the structural similarity index with respect to the denoised image and original image. Recently, a 3D dense U-net model was proposed to process 3D US B-mode data from clinical US systems [46] . Apart from achieving a suppression and mean preservation index similar to that of traditional denoising, the running time was also reduced by two orders of magnitude. 2) Deconvolution: Deep neural networks have also found applications in ultrasound image restoration. Perdios et. al. [47] proposed to exploit stacked denoising autoencoders (SDA), and successfully applied them to the recovery of structured signals, including US images. They explored both a linear measurement case where a known Gaussian random matrix was used as the measurement matrix (SDA-CNL) and a nonlinear measurement case where the weight matrices and bias vectors were learned (SDA-CL). It was shown that a 4-layer SDA-CL outperforms a state-of-the-art compressed sensing algorithm without the need to tune any hyper-parameter. While increasing the quality of the reconstructed image is the main objective, reducing the calculation time is also an important requirement in applications. Yoon and Ye [48] proposed a novel DL approach that interpolates the missing RF data by utilizing the sparsity of the RF data in the Fourier domain. The algorithm effectively reduced the data rate without sacrificing the image quality and is also widely applicable. The CNN can be trained using a RF data measured by the linear array transducer for a particular organ, and can be extended for other types of transducers and/or different body parts. The key to image reconstruction is to find a good sparse representation of the image to be reconstructed (which is also true for model-based approaches). DL methods assume that the representation is provided by deep neural networks, so it is nonlinear. The model parameters of the representation must be learned from a large amount of data. Hybrid approaches were therefore proposed. In hybrid approaches, neural networks are normally used to learn the prior of the observed data such that the size of the required dataset can be reduced. There are obvious theoretical connections between DL networks and traditional iterative algorithms. Jin et. al. [49] explored the relationship between CNNs and iterative optimization methods for inverse problems where the normal operator associated with the forward model is a convolution. The proposed method, which is called FBPConvNet, combined filtered back projection (FBP) with a multiresolution CNN. The direct inversion realized by FBP encapsulated the physical model of the system, but led to artifacts when the problem is ill-posed; the CNN combined multiresolution decomposition and residual learning in order to learn to remove these artifacts while preserving image structure. The structure of the CNN was based on U-Net [50] , with the addition of residual learning. This approach was motivated by the convolutional structure of several inverse problems in biomedical imaging, including CT, MRI, and diffraction tomography (DT). Chang et. al. [51] proposed a general framework to train a single deep neural network that solves arbitrary linear inverse problems. They observed that in optimization algorithms for linear inverse problems, signal priors usually appear in the form of proximal operators. Thus, the proposed network acted as a proximal operator for an optimization algorithm and projected similar image signals onto the set of natural images defined by the decision boundary of a classifier. The learned projection operator combined the high flexibility of deep neural nets with the wide applicability of traditional signal priors. This has the potential to lower significantly the costs involved in the design of specialized hardware and medical US is a clear potential beneficiary. This section presents the latest advances in computational imaging and image analysis methods utilising LUS. The LUS image processing methods are grouped into two major categories, specifically into the long developed model-based approaches and the data-driven (or learning-based) methods. In subsection IV-A, we talk about LUS line artefacts identification in a model-based way. Subsection IV-B reviews datadriven approaches that have been developed in clinical LUS interpretation including artefact localization, segmentation and so on. Most tasks of LUS analysis are related to the identification of line artefacts. An added benefit of model-based approaches is that they constitute an unsupervised framework for the labelling of LUS images (e.g. B-lines) in cases where annotated data are not available for machine learning. To mask line artefacts on LUS images, the authors in [21] generated a normalized gray scale map that can be used for delineation of different structures in the images. They used a random walk method to delineate the pleural line, and then excluded the upper pleural region before identifying Blines. This was achieved by an alternate sequential filtration, and subsequently applying the top-hat filter to ensure that Blines are laterally detached. Finally, a Gaussian model was fitted to each detected B-line, and the peak point of the fitted Gaussian models corresponding to the B-lines are calculated and used to accurately determine the position of B-lines. Blines were then overlaid on the B-mode images. Similarly, to implemented automatic pleural line detection, the authors of [23] masked a LUS image with expectation maximization (EM) based thresholding, after which hidden Markov model (HMM) and Viterbi algorithm (VA) were used to highlight the pleural line. Line detection in LUS can also be posed as an inverse problem, whereby the line information (position and orientation) is estimated from the observations. The solution however may not be unique, or it may be highly sensitive to variations in the data, which illustrates the ill-posedness of the problem. Regularization with specific penalty functions (or by employing specific priors if the problem is addressed in a Bayesian framework), is then necessary to solve such problems. Normally, the regularization term is related to the prior information of the parameter to be estimated [33] . In the literature, there are three main categories of such approaches, which include (i) statistical methods, (ii) regularized geometric modeling methods, and (iii) methods based on sparse representations [52] or, sometimes, a combination thereof. In the following, we discuss the most widely studied inverse problems in conjunction with LUS image analysis. In the very first work adopting an inverse problem formulation, Anantrasirichai et. al. [53] proposed an innovative way of detecting line artefacts in LUS images by employing the Radon transform, converting an image into a space of lines, and solving the following optimization problem: where R represents Radon transform, α and β are regularisation constants andx is the set of detected lines. The ADMM algorithm was used to solve this optimisation problem, offering a fast convergence rate. The scheme firstly detected the pleural line in order to locate the lung space. Then, the local peaks of the Radon transform were detected and line-type classification was done following clinical definitions, in the spatial image domain. B-lines, A-lines and Z-lines were hence successfully identified. In [54] , the authors further extended the method by combining the Radon transform with the PSF of the US acquisition system in a single equation thereby achieving line detection and deconvolution simultaneously. To enhance line detection performance and the visualization of restored lines, they included an additional convolution factor in the Radon transform domain with an unknown blurring kernel. The penalty function employed was the p -norm with norm order values of (0 ≤ p ≤ 1) in order to promote sparsity in the Radon space. Extending the above work in the context of evaluating COVID-19 patients, Karakus et. al. [55] improved the line detection performance by regularizing the solution using the Cauchy proximal splitting (CPS) algorithm [56] to promote statistical sparsity by utilising the Cauchy-based penalty function. Compared to conventional model-based approaches, ML based methods can provide improved performance in medical image analysis tasks, since they are able to capture more complex patterns in the data. ML approaches are often driven by pattern-contained data which are generally required for training and testing purposes. Among various ML techniques, DL, which is also a sub-class of ML, dominated the image processing literature during the last decade. DL approaches have three important characteristic properties [58] : (i) features are learned from the data instead of being handcrafted, (ii) features are abstracted in a hierarchical way from lower level to higher level, and (iii) the number of non-linear feature transformation layers is at least two. Over the past few years, DL techniques have rapidly gained attention in the field of medical imaging applications, ranging from image analysis and interpretation to image formation and reconstruction [59] . A review by Liu et. al. [60] described several popular DL architectures used in general medical US analysis, including (i) supervised deep networks or deep discriminative models, (ii) unsupervised deep networks or deep generative models, and (iii) hybrid deep networks. The authors point out that the current basic models are mainly convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep belief networks (DBNs), autoencoders (AEs), and variants thereof. They also summarized and discussed the application of these models in detail for various specific tasks in US image analysis. These include (i) traditional diagnostic tasks, such as classification, segmentation, detection, registration, biometrics, quality assessment, and (ii) emerging tasks like image-guided interventions and treatments. They stressed the main issue of such approaches, which is the generally small number of datasets available. One possible solution is to perform transfer learning, which can be either crossdomain or cross-modal. The former is the most common way to accomplish a variety of tasks in medical US analysis, but its source and destination are not within the same domain. However, the solution to train models on natural images might not be the optimal one and therefore domain-specific models in medical imaging are normally considered to enhance performance. In this sense, cross-modal transfer learning may be a wiser choice. Later, Razzak et. al. [61] discussed the application of ML/DL in US imaging, outlining the main challenges and solutions. They highlighted the importance of future ML systems to provide real-time feedback to the sonographer during image acquisition. It will also be important to replace manual ROI selection with automatic ROI selection or at least with a hybrid solution involving the human in the loop. After discussing important reviews in this area, in the sequel we conclude this chapter by discussing data-driven methods and the three main classes of learning approaches, which are fully-supervised, weakly-supervised and transfer learning. 1) Fully-supervised Learning: Various studies in the literature have shown that the data-driven algorithms are effective tools for medical ultrasound image analysis, and they have already started to make an impact in LUS image analysis. As a first example, the work in [24] implemented a fullysupervised method, where a total of 1450 pneumonia and 1605 normal lung images were analyzed with a three-layer feed-forward neural network. The method combined image processing and vector classifiers to identify consolidations of pediatric pneumonia. After the identification of the pleural line and the removal of skin/soft tissue, feature extraction was carried out based on the analysis of brightness distribution patterns presented in rectangular segments from the images. In the end, the method was able to correctly identify pneumonia infiltrates with a sensitivity of 90.9% and a specificity of 100%. Later, Wang et. al. [25] used a generic CNN structure to perform real-time lung B-line quantification to diagnose and quantify pulmonary edema. The training was implemented on a small dataset composed of 4864 clinical LUS images. The approach used (i) data augmentation to increase the variety of the data, (ii) dropout to avoid over-fitting, and (iii) rectified linear units (ReLU) to realize a faster training speed. Despite an apparently low 43.4% absolute accuracy in the test set, the intra-class correlation of 0.791 indicates substantial agreement of the neural network algorithm with the human-identified Bline counts. The B-line identification and clinical classification time was around 0.1s, nearly reaching real-time processing. Baloescu et. al. [62] proposed to use a relatively shallow custom-made network (CNN) architecture with 3-D filters (called 3D-CsNet) for the purpose of automatically distinguishing between the presence and absence of B-lines as well as assessing B-line severity. The backbone of 3D-CsNet is a CNN consisting of eight intermediate layers followed by two fully connected layer. As it alleviates the need for pre-training, the proposed algorithm is fast, employing few trainable parameters, and it is flexible and easy to deploy. However, this method has a weaker performance on multiclass severity rating (0.65 kappa) compared to its binary rating (0.88 kappa), because a higher number of categories may result in lower agreement when it comes to DL performance for the same amount of data. To improve the diagnosis efficiency, the approach introduced by Kulhare et. al. in [63] automatically segmented LUS features in simulated animal models with six single-class single shot detection (SSD) neural nets, making it suitable for ondevice inference tasks. However, the success of application to human LUS images has not been investigated. Motivated by the COVID-19 pandemic, researchers have also been enthusiastically investigating various solutions to the application of the COVID diagnosis. McDermott et. al. [64] and Soldati et. al. [30] have illustrated the applicability of LUS imaging to COVID-19 symptoms detection. By comparing LUS protocols and image features, the former work suggested the possibility of aiding in interpreting LUS images autonomously or semi-autonomously, and the latter clarified the urgent need of the diagnostic and prognostic role of LUS in COVID-19. One of the supervised method examples is the quantitative and automatic LUS scoring system developed by Chen et. al. [65] , using multi-layer fully connected neural networks (FNNs) for evaluating COVID-19 pneumonia. The authors introduced a curve-to-linear conversion process, thereby reducing the feature extraction problem from two dimensions to one dimension. One and two-layer FNNs with hidden nodes were used to learn the data features. ADAM optimizer was applied to fit the data with ReLu as the activation function. With all the settings, the neural network with 128×256 neurons gave the highest accuracy of 87%. 2) Weakly-supervised Learning: Even though the aforementioned approaches have brought improvements in LUS image interpretation, it cannot be denied that they require a high amount of labelled LUS images for training. This limitation has motivated researchers to develop weakly-supervised learning algorithms, where smaller amount of training samples are required. Van Sloun and Demi [66] explored weaklysupervised DL techniques, requiring only a single label per frame for training, and exploited gradient-weighted class activation mapping (grad-CAM) [67] to perform B-line localization. The network has been applied and tested both in-vitro and in-vivo. Notably, the proposed method can also serve as a region-of-interest selector for further quantitative analysis of segmentation in the LUS data that contain the most relevant information. Subhankar et. al. [68] presented an extended and fullyannotated version of the ICLUS-DB database 4 . They presented a deep network, derived from spatial transformer networks (STN) [69] . Two transformed versions of the input image are introduced to localize pathological artifacts. Then a feature extractor generates the final prediction. The network simultaneously predicts the disease severity score associated to an input frame and provides localization of pathological artefacts in a weakly-supervised way by inferring it from simple frame-based classification labels. In [68] , a method based on uninorms was used to effectively aggregate the frame score at video level. An F1-score prediction of 65.1 an was reported when performing the prediction at frame level. For video-based score prediction, because of the low inter-annotator agreement and the small number of videolevel samples annotation, an F1-score of 61%, a precision of 70% and a recall of 60% were reported. When evaluating segmentation results, the pixel-wise accuracy reaches 96% and a binary Dice score is 0.75. Tsai et. al. [70] further extended the idea of STN to Regularised Spatial Transformer Network (Reg-STN) via implementing a video-based (weakly supervised) and a frame-based (supervised) labelling approach. The video-based labelling approach reaches 91.12% accuracy for 10-fold cross validation, and the frame-based reaches 92.38%. This was the first attempt at full automation of LUS evaluation of lung pathology, and thus proposed an evaluation tool specifically for COVID-19. Inspired by the idea of support vector machines (SVM), which usually does not require a large number of training data, Veeramani and Muthusamy [71] proposed a method called relevance vector machines (RVM) to detect abnormalities 4 https://iclus-web.bluetensor.ai. in LUS images. Specifically, through pre-processing by an adaptive median filter and feature extraction, a complete local binary pattern mask was generated. Then, the binary RVM decides whether the given input LUS image corresponds to a healthy or unhealthy subject. If abnormal, the images will then be classified by the multi-level RVM into one of the lung diseases considered. This method finally leads to a classification accuracy of above 90% and a 100% specificity. 3) Transfer Learning: Transfer learning (TL) in medical tasks helps deep learning models achieve better performances when there are scarce medical images, and can also be defined as a solution to reducing training time. Horry et. al. [72] compared the performance of 8 widespread network structures, which are amenable to transfer learning for COVID-19 inspection with 3 modalities, including X-ray, CT and LUS. The results showed that their DL models are suitable for performing contrast enhancement on LUS images. The VGG19 outperforms all other models. The work in [73] used two pre-trained V-Unet and X-Unet models on 200 and 400 images respectively. In the former structure, the last three fully connected layers of a typical symmetrical U-Net structure were excluded and new layers for the expanding path were added, so that the VGG16 can be mimicked. The expanding path works reversely to the VGG16, with up-sampling through deconvolution by transposing convolutional layers. Then the V-Unet was pre-trained using ImageNet. The X-Unet is constructed by a U-Net pre-trained with grayscale salient object images. Almeida et. al. [74] proposed the use of MobileNets (lightweight neural networks) to support the diagnosis of COVID-19. MobileNet was also pretrained on ImageNet, substituting the last layer for a fully connected layer with a ReLu activation and the output layer with a softmax activation. Multitask learning was achieved by training different MobileNet models for each pathological indication. Even with limited training data, the MobileNet achieved accuracy values above 95% for all pneumonia indications. Compared with pre-training on real-life objects, for example ImageNet, the pre-training on LUS images may help detecting specific lung patterns such as B-Lines. Born et. al. [75] published the first dataset of lung POCUS recordings of COVID-19, pneumonia, and healthy patients. The collected data is heterogeneous but was pre-processed manually to remove artifacts and checked by a medical doctor for quality. They also proposed a model based on a pre-trained CNN (POCOVID-Net) on the available data and evaluated using 5-fold cross validation. The POCOVID-NET contains a convolutional part in the form of VGG16 followed by one hidden layer with ReLu activation, and batch normalization followed by the output layer with SoftMax activation. The results reveal the efficiency in detecting COVID-19 by reaching a sensitivity of 0.96, a specificity of 0.79, and F1-score of 0.92. Xue et. al. [76] introduced the idea of contrastive learning into severity assessment through LUS images, which falls under the transfer learning umbrella. The framework includes two modules: (i) a dual-level supervised multiple instance learning module (DSA-MIL) that leverages supervision from both the LUS zone scores and the patient severity of pneumonia; and (ii) a modality alignment contrastive learning module (MA-CLR) that combines representations of LUS and clinical information without dropping discriminative features. A staged representation transfer (SRT) strategy was introduced to train the nonlinear mapping, leveraging the semantic and discriminative information from the training data, providing an accuracy of 75% for 4-way severity assessment and 87.5% for two-way classification. Fig. 7 . Examples of vectors in specific regions used to compute brightness profiles: healthy (green), rib-bone (blue), and pneumonia (red). [24] . Table V provides a summary of all the aforementioned methods. For the model-based approaches, the most obvious advantage is the lack of need for training, and that a large dataset is not required. Given a specific task, the model-based methods are able to show good performances in terms of accuracy as the feature extraction algorithms are purposely designed by experts. Some rapid-converging algorithms enable the analysis of LUS image sequences in a relatively short amount of time, but it is still hard to achieve real-time processing. Meanwhile, being inherently explainable, the effects of the algorithms can be intuitively understood in each processing step, so they are straightforward to modify. Simple linear regression (SLR) and robust linear regression (RLR) [57] Provides a reliable operator-independent assessment of EVLW in ARDS, which is comparable or superior to previous scores. Not suitable for the case when multiple B-lines coalesce into a single white line. Hidden Markov model, Viterbi algorithm [23] Average overall accuracy in detecting the pleura is 84% and 94% for convex and linear probes, respectively. -Radon transform based inverse problem solved by ADMM [53] , [54] Improves the performance of B-line detection approaches by up to 50%. Non real-time. Radon transform based non-convex optimization with Cauchy-based penalty function [55] Up to 88% detection accuracy on 12-segment LUS for 9 COVID-19 patients. Non real-time. Characteristic vector classifiers [24] 90.9% Sensitivity and 100% specificity for pneumonia infiltrates identification. Only severe and radiographically evident cases are analyzed; The data only come from a single patient; The algorithm requires validation for individual ultra-sonographers. A basis CNN with data augmentation and dropout to avoid over-fitting [25] Requires littele computing power; Low number of parameters; Nearly real-time; Adaptive to other clinical tasks. Lack comet-rich samples; Labeled by a single observer; Manually count the number of B-lines. A small number of trainable parameters; Effective in poorer quality datasets. Deficient categories of the training data leads to a weaker performance on multiclass severity rating compared to binary rating. Single shot neural net [63] Light-weight network structure makes it suitable for on-device inference tasks. The application to human LUS images has not been investigated. multilayer FNNs [65] The work is not dependent on arbitrary or humandecided thresholds and is capable of getting robust classification result based on limited training data. The results are evaluated on visual determination of the Bscore and no artefacts are considered other than B-lines. CNN for B-line detection and weakly-supervised localization through CAM [66] Nearly real-time; the algorithm can also serve as a region-of-interest selector for further quantitative analysis of segments. The characterization of adequate phenotyping of various lung pathologies is needed. STN based deep network [68] Extended and fully-annotated version of the ICLUS-DB database. The precise demographics of the patient group in the database remain unknown; The possible inclusion of false positive cases; Noisy labels in the database. multi-level RVM [71] Classification accuracy of above 90% and 100% specificity when tested on the dataset. The size of the dataset is not known. SVM [23] The need of small dataset by using SVM. The learning stage is time-consuming. V-Unet and X-Unet [73] Short training time; Accurate when scare medical images. No regularization may lead to over-fitting to US images only; Performance is dataset-dependent. POCOVID-Net [75] The first dataset of lung POCUS recordings of COVID-19, pneumonia, and healthy patients. The dataset is restricted to convex probe. A DSA-MIL module to predict the patentseverity and a MA-CLR for combination ofthe LUS data and the clinical information [76] Issues related to data heterogeneity, multi-modality and non-linearity are alleviated. The distribution of lung zone information is ignored. However, considering the task-oriented nature of the modelbased methods, the ability of generalization can be limited in complex clinical cases. Therefore, data-driven approaches are investigated. In DL methods, network parameters need training through a large amount of data, so as to achieve the purposes of object detection, recognition, segmentation and so on. When enough labeled data are available, DL can achieve or even exceed the performance of humans [77] . The supervised learning approaches have reached the accuracy required for clinical diagnosis [78] . However, for LUS, it is usually difficult to acquire a large number of multi-category data, so attempts to design weakly-supervised and transfer learning methods have been made. For transfer learning, the network trained with LUS images will have a better performance than that trained with natural images [75] . In clinical scenarios, the high detection efficiency is a must, and the data processing speed should be fast. Therefore, real-time tasks and computationally efficient ML approaches have also been studied. From the literature reviewed above, the importance of LUS in assisting clinical diagnosis has been demonstrated thoroughly. However, although LUS has a proven capability to intercept alterations along the lung surface and value in treatment monitoring, it is still facing challenges in real application. Despite the fact that model-based methods can fulfil the aim of line identification, their generalization ability is not as good as data-driven approaches such as ML techniques. But in future researches, one should not neglect the inherent interpretability of the classical algorithms, which may help alleviate the black box problem of ML/DL techniques. As the application of data-driven approaches is predominant in the medical field, data standardization becomes an essential problem. This concept concerns various aspects of LUS image processing, such as data collection, data labelling, and algorithm validation etc. So far, there is no regularization in LUS image collection. Therefore the quality and the form of the images are dependent on the equipment and the operator. This inconsistency in data collection along with the lack of labeling standards brings difficulties in providing an accurate ground truth of the lung condition, because the diagnosis results varies among clinicians. Without a well-labelled dataset to rely on, the output of the trained network would not be trusty enough in clinical cases. Accordingly, gold rules for setting up standardized datasets are urgently needed. What's more, to maximize working efficiency, the requirement on realtime artefact detection is also highly expected by clinicians. Even though the challenges mentioned above have not been adequately tackled yet, several prospective data-driven techniques in medical image processing are worth studying. For example, transfer learning is a feasible solution to the lack of suitable datasets. A pretrained network can be first obtained on ImageNet, and then be used for LUS image training. However the performance has not been demonstrated to be as good as targeted training on medical image datasets [60] . More recently, looking at current trends in the ML community with respect to DL, we identify as key area which can be relevant for LUS image processing that of self-supervised learning. Different from fully supervised learning, self-supervised learning does not require manual labeling since it generates them by itself [79] . In this sense, self-supervised learning can be considered as a subset of unsupervised learning methods. Self-supervised learning methods are attractive as they have been able to utilize unlabelled data to learn the underlying representations. This is where self-supervised methods play a vital tole in fuelling the progress of DL without the need for expensive annotations and learn feature representations where data provide supervision [80] . The successful application of self-supervised learning in MRIs [81] sets a hope in its usage in other medical modalities including LUS. Moreover, by compacting light-weigh network structures, the computational efficiency will be possibly further improved. Clinical applications of LUS have become more and more numerous. In this article, we reviewed recent research progress in computational LUS image reconstruction and analysis, including model-based methods as well as data-driven methods. The approaches presented in this article all show the possibility of assisting disease diagnosis by detecting and quantifying line structures, especially through B-lines. The model-based methods are unsupervised and explainable, whilst the datadriven methods provide better generalization and more rapid development. In future studies, the combination of the two methodologies can be a trend, as a result of which the requirement of labelled dataset can be relaxed while keeping the performances of the algorithms. Furthermore, the availability of the LUS dataset is improving, which will likely nurture to algorithmic development. Lung ultrasound in the critically ill Application of lung ultrasound during the covid-19 pandemic: A narrative review Lung ultrasound: a valuable tool for the assessment of dialysis patients with COVID-19 Lung ultrasound imaging, a technical review Forum of international respiratory societies. the global impact of respiratory disease -second edition Lung disease in the UK Lung ultrasound: The future ahead and the lessons learned from covid-19 Robust mean-squared error estimation in the presence of model uncertainties Clinical Analysis of Ultrasound Diagnosis of PULMONARY DISEASES The Second Military Medical University Press Development, optimization and clinical evaluation of algorithms for ultrasound data analysis used in selected medical applications Application of lung ultrasound in critical care setting: A review Determination of a potential quantitative measure of the state of the lung using lung ultrasound spectroscopy The role of ultrasound lung artifacts in the diagnosis of respiratory diseases On the influence of imaging parameters on lung ultrasound b-line artifacts, in vitro study Real-time multifrequency ultrasound imaging for quantitative lung ultrasound-first clinical results Proposal for international standardization of the use of lung ultrasound for patients with covid-19: a simple, quantitative, reproducible method Simplified 8-site lung ultrasound examination to assess fluid overload in children on haemodialysis B-line detection and localization by means of deep learning: preliminary in-vitro results Pleural line and b-lines based image analysis for severity evaluation of covid-19 pneumonia Automatic detection of b-lines in in-vivo lung ultrasound Devices, systems, and methods for diagnosis of pulmonary conditions through detection of b-lines in lung sonography Automatic pleural line extraction and covid-19 scoring from lung ultrasound data Automatic classification of pediatric pneumonia based on lung ultrasound pattern recognition Quantifying lung ultrasound comets with a convolutional neural network: Initial clinical results Finding covert fluid: methods for detecting volume overload in children on dialysis Lung ultrasound: a novel technique for detecting fluid overload in children on dialysis What's new in lung ultrasound during the covid-19 pandemic Pointof-care lung ultrasound in patients with covid-19 -a narrative review Is there a role for lung ultrasound during the covid-19 pandemic? Advances in the clinical application of lung ultrasonography Novel bayesian multiscale method for speckle removal in medical ultrasound images Inverse problems in medical ultrasound images -applications to image deconvolution, segmentation and super-resolution Despeckling algorithm for removing speckle noise from ultrasound images Speckle reducing anisotropic diffusion A novel speckle noise removal algorithm based on admm and energy minimization method Distributed optimization and statistical learning via the alternating direction method of multipliers Block-wise ultrasound image deconvolution from fundamental and harmonic images Ultrasound image deconvolution using fundamental and harmonic images Statistical methods for analysis and processing of medical ultrasound: applications to segmentation and restoration Iterative reconstruction of medical ultrasound images using spectrally constrained phase updates Joint blind deconvolution and robust principal component analysis for blood flow estimation in medical ultrasound imaging Deep learning in ultrasound imaging Towards ct-quality ultrasound imaging using deep learning Speckle Noise Removal in Ultrasound Images Using a Deep Convolutional Neural Network and a Specially Designed Loss Function 3d b-mode ultrasound speckle reduction using deep learning for 3d registration applications A deep learning approach to ultrasound image recovery Deep learning for accelerated ultrasound imaging Deep convolutional neural network for inverse problems in imaging U-net: Convolutional networks for biomedical image segmentation One network to solve them all -solving linear inverse problems using deep projection models Ill-posed problem or inverse problem in image processing Automatic bline detection in paediatric lung ultrasound Line detection as an inverse problem: Application to lung ultrasound imaging Detection of line artefacts in lung ultrasound images of covid-19 patients via non-convex regularization Convergence guarantees for nonconvex optimisation with cauchy-based penalties Quantitative lung ultrasonography: a putative new algorithm for automatic detection and quantification of b-lines Machine learning in acoustics: theory and applications Deep learning in medical ultrasound-from image formation to image analysis Deep learning in medical ultrasound analysis: A review Deep learning for medical image processing: Overview, challenges and future Automated lung ultrasound b-line assessment using a deep learning algorithm Ultrasound-based detection of lung abnormalities using single shot detection convolutional neural networks," in Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and Navigation Sonographic diagnosis of covid-19: A review of image processing for lung ultrasound Quantitative analysis and automated lung ultrasound scoring for evaluating covid-19 pneumonia with neural networks Localizing b-lines in lung ultrasonography by weakly supervised deep learning, in-vivo results Grad-cam: Visual explanations from deep networks via gradient-based localization Deep learning for classification and localization of covid-19 markers in point-of-care lung ultrasound Spatial transformer networks Automatic deep learningbased pleural effusion classification in lung ultrasound images for respiratory pathology diagnosis Detection of abnormalities in ultrasound lung image using multi-level rvm classification Covid-19 detection through transfer learning using multimodal imaging data Transfer learning u-net deep learning for lung ultrasound segmentation Lung ultrasound for pointof-care covid-19 pneumonia stratification: computer-aided diagnostics in a smartphone. first experiences classifying semiology from public datasets Pocovid-net: Automatic detection of covid-19 from a new lung ultrasound imaging dataset (pocus) Modality alignment contrastive learning for severity assessment of covid-19 from lung ultrasound and clinical information Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images Machine learning approach on healthcare big data: a review Self-supervised learning: The dark matter of intelligence Self-supervised visual feature learning with deep neural networks: A survey Self-supervised learning for spinal mris