Rethinking Ultrasound Confidence Maps

Author: Stephan Baller

Advisor: Vanessa Gonzalez Duque​

Introduction

Ultrasound Imaging is a very popular non-invasive technique used in a variety of medical cases to gather visual information of a closed body. The technology behind it comes with certain weaknesses though, leading to distorted representations, bad image quality and possible misinterpretations as a consequence. To counteract such casualties, a variety of image processing techniques can be applied to enhance image quality or help identify objects and features. In the scope of this blog post, we'll specifically focus on confidence maps as powerful tool to visualise the certainty associated with the information content for different regions in an image. The goal is to get an insight into the validity of the information we get for each pixel. confidence maps are therefore particularly useful as additional input for Convolutional Neural Networks (e.g. for ultrasound image classification)^[1] and other advanced image processing techniques. We'll first go through the challenges we face when calculating confidence maps reflected by the common unwanted effects in ultrasound imaging and then review the approaches from the following two papers:

• "Ultrasound Confidence Maps of Intensity and Structure Based on Directed Acrylic Graph and Artefact Models"^[2]
• "Weakly Supervised Estimation of Shadow confidence maps in Fetal ultrasound Imaging"^[1].

Unwanted Effects in Ultrasound Imaging

As part of the calculation of a confidence map, it's important to consider all unwanted effects that can emerge through the technology and how they affect other parts of the image.

Ultrasound Imaging generally works by emitting and receiving acoustic pulses by a transducer and analysing the echo coming from the intermediate tissue. The origin of the reflected signal can be derived from the direction in which the transducer is transmitting the pulse and the time passed from transmission and capture of the signal. The more impedance the intermediate tissue has, the higher the amplitude of reflected sound wave will be. Depending on the measurement, a brightness value can then be assigned at the respective position in the resulting image. This is of course just a very simplified explanation. In reality this process includes many additional steps.^[3]

[10] Example ultrasound images with speckle noise(both), downward attenuation(both), shadows(blue regions left) and reverberation artefacts (blue lines right)

Speckle Noise

As is the case for many coherent imaging systems (systems in which waves can interfere with each other), the sensor unintentionally picks up unwanted signals that may come from the environment or the transducer itself. This leads to random overlaying patterns in the image. Speckle Noise in particular is the noise generated by the random scattering of transducer sound waves through the tissue. We can see this effect in both of the above images.

Acoustic Shadows and Downward Attenuation

Highly reflective tissue or materials reduce the amount of transmitted sound waves that would pass to tissue more distant to the transducer. This leads to so-called shadowing regions that are particularly prune to noise signals coming from the surrounding tissue. Since sound waves will be dampened while passing through any kind of material, even a body with equal impedance will produce an image with a continuously darker growing shadow. This effect is called downward attenuation and can also be seen in the above images that both get darker in downward direction. An example for shadows can be seen inside the blue coloured circles in the left image.

Reverberation Artefacts

In addition to shadowing regions, two reflective objects can also cause reverberation artefacts when light bounces back and forth between them, causing a repeating pattern. In the right image they are highlighted with blue color and generally appear as repeating pattern with equal distance.

Methods for CM Calculation

For CM calculation we'll reproduce the steps described in the two papers "ultrasound confidence maps of Intensity and Structure Based on Directed Acrylic Graph and Artefact Models"^[2] and "Weakly Supervised Estimation of Shadow confidence maps in Fetal ultrasound Imaging"^[1]. The first paper will give us an insight into a traditional computational approach, while the second paper is using Convolutional Neural Networks. For each paper we'll go through the proposed method and their results.

Directed Acrylic Graph and Artefact Models

The approach used here is inspired by another paper^[4] which is itself based on the random walks segmentation algorithm^[5] and promises several quality improvements in comparison. Furthermore, a structural confidence map is proposed that gives insight into the anatomic structural boundaries, which we'll go into detail later on.

Speckle Noise Denoising

As first step we have to get rid of the speckle noise pattern. There are many filters that can take care of this very well like the Gaussian filter for blurring. By using common blurring algorithms though, the issue arises that the gradient magnitude decreases depending on the intensity of the blur. In other words, blurring the image doesn't preserve the edges and homogeneous regions, making the detection of objects and shadows even more difficult. Therefore, we apply an anisotropic diffusion based algorithm^[6] that is able to smooth out homogeneous regions while preserving edges. Explaining the details behind this algorithm would go beyond the limits of this article, so we'll just do a brief overview on the equations used and rather focus on getting a visual understanding of how it works.

[9] Original image on the left, right image reproduced using authors algorithm; Denoised image on the right

$//<![CDATA[ \begin{array}{l}\mathrm{q\,= \sqrt{ \frac{\frac{1}{2} \left( \frac{\nabla I}{I} \right)^2 - \frac{1}{4^2} \left( \frac{\nabla^2 I}{I} \right)^2}{\left[ 1 + \frac{1}{4} \left( \frac{\nabla^2 I}{I} \right) \right]^2} }}\end{array} //]]>$​​  (1)

$$\text{\hspace{1em}(2)}$$

$//<![CDATA[ \begin{array}{l}\mathrm{c(q)\,= \frac{c_{\text{canny}}}{1 + \left| q^2 - q_0^2 \right| / \left[ q_0^2 (1 + q_0^2) \right]}}\end{array} //]]>$ ​​(3)

The main idea is to calculate the Diffusion Coefficient c (see equation 3) for which blurring will be more intense for high values (i.e. homogeneous regions with low gradient magnitude) and less intense for low values (i.e. edge regions with high gradient magnitude). That way surfaces become more visible as we can see in the above figure.We therefore calculate the local homogeneity $//<![CDATA[ \begin{array}{l}\mathrm{q}\end{array} //]]>$ of each pixel using the Instantaneous Coefficient of Variation (ICOV, see equation 1) and compare it to the local Coefficient of Variation (COV, see equation 2)  $//<![CDATA[ \begin{array}{l}\mathrm{q_0}\end{array} //]]>$. The canny edge detector returns an image in which only detected edges are present, for which when multiplied with the rest of the equation the resulting diffusion coefficient is additionally enforced to be lower in edge regions. For each iteration, brightness and contrast are adjusted corresponding to the original image. As a result, we see patches form in the denoised image that intensify for every step. A more detailed explanation can be found in the paper "Speckle reducing anisotropic diffusion".^[6]

Ultrasound Intensity Confidence

In an ultrasound image where sound waves travel downwards into the tissue, each pixel's confidence value decreases depending on how the sound waves interact with objects represented by the pixels above. We can therefore estimate a confidence map  $//<![CDATA[ \begin{array}{l}\mathrm{C}\end{array} //]]>$ by taking the original image and confidence values of a pixel row  to calculate the confidence values for every pixel  $//<![CDATA[ \begin{array}{l}\mathrm{C(i+1,j)}\end{array} //]]>$ in the next pixel row. This procedure can be modeled as acyclic Graph as depicted in the figure below.

Source: Original paper; Directed acyclic graph modelling the procedure

$$\$//<![CDATA[\; \backslash begin\{array\}\{l\}\backslash mathrm\{C(i+1,j)\backslash ,=\; \backslash sum\_\{k=-\backslash kappa\}^\{\backslash kappa\}\; \backslash psi(k)\; w\_\{i,j+k,k\}\; C(i,j+k)\}\backslash end\{array\}\; //]]>\$\text{\hspace{1em}(4)}$$

Trivially, the pixels in row $//<![CDATA[ \begin{array}{l}\mathrm{i}\end{array} //]]>$ that are closest have the most impact on the directly underlying pixels. For that reason we apply a weight  $//<![CDATA[ \begin{array}{l}\mathrm{\psi(k)}\end{array} //]]>$ to each pixel  $//<![CDATA[ \begin{array}{l}\mathrm{j+k}\end{array} //]]>$ in a predefined range of  $//<![CDATA[ \begin{array}{l}\mathrm{-k}\end{array} //]]>$ and $//<![CDATA[ \begin{array}{l}\mathrm{k}\end{array} //]]>$ based on the normal distribution. The two pixels at each side of the range (where  $//<![CDATA[ \begin{array}{l}\mathrm{k=\pm\kappa}\end{array} //]]>$) are weighted with the remaining out-of-range pixels weights so that all weights within the range sum up to 1:

$//<![CDATA[ \begin{array}{l}\mathrm{\psi(k)\,= \begin{cases} \Phi\left(\frac{k+0.5}{\sigma}\right) - \Phi\left(\frac{k-0.5}{\sigma}\right), & k \neq \pm\kappa \\ \left(1 - \sum_{k=-\kappa}^{-\kappa+1} \psi(k)\right)/2, & \text{otherwise} \end{cases}}\end{array} //]]>$​ (5)

The second weight scales according to the row number and the adjusted relative gradient  (with direction $//<![CDATA[ \begin{array}{l}\mathrm{d}\end{array} //]]>$) to model both downward attenuation and diffraction of sound waves:

$$\text{\hspace{1em}(6)}$$

We won't go into further detail for this equation, as further mathematical tricks come into play here (like the Beer-Lambert-Law-adjusted relative gradient which could fill an entire block post for itself)^[7]. Continuing with the example figure from above, the calculated confidence map looks as follows (right image):

[9] Original image on the left, right image reproduced using author's algorithm; Confidence map on the right calculated using the authors code

Detecting Needle and Reverberation Artefacts

To model needles and reverberation artefacts, the authors make use of a machine learning approach from the here referenced paper. The algorithm returns two arrays of pixels belonging to needles and reverberation artefacts, which are taken into account during the previously described calculation of the confidence map. Needles are highly reflective, so the relative gradient values are set to maximum value, decreasing underlying confidence values as consequence. Reverberation artefacts are artificially generated and shouldn't affect the underlying confidence values, so the relative gradient values are set to 1 in the respective region.

Source: Leftmost image from the paper and the others reproduced using author's algorithm; From left to right: original image, reverberation detections, needle detections, newly calculated confidence map

Structural Confidence Map

$//<![CDATA[ \begin{array}{l}\mathrm{\Gamma(i,\,j) = \frac{C'(i, j)}{R(i, j)} \in [0.0, 1.0]}\end{array} //]]>$ (7)

Additionally, the authors propose a "structural confidence map" with which anatomical boundaries can be easier differentiated from artefacts. This is achieved by comparing the confidence map to a reference confidence map like in eq. 7. The reference confidence map is calculated by applying the previously described steps to an "empty" image (with same dimensions as the original image), so that it produces a map with maximum confidence values for each row. As noise can lead to confidence values higher than in the reference confidence map, the respective values are adjusted to the maximum possible value (i.e. the value in the respective row of the reference confidence map). 

Results

The results are centered around the tasks of identifying shadow and reverberation artefacts, detecting loss-of-contact between the transducer and skin surface, and image compounding. The experiments were conducted on a variety of subjects, including chicken breast, a live pig, and an anthropomorphic phantom, using a UF-760AG Fukuda Denshi machine. The computations were performed using Python and NumPy on an Intel Core i5-8279U processor. The algorithm demonstrated the capability to infer appropriate confidence values for challenging regions, such as areas with shadows, reverberations, and artefacts. This was tested on 20 images where specific patches representing these regions were manually identified and labeled.

Source: Original image from the paper; patches A (red & cyan), B (green & magenta) and C (white & yellow), original image, confidence using ^[5], confidence, structural confidence

In the tested images, three types of patches were identified

• Patch A (red & cyan): Region above actual tissue, above surfaces causing shadows or artefacts.
• Patch B: (green & magenta): Shadowed or artefact region.
• Patch C: (white & yellow): Region without shadows or artefacts, in the same horizontal line as Patch B.

Qualitative Results: The confidence map by previous methods (referenced as ^[5]) was found to be overly sensitive to abrupt changes in gradient, resulting in low confidence values across large image areas. In contrast, the algorithm developed in this paper was more robust against such gradient changes. It produced intensity confidence values that decayed more slowly with depth, except in shadow or artefact regions. Notably, the algorithm effectively modeled sound attenuation through needle and vessel walls, assigning lower confidence values to artefacts and shadow regions and identifying regions where the probe detached from the surface.

Source: original image from the paper; Ranges and median values for A, B and C for a set of 20 test images.

Quantitative Results: The median values of the confidence in different patches of 20 images were calculated. The intensity and structural confidence values follow the pattern: $//<![CDATA[ \begin{array}{l}\mathrm{C_{int}(A)\,> C_{int}(C) > C_{int}(B)}\end{array} //]]>$​ and  $//<![CDATA[ \begin{array}{l}\mathrm{C_{str}(A)\,\approx C_{str}(C) >> C_{str}(B)}\end{array} //]]>$. The structural confidence median values were about 0.6, much lower than the values in patches A and C, which were close to 1. This indicates that the structural confidence successfully differentiated artefact patches from non-artefact patches. Moreover, the intensity confidence had the lowest value in patch B and highest in patch A and C, successfully modelling the attenuation in contrast to the original random walks approach (denoted in the image as intensity, [8]).

The algorithm developed in the first paper shows a significant improvement in handling shadow and reverberation artefacts in ultrasound images. Its ability to produce more accurate confidence values across various regions demonstrates its potential for enhancing ultrasound image analysis and interpretation.

Weakly Supervised Estimation of Shadow Confidence Maps

In contrast to the previous more traditional approach, the authors of this paper introduce a machine learning method that is based Convolutional Neural Networks. This comes with the advantage that we don't have to care about each unwanted ultrasound imaging effect individually, as the neural network learns their effect on their effect on the confidence the values automatically. The main challenge we face here is to come up with a CNN architecture and training process, which we'll go into detail now.

Training Process and Architecture

Source: Original image from the paper; Training process and architecture

For a CNN to produce decent results we usually need a large dataset of very distinctive and varied data. Creating a dataset with pixel-wise shadow annotations is very time intensive though, for which only 10% of the images $\$//<![CDATA[\; \backslash begin\{array\}\{l\}\backslash mathrm\{X\}\backslash end\{array\}\; //]]>\$$ in the here used dataset contain shadow map annotations  $//<![CDATA[ \begin{array}{l}\mathrm{Y^S}\end{array} //]]>$. This issue is addressed with a weakly-supervised learning approach using a shadow-seg module together with a transfer function to first obtain a better dataset with which the final confidence estimation network can be trained. For calculating this dataset the shadow-seg module estimates binary shadow maps  $//<![CDATA[ \begin{array}{l}\mathrm{\hat{Y}^S}\end{array} //]]>$ that are then transformed into probabilistic confidence maps using the transfer function  $//<![CDATA[ \begin{array}{l}\mathrm{T(x,\,\hat{Y}^S, Y^S)$}\end{array} //]]>$. The details will be explained in the following.

Shadow-seg module

Main goal of the shadow seg-module is to get a decent binary confidence map prediction network from a dataset with weakly annotated images. As shown in other papers^[8], this can be achieved by first only using global image level labels to train a classification network ( $//<![CDATA[ \begin{array}{l}\mathrm{X|L=1}\end{array} //]]>$ if it contains a shadow,  $//<![CDATA[ \begin{array}{l}\mathrm{X|L=0}\end{array} //]]>$ if not). As the layers of the classification network develop to recognise shadowing features in the image, its weights can be reused for the shadow segmentation network to predict the anticipated binary shadow maps  $//<![CDATA[ \begin{array}{l}\mathrm{\hat{Y}^S}\end{array} //]]>$. It employs an encoder-decoder architecture for which the decoder mirrors the feature encoder that has equal structure to the classification network. Training the network on the limited images with shadow annotations is sufficient to acquire results due to the pre-trained weights from the classification network.

Transfer function

A new dataset can then be created with the transfer function $//<![CDATA[ \begin{array}{l}\mathrm{T}\end{array} //]]>$ estimating new confidence maps using the shadow regions from the input images  $//<![CDATA[ \begin{array}{l}\mathrm{X}\end{array} //]]>$, the binary shadow maps from the ground-truth data  and the estimation from the Shadow-seg module $//<![CDATA[ \begin{array}{l}\mathrm{\hat{Y}^S}\end{array} //]]>$. We assume that the transition in brightness values away from the median ​ to the maximum and minimum value in the shadow region ( $//<![CDATA[ \begin{array}{l}\mathrm{I_{max}}\end{array} //]]>$ and $//<![CDATA[ \begin{array}{l}\mathrm{I_{min}}\end{array} //]]>$) reflect the change in confidence.

$//<![CDATA[ \begin{array}{l}\mathrm{T(x_{ij}\,| x_{ij} \in X_{FP}) = \begin{cases} \frac{I_{x_{ij}} - I_{min}}{I_{mean} - I_{min}}, & \text{if } I_{min} \leq I_{x_{ij}} < I_{mean} \\ \frac{I_{mean} - I_{x_{ij}}}{I_{max} - I_{mean}}, & \text{if } I_{mean} < I_{x_{ij}} \leq I_{max} \\ 1, & \text{if } I_{x_{ij}} = I_{mean} \end{cases}}\end{array} //]]>$​​

If $//<![CDATA[ \begin{array}{l}\mathrm{I_{mean}\,= I_{min}}\end{array} //]]>$:

$//<![CDATA[ \begin{array}{l}\mathrm{T(x_{ij}\,| x_{ij} \in X_{FP}) = \begin{cases} \frac{I_{x_{ij}} - I_{mean}}{I_{max} - I_{mean}}, & \text{if } I_{mean} < I_{x_{ij}} \\ 1, & \text{if } I_{x_{ij}} = I_{mean} \end{cases}}\end{array} //]]>$

The values for the true positive and true negative regions can stay the same. For the false positive and false negative regions we calculate the normalised distance from the mean value of the true positive region $//<![CDATA[ \begin{array}{l}\mathrm{I}\end{array} //]]>$ in the original image to get a confidence estimation in the respective shadowing regions.

Confidence Estimation Network

Finally, the confidence estimation network can be trained on the newly generated confidence map dataset. Similar to the segmentation network, it employs an encode-decoder architecture, but is trained from scratch on the images produced by the transfer function.

Results

Source: Original image from the paper; Qualitative comparison of different approaches

In the second paper various methods were evaluated based on several metrics, including Mean Squared Error (MSE), Soft DICE, and Intraclass Correlation (ICC). The methods compared include:

• RW^[5]: A random-walks method the first paper is based on.
• RW*^[5]: A variation of the random-walks method.
• Pilot: A prior work by the authors or collaborators.
• Baseline: The basic version of the proposed method without additional enhancements.
• Proposed: The authors' newly developed method.
• Proposed+AG: The proposed method with the addition of Attention Gates.
• Weak GT: Weak Ground Truth referring to a baseline or comparison standard in the study.

Quantitative comparison of different approaches used on two different datasets

The results show that the proposed methods (both the standard and Attention Gates-enhanced versions) outperform existing methods and the baseline in terms of shadow confidence estimation of curved array ultrasound images and. This is indicated by lower MSE and higher Soft DICE and ICC scores, suggesting better accuracy and consistency in shadow detection compared to other methods. As the way the Random Walks approach was evaluated, it detects big parts of the lower part in the image as shadows due to its high focus on downward attenuation. Consequently, there are just few false positives and a high recall value as a result.

Review and Comparison

The objectives of both papers are mostly the same, but differ in a few details which poses some challenges when comparing them. Firstly, the inputs for the proposed algorithms differ slightly. While the neural network approach is primarily trained on curved array ultrasound images, the random-walks algorithm produces the best results for linear array images. Furthermore, the approach of the first paper additionally specialises on reverberation and artefact detection while the second paper approach is implementing general shadow detection for which its capability of taking artefacts into account is highly dependent on the dataset it was trained on. Thus, the qualitative and quantitative measurements used in the second paper hardly reflect how the two approaches compare to each other.

Open Challenges

Confidence maps are a powerful aid for interpreting ultrasound or other kinds of sonographic images in many clinical and engineering applications, especially for inexperienced professionals^[1]. For clinical applications, the use of intensity confidence maps for automatic positioning of robotic ultrasound probes has been investigated and described in a paper published in 2020^[11]. It shows that confidence maps can be successfully used to optimise probe orientation.

Personally, I see huge potential in combining the two approaches and use an unsupervised approach to train a neural network on producing the same results as the algorithm from the first paper. The results from the Neural Network approach already show that confidence map calculation can be done efficiently in real time with few computational resources. Combining both methods could therefore potentially outpass them both in accuracy and performance, without the need for creating time intensive annotations by hand.

1. Meng, Qingjie, Matthew Sinclair, Veronika Zimmer, Benjamin Hou, Martin Rajchl, Nicolas Toussaint, Ozan Oktay, et al. “Weakly Supervised Estimation of Shadow Confidence Maps in Fetal Ultrasound Imaging.” arXiv.org, November 20, 2018. https://arxiv.org/abs/1811.08164v3.
2. Hung, Alex Ling Yu, Wanwen Chen, and John Galeotti. “Ultrasound Confidence Maps of Intensity and Structure Based on Directed Acyclic Graphs and Artefact Models.” arXiv.org, November 24, 2020. https://doi.org/10.1109/ISBI48211.2021.9433862.
3. Karamalis, A. “Ultrasound Confidence Maps and Applications in Medical Image Processing,” 2013. https://www.semanticscholar.org/paper/Ultrasound-confidence-maps-and-applications-in-Karamalis/19997d39447e570c7030a214eb4d81e3669ffd1f.
4. Karamalis, Athanasios, Wolfgang Wein, Tassilo Klein, and Nassir Navab. “Ultrasound Confidence Maps Using Random Walks.” Medical Image Analysis 16, no. 6 (August 2012): 1101–12. https://doi.org/10.1016/j.media.2012.07.005.
5. Grady, L. “Random Walks for Image Segmentation.” IEEE Transactions on Pattern Analysis and Machine Intelligence 28, no. 11 (November 2006): 1768–83. https://doi.org/10.1109/TPAMI.2006.233.
6. “Speckle Reducing Anisotropic Diffusion | IEEE Journals & Magazine | IEEE Xplore.” Accessed January 10, 2024. https://ieeexplore.ieee.org/document/1097762.
7. “The Beer-Lambert Law | Journal of Chemical Education.” Accessed January 10, 2024. https://pubs.acs.org/doi/10.1021/ed039p333.

8. Meng, Qingjie, Christian Baumgartner, Matthew Sinclair, James Housden, Martin Rajchl, Alberto Gomez, Benjamin Hou, et al. “Automatic Shadow Detection in 2D Ultrasound Images.” In Data Driven Treatment Response Assessment and Preterm, Perinatal, and Paediatric Image Analysis, edited by Andrew Melbourne, Roxane Licandro, Matthew DiFranco, Paolo Rota, Melanie Gau, Martin Kampel, Rosalind Aughwane, et al., 66–75. Lecture Notes in Computer Science. Cham: Springer International Publishing, 2018. https://doi.org/10.1007/978-3-030-00807-9_7.

9. “Ultrasound-Edge-Shadowing-Artefact-768x427.Png (768×427).” Accessed January 16, 2024. https://pocus101.b-cdn.net/wp-content/uploads/2020/06/Ultrasound-Edge-Shadowing-Artefact-768x427.png.

10. “Ultrasound Artefacts for Beginners – Critical Care Transport at Michigan,” July 4, 2019. https://survivalflighteducationblog.com/ultrasound-artefacts-for-beginners/.

11. Jiang, Zhongliang, Matthias Grimm, Mingchuan Zhou, Javier Esteban, Walter Simson, Guillaume Zahnd, and Nassir Navab. “Automatic Normal Positioning of Robotic Ultrasound Probe Based Only on Confidence Map Optimization and Force Measurement.” IEEE Robotics and Automation Letters 5, no. 2 (April 2020): 1342–49. https://doi.org/10.1109/LRA.2020.2967682.