Automated Semantic/Anatomical Vessel labeling

Abstract

Automated semantic or anatomical vessel labelling describes the procedure of relating vessel branches or segments to their corresponding anatomical description. This is achieved in order to aid a reliable diagnosis of artery diseases, treatment planning as well as being a precondition for many computer aided medical procedures. From its many use cases it becomes clear that a robust labelling method is essential. Yet, to attain a robust classification, meaning achieving accurate results and as little misclassifications as possible, is still a challenge in vessel labelling, especially in coronary vessel labelling. This is due to the huge variability in the human anatomy and a corresponding lack of samples. Within the past years some of the intuitive labelling methods such as knowledge based, geometric based and graph matching based, have been extended to deep neural networks in order to achieve better results. Some of these methods include the so called TreeLab-Net, CPR-GCN and CorLab-Net learning approaches. Those three methods, developed between the years 2018 and 2021, correlate in some aspects and built upon each other in other aspects. The working and results of named methods will be discussed in this post. In the end a personal review is added and ideas for further research are given.

Introduction

Patients with artery diseases depend on a reliable diagnosis as well as treatment planning and surgery. Vessel labelling is one of the many steps necessary to achieve good results in all these mentioned use cases. But also in creating medical reports, writing statistics of anatomical branches and visualization of regions of interest for research, vessel labelling is of huge interest.

The major challenge in vessel labelling is robustness. A robust labelling system achieves a high accuracy and includes little misclassifications. In this post I will focus on coronary vessel labelling because the challenge here seems to be rather prominent. The coronary arteries are named according to different branches as well as left and right heart domain (see Figure 1). Main branches include right coronary artery (RCA), left main (LM), left descending artery (LAD), left circumflex artery (LCX). Bifurcating from the main branches are several side branches such as obtuse margin (OM), diagonal artery (D), septal branch (S), posterior descending branches (L-PLB/R-PLB) and right posterior artery (R-PDA) (see Figure 2). Between different individuals, branches may be missing or vary in length and position.

Additionally, image quality and segmentation can have an impact on robustness. Depending on the visual representation and segmentation circles might appear. Linkage problems, broken vessel segments and noise might make labelling more difficult as well [3].

Focusing on coronary labelling is a personal choice, many of the methods mentioned in the following paragraphs can be transferred to other regions of the body such as for abdominal and head and neck vessel labelling.


Figure 1: Labelled branches of two patients. Each colour indicates a different branch. a and b showing positional information of coronary vessels. Image c showing a comparison of two patients where the left set of branches is more complete than the right set of branches. [1]


Figure 2: Visualization of coronary arteries for multiple patients. Left set of branches is more complete than right set of branches. [2]

Intuitive Labelling Methods

Most intuitive vessel labelling methods can be split into several categories including knowledge based vessel labelling, geometric feature based vessel labelling and graph matching based labelling. Recent approaches built upon these previously researched and applied methods in order to improve them and include more complex structures.

Knowledge Based Labelling Methods

In the case of knowledge based labelling, first a base model from many statistical examples is developed. This model defines hard rules for labelling. Once it is set an algorithm to match the current case to the reference model can be applied. An example procedure can be seen in Figure 3. After segmenting the coronary tree (mostly from a CT image) the main branches are identified by matching them to a predefined or labelled model and then side branches are matched. The rather obvious problem with knowledge based methods is that they do not account for the huge variability in human anatomy [4].


Figure 3: Work flow graph of a knowledge based coronary artery vessel labelling method. [5]

Geometric Feature Based Labelling Methods

In order to overcome the aforementioned challenge, geometric feature based labelling can be applied. A simple model is trained with many examples. Through the training process it can adapt to some variability. Common intrinsic features of the model are length, curvature or diameter of vessels. Whereas common extrinsic features are direction and position of the vessels. The two main challenges of this approach include identification of branches and finding the optimal features to achieve the best accuracy. Typical employed models are multivariate Gaussian, AdaBoost or a support vector machine.

Still, one problem occurs. If different branches have a similar geometry they will be labelled in the same class. This may lead to overlapping classification [6].


Figure 4: Work flow graph of a geometric feature based coronary artery vessel labelling method. [7]

Graph Matching Based Labelling Methods

In order to further improve classification graph matching based labelling is introduced. This labelling method is often utilized in addition to the previous mentioned methods. Instead of using a discrete model now a probabilistic model, a Bayesian network, is employed. The main strategies are to include a hidden Markov model or Markov random field. The major challenge in this case of labelling is to achieve a stable distribution [5,9].


Figure 5: Images showing the graph structure of retinal vessels. [8]

Recent Approaches - Methodology

Based on these base groups of approaches, new technologies have been proposed within recent years. Three of these methods will be explained in detail in the following section.

Bidirectional Tree LSTM (2018)

TreeLab-Net implements a framework of the form of a tree structured recurrent neural network embedding a deep bidirectional long-short-term memory network (LSTM).


Figure 6: Processing pipeline of TreeLab-Net architecture. [10]

The pipeline shows that in the first step the centrelines are extracted with a commonly used aorta segmentation method from a coronary computed tomography angiography (CCTA) image.

Afterwards a spherical coordinate transform is applied. This is done in order to normalise the large variance of centreline points in the cartesian coordinate system. In the resulting image the aorta is always on top and all images have a common origin. The image is created by computing the spherical coordinates for all points. Hereby the origin is the bounding box center of all points. The vector pointing to the north pole direction from the origin, is found by applying a rotation matrix to the z-axis in cartesian coordinate system. The new x- and z-axis are the angles. You can imagine this view to be similar to looking at a 2D world map.

Any part of artery between two bifurcations is defined as a vessel segment. Those segments each contain several features which are extracted from the original 3D image as well as the 2D transform.

Vessel segment features include:

Ground truth label (assigned manually),
Predicted label,
Angles between two connected segments both in 3D and SCT2D space (from directional vector of each segment),
SCT2D coordinates of first point, center point and ending point
Directional vectors in SCT2D space, including directions between start and end points


Figure 7: Close up look at the processing of the neural network itself and its input structure. [10]

The classification happens within TreeLab-Net. The tree can handle arbitrary numbers of segments and through its internal structure incorporates parent child relationships. Each segment, taken from a, is a node, in b. The network itself consist of three main modules: An multilayer perceptron (MLP) acting as a feature encoder to transform features into vectors of same length and weights Z, the Bi-TreeLSTM including long term learning information and a softmax layer for the final prediction (see Figure 7). In order to include topological dependencies, segment information from up and downstream, not only a standard LSTM (as it can only handle sequential information) but a tree structure is used.


Figure 8: detailed structure of a single node and its corresponding calculations.

In order to update a hidden state, we take the previous set of child nodes and states.

Each node contains input gates, output gates, memory gates and has a single forget gate for each child to keep information from each child if important. Memory cell updates depend on the child nodes. In the end the sum of all child nodes is computed.

The bidirectional extension to the network is achieved by adding a hidden state vector in the up-to-down direction. The two final stream vectors are simply concatenated in the final version of the update rule.

CPR-GCN (2020)

The second method, is the proposed CPR-GCN, short for conditional partial residual graph convolutional network. It considers not only position information but all CT data and consists of 2 major parts, partial-residual graph convolutional network and a condition extractor.


Figure 11: Processing pipeline of the CPR-GCN network. [2]

Again the centrelines are extracted using an automated coronary artery extraction system.

TreeLab-Net makes a 3D to 2D transformation to normalise the data, but there is a problem: due to angles being periodic, small shaking might introduce errors. In order to avoid this this approach uses a S^2 manifold to represent the angles. Similar to the TreeLab-Net the coordinates are thereafter transformed.


Figure 12: Feature extraction from vessel segments by sub-sampling and applying [2]

First any spatial features are extracted by running a 3D CNN on the data and afterwards any features concerning the tube like shape, meaning they describe the sequential relations, are extracted through a BiLSTM.

separate branches where centrelines end or branches bifurcate (if children branches close then consider same starting point)
sub-sample control points with the same length
extract features from the cubes by applying 3 Layers of 3D CNN and 3D maxpooling before being transferred to the BiLSTM


Figure 13: Extension of the update rule to include a residual connection. [2]

The layerwise propagation rule for the classification can be seen in Figure 13. As input is the Adjacent matrix of the graph A, the layer wise Weights, the feature vector as well as the previous hidden state are used. The output after the update is the next hidden state. In order to improve performance, some updates are made. These include for example a partial residual block in the GCN, which aids to put weight on the certain features and absorb less relevant ones to be used as conditions. Now not only the input direction but the entire residual part is changing the update the the next step.

CorLab-Net (2021)

Lastly the proposed CorLab-Net network will be explained. The network includes inter-organ relations, meaning between vessels and heart chamber, by introducing a spatial distance field as well as intra-artery spatial dependencies between vessel points and joint using a morphological distance field. The network consists of three main modules.


Figure 14: Processing pipeline of CorLab-Net. [12]

3 main modules:

anatomical distance field: The anatomical distance field consists of the points of coronary arteries and surface points of heart chamber. The minimal distance from each point of branch to the surface is calculated and combined with the information of its placements in the coordinate system to create the distance field (see Figure 15).
Figure 15: Calculation of the distance field. [12]
Morphological distance field: It is assumed that different points influence the network more or less strong. Joint points for example may include more information than points being part of a middle segment. Thus first a landmark detection point cloud network is applied to detect joint points. Then the distance in all directions from other points to the joint points is computed to create the morphological distance field.
Point cloud network: Both features fields are used for classification forming an enriched feature space. This is one of the major parts in this network to aim at high results. The network itself is evolved from PointNet++, but in order to achieve a consistent labelling across neighbouring points GCN modules are incorporated.
Figure 16: Depiction of the structure of the point cloud network. [12]