Data Fusion in Non-Destructive Testing

Tobias Vogt, winter semester 2020/21

Physical limitations of individual sensor measurements in non-destructive testing (NDT) often lead to a lack of validity when evaluating the underlying data. Data Fusion allows for an automized evaluation of a test subject by combining information sources (e.g. different testing methods) and therefore increasing test accuracy. ^{[1] [2]}

Motivation

Although the usage of data fusion in non-destructive testing as a research topic was first introduced in 1996 by X.E. Gros ^[3], the concept of data fusion itself has been around in nature for thousands of years. Animals and humans recognize their environment by combining information from their body senses. Nature has found a way for a human brain to combine signals from its five body senses into an interpretable and dynamic model of the world. The ability to fuse multi-sensory data allows us to evaluate situations quickly and precisely ^[4].  

The field of data fusion opens up promising opportunities for progress in many research disciplines and has received unprecedented attention from the NDT community in recent years. Missing a critical defect in a non-destructive test situation might lead to catastrophic consequences, thus creating a need for highly reliable measurements. Given the fact that different NDT techniques have very different strengths and weaknesses (e.g dye penetrant inspection is useful for surface defect detection, while radar inspection displays volumetric information), combining techniques can provide a more complete representation of the inspected specimen.
It is also worth mentioning that the available and ever increasing computational power enables new opportunities in complex data fusion tasks.^[5]

Fundamentals of data fusion

Described below are the technical fundamentals of data fusion. Chapter 2.1 elaborates on how data fusion takes place on different data levels while chapter 2.2 provides an overview of how multiple sensor data sets can be configured in a fusion.

Abstraction levels of data processing in data fusion

When referring to data fusion, a categorization is typically made based on the abstraction level of the underlying data. The three levels widely used are raw data fusion (or low-level fusion), feature-level fusion (or intermediate-level fusion) and decision fusion (or High-level fusion). The fusion on a higher level is usually more efficient, however it might not be as effective as it goes hand in hand with data compressing and loss of information. Lower-level data fusion is a more complex undertaking that delivers results closer to the truth, but harder to interpret. ^[1][4]
Figure 1 provides an overview of the different data levels

As the name suggests, raw data fusion involves the combination of signals from individual sensors. There are numerous approaches to merge this kind of data, but an essential prerequisite is the time synchronization and registration of the data. Registration describes the process of aligning the coordinate system of each individual sensor to a global coordinate system. ^[1]
The approaches can be divided into:

• Feature-based approaches, such as the weighted average or the Kalman-Filter.

• Probabilistic approaches, such as Bayesian Statistics or the Maximum-Likelihood estimation.

• Fuzzy-based methods which can incorporate user knowledge and are therefore support subjective results

• Neural-network approaches, that show great strengths in specifically defined fusion environments ^[1]

The goal of raw data fusion is the enhancement of data quality which is expected to be more informative than the input data. ^[4]

Diverse data types with latent information content are fused on the feature level. This level is useful when a lack of temporal and spatial coherence does not allow the fusion on the raw data level. Nevertheless fused raw data can also serve as a valid input. The information is condensed by extracting signal properties from the raw data, called features. Such features can be amplitudes, runtimes, frequency spectra, integrals or differentials of measurement curves. The combined effort from human preliminary work of the feature determination and the high abstraction capability of fusion algorithms, make feature-based-fusion approaches to be considered as particularly powerful.^[2]

The highest level of data abstraction takes place on the decision level fusion. Here, data already is available in a form of logical objects. The underlying raw data can be completely diverse and a fusion takes place through boolean functions or expert-votings which then provide a decision. An Example: The existence of a blowhole in a steel beam can be considered as a typical logical object. Different NDT-techniques have different capabilities to detect the blowhole. On a decision level a boolean function evaluates if the blowhole exists or not by combining the probabilities from each NDT-technique. ^[2]

 Depending on the use case, fusion is not limited to only one level of data. Feature level and decision level fusion can rely on lower level results, therefore increasing the degree of automation of data evaluation. ^[1]

Sensor Configuration types in data processing

The way multi sensor networks are configured yields towards different methods of data linkage or integration. Generally, multiple data sets either deliver redundant or complementary information. Data redundancy can increase measurement reliability or the signal-to-noise ratio while complementary data sets (e.g. from heterogenous sensors) can provide a new perspective on a specimen. Complementary data leads to an extended response range compared to single sensor measurements. This can enable the inspector to detect not seen before features of a specimen, thereby eliminating ambiguity given by incomplete information from the individual sensors. ^[1]

Configuration concepts of multi sensor networks are often divided into competitive fusion, complementary fusion and cooperative fusion. These three types are displayed in Figure 2 and are described as follows:

Competitive data fusion, also called concurring data fusion describes the integration of data from independent measurements of  type identical (homogenous) sensors. This can either be achieved by using two or more homogenous sensors or by using a single sensor and taking measurements at different instants. An example for this kind of integration would be the accumulation of pictures from a camera under the same circumstances to reduce image noise. Competitive data fusion is used to provide robustness to a measuring system.
In Figure 1, Sensor 1 and 2 are setup up in a competitive configuration as they observe the same property in the environment space (Property A). ^[1][4]

Complementary data fusion describes the data integration of multiple sensors that do not necessarily depend on each other as they observe different properties of an environment. The main goal of this kind of fusion is the closing of information gaps, or the creation of a more complete environment image respectively. Linking sensors with different measuring ranges or working principles is a typical use case of complementary data fusion. It is also widely used in image processing applications where the goal is to fuse multiple surveillance cameras or multidirectional cameras of modern cars.  
In Figure 1, Sensor 2 and 3 are used in a complementary fusion. As both Sensors measure different properties of the environment (A and B), the resulting data is a combination of both properties.^[1][4]

Cooperative data fusion is considered as the most difficult fusion to design. Here, information from two independent sensors is derived to extract new information that a single sensor cannot measure. It is especially difficult, because the fused data is very sensitive to inaccuracies of the preprocessed data. Therefore, in contrast to the configuration concepts explained before, cooperative data fusion is generally less reliable and accurate. The calculation of 3D information from two 2-dimensional images from slightly different angles is an example for this kind of fusion.
In Figure 1, Sensor 4 and 5 are set up to enable a cooperative data fusion of property C in the environment space. ^[1][4]

Exemplary procedure

In order to get a better impression of the steps involved in a typical data fusion process, this chapter will summarize a case study conducted by Heideklang and Shokouhi in 2013.^[5] 
In the researchers’ work, a specimen in form of a steel block with differently sized notches (in width and depth) was examined using three non-destructive testing methods. This idealized test specimen was chosen to analyze the basic characteristics of different data types. Figure 3 shows a schematic replica of the specimen.

The individual NDT methods chosen were:

1. Eddy Current (ET): An electromagnetic testing method used to examine electrically conductive material for defects and imperfections in near surface areas.^[6]

2. Giant Magnetoresistance (GMR): A testing method sensitive to the magnetic stray field of the specimen.^[5]

3. Active Thermography testing (TT): An infrared testing method that measures the thermal radiation emitted by the specimen.^[6]

As each testing method is an imaging technique, the fusion can be conducted based on the image pixels. Before fusing, the result of each measuring technique needs to go through a number of preprocessing steps such as noise reduction or de-trending. This is followed by the image registration (the alignment of the images along a global coordinate system). The different resolutions of the measurement images must be taken into account and normalization must be applied accordingly.^[5]

Figure 4 provides a schematic representation of an excerpt of the preprocessed images from Heideklang and Shokouhi’s case study. Notably, all NDT methods were able to detect the deepest notches, or expressed differently, the most severe damages. However, their performance differs and each technique has its own strengths and weaknesses. ET is sensitive to all notches, yet the resolution is rather poor. GMR shows clearly defined peaks and a high resolution, yet the slighter damages remain undetected. TT combines a high resolution with a powerful damage detection, but at the price of an increased false alarm rate. These diverse qualities make the measurements excellent candidates for a successful complementary data fusion that delivers more meaningful results on a pixel and decision level.^[5]

The case study proposes multiple fusion algorithms, the most performant one being a wavelet based fusion. In essence, this technique recognizes contrast changes in images and maps coefficients to the corresponding image areas. The sensitivity of the coefficients depend on the type of wavelet transformation applied (here a discrete wavelet transform using a Daubechies 4 wavelet). In a fusion, the coefficients from each modality are then combined, e.g. by calculating the mean value or combining the maximum values.^[5]

Figure 5 shows a sketch of the pixel level fusion result. Here, all notches appear with a higher spatial resolution compared to the individual ET measurement and the detection ratio is higher compared to GMR. The background noise of the fused image is more noticeable than in GMR and TT which originates from the low resolution ET measurement. The increased false alarm rate of the TT measurement results in slightly visible image artifacts.^[5]

The researchers have also implemented a classifier which automatically detects faults based on the wavelet coefficients from each modality. This marks a classic application of decision based fusion. The performance of the model is then determined by calculating the true positive rate (TPR) of the detected defects. The wavelet fusion reaches a TPR of approximately 0.58 compared to individual NDT methods with TPRs between 0.39 and 0.49, thus indicating a positive effect of data fusion.^[5]

Opportunities and challenges of data fusion

Opportunities

W. Elmenreich has summarized the most prominent opportunities/advantages of multi sensor data fusion in comparison to single sensor systems ^[4]:

1. Reliability: Reliability of the system greatly profits from an increasing number of sensing units as this leads to data redundancy.

2. Extended spatial coverage: This advantage especially refers to complementary data fusion as described above. Multiple sensors can be set up in a way that they measure or monitor different aspects of a subject and therefore extend the spatial coverage.

3. Increased confidence: The main driver for competitive fusion is to increase the measurement confidence. In a dual sensor setting, a measurement of one sensing unit is confirmed by the other one covering the same domain.

4. Reduced ambiguity and uncertainty: Ambiguous interpretations of a measurement feature can be narrowed down to a clearer feature interpretation

5. Robustness against interference: By increasing the dimensionality of the measurement space, interference can be reduced which leads to a better signal to noise ratio.

6. Improved Resolution: When fusing multiple independent measurements of the same property, the resolution of the output value is generally better than the resolution of each independent input.

7. Reduction of complexity: In a standardized environment with tuned fusion parameter, automized data fusion can greatly compress the output data and therefore reduce the complexity of a given problem.

Challenges

Different researchers have come across multiple challenges or limitations of multi sensor data fusion. Depending on the application field, these might have a great or minor influence on the fusion. Some noteworthy challenges include:

1. Data fusion is still largely performed on the expert level. While generally, properly tuned multi modal NDT setups increase the confidence of a measurement, the fusion algorithm and measuring methods of choice determine, how much potential information can be extracted. Choosing the right influencing parameters is far from trivial, thus making data fusion a complex undertaking.^[5]

2. Information overload from multiple sensor databases. Data uncertainty, conflicting data, data correlation and spurious data complicate the search for patterns in multiple data sets. Without sufficient data fusion and compression methods developed in an academic environment, multiple sensor setups in real life applications may overcomplicate data extraction and interpretation.^[7]

3. In construction engineering, the efforts of mobilizing and demobilizing the data fusion technology are a challenge that may not be underestimated. Applying data fusion in a constantly developing environment makes it hard to automize fusion algorithms. Here, the fusion mostly takes place on the decision level in form of human interpretation.^[7]

New approaches to data fusion

In recent years, the enhancements in computing power opened up new approaches for data fusion in the area of deep neural networks. In comparison to other machine learning methods, the input data for deep learning networks does not need to be handcrafted and carefully procured. Neural networks gain their strength from large amounts of training data. As gathering labeled training data, i.e. data that has been evaluated towards damages or anomalies manually, is a very elaborate procedure, the usage of unlabeled training data is a common practice. Here, the neural network finds patterns in training data that later can be labeled as certain types of damages or fatigue symptoms.^[8]

Besides these approaches which tackle the fusion of data, researches have also pointed out the significance of high quality input data. Quality metrics such as sensor resolution, signal-to-noise ratio, completeness, robustness or timeliness are often not considered in data fusion applications. The performance of fusion algorithms can greatly profit from high quality training data.^[8]

References

1. Ruser, H. & Puente León, F. Methoden der Informationsfusion - Überblick und Taxonomie. in Informationsfusion in der Mess- und Sensortechnik (eds. Beyerer, J., Puente León, F. & Sommer, K. D.) 1–20 (Univ.-Verl, 2006).

2. Völker, C. Datenfusion zur verbesserten Fehlstellendetektion bei der zerstörungsfreien Prüfung von Betonbauteilen. (2017) doi:10.22028/D291-26841.

3. Gros, X. E. NDT data fusion. (Arnold ; John Wiley, 1997)

4. Elmenreich, W. An Introduction to Sensor Fusion. (2002).

5. Heideklang, R. & Shokouhi, P. Application of data fusion in nondestructive testing (NDT). in Proceedings of the 16th International Conference on Information Fusion 835–841 (2013).

6. Dr. Große, C. U. Einführung in die Zerstörungsfreie Prüfung im Ingenieurwesen - Grundlagen und Anwendungsbeispiele. (ZfP Lehrstuhl für zerstörungsfreie Prüfung, 2020).

7. Shahandashti, S. M. et al. Data-Fusion Approaches and Applications for Construction Engineering. J. Constr. Eng. Manage. 137, 863–869 (2011).

8. Wu, R.-T. & Jahanshahi, M. R. Data fusion approaches for structural health monitoring and system identification: Past, present, and future. Structural Health Monitoring 19, 552–586 (2020).