Printer Friendly

A pixel-level method for multiple imaging sensor data fusion through artificial neural networks.


Studies have shown the capability of multiple imaging sensor data fusion to combine different images of the same object from different imaging sensors to produce a resultant image with improved visual perception and enhanced features for intelligent decision support (Blum and Liu, 2006). Multiple imaging sensor data fusion has been used in remote sensing (Pohl and Genderen, 1998), medicine (Laliberte et al., 2003), military surveillance (Tsagaris and Anastassopoulos, 2005), agriculture (Leinonen and Jones, 2004), and human body detection (Han and Bhanu, 2007). Available methods of multiple imaging sensor data fusion are at pixel level, feature level and symbolic level (Blum and Liu, 2006). Most research on multiple imaging sensor data fusion was actually at pixel level. This paper presents a new method of data fusion from multiple imaging sensors at pixel level for Non-Destructive Inspection (NDI). NDI is one of the major application areas of multisensor integration and data fusion (Gros, 1997). For corrosion detection in aging aircraft panels, the NDI imaging methods, such as eddy current and ultrasound, were studied (Rebbapragada et al., 1999; Palakal et al., 2001). In order to increase the accuracy of the detection from an individual sensor, a NDI data fusion method was developed for study of aging aircraft structures (Forsyth et al., 2002). Although research and applications of NDI data fusion have been conducted in various areas, a general data fusion system model capable of handling various applications is very difficult, if not impossible, to design (Gros, 1997). Various data fusion models under the general concept are necessary for each specific area of research and application. The purpose of this research was to develop and apply a new method of data fusion from multiple imaging sensors through Artificial Neural Network (ANN) classifiers. The work was originated from and applied to NDI to identify the corroded spots on the aircraft panel specimens. The developed method is expected to have broad applications wherever multiple imaging sensors inspect the similar specimens.


1.1 Imaging Sensors and Images

NDI is a method for materials characterization. There exists a wide range of non-destructive testing methods to help examine various problems and defects in different kinds of materials and under varying circumstances. Typical NDI methods include liquid penetrant, magnetic particle, eddy current and radiographic inspection, ultrasonic inspection, tomography, and real-time X-ray radiography. With the development of digital image processing technology, these NDI methods have been effectively and widely used in the materials industry.

X-ray radiography is a costly NDI method used to inspect material and components with differential adsorption of penetrating radiation. Each specimen under evaluation absorbs different amount of radiation because each individual is different in density, thickness, shape, size, and even absorption characteristics. The unabsorbed radiation that passes through the specimen is recorded as the indication of internal and external conditions that appear as variants of black and white gray scale contrasts on exposed film, or variants of color on fluorescent screens.

Eddy current is used to detect surface cracks, pits, subsurface cracks, corrosion on inner surfaces, and to determine alloy and heat-treat condition. With an eddy current instrument eddy currents are induced in a specimen when an alternating current, which is generated by the instrument, is applied to a test probe. The alternating current in the probe induces an alternating magnetic field in the article which causes eddy currents to flow in the specimen. Then, the instrument processes the signal acquired from the probe into the designated magnitude and format to display.

Ultrasonic inspection is an acoustic method of NDI that uses sound energy moving through the specimen for detection of cracks and defects. Ultrasonic inspection instruments utilize ultrasound to examine the internal integrity of metals, plastics, and composite materials. The sound waves essentially "bounce-off" internal defects. The ultrasound transducer/receiver probe can be manually maneuvered. The probe is placed in contact with the part to be inspected with an oil or gel-layer acting as a carrier of the acoustic signal. The probe is moved, and the acoustic response changes. The size and position of internal flaws can be determined by the height and position of the "peaks" observed on an image.

This research used X-ray NDI image data as the benchmark and evaluated eddy current and ultrasonic NDI image data and the fusion of the data from the two different NDI imaging sensors in detection of corroded spots on the aircraft panel specimens. For this research, a total of 30 aircraft corrosion color bitmap images were obtained. Fifteen of them were eddy current images while another fifteen were ultrasonic images.

1.2 Artificial Neural Networks

An ANN is an information processing paradigm that is inspired by the behavior of biological nervous systems such as the human brain (McCulloch and Pitts, 1943; Hebb, 1949; Rosenblatt, 1958; Minsky and Papert, 1969; Grossberg, 1976; Hopfield, 1982). ANNs deduce the essential features of neurons and their interconnections using a computer program to simulate these features. In engineering, an artificial neuron is a unit that has multiple inputs and one output. The neuron functions both in training and testing. In training, the neuron is iteratively taught to fire or not fire for particular input patterns. In testing, when a trained or similar input pattern is detected by the neuron, the associated output will become the current output. In this way, like a human being, the neuron learns by example. An ANN is composed of a large number of highly interconnected neurons, information processing elements. These neurons work together to solve specific problems like a human brain does.

ANNs learn by example and find the solution of the problems by themselves. Therefore, ANNs often outperform conventional statistical and mathematical methods in solving complex problems. ANNs have been widely used in pattern recognition and data classification for solving problems in engineering and scientific research.

In order to accurately identify the corroded spots on the aircraft panels, ANN classifiers were developed. With the 30 eddy current and ultrasonic images, the ANN classifiers were trained to determine the ability to differentiate the corrosion data into two classes (threshold at 10% corrosion level), and to verify ANN classifiers with a real corrosion sample. The two-class classification of corrosion data is of industrial interest. It basically uses 10% corrosion level as the threshold to form a binary problem: yes or no, i.e. the corrosion level is either higher or lower than 10%.

For ANN training, the RGB (Red-Green-Blue) values of the color images were converted into hue values. The converted hue matrices were used for the development of ANN classifiers. The feature vectors were formed by shifting a window across the hue matrices (Figure 1). The window shift starts column-wise by moving one column at a time. When the window moves over all the columns, it shifts down one row and goes back to the first few columns to start the next round column-wise shift. After continuing this process, the window will move over the entire image to extract image feature parameters. The size of the window can be changed to allow the ANN classifiers to differentiate the data at different resolutions. The feature vectors are the input to the ANN classifiers for ANN training, cross-validation, and testing.


Multilayer perceptrons (MLPs) were trained to classify the aircraft panel corrosion image data. MLPs are the most popular neural network architecture. They are layered feedforward networks training with the algorithm of backpropagation (Rumelhart et al., 1986). In this research, the MLPs with one hidden layer were used (Figure 2).



Typically, through training, ANNs may memorize individual exemplars rather than the trend(s) in the data set as a whole. This is so-called over-training. The over-trained ANNs typically pass by a point at which the performance of the networks on new data starts to deteriorate. Cross-validation is an effective method to avoid over-training. With this method, a data set will be divided into three non-overlapped subsets: one for training, one for cross-validation and one for testing. The cross-validation subset is for training termination. In training computing, the cross-validation method monitors the error on a data set independent from the training data set in attempt to avoid over-training and terminate training when the error gets to the point at which the error begins to increase. The point is considered containing the settings and values of best generalization of ANN training. Figure 3 shows that with the method of cross-validation the training process can be terminated at an optimal point, t*, without over-training and save training time even though the training error still decreases.

In order to quantify the classes of the aircraft panel corrosion image data after ANN training, the method of minimal distance, or norm, was used. This method looked for minimal geometric distance between the image feature vectors and the vectors representing the desired classes:

[Minimal_Distance.sub.i,j*] = min ([parallel][[O.sub.i].bar] - [[C.sub.j].bar][parrallel]) (1 [less than or equal to] i, j [less than or equal to] n) (1)

[parallel][[O.sub.i].bar] - [[C.sub.j].bar][parrallel] = [(<[[O.sub.i].bar] - [[C.sub.j].bar], [[O.sub.i].bar] - [[C.sub.j].bar]>).sup.1/2] (2)

where n is the number of classes of the ANN classifier. In this research n = 2 (two classes); <[], []> represents the inner product of the vectors [] and []; and [[O.sub.i].bar] = [[[y.sub.1], [y.sub.2], ..., [y.sub.i], [y.sub.n]].sup.T] is the output vector from the ANN classifier representing that the ith class is the desired class of the vector corresponding to the desired class vector [[C.sub.i].bar] = [0, 0, ..., 1, ..., 0].sup.T] in which the ith element is 1 and the rest are zeros.

In ANN training equations (1) and (2) are implemented. For each vector like [[O.sub.i].bar] the distances between [[O.sub.i].bar] and [[C.sub.1].bar], [[C.sub.2].bar], [[C.sub.n].bar] are calculated. If the distance between [[O.sub.i].bar] and [[C.sub.i].bar] is minimal, i.e. [j.sup.*]=i in equation (1), the output vector [[O.sub.i].bar] is correctly classified; otherwise it is misclassified.

1.3 Imaging Sensor Data Fusion

Based on the classification of aircraft panel corrosion image data from two different NDI image sensors through ANN classifiers, the integration or fusion of the classified results has potential to overcome the limit of an individual image sensor and to have more accurate quantification of the data classification.

In general, the method of multiple image sensor data fusion can be established with two merge functions: merge of classification signature matrices and merge of outputs of ANN classifiers for different image sensors to the same specimen. By the two merge functions, the fused result should be better than the one from an individual image sensor since the fused one integrates what any individual captured.

Each of the NDI images, as the input to the ANN classifiers, contains two types of pixels: non-zero pixels representing the degree of corrosion with their values and zero pixels representing no corrosion with 0. The non-zero pixels are included in the image ROI (Region of Interest). The zero pixels are excluded from the image ROI. Therefore, in feature extraction from each image, the windowed extraction process should be monitored to skip the window(s) containing many more zero-value pixels than the ROI pixels, i.e. non-zero pixels in the window(s). By doing so, a signature matrix can be generated after window scanning over the image to indicate what each of the pixels is:

S = [[s.sub.ij]] (1<i<i_row; 1<j<i_col) (3)

[s.sub.ij] = 0 or 1 or 2 (4)

where i_row is the number of image rows; and i_col is the number of image columns. In the two equations, [s.sub.ij] = 0 indicates that the corresponding pixel, i.e. the pixel (i, j), in the image shows no corrosion. [s.sub.ij] = 1 indicates that the corresponding pixel in the image shows a certain degree of corrosion, but this pixel will be excluded from the ANN classification since in image feature extraction the window containing this pixel is skipped. [s.sub.ij] = 2 indicates that the corresponding pixel in the image also shows a certain degree of corrosion, and this pixel will be included in the ANN classification since the window containing this pixel is included to form an image feature vector as the ANN input.

Each image from an image sensor produces a signature matrix from image feature extraction. If one supposes that there are L images from L different image sensors, then, the L images will produce L signature matrices after image feature extraction: [S.sup.1], [S.sup.2], [S.sup.L]. In order to conduct data fusion from different image sensors, these signature matrices should be merged:

M = [[m.sub.ij]] (5)

[m.sub.ij] = 0 or 1 or 2 (6)


[m.sub.ij] = 0 when all [s.sup.k.sub.i,j] = 0 (7)

[m.sub.ij] = 1 when [s.sup.k.sub.ij] = 0 and 1 (8)

[m.sup.ij] = 2 when at least a [s.sup.k.sub.ij] = 2 (9)

where k = 1, 2, ..., L.

An ANN classifier is developed for each image from each of the image sensors. The outputs of the ANN classifiers should be merged to fuse the data from different image sensors. If one supposes that [[bar.O].sup.j.sub.1], [[bar.O].sup.j.sub.2], ..., [[bar.O].sup.j.sub.N] (j = 1, 2, ..., L) are the output vectors of the ANN classifier for image sensor j, then the outputs of the L ANN classifiers should be in the form:

[[bar.O].sup.f.sub.i] = [L summation over (j=1)] [c.sub.j][[bar.O].sup.j.sub.i] (10)

where N is the sample size; and [c.sub.1], [c.sub.2], ..., [c.sub.L] are the output merge coefficients which can be determined by assigning values or optimization.

In general, the image ROIs vary from sensor to sensor even when they visualize the same specimen. Therefore, in general, the sequential relationship of image feature vectors from different image sensors is different. When merging the outputs of the ANN classifiers, the sequential relationship between the outputs of the ANN classifiers of different image sensors has to match. The signature matrices are useful to help find the match of the sequential relationship.

With the method of image sensor data fusion as described above, a general procedure of data fusion for L images from L image sensors respectively can be formulated as follows:

1. For each image from an image sensor, specify the size of window and perform image feature extraction by window scanning over the image to produce two sets of matrix data related to the image: feature vectors and a signature matrix;

2. Generate the output vectors with the input of the image feature vectors through the ANN classifier;

3. Produce the classified image by data post-processing of ANN classification;

4. Merge the output vectors of the L ANN classifiers with corresponding signature matrices;

5. Produce the final classified image by data fusion.

The flowchart of this procedure is shown in Figure 4. This is a generic procedure. Specifically for NDI aircraft panel corrosion in this research, we focused on the data fusion of two NDI image sensors, eddy current and ultrasound with L = 2.

In this research image processing including window shifting in generating feature vectors and signature matrices and classification post-processing including generation of classified images, merge of output vectors of ANN classifiers and generation of the final classified image of data fusion were programmed with Borland C++ 5.0 (Borland, Austin, Texas). Training, validation and testing of ANN classifiers were conducted with NeuroSolutions 4.0 (NeuroDimension, Inc., Gainesville, Florada).



One-layer MLPs were trained, cross-validated, and tested to develop ANN classifiers for aircraft panel corrosion image data. For classification, the input vectors were generated by shifting non-overlapped square windows over the images. The sizes of the square windows were from 2 x 2 to 12 x 12.

Before ANN training, the fifteen hue matrices from eddy current images and ultrasonic images respectively were evaluated statistically in terms of the two classes. In order to preserve the statistical variation over the hue matrices in each class, the pixel values from the non-overlapped square windows were used directly as the values of the feature parameters.

After ANN classifiers were trained, validated and tested, they were verified with an eddy current image and an ultrasonic image. After image processing, both images showed blank areas that represented no corrosion. In order to remove the impact of the blank areas in image feature extraction, a procedure was established to produce a feature vector from a window only when the window contains more than a part of non-zero pixels. In producing the feature vector the zero pixels were reassigned the value of the mean of the non-zero pixel values of the window. The part of non-zero pixels in the window was defined as thresholding value. In the verification it was found that the larger the feature window size, the more non-zero pixels were unexpectedly filtered out. Therefore, the thresholding value could be set with the increase of the window size, for example, for window size 2 x 2, the threshold was 1/4; for the window 5 x 5 or 6 x 6, the value could be 7/8; and for the window 10 x 10 or 12 x 12, the value could be 15/16.

The data fusion to merge the data from image processing and ANN binary classification originated from two NDI image sensors: eddy current and ultrasound. The two different NDI images visualized the same area of an aircraft panel corrosion specimen. A NDI X-ray image that visualized the same area as the two images did was used as the benchmark to evaluate the performance of each of the image classification and data fusion. Table 1 shows the pixel distribution over the X-ray image. From the table it can be derived that 2.78% of the pixels in the image ROI represent low corrosion ([less than or equal to] 10%) while 0.28% of the ROI pixels represent high corrosion (>10%), which means that about 90% corrosion pixels in the image ROI represent low corrosion.

In data fusion, the window size of image feature extraction was incremented from 2 x 2 to 12 x 12. Therefore, the data fusion can be technically described as the merge of the two class eddy current and ultrasonic corrosion image data of ANN classifiers in a multi-resolution feature extraction.

The features from the eddy current corrosion image and the ultrasonic corrosion image were extracted with the increment of window size from 2 x 2 to 12 x 12 for ANN binary classification. The extracted feature vectors were used to go through two developed ANN classifiers at different window sizes. At each window size, the outputs of the two ANN classifiers were merged and then a new classified image was produced with the merged signature matrix. The new classified image was the result of data fusion of eddy current and ultrasonic images. The performance was evaluated by comparing the class signature in classified images with the corrosion signature in the processed X-ray corrosion image pixel by pixel.

Figure 5 shows all classified images of eddy current, ultrasound and data fusion at different window sizes in image feature extraction. Figure 6 and Figure 7 show the plots of the classification rates of the eddy current image, the ultrasound image and image data fusion of high and low corrosions respectively. From the images and plots in the figures, it can be found that the eddy current data could consistently be classified to detect both high and low corrosions but the classification could indicate clearly the pattern of spot welds in the area of the corrosion specimen although many pixels in the eddy current image were misclassified. It is also found that the ultrasonic data could only be consistently classified to detect low corrosion and the classification could indicate clearly the pattern of rivets and spot welds in the corrosion specimen; many pixels in the ultrasonic image were misclassified at the same time. Also many other pixels were skipped due to window shifting over the images in feature extraction. The figures also show that the data fusion consistently enhanced the ANN classifications and presented a better match with the X-ray corrosion image in detection of low and visualization of patterns in the corrosion specimen.




Figure 8 shows the plots of the classification rates of the eddy current image, the ultrasound image and image data fusion combining high and low corrosions together. The plots in the figure indicate that the data fusion consistently enhances the ANN corrosion classifications with eddy current and ultrasonic data in the combined classifications with incremented window sizes from 2 x 2 to 12 x 12 in image feature extraction.


In summary (Table 2), image data fusion improved the ANN classification rates of the eddy current image by 12.6% and 12.21% in average for low corrosion and overall corrosion classification respectively, and it improved the ANN classification rates of the ultrasonic image by 28.88% and 32.18% in average for low corrosion and overall corrosion classification respectively.


This research has developed a method of data fusion of ANN classification from multiple image sensors for NDI. With the method, data fusion was implemented over ANN classifications of eddy current and ultrasonic images compared with X-ray benchmark image for detection of aircraft panel corrosion. The results have proven that the method is effective and data fusion is very promising in enhancement of individual classifications from different image sensors for corrosion NDI. Original eddy current and ultrasonic image ROIs were limited in characterization of aircraft panel corrosion. Data fusion classified consistently better in overall and low corrosion pixels, which are 90% of all corrosion pixels. It certainly performed better with a high performance configuration of image sensors, image processing, image feature extraction, and ANN image classification.


Mention of trademark, vendor, or proprietary product does not constitute a guarantee or warranty of the product by the USDA and does not imply its approval to the exclusion of other products that may also be suitable.


Blum, R. and Z. Liu. (2006). Multi-Sensor Image Fusion and Its Applications. CRC Press.

Forsyth, D. S., Z. Liu, J. P. Komorowski, and D. Peeler. (2002). An application of NDI data fusion to aging aircraft structures. 6th Joint FAA/DoD/NASA Conference on Aging Aircraft, Sept. 16-19, 2002, San Francisco, CA. U.S.A.

Gros, X. E. (1997). NDTData Fusion. Arnold, London, Great Britain.

Grossberg, S. (1976). Adaptive pattern classification and universal recording: I. parallel development and coding of neural feature detectors. Biological Cybernetics. 23, 121-134.

Han, J. and B. Bhanu. (2007). Fusion of color and infrared video for moving human detection. Pattern Recognition, 40, 1771-1784.

Hebb, D. O. (1949). The Organization of Behavior. John Wiley & Sons, New York.

Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences. 79, 2554-2558.

Laliberte, F., L. Gagnon, and Y. Sheng. (2003). Registration and fusion of retinal images-an evaluation study. IEEE Transactions of Medical Imaging. 22(5), 661-673.

Leinonen, I. and H. Jones. (2004). Combining thermal and visible imagery for estimating canopy temperature and identifying plant stress. Journal of Experimental Botany. 55(401), 1423-1431.

McCulloch, W. S., and W. Pitts. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics. 9, 127-147.

Minsky, M. L. and S. A. Papert. (1969). Perceptions. MIT Press, Cambridge, MA.

Palakal, M. J., R. M. Pidaparti, S. Rebbapragada, and C. R. Jones. (2001). Intelligent computational methods for corrosion damage assessment. The American Institute of Aeronautics and Astronautics Journal. 39(10), 1936-1943.

Pohl, C. and J. L. V Genderen. (1998). Multisensor image fusion in remote sensing: concepts methods and applications. International Journal Remote Sensing, 19(5), 823-854.

Rebbapragada, S., M. J. Palakal, R. M. Pidaparti, and C. R. Jones. (1999). Corrosion detection and quantification using image processing for aging aircraft panels. The Third Joint FAA/DoD/NASA Conference on Aging Aircrafts, Albuquerque, NM. U.S.A.

Rosenblatt, F. (1958). The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review, 65, 386-408.

Rumelhart, D.E., G. E. Hinton, and R. J. Williams. (1986). Learning internal representations by error propagation. In: D. E. Rumelhart and J. L. McClelland (Eds.), Parallel Distributed Processing: Explorations in the Microstructures of Cognition, vol. I. MIT Press, Cambridge, MA (Chapter 8).

Tsagaris, V. and V. Anastassopoulos. 2005. Fusion of visible and infrared imagery for night color vision. Displays, 26, 191-196.

Huang Yanbo (1) *

Lan Yubin (2)

Hoffman W. C. (3)

Lacey R. E. (4)

(1) United States Department of Agriculture, Agricultural Research Service, Crop Production Systems Research Unit, Stoneville, Mississippi, USA

* Corresponding Author. Email:

(2) United States Department of Agriculture, Agricultural Research Service, Areawide Pest Management Research Unit, College Station, Texas, USA

(3) United States Department of Agriculture, Agricultural Research Service, Areawide Pest Management Research Unit, College Station, Texas, USA

(4) Texas A&M University, Biological and Agricultural Engineering Department, College Station, Texas, USA

([dagger]) Received March 11, 2011; accepted June 1, 2011.
Table 1
Pixel Distribution over the X-Ray Image
for Binary Classification

 Number Percentage

Number of Non-Corrosion 226,548 96.94%
 Pixels in the Image
Number of Low Corrosion 6,501 2.78%
 ([less than or equal to]
 10%) Pixels in the Image
Number of High Corrosion 660 0.28%
 (>10%) Pixels in the Image
Total 233,709 100%

Table 2
Percent Improvement of Binary ANN Corrosion Classifications of Eddy
Current and Ultrasonic Images and Data Fusion at Different Window
Sizes for Image Feature Extraction

Window Size Low Corrosion Classification Overall

 Data Fusion Data Fusion Data Fusion
 over over over
 Eddy Current Ultrasound Eddy Current

 2 x 2 13.95% 25.13% 13.50%
 3 x 3 14.49% 26.98% 13.72%
 4 x 4 13.83% 24.44% 13.44%
 5 x 5 12.67% 25.76% 12.27%
 6 x 6 18.71% 32.62% 17.46%
 7 x 7 11.07% 30.00% 10.88%
 8 x 8 11.10% 31.67% 11.03%
 9 x 9 12.83% 32.78% 12.12%
 10 x 10 9.58% 28.95% 9.92%
 11 x 11 11.09% 30.63% 10.77%
 12 x 12 9.27% 28.75% 9.21%
 Average 12.60% 28.88% 12.21%

Window Size Overall

 Data Fusion

 2 x 2 29.40%
 3 x 3 30.86%
 4 x 4 28.93%
 5 x 5 29.58%
 6 x 6 35.53%
 7 x 7 32.22%
 8 x 8 34.47%
 9 x 9 35.82%
 10 x 10 31.59%
 11 x 11 33.83%
 12 x 12 31.76%
 Average 32.18%
COPYRIGHT 2011 Canadian Academy of Oriental and Occidental Culture
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2011 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Huang, Yanbo; Lan, Yubin; Hoffmann, W.C.; Lacey, R.E.
Publication:Advances in Natural Science
Article Type:Report
Geographic Code:1USA
Date:Jun 30, 2011
Previous Article:Primary research on bionic design of multi-surface solar concentrator based on the flower structure.
Next Article:Prediction of world crude oil price with the method of missing data.

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters